PAC-FNO: Parallel-Structured All-Component Fourier Neural Operators for Recognizing Low-Quality Images
Authors: Jinsung Jeon
Hyundong Jin
Jonghyun Choi
Sanghyun Hong
Dongeun Lee
Kookjin Lee
Noseong Park
Venue: ICLR 2024 poster (2024)
Keywords: Low-quality image
Visual recognition
Classification
Low-resolution
Input variations
Primary area: general machine learning (i.e., none of the above)

Abstract: A standard practice in developing image recognition models is to train a model on a specific image resolution and then deploy it. However, in real-world inference, models often encounter images different from the training sets in resolution and/or subject to natural variations such as weather changes, noise types and compression artifacts. While traditional solutions involve training multiple models for different resolutions or input variations, these methods are computationally expensive and thus do not scale in practice. To this end, we propose a novel neural network model, parallel-structured and all-component Fourier neural operator (PAC-FNO), that addresses the problem. Unlike conventional feed-forward neural networks, PAC-FNO operates in the frequency domain, allowing it to handle images of varying resolutions within a single model. We also propose a two-stage algorithm for training PAC-FNO with a minimal modification to the original, downstream model. Moreover, the proposed PAC-FNO is ready to work with existing image recognition models. Extensively evaluating methods with seven image recognition benchmarks, we show that the proposed PAC-FNO improves the performance of existing baseline models on images with various resolutions by up to 77.1% and various types of natural variations in the images at inference.

Published as a conference paper at ICLR 2024
PAC-FNO: PARALLEL-STRUCTURED ALL- COMPONENT FOURIER NEURAL OPERATORS FOR RECOGNIZING LOW-QUALITY IMAGES
Jinsung Jeon', Hyundong , Jonghyun Choi', Sanghyun Hong', Dongeun Lee", Kookjin Lee", Noseong Park
Yonsei University, "Seoul National University, 'Oregon State University, "Texas A&M University Arizzna State University, KKAIS
{jjsijs0902, tuzi04)8yonsei. ac kr. jonghyuncho @@enn ac kr sanghyun. wwwg@oegoosstteeeeee.eeoeooeeoe. edu, dongeun. kookjin leelasu edu, noseong@ka ist kc
edu,
ABSTRACT
A standard practice in developing image recognition models to train a model on a specific image resolution and then deploy it. However, in real-world infer- ence, models often encounter images different from the training sets in resolution and/or subject to natural variations such as weather changes, noise types and com- pression artifacts. While traditional solutions involve training multiple models for different resolutions or input variations, these methods are computationally expensive and thus do not scale in practice, To this end, we propose novel neu- ral network model, parallel-struce strruturdd and all-component Fourier neural operator (PAC-FNO), that addresses the problem. Unlike conventional frewwrd neu- ral networks, PAC-FNO operates in the frequency domain, allowing it to handle images of varying resolutions within a single model. We also propose a two- stage algorithm for training PAC-FNO with minimal modification the origi- nal, downstream model. Moreover, the proposed PAC-FNO is ready to work with existing image recognition models. Extensively evaluating methods with seven image recognition benchmarks, we show that the proposed PAC-FNO improves the performance of existing baseline models on images with various resolutions by up to 77.1 % and various types of natural variations in the images at inference.
INTRODUCTION
Deep neural networks have enabled many breakthroughs in visual recognition (Simonyan Zisser- man 2014 He et al.) 2016 Szegedy et al. 2016 Krizhevsky al. 2017 Dosovitskiy et al. [020]] Liu et af.] 2022). common practice of developing these models is to learn a model on training images with fixed input resolution and then deploy the model to many applications.
In practice, when these models are deployed to real world, they are likely to face low-quality in- puts at inference, e.g., images with resolutions different from the training data and/or those with natural input variations such as weather changes, noise types, and compression artifacts. The use of such low-quality inputs significantly degrades the performance of visual recognition models. For example, Figure shows that the ConvNeXt models (Liu et al. (2022) trained on ImageNet- 1k (Rus-) sakovsky et al. 2015) suffer from (top-1) accuracy degradation when their inputs are of low-quality.
Imagehit-1k
Imageket-CPP
Figure 1: Performance improvements by PAC-FNO in ImageNet-Ik and ImageNet-C/P (Fog). PAC- FNO improves the top-1 accuracy of ConvNeXt mod- els when low-quality inputs are used, compared the 'resize' baselines which is resize-and-feed using inter- polation. ImageNet-C/P (Fog) is dataset on which the 'fog' variation is applied to the inputs.
82
180
60
40
# 20
PAC PMO
RAC-FND Resioc 128
Rosize
0
Resolutions

---

Published as a conference paper at ICLR 2024
A naive approach for using low-resolution images is to resize-and-feed them using interpolation methods. However, the figure shows that even when we use the best-oerg interpolation method, the model's performance degrades significantly (i.e., the images with a x2x32 32 resolution resized to 224 x x24 decreases the top-1 accuracy by 58%). Similarly, we observe the performance decrease in the ConvNeXt models if the images in weather changes such fog. It becomes much worse when multiple variations are combined; for example, resizing x2x32 inputs containing the foggy weather condition leads to a decrease in the top-1 accuracy of the ConvNeXt models by 87%.
Here, we propose a novel architecture that can significantly alleviate the performance degradation of visual recognition models under low-quality inputs.
(1) Unlike the prior work (Geirhos et al.) 2018 Wang et al.) [020] Zhu et al. 2020; Yan et al., 2022albb, our method addresses both the low- resolutions and input data variations at the same time. (2) Compared to existing methods (Haris) et al. 2018 Anwar & Barnes 2020) that require training multiple super-resolution models to ac- commodate different input resolutions (and/or input data variations), we only need a single model to support these input quality degradations.
To address both issues, we develop parallel-structured and all-component Fourier neural operator (PAC-FNO). By operating in the frequency domain, PAC-FNO is able to learn the semanties of images in various resolutions and/or natural variations for challenging image recognition with a single model. The proposed PAC-FNO is ready work with deep neural visual recognition models to improve their performance.
We perform a comprehensive evaluation of PAC-FNO with seven image recognition benchmarks under various input quality degradation. In the evaluation, PAC-FNO demonstrates superior perfor- mance over other baselines in multiple variations such as weather changes at low resolution, with performance increase of at least 1 % at a 32x32 resolution.
Contributions. We summarize contributions follows:
We propose PAC-FNO, a novel neural network architecture that operates in the frequency do- main and offers resilience to image quality degradation, such as resolution changes and/or natu- ral variations, a visual recognition model typically faces in real-world settings.
Our design choices of PAC-FNO can offer such resilience with single model, able to attach to a downstream visual recognition model. Not only does this handle multiple input variations at once, but the approach also minimizes the changes in the downstream model during fine- fine-tuning.
We perform an extensive evaluation of PAC-FNO with seven benchmarking tasks. We show that existing image recogmition models, fine-tuned with PAC-FNO, can handle any input resolution degradation and are resilient to input variations that occur in real-world deployment settings.
2 PRELIMINARIES AND RELATED WORK
Visual recognition with low-quality images. Although low-quality images can be characterized by both low resolution and quality degradation, previous studies have only considered: i) resolution degradation or ii) quality degradation due to noise and weather changes, name few. For low- resolution images, they design a specific network for low-resolution classification (Zhu et al. 2020) Yan et al., 2022alb) b) or attach additional model that can handle low-resolution images, such as super-resolution (SR) models, in front of the pre-trained classification models (Cai et al.) 2019). For quality degradation, approach solving this problem is to images for training, which makes the model robust to image variations (Geirhos et al.) 2018 Wang et al.) 2020). Although they are successful in handling low resolution and input variations, respectively, our knowledge, there is no method that considers the co-existence of resolution degradation and input variation
Fourier neural operators (FNOs). The FNOs achieve remarkable performance in solving PDFs with small computational costs and have been adopted by many other applications (Guibas et al., 2021 Li al. 2020a b. Rahman et al., 2022). First of all, FNOs are mathematically defined under the infinite dimensional continuous space regime and for this reason, they can process various res- olutions of the continuous space without model changes. However, since the infinite dimensional continuous function of FNOs cannot be implemented in modern computers, the real implementation of FNOs follows the following quantized version:
h = h= oF (Ro O ((F(b()))) + he-iW),
(I)
2

---

Published as a conference paper at ICLR 2024
[x] Fourier Neural Operator (FNO)
[b) U-shaped FNO
(c) Adaptive FNO
(4) PAC-FNO (Ours)
Block
Mack
XC- PAD
h
Fourier Transfonn
Frassannn Componett
Foutier Transtrn
Fountor Transfomm
Fourler Transform
Connolaton
Connollttoo bo
Connolation
1x4
Pana Fibbr
Learnable Filerr
Learaale Filler
ASC
A6C
19
Inverse Fourter Tranaform
Inverne Transfor
Invercan Transform
FNO Block
FND Block
A-FNO Block
AC ANO Block
Figure 2: Comparison of parallel-structured and all-component FNO (PAC-FNO) with existing
FNOs. (a) illustrates the vamilla FNOs. Each FNO block contains an ideal low-pass filter, a learnable filter (Rş) operating in the frequency domain, and a 1 x 1 convolutional operator (W). FNOs can have series of these blocks. (b) shows U-shaped FNO (UNO), connecting the FNO blocks in U-shape. (c) depicts Adaptive FNO, which replaces the learnable filter in the FNO block with adaptive global convolution (AGC). Our PAC-FNO shown in (d) uses all frequency components by removing the low-pass filter of the FNO block and runs forward with the AC-FNO block in parallel. (architecture advances are shown from (a) to (d) red).
where hy E x is a hidden vector at f-th layer. The kernel operator F(s) is a Fourier transform that transforms the hidden vector into the frequency domain. is ideal low-pass filter that removes signal components whose frequencies are greater than a threshold from the hidden vector. O denotes Hadamard product with a learnable filter Rg in the frequency domain. The kernel operator (n) is an inverse Fourier transform. W is 1 X 1 convolutional operation. a is a non-linear activation function. For efficiency, FNOs restrict the size of the parameter Ro by using an ideal low-pass filter. Then, Ra E CC XC x H, xW, W. where Cin (Cont) is the channel size of the input (output) hidden vector and H,W,,) means the height (width) after the ideal low-pass filter in the frequency domain (cf. Figure (a)).
FNOs for visual recognition. FNOs' resolution invariance attractive for visual recognition. Recently, applying FNOs to visual recognition has received attention (Guibas et al. 202 Rahman et al. 2022; Viswanath et al., 2022 Pan et al., 2022; Wu et al. 2023), such as replacing the Vi- sion Transformer's 's token mixer or detecting embers during wildfire, etc. Among them, U-shaped FNOs (Rahman et al.] 2022) change the block configuration of FNOs, which is serial in FNOs, to an encoder and decoder structure using skip connection (cf. Figure (b)).
In addition, adaptive FNO block (Guibas et al. 2021) replaces a token mixer of the Vision Trans- former's self-attention (Dosovitskiy et 2020 by mixing tokens in the frequency domain. They improved memory and computational efficiency by replacing the learnable filter in the FNO block, which is not suitable for high-resolution input, with small-sized filters interpreted by the adaptive global convolution (AGC) operator. They parallelized AGCs to increase computational efficiency. Here, for a fair comparison, the A-FNO block is set the same structure the FNO (cf. Figure 2 (c)). As briefly reviewed here, FNOs are used as one part of the visual recognition model (e.g., token mixing), and their use in visual recognition is still the early stage of research.
PAC-FNO: PARALLEL-STRUCCTRRE AND ALL-COMPONENT FOURIER NEURAL OPERATORS
Despite the success of FNOs in visual recognition (Rahman et al. 2022 Viswanath et al. 2022
Wu et al. 2023), there significant challenges in applying it to image classification. |aan et al. (020)) shows that there is a trade-off between model performance and generalization depending on image frequency components captured by the neural network in image classification. If the model
3

---

Published as a conference paper at ICLR 2024
Frequency Components
AC-FNO Blook
AN Block
h
b
PAC-FNO 0g
AC-FNO
Zero Pard-aadding
hj
by
mm0
AC-FNO Block
FNO Block
Pre-trained Backbone Model
www.t
by
B4
Lincer Layer
Convolution
1xt
Interpolation
Norm
(a) AC-FNO Block
(b) PAC-FNO
(c) Overall Workflow
Figure 3: Parallel-structured and all-component Fourier neural operator (PAC-FNO) archi- tecture detail. (a) AC-ENO blocks use all frequency components and rely on Zero-padding
and Interpolation to construct the images for the target resolution. (b) Contrary to previous FNOs, PAC-FNO consists of multiple AC-FNO blocks in a parallel manner. by is (4i, 1-tth hidden vector (4 E {O..... m).4 14 E (O....,,ul ne)). In (c), DR is a set of image datasets with different resolutions, h is a hidden vector processed from PAC-FNO, and is predicted class (see 33 1 and 3.2 for more details about PAC-FNO).
captures wwwffreqeency components, the accuracy of the model increases, whereas if it captures higu-freyeency components, generalization improves.
However, to reduce the complexity of post- poot-processiig in FNO, an ideal low-pass filter is used after the Fourier transform (Li et al., 2020a), which removes high-frequency components that represent detail in the image. We argue that the low pass filter removes useful information for generalization of the model and harms the image classification accuracy. Thus, we propose a FNO block without the low pass filter, and call it 'all component' FNO (AC-FNO). We then propose to stack the AC-FNO blocks in parallel structure to increase the capacity of encoding spatial information of images, call the whole structure as 'parallel' AC-FNO or PAC-FNO. We depict its architecture in Figure
3.1 AC-FNO BLOCK
When solving PDEs describing fluid dynamics with FNOs, the input signal is assumed periodic in the frequency domain. For efficiency, non-significant parts are removed using a low-pass filter (in other words, high-frequency information is removed). However, in the case of images, high- frequency information sometimes plays an important role in image classification, especially when detailed information is required (type of bird, type of car, etc.). To this end, we propose AC-FNO block without any band pass filters. Formally, our AC-FNO block is written as:
o
F(he-1))) & + W)),
where Rg E C3x3xH, W. denotes learnable filter that maintains 3 channels of the image where H, and W, denotes the height and the width of the image (resolution) in the frequency domain, X is an optional zero-padding, and is optional interpolation scale up the processed outcomes for low- resolution inputs only- - the sizes of the processed outcomes are scaled up to the target resolution for which the backbone model was trained. Therefore, the AC-FNO block allows the processing of images in various resolutions while maintaining all frequency components (cf. Figure (a)).
3.2 PARALLEL CONFIGURATION OF AC-FNO BLOCKS
We now propose to configure AC-FNO blocks a parallel structure to increase the capacity to learn various types of input variations. Previous FNO models have serial structure, and the first layer of the model, which is directly related to data, consists only one block. It has too small a capacity to consider all components of the data. Therefore, increase the capacity of the first layer with the parallel configuration of AC-FNO blocks, allowing it to utilize all frequencies of the image, including high-frequency and low- frequencccc components.
4

---

Published as a conference paper at ICLR 2024
There are AC-FNO blocks, where is the number of stages and is the number of parallel AC-FNO blocks in a stage. We then concatenate all the m processed outcomes with linear layer, followed by batch normalization. The overall flow of the AC-FNO block can be written as follows:
h A.C-FNO-Blockk (h 11). ey. hy BN(Linear(h)) h ...... h))), h RR.H.XWW
(3)
(4)
where AC-FNO-Block is the (-)th AC-FNO block in Eq. 2 h R is the last stage of the hidden vectors where H2 (W) is the height (width) of the target resolution, Linear denotes the linear layer that transforms from 3 x channel to 3 channel, and BN denotes batch normalization layer. hy is an input to the following backhone model. See $4.3 for the impact of PAC-FNO configurations input variation.
3.3 TWO-STAGE LEARNING ALGORITHM
To leverage the pre-trained model in training PAC-FNO model, propose two-stage training al- gorithm for stable training. Specifically, we jointly train the PAC-FNO and the pre-trained backbone model together using only the target resolution dataset (Dtarget) for which the backbone model was trained. Since the pre-trained backbone model may not able to fully understand the hidden space created by the PAC-FNO, we train them in a two-stage algorithm.
In the first phase, if the PAC-FNO and backbone model are well harmonized ! for target resolu- tion, the second phase begins. In the second phase, fine-tune the well-harmonized model with images in low resolutions to generate a unified hidden space for all resolutions. If we use three low- resolution datasets with different resolutions for training, such as 32, 64, and 128. then R in DR has a set of (target, low,, low2, low3]. In AC-FNO blocks, interpolation and zero- pero-padding operations are optionally performed on low-resolution images to match the size of the processing result. Then, hy, the output of PAC-FNO, is hidden vector that represents commonalities in images of various resolutions and is used as input to pre-trained backbone model to predict the class (i).
In both stages, we use the cross-entropy loss for training. See $4.3 for the effectiveness of our pro- posed two-phot swo-phase training algorithm. We describe the two-stage training algorithm in the Appendix (Algorithm[] D for the sake of space.
In addition, the computational cost of PAC-FNO is small since the number of parameters in the PAC- FNO is 1-13% of that of the backbone model. The parameters of PAC-FNO for various backbone models are reported in the Appendix $.2
4 EVALUATION
For empirical validations, we evaluate PAC-FNOs and other state of the arts for low-quality visual recognition. After we train them with a pre-trained backbone model, we evaluate the model low- resolution and input variations tasks the same time. We then investigate the benefit of proposed components by ablation studies, and analyze the PAC-FNO further by sensitivity studies. Detailed setup is found in Appendix SF.
Datasets. We use seven image recognition benchmark datasets to evaluate PAC-FNO. For low- resolution tasks, six image recognition benchmark datasets are used to evaluate PAC-FNO: ImageNet- 1k (Russakovsky et al.) 2015). Stanford Cars (Krause al.] 2013), Oxford-IIIT Pets (Parkhi et 20.12))). Flowers (Nilsback & Zisserman 2008). FGVCrirrerat (Maji et al.) 2013). and Food- 101 (Bossard et al.) 2014). For imput variation task, we use ImageNet-CrP (Hendrycks & Dietterich 2019) which is image dataset containing 19 common corruptions and perturbations.
Backbone architectures. We use four image classification models pre-trained on ImageNet- 1k: two convolutional neural networks, ResNet-18 (He et al., (016)) and Inception-V3 (Szegedy) al. 2016). Vision Transformer (ViT) (Dosovitskiy al. 2020) for a different architecture, and ConvNeXt-Ting (Liu al. 2022).
WWell harmonized means that the performance of the model combining PAC-FNO and the pre-trained back- bone model is similar that of pre-trained model at the target resolution.
5

---

Published as a conference paper at ICLR 2024
Table 1: Performance of PAC-FNO on the low-resolution tasks using ImageNet-Ik. We report top-1 accuracy on low-resolution images generated from ImageNet-1k.
Model
Method
Resolution
Model
Method
Resolution
32
56
64
112
128
224 299
28
64
112
224
299
Resize 16.7
12,7145,,
55,7
63.7 69.8
Resize
41.647.4
76.8
78.1
81.1
Fine-tune
39.8
69.6773
78.437111
Fine-tune
1,01 2.15
10.2
16.3
34.5
50.665.0
DRIN
3.75
42.0
77.0
81.1
ResNet- 18
DRIN 0.22
DRPN 31.6
55.6
17.1
67.5
62.8
69.8
69.88
VIT-B16
DRPN
FNO
39.1
52.2
66.2666662661.
73.0
77.1
80.0
79.7
81.1
PNO
40.245.259. 61.5
67.6 68,5
UNO
43.7
50.666..77 70.3
77.9 78.8 80.2
UNO
38.9445.358.9960
67.1 68.1 69.4
A-PNO
54,9
72.3 74.1 78.3 78,9 79.5
A-FNO 43,5 60.2 62.5 66.9 67.5 68.5
PAC-FNO
46.6 69.0
77.7 78.6 79.6
PAC-FNO
42.7 47.7 60.5 62.8 68.3 69.1 70.2
Resize 24.5 60.7
75.9 77.0
Resize 16.7 22.0 48.6 53.9 79.5
77.3
Fine- Tune-tme 40.2
65.8
76.0
80.7
81.8
Fine-tune 39.6 63.7 69.8 73.9
77.5
DRLN 0.30
25.1
71.8
82.3
Inception-Ve V3
FNO 48.9 68.5 70.2 74.9 76.5
78.4
CousNcX:-Tiny
DRPN 40.7
FNO 42.8
03.48..9 67.4
68.2
69.8
76.5
79.4
77.4
79.2
82.5
UNO 42.248.1 65.4 68.0 75.7
77.3
UNO
58.662.9
75.5
79.1
A-FNO
39,6 59.6 62.7
72.
74.9
A-FNO
43.3
68.1 70.5
75.9 76.6 78.1
PAC-FNO 54,9 68.8 70.7
76.1 76.9
78.4
PAC-FNO
58.9 63.2 74.5 80.2
80.7
Baselines. We use three baseline types: i) resizing, ii) super-resolution (SR) models, and iii) Fourier neural operator models. Resizing includes Resize and Fine-tune methods that do not re- quire additional networks. Resize is method that directly feeds the resized images to preitrdi classification model using interpolation, while Fine-tune is method of fine-tuning the pre-trained classification model with the resized images. SR models include DRLN (Anwar & Barnes 2020 and DRPN (Haris et al. 2018), and these two models are representative models capable of up to times super-resolution. With SR models, low-resolution images are upscaled to the target resolution and fed into a pre-trained model. Fourier neural operator models include vanilla FNO (Li et al.) 2020a), UNO (Rahman al. 2022), and A-FNO (Guibas et al. (02T). These models are most similar to our PAC-FNO and are trained by our proposed training method for fair comparison.
Metrics. We use two metrics for the evaluation: top-1 accuracy and relative accuracy. Relative accuracy is defined the ratio of model's accuracy in low quality to that in the original resolu- tion. This metric enables us to compare the effectiveness of methods in the same resolution across different datasets and models.
Image degradation types. We generate res-uuouttionn images by resizing and cropping the orig- inal images. We consider resolutions in {28, 32, 56, 64, 112, 128, 224, 299), which are frequently used in image classification. For training we use [224] or (299) as target resolution depending on the backbone model along with {32, 64, 128] low resolution. During inference, seven resolu- tions are used, including three datastes not used for training. Super-resolution models can only be upscaled by 2, 4, or 8 times, so they can be inferred only at resolutions of 28, 56, and 112.
4.1 LOW-RESOLUTION IMAGE RECOGNITION
Table shows the top-1 accuracy results of ImageNet-Ik: lk. In most cases, PAC-FNO shows good performance. In particular, PAC-FNO shows the best performance not only at low resolution but also at the target resolution of 299 in Inception-V3, We also find that in Con-NeXT-Tiny, PAC-FNO shows the best accuracy at all low resolutions, even if the result at the target resolution is lower than the Resize method (81.5 vs. 82.5). This suggests that PAC-FNO shows good performance in terms of relative accuracy which we proposed. The results of relative accuracy are in the Appendix F
For fine-grained image datasets, ConvNeXt-Tiny pre-trained with ImageNet- 1k is fine-tuned for each dataset with a target resolution. After that, the same algorithm used in ImageNet- 1k is applied to the fine-tuned ConvNeXt-Tine model. Table the experimental results for four fine-gra grained datasets. The proposed method, PAC-FNO, shows good performance at low resolutions by a large margin for all datasets.
In particular, among the Fourier neural operator models, only the PAC-FNO shows better perfor- mance than Fine-tune method. This represents the effectiveness of the ideal low-pass filter in the FNO block, which removes detailed image signals that play an important role in classification in the fine-grained dataset. In Food-101 the target resolution accuracy is 2% lower than Fine-tune method,

---

Published as a conference paper at ICLR 2024
Table Performance PAC-FNO on the low- Table 3: Performance of PAC-FNO on the resolution tasks using fine-grained datasets.
input variation tasks. We report top-1 accu- We report top-1 accuracy for the ConwNext-Ting racy for the ConvNext-Tiny model on four nat- ural input corruptions, chosen from ImageNet- C/P (Hendrycks & Dietterich 2019).
model on low-resolution images generated from fine-grained datasets.
Dataset
Method
Resolation
Variation
Method
Resolution
28
32
56
64
112
128
224
24
32
54
64
128
224
Resize 29.4
36.4
70.1
77.0
91.2
93.7
Resize
8.21 10.8
27.2
31.9
52.3 58.4
Fine-tune
32,6
73.2
99.3
93.8
Fine-tune
28.2
47.5
51.4
62.2
63.0
DRLN
3.37
36.8
88.1
93.7
DRLN
0.16
6.54
33.5
58.4
Oxford-IIIT Pets
DRPN
FNO
41.5
19,1340006
84.2
70.2
92.6
86.7 90.4
91.4
Fog
DRPN
FNO
0.67
18.4
38,9 43.5
0.99
56.6
1.32
58.4
60.3
58.4
UNO 11.1
80.1 83.8 90.3
UNO
24.629.77
46.8
49.5
59.2 60.4 59.0
A-FNO
27.0 63.5 70.2 85.2 86.9 89.1
A-PNO
11,0
33.4 38.3 51.9119999 53.6 53.3
PAC-FNO
73.4 86.4777.366..7777 90.5
91.1
91.7
PAC-FNO
30.4 48.2 51.7 60.1 61 62.8
Resize 47.547..75.3
82.
93.7 95.9
Resize
22.028.2
52.7 56.8
69.9
73.5
Fine-tune 44.3
95.9
Fine-tune 42.848.9
65.8 68.1
73.5 74.4
75.7
DRIN 9.92
53.4
91.3
95.9
DRLN 6.64
35.71
56.1
58.4
Flowers
DRPN 64.5
89.0
95.0
Brightness
DRPN 3.09
1.08
8.01
58.4
FNO
25.3328665.9
73.0
93.7
96.6
FNO 40.9
59.4 62.6
70.07
UNO
23,230.2 7 90.8 92.6
96.1
UNO 55.6
66.7 67.9 17.9714720 72.1 72.8
A-FNO
26.8 33.663.77 70.0 86.2
88.8
91.9
A-FNO
37.643.2 59.4 62.5 68.8 69.8 70.0
PAC-FNO
74.1 78.0 89.3 93.6
94.2 94.4
PAC-FNO
56.5
68.2 69.8 73.4 74.1 74.7
Resize 2.76 27.3 41
4
7
74.3
79.7
Resize
22.9
40.7
44.1
56,3 58,9 62.6
Fine-tune 8.5 16.2
58.3
77.2
76.0
79.8
Fine-tune 35,6
52.5 54.8
61.8 63.0
64.6
DRLN 1.14
13.1
68.9
79.7
DRLN 1.85
17.5
31.3
62.6
FGVC Aircraft
DRPN 18.4
FNO 26.3332394480
59.6
77.0
74.6 75.7
80.7
79.7
Spatter
DRPN 0,83
FNO 28,33199
44,9
0.36
47.8
0.34
62.4
62.6
UNO 1.65 7..7099.772..
73.0 76.4
81.6
UNO 40.2
50.5 52.2 58.8 59.8
A-FNO
6.78
A-FNO
32.0
45.8 48.4 57.4 59.2
PAC- 37.7 69. 78.6 81.1
PAC-FNO
43.2 46.0 54.5 51.4 62.3 64.0
Resize 40.1 48.1 76.2
80.2
88, 89.2
91.1
Resize
17.022.6
47.5
52.2
62.66777.111
Finc-tune 50,2 54.1 80.3
82.0
88.3
89.6
91.2
Fine-tune
35.641.7
61.4
71.471..2255722 74.3
DRLN 7,06
39.1
86.8
91.T
DRLN 1.26
19.4
44.3
73.5
Food-101
DRPN 54.8
FNO 43.651.779881 01..08888.22
81.6
88
59.5
89.0
90.8
91.1
Saturate
DRPN 0.78
FNO 30.3
55.7
0.56
61.3066.3688.3
0.72
73.5
70.5
UNO 50.5 75.5 78.3
85.2 86.0
87.9
UNO
45.8
50.7 63.5 65.3
71.2 70.8
A-FNO 46.051.72.22 75.8
84.8 88.2
A-FNO
37,943.3
59.3 62.1 68.2 69
69.3
FNC 74.9 85.5 86.9
89.2
89.7 90.4
PAC-FNO 44.2 56.5 64.5 71.9 74.1 74.7
but PAC-FNO shows the highest accuracy in all other low resolutions. Compared to the Fine-tune method, accuracy increased by 48.3%, 43.8%, 6.48%, and 5.98%, at 28, 32, 56, and 64 resolutions, respectively. Results for the rest of the aeningr datasets and best hyperparameter settings are in the Appendix $F4 and SE Additionally, we conducted a fine-grained classification experiment Oxford-IIIT Pets with the ViT model (cf. Appendix F.4)
4.2 RESILIENCE TO NATURAL INPUT VARIATIONS
We now test the resilience of input data variations in test-time, such as weather changes, that likely occurs in real-world deployment settings. We run experiments with ImageNet-C/P (Hendrycks & Dietteric 2019). ImageNet-C/P has 19 noises that can be used test resilience against image qual- ity degradation. Table summarizes the top-1 accuracy of PAC-FNO under four input variations- fog, brightness, spatter, and saturate.
PAC-FNO shows superior performance in most cases, especially at low resolution. Only the Fine- tune method shows slightly better performance than PAC-FNO at target resolution of 224 in fog and brightness. However, the Fine-tune method in fog has a greater performance degradation than PAC-FNO at low resolution. In particular, at 32x32 resolution, there is a 7.6% performance dif- ference (35.6% vs. 43.2%) between Fine-tune and PAC-FNO. Table 3 shows that the resolution in- variant models show good performance at low resolutions and Resize and Fine-tune methods show good performance relatively high resolutions. In the case of SR models, performance is com- pletely reduced at low-resolution input variation. Our proposed PAC-FNO is more resilient to input changes regardless of resolution than other baseline methods. Results for the rest of the fifteen input
7

---

analysis of Published as a conference paper at ICLR 2024
variations are in Appendix FF for the sake of space. We also show in Appendix F.8 how the parallel structure of PAC-FNO affects input variation.
4.3 ABLATION STUDIES
We now investigate the effect of PAC-FNO's contribution. We conduct experiments for parallel configurations of PAC-FNO and two-stage training algorithm. The results of the ablation study of PAC-FNO with low and high pass filters were reported in Appendix F for the sake of space.
Configurations of PAC-FNO layer. We in- vestigate the benefit of our parallel structure to increase the capacity to learn various types of input variants by comparing the performance of the parallel and serial configurations using eight AC-FNO blocks so that total number of AC-FNO blocks are the same but in difference configuration.
Serial (nmB, mm1)
Parallel (nm2. mm4)
60-
70
22.
Top-1 Acc. (%0
60
39.1%
50
ImageNet-1k
ImageNee-1k
20-
Imagelet CIP (Fog)
ImageNet CIP (Fog)
224
128
64
32
224
128
64
Resolutions
For the experiment, ConvNeXt- Tiny is used for the backbone network with ImageNet- 1k and ImageNet- with fog degradation. Figure 4 shows that the parallel configuration increases performance in terms of accuracy compared to the serial configuration with the same number of AC-FNO blocks on both datasctes at all res- olutions. It implies that the parallel structure has clear benefit to better encode the images.
Figure 4: Benefit of parallel structure. We show the top-1 performance of PAC-FNO models with different configurations. n and m are the number of blocks in series and parallel, respectively. Note that the total number of AC-FNO blocks is the same for both configurations. We use ImageNet- Ik and ImageNet-C/P (Fog). Compared to serial configuration, our proposed parallel configuration shows less performance degradations at target res- olution (39.1% % vs. 22.9%)
In addition, ssows the resilience to in- put variations depending the PAC-FNO con- figurations. In target resolution (224), the performance decrease is reduced in parallel configu- ration compared to serial configuration such as 22.9% (81.5% - 62.8%) in parallel and 39.3% (79.1% - 48.0%) in serial configuration. As expected, parallel architectures seem to consider more frequency components than serial architectures.
Two-stage Training Algorithm for PAC- FNO. We also conducted an ablation study on
80
Top-1 Acc. (96)
our proposed two-stage training algorithm. We compare the two-stage training algorithm with
60
i) only the first training stage with the target res-
40
olution and ii) only the second training stage
128
with resouuuuuus without the first training
20
224
stage.
First stage only Second stage only Two- algorithm Figure 5: Benefit of two-stage training algo- rithm. We show the top-1 accuracy of the abla- tion study of the two-stage algorithm. 'First stage only' refers the first stage, and 'Second stage only' refers to model that was trained only with the second stage, and "Two-stage algorithm' refers to a model that was trained by our two-stage training algorithm. We use ResNet- 18 in ImageNet-Ik.
We show the result in Figure 5 As shown in the figure, our two-stage training algorithm has clear benefit. 'First stage only', which does not use low-resolution images but only trains with target resolution, shows perfor- mance decrease at low resolution. 'Second stage only', which is trained directly with low- resolutions and target resolution without first stage, shows higher performance than 'Fisst stage only' at tow-resolutions, but it shows that performance does not increase as the resolution in- creases, but that performance is good at intermediate resolutions. Without our full training scheme, the model does not converge well and the accuracy in its target resolution is severely degraded.
model that was trained only with
4.4 SENSITIVITY STUDIES
We then investigate the performance according to the PAC-FNO configuration and the number of trained resolutions.
8

---

Published as a conference paper at ICLR 2024
80
66 06.08
20.00 10.00
0611
76.12
$8.06
70.17
Figure 6: Results for the number of stages n and blocks m in PAC-FNO.
683
69
Top ACc. 186
70-
02 000
62.78
Top (18)
70
6225
60-
80
We vary 71 and m from {1, 2, 4), respec- tively. and show the model performance in low-resolution inputs.
30-
46.40
m==2
m=1
50-
15.11
n=2
40
32
64
128
224
32
64
128
224
Resolutions
Reselutions
(a) ImageNet- (n 1)
(h) ImageNet-Ik (m om=2
Table 4: Results for a number of seen resolutions during training, We show the top-1 accu- racy of ResNet-18 models according to the number of resolutions during training time. The image resolutions we used are on the left, and the inference resolutions we test are on the right. We use ImageNet- lk.
Training Resolution
Resolution
Training Resolution
Resolution
48
128
(2, 128, 126,
47.7
57.8
62.
67.1
69.1
700
70.0
702
(32,220)
53.0
55.5
39.3
64.1
06.4
68.0
69.0
693
Sensitivity study of PAC-FNO layer. We show the impact of changing the number stages (11) and the number of AC-FNO blocks (m) constituting the PAC-FNO. While one of the and m is controlled, the other parameter is manipulated, varying as 2, and 4. For the experiment, ResNet-18 is used for the backbone network with ImageNet-Ik.
As shown in Figure 6 the appropriate number of stages and AC-FNO blocks for ImageNet- Ik is and = 2. and show differences depending the backbone models and datasets, but m 2 2 in all cases. In other words, we found that our proposed parallel configuration of AC-ENO blocks is effective in low-resolution image recognition. The best hyperparameters for PAC-FNO configuration are reported in the Appendix
Number of resolutions during training. We show the performance of PAC-FNO depending on the number of resolutions used for training. We use two models for comparison and report top-1 accuracy in Table One is a model trained with resolutions frequently used in image classifica- tion, such as {32, 64, 128, 224), and the other one model trained with extreme resolutions of (32, 224]. Both models show meaningful results even for resolutions that are not used for training.
However, the model trained with a wider number of resolutions shows better performance except for the results at 32 x 32 resolution. This shows that better performance can be achieved in real-world applications with various resolutions without process to resize to the target resolution. In addition, this result shows the efficacy of our neural operator-based mechanism which has strong point in utilizing images in real-world applications.
5 CONCLUSION
We proposed parallel-structured and all-component Fourier neural operators (PAC-FNOs) for visual recognition under low-quality images. To this end, design i) an AC-FNO block without any band pass filters and ii) a parallel configuration AC-FNO blocks with increased capacity to learn different types of input variations.
In addition, we also propose a two-stage training algorithm for the training of the stable PAC-FNO model. As result, PAC-FNO provides two advantages existing methods: (i) It can handle both low-resolution and input variations typically observed in low-quality images with a single model; (ii) One can attach PAC-FNO to any visual recognition model and fine-tune In the evaluation with four visual recognition models and seven datasets, we show that PAC-FNO achieves high accuracy for various resolutions and input variations. In particular, it shows the highest accuracy on low- quality inputs that combine multiple effects (e.g., low-resolution images in foggy weather).
Future Work. An interesting line of work is to apply PAC-FNO in complex real-world settings, e.g., where multiple degradations of input can occur (e.g., the low-resolution surveillance camera mixed with motion blur and fog).

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
Our approach fine-tunes PAC-FNO and pre-trained image recognition models together, allowing a single model to handle all input resolution degradation and be resilient input changes that occur in real-world deployment settings. In the process of fine-tuning the carbon emissions of the machine for training and running, it is essential. However, this can significantly reduce carbon emissions compared to approaches that require training and running resolution degradation model and an input variations model separately.
REPRODUCIBILITY STATEMENT
For reproducibility, we attached the source codes in supplementary materials. There detailed descriptions for experimental environment settings, datasets, training processes, and evaluation pro- cesses README. md. In the Appendix S
we also list all the detailed hyperparameters.
ACKNOWLEDGMENTS
This work was partly supported by the NRF grant 2No.2022RL. 0022RLA2240023000 10%) and IITP grants 10.22020-0-0136 (20% Yonsei AI), N0.2002--00133 (10%, SNU AD,, 1--2022--0000 (10%), No.2022-0-008 (40%), No.2021-0-0200 ment (MSIT). Sanghyun is partially supported by the Google Faculty Research Award. Work is done when J. Choi and N. Park with Yonsei University. J. Choi and N. Park are co-corresponding authors.
(10%, AI Innov. Hub)) funded by the Korea govern-
REFERENCES
Saeed Anwar and Nick Barnes. Densely residual laplacian super-resolution. JEEE Transactions on Pattern Analysis and Machine Intelligence, 44(3): 192-1204, 2020.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. ponents with random forests. In European Conference on Computer Vision, 2014.
Food-101 mining discriminative com-
Dingding Cai, Ke Chen, Yanlin Qian, and Joni-Kristian Kämäräinen. Convolutional low-resolution fine-grained classification. Pattern Recognition Letters, 119:166-171, 2019.
Patryk Chrabaszez, llya Loshchilov, and Frank Hutter. downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint arXiv:1707.08811 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 / 1929, 2020.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cans are biased towards texture; increasing shape bias im- proves accuracy and robustness. arXiv preprint arXiv: 12233,, 2018.
John Guibas, Morteza Mardani, Zongyi Li, Andrew Tao, Anima Anandkumar, and Bryan Catan- zaro. Adaptive fourier neural operators: Efficient token mixers for transformers. arXiv preprint arXiv:2111.13587 2021.
Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita. Deep back-projection networks for super-resolution. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 16641666660 1673, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. arXiv preprint arXiv: 1903. /2261, 2019.
10

---

Published as a conference paper at ICLR 2024
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- nne-graindd categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR- 13), Sydney, Australia, 2013.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprini arXiv:2010.08892 2020a.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differ- ential equations. arXiv preprint arXiv:2003.03385, 2020b.
Zhuang Liu, Hanzi Mao, Chao- Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1 11976-18 1986, 2022.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.515/ 2013.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008
Hongyi Pan, Diaa Badawi, Chang Chen, Adam Watts, Frdem Koyuncu, and Ahmet Enis Cetin. Deep neural network with walsh-hadamard transform layer for ember detection during a wildfire. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2022.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In JEEE Conference on Computer Vision and Pattern Recognition, 2012.
Md Ashiqur Rahman, Zachary Ross, and Kamyar Azizzadenesheli. U-no: U-shaped neural oper- ators. arXiv preprint arXiv:2204. / 127, 2022.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJVV,, 115(3): 2015. doi: 10. 10.10077/111233-15---8181
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. arXiv preprint 1rXiv:1409.1530 2014.
Christian Szegedy. Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826. 2016
Hrishikesh Viswanath, Md Ashiqur Rahman, Rashmi Bhaskara, and Aniket Bera. Nio: Lightweight neural operator-based architecture for video frame interpolation. arXiv preprint arXiv:2211.10777 2022.
Haohan Wang, Xindi Wu, Zeyi Huang, and Eric Xing. High-frequency component helps explain the generalization of convolutional neural networks. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 6684-8694, 2020.
Hao Wu, Wei Xion, Fan Xu, Xiao Luo, Chong Chen, Xian-Sheng Hua, and Haixin Wang. Past- net: Introducing physical inductive biases for spatio-temporal video prediction. arXiv preprint arXiv:2305.1142.1 2023.
Tiantian Yan, Haojie Li, Baoli Sun, Zhihui Wang, and Zhongxuan Luo. Discriminative feature min- ing and enhancement network for low low-resolution fine- grained image recognition. IEEE Transac- fions Circuits and Systems for Video Technology, 32(8)=5319-5330 2022a.

---

Published as a conference paper at ICLR 2024
Tiantian Yan, Jian Shi, Haojie Li, Zhongxuan Luo, and Zhihui Wang. Discriminative information restoration and extraction for weakly supervised low-resolution fine-grained image recognition. Pattern Recognition, 127: 127:108629, 2022b.
Fanghua Yu, Xintao Wang, Mingdeng Cao, Gen Li, Ying Shan, and Chao Dong. Osrt: Omnidirec- tional image super-resolution with distortion-aware transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13283-13292, 2023.
Xiaobin Zhu, Zhuangzi Li, Xianbo Li, Shanshan Li, and Feng Dai. Attention-aware perceptual enhancement nets for low-resolution image classification. Information Sciences, 515:233-247, 2020.
12

---

Published as a conference paper at ICLR 2024
APPENDIX
A
VARIOUS RESOLUTION IMAGES
There are no image datasets prepared for various resolutions. Therefore, creating images in various resolutions through appropriate resizing method is important our task. Chrabaszcz et (2017) showed that low-resolution images resized by some interpolations have the characteristies of the original image. Among various interpolation methods, the bilinear interpolation method retains the most characteristics of the original images. Therefore, we constructed low-resolution image datasets applying the same bilinear interpolation method the original dataset.
After constructing the low-resolution datasets, we examined the performance decrease with various check the performance decrease, the low-resolution images were resized back to the target resolution (224) with various interpolation methods. Table that the bicubic interpolation has the smallest performance decrease. As mentioned Section I images resized by the bicubic interpolation were used as input to various pre-trained models.
pre-trained models.
Table Classification accuracy drops when scal- ing up low-resolution images, i.e., 32, 64, 128, to target resolution, i.e., 224 or 299, using existing image interpolation methods. We report the best interpolation method's ' results.
Table 5: The top-1 accuracy of pre-trained ConvNeXt-Tiny in various resolutions when re- sized up with each interpolating method.
Resolution
Method
Ace. (%)
64
128
224
(8)
Nearest
Top-1
0.9
18.17
58.27
82.52
Bilinear
Top-1
29.47
60.36
76.98
82.52
ResNeci-18
Relaiive
Bicubic
Top-1
34.51
64.88
77.27
82.52
Inception
Top-1
Arca
Top-1
0.9
34,92
74.42
82,52
Top-1
Nearest-exact
Top-1
0.9
18.27
58.27
82.52
Relaiive
09.37
ComNeeXi-Ta Tany
77.27
B DETAILED DESCRIPTION OF DATASETS
The following are detailed descriptions of the datasets used in our experiments. The number of classes, train images, and test images are organized in Table
B. IMAGENET-IK
ImageNet- Ik (Russakovsky al. 2015) the most widely used dataset in image elassification along with iNaturalist. In particular, the most widely used dataset, ImageNet Large Scale Visual Recog- nition Challenge 2012 (ILSVRC2012), contains 1000 categories of 1.2 million images. ImageNet is built with the WordNet hierarchy which means each category is explained by several words or phrases. Thus, its goal is to offer set of qualified images in the same hierarchical words or phrases.
B.2 FINE-GRAINED DATASETS
Fine-grained Datasets contain classes from a single category, such as cars, birds, flowers, aircraft, or food, and these more difficult to classify where detailed elements must be considered.
Stanford Cars Stanford Cars (Krause et al. 2013) is a dataset containing car images. There are 196 classes with information make, model, and year (e.g., 2012 BMW M3 coupe).
Oxford-IITT Pets Oxford-IIIT Pets (Parkhi et al. 2012) contains images of cats and dogs with 37 categories. The label consisted of species and breeds. This dataset also provides bounding box for segmentation tasks.
Flowers Flowers (Nilsback & Zisserman, 2008) includes 102 categories of flowers that mainly found in the United Kingdom. Even flowers belonging to the same categories have large variations in pose, size, and light.
13

---

Published as a conference paper at ICLR 2024
FGVC Aireraft FGVC Aireraft (Maji et al. 2013) is used in the fine- fine-grained recognition challenge 2013 (FGComp). Aircraft with various sizes, purposes, driving forces, and other features are classi- fied by those fine- tine-grained features. The 100 categories of aircraft include the make name of the model series.
Food-101 Food-101 (Bossard et al. 2014) contains 101 kinds of food images, which were originally provided to be used for RandomForest. The number of images is much larger compared to other fine- grained datasets. The images in this dataset are already rescaled and contain noises ranging from color intensity to wrong labels.
Table 7: The number of classes, train images and test images.
Datasets
classes
train images
test images
ImageNet-Ik
1000
1,281,167
50,000
Stanford Cars
196
8,144
8,041
Oxford-IIIT Pets
3,680
3,669
Flowers
102
1,020
6,149
PGVC Aircraft
100
3,334
3,333
Food- 101
101
75,750
25,250
C TRAINING ALGORITHM
We describe two-stage training algorithm in Algorithm. We first stage for training with tar- get resolution in the first while loop, followed by the second while loop for training with various resolutions.
Algorithm 1: 2-stage Training of PAC-FNO
Input: Target resolution data Diargets Number low-resolutions N. Low resolution data (Diway. Divay Class label y. Prodicted class label å. First phase iteration number K ficsis Second phase iteration number K second. PAC-FNO parameters Pre-trained backbone model. parameters Op. cross entropy loss CE
Initialize Po;
k 4
/+ First training stage
while K
for each mini-batch B S Diarget do
(9:) Backhone(PAC-FFNN 0.):0ak
Train and 0, CE(L9) E 1
end
Second training stage Karcone do for / 4- to N do
for each mini-batck
Diew, do (9:) +- Backbene(PAC-FFOOO (..)
Train 0, with CB(1S, 11 ly: 1()))
for each mini batch B _ Drarget do
19: [9 Bacboon(PACCFNOOBBB.. 0.)00,):
Train
R (E( ():
cnd
D BACKBONE MODELS
We propose plug-in in module for multi-scale classification. Therefore, applied various pre- trained classification models from CNN-based to ViT-based classification models. All classification models were trained with ImageNet-14 -k from scratch with the settings in Table TORCHVISION provides all such pre-trained models.
ResNet-18 Residual network (ResNet) is model that applies the concept of residual connection to CNN architectures. ResNet-18 18 (He et al. 2016) consists of 18 convolutional blocks and residual layers. It is the most fundamental model image classification.
14

---

Published as a conference paper at ICLR 2024
Table 8: Recipe for the pre-trained models provided in TORCHVISION
Model
ResNect-18
ViT-B16
ConvNeet--iny
Epochs
90
300
600
Batch size
32
512
128
Optimizer
sgd
adamw
adamw
Learning rate (Ir.)
0.1
3e-3
le-3
Weight decay
le-4
0.3
0.05
Ir. scheduler
steplr
cosineannealinglr
cosineannealinglr
warmup method
linear
linear
Ir. warmup epochs
Ir. warmup decay
0.033
0.01
Amp
O
Random erase
0.1
Label smoothing
0.11
0.1
Mixup alpha
0.2
0.2
Auto augment
ta wide
Clip grad
Ra sampler
Cutmix alpha
1.0
1.0
Model-ema
O
O
Norm weight decay
0.0
Train crop size
224
224
176
Test resize size
256
256
232
Test crop size
224
224
224
Ra reps
Inception-V3 Inception networks (Szegedy et al.) 2016) are one of the most popular classification models, which stacks deep convolutional layers in efficient way. They proposed techniques such as the concatenation of convolutional layers with different kernel sizes, kernel decomposition, and backpropagation with auxiliary classifier for efficient training. We used the pre-trained Inception- V3 model provided by TORCHVISION, and there is no training recipe for training from scratch.
VTT-B16 Vision Transformer (Dosovitskiy al. 2020) (VIT) is model that uses transformers based on a self-attention architecture in the image domain. It divides an image into patches and calculates all patch-by-pate -itt-hhypppcchhhc relationships through self-attention. Because of these calculations, much memory and training time are required. Among those ViT-based models, ViT-B 16 has the smallest model size and divides image into 16 patches.
ConyNeXt-Tinn Tiny ConvNeXt (Liu et al. 2022) is one of the most recent models with good per- formance using only convolutional networks. shows better performance with fewer parameters compared to VIT. To achieve good performance, it utilizes several advanced training schemes for convolutional networks. ConyNeXt-Timg refers the smallest model size in the ConvNeXt family.
E HYPERPARAMETERS
In Tables and 10 we list the key hyperparameters our experiments for each dataset. Our Appendix accompanies some trained checkpoints and one can easily reproduce.
Table The best hyperparameter of our main experiments (ImageNet-1k)
ImageNer-tt Ik
ResNet-18 18 Incepsion-V3 Vision Transformer ComNNeXi-Tan Tiny
# of panillel AC-FNO blocks (ro)
2
2
stages
First phane training
le3
le3
2o-4
Second plose miining
16-6
166
16-6
26-6
15

---

Published as a conference paper at ICLR 2024
Table 10: The best hyperparameter of our experiments on the fine-grained datasets.
StunfordCrrs Oxfod-HIIT Flowers
Food-101
of ponalle AC- blocks (m)
2
2
of (r)
1
Firs plase training
le:3
le3
1e3
1e3
1e:
phuse training
le-5
F EVALUATION
F.1 EXPERIMENTAL SETUP
We run our experiments on a machine equipped with Intel i9 CPUs and Nvidia RTX A5000/A6000 GPUs. We implement PAC-FNO using Python v3.8 and PyTorch v1.12.
F.2 NUMBER OF PARAMETERS OF PAC-FNO
We report the number of parameters of PAC-FNO according to backbone models and datasets.
Table 11: The number of parameters according Table 12: The number of parameters according to to backbone models.
datasets
InmoeNet-I ResNet-18 Inception- V3 Vision Transformer ConvNeXi- Tiny ConNNeXi-Tag Taay StanfordC'ars Oxford-IIIT Hlwers KGVC Aireraft Food-101
PAC-FNO 1091M
+1.61M
+0.91M
+3.65M
PAU-FNO
+3.65M
+3.65M
+3.65M
+3.65M
+3.65M
parameters
panamelars
F.3 ADDITIONAL SUPER-RESOLUTIO METHODS
In this section, we report experimental results with the additional latest super-resolution bascline. We fine-tune pre-trained classification model with low-resolution images that are upscaled by the super-resolution model. We note that we used the latest super-resolution model.
In Table 13, combining the super-resolution and fine-tune methods shows better performance than simply upscaling low-resolution images to the target resolution using the super-resolution model. Even at 32 resolution, it shows slightly better performance than PAC-FNO. However, this super- resolution method has two major drawbacks. The first drawback is the model size. DRPN's x8 upscaling model has similar number of parameters the the pre-trained model size, i.c. 23.2 v8.28.59M. Second, a model is needed for each resolution. In other words, upscaling models for x8, x4, and x2 needed to handle 28, 56, and 112 resolutions, respectively. In contrast, proposed PAC-FNO can handle images of all resolutions with an additional 3,65M network and shows good performance.
Additionally, we verify the superiority of PAC-FNO by providing a comparison with a method equipped with the latest super-resolution model. OSRT (Yu et al. (023) is state-of-the-ar model in the super-resolution domain but it does not support x8 upscale. Therefore, we only use the x2 and x4 upscale models of OSRT. As result, PAC-FNO shows better performance than OSRT and OSRT (fine-tune) methods.
Table 13: Results with the additional latest super-resolution method. For the experiment, we used ConvNeXt-Te Tiny as a pre-trained model. (Left: ImageNet-Ik, lk, Right: ImageNet-C/P Fog)
Mudel
Mihhod
Maric
28
112 128 224 Se
64
Medal
Mahhod
Maric
28
32
Sn
64
112
128 224
Rmuze
Arp 183
17. TES
7511
Rouze
Arpl-Ac (81
8.12
T0N
11.7
10's
Fise-une
Tipl-Ace (9)
40.2
62.
65.5
76.0
764
10.7518
Fise-wne
Aepl-Aes (9)
23.2
28.2
47.5
51,4
6100
62.263.0
DRPN
Pocanttesss
Arp
63.1
ImpinAre 181
23,21
0357
1100
1.00
DRPN
CondNeXi- [Fite
OSKT
*f Parameters (M)
Topt-Aus (S)
Parameters
11.93
61.4
11.79
754
125
Condleei-Tleg
OS87
of Patameers (MI
TeprsAre (5)
Parameters
10.33
11.93
1944
11.79
5.85
37.9
984
OSKT
Topl-Ace (96)
78.2
78.4
61.2
OSRT
Topi-Ass (9)
42.3
5644
59.4
(Fise (usc)
(Fiuc-lunc) Paramcioon
PAC-FNO
AopitAAc
PAC-FND Topl-Are (15)
3.65
16

---

Published as a conference paper at ICLR 2024
F.4 ADDITIONAL FINE-GRAINED DATASETS
Tables and 15 show the performance of the remaining fine-grained dataset, Stanford Cars. PAC- FNO shows good performance in most cases, especially at low-resolution compared to other models. Tables [6] and 17 show the performance of the Oxford-IIIT Pets with the ViT model. PAC-FNO shows good performance than FNO at all resolutions.
Table 14: Top-1 accuracy on low- resolution.
Table 15: Relative accuracy on low row-rusouuiion.
Dataset
Mothod
Resoluttin
Datnset
Method
Resolution
28
32
56
64
112
128
224
28
32
56
64
112
128
224
Resize
22.5
66.3
73,6
89.6
91.5
Resine
16.1
34.5
72.4
96.4
97.9
Fine-tune
12.3
66.5
70.5
88.5
91.2
92.4
92.6
Fine-tune
13.6
71.9
76.
95.6
98.5
99.8
1001
DRLN
1.88
36.1
$7.0
91.5
DRIN
2.16
39.4
95.1
100
Standfford Cars
DRPN
FND
41.5
11.5
19.2
84.2
67.8
753
91.0
90.9
92.6
03.9
91.5
Standford Cars
DRPN
INO
51.9
12.7
20.3
92.1
72.2
80.2
96.8
99.3
98.6
100
100
UNO
57.4
66.2
85.9
67.8
91.6
92.1
92.3
71.3
99.2
99.8
A-FNO
80.3
PAC
76.6
89.0
92.8
PAC-FNO
75.3
81.9
98.2
96.3
09.0
99.3
Table 16: Top-1 accuracy on low-resolution.
Table 17: Relative accuracy on low-resolution.
Dataset
Method
Resolution
Model
Method
Resolation
32
128
125
Ostovd-IU Pets
PAC-FNO
FNO
26.3
40.3
33.0
46.2
38.1
69.0
64.8
72.5
86.8
K2.9
89.2
85.8
91.3
92.2
Oxford- Pers
PAC-FNO
FNO
43.7
28.8
50.1
26.1
63.6
74.8
71.0
78.6
90.8
94.1
94.0
96.7
100
100
F.5 ADDITIONAL METRIC FOR IMAGENET-IK K AND FINE-GRAINED DATASETS
Table 18 and 19 show the relative accuracy of low-resolution images generated by ImageNet- k and fine-grained datasets. In ImageNet- 1k, the performance of PAC-FNO is the best in most of the datasets, and especially in lower resolution. In fine-grained datastes, PAC-FNO shows good performance in all cases. other words, PAC-FNO works very well for low-resolution image classification.
Table 18: Relative accuracy on low-resolution Table 19: images generated from ImageNet-Ik
Relative accuracy on low-resolution images generated from Fine- -eeeegraindd datasets
Model
Method
Resolution
Datase
Resolation
28
32
56
128 224 299
112
128
Resize
23.7
100
Resize 31.4
74.8222
95.9
100
1.6 3.3
15.7
53.1
100
Fine-tune 34.8 43.8
78,0445
97.097.5
1000
21.5
DREN
3,6
39.3
94.0
100
R1
DRPN
FND
45.3 79.7
57.3645543887.666.3
96.47777
96.7
100
100
Oxtoed HITT
DRIN
FNO
44.3
89.9
98.8
100
UNO 57.565,331,988333
96.7 100
63.272.387.999
A.FNO
PAC-PNO-0X 86 89.5
PAC
Resise 21.628.562..69 7
79.9 99.9 92.9
100
Finestme
Resize
46.3
96.5
96.097.7
Finz-tune 51.1 61/66...2.29 90, 94.1 94.6
100
DRLN
10.3
35.7
95.2
100
Inception
FND 62.456.933744 89.3 05.3
54.6662.284.6
T00
Flowers
DRPN
FNO
26.3
67.3
144.66627769449999
92.8
25.6
99.1
100
100
A-FNO
44.6 79.6699633333 83.7 91,8 96.3
100
UND
24.1 31.467.1 74.6 94.55449
100
MMC-FXX6229900007222 90.2 97.1 98.1
100
ANO
29.2 36,6 76.2 93.8 96.6 100
Resize 51,3
58,455 9110 96.3
100
PAC-FNO 78L5 82.6 92,8944 99.8 100
49.1 59.8
Resize 3.1 3.5 34,352,489) 89.1 19.2 100
DRLN
4.6
51.8
94.9
100
Fine-tuee 20.3550.5
96.795.2 100
BI6
DRPN
PNO
49.1
54.163.184.7
61.1
(9.158883. 86.3
90.0
98.6
V6.7
97.9
9W.
100
100
RGVC Airemat
DRLN
DRPN 23.1
32364.1
2.1
11.499 TIX
1954
96.6
I00
PAC-PNO 58.5 65.5 36.7 89.9
A.PNO
69.1
93.2
97.6
99.2
98.7 100
PAC-FNO 25.6826685.6
A.FO
92.9
Resize 33.341.873. 78.7
90.9
93.7 100
Resive 44.052.833.888000
Fine 49.1 76.2 80.4
93.9933..3999999 98.7
100
Fine cune 55.059..388.00008
100
DREN
7.7
42.5
TIK
CoevNeXi-Tiny
DRPN 49.3
PND 54.061.788.11
82.7
88
96.6 97.7 100
96.2
100
Fadd-101
DRPN
FNO
6012
48.0
56.50.66588888 68..588889...
89.6
97. 17.0 100
65.3
100
UNO
72.7
72.79199149.
98.1
100
UNO
57.5 64.5 $5.9 99.1 96.997.8 100
61.8
68.566.577.220
A-FNO
52.2
01.256661..9955 95.995500000 05.099..111 100
PAC-PNO
72.3 77.5 91.4
93.5
98.4
PAC-TNO
82.9 96,1 94,6 96.1 99.7 100
17

---

Published as a conference paper at ICLR 2024
F.6 ADDITIONAL NATURAL INPUT VARIATIONS
Table 20 shows the remaining input variation results of ImageNet-C/P Hendrycks & Dietterich 2019 For the remaining input variations, we report results at 32. 64, 128, and 224 resolution, ex- cluding SR models whose performance appears to be meaningless. In most cases, PAC-FNO shows good performance at 32 and 64 resolution, and Fine-tune shows good performance at 128x x 128 and 224 resolution. However, at 128 and 224, there is performance difference of up to 5% (Glass noise 44.2% vs. 39.5% at 128 x 128 resolution), but at 32 and 63, there a performance difference of up to 47% (Pixelate 44.2% vs. 39.5% at 32 x 32 resolution). In other words, PAC-FNO shows good performance by large margin at low-resolution.
Table 20: Performance of PAC-FNO on the input variation tasks. We show the top-1 accuracy of the ConvNext- Tiny model on remaining input variations, chosen from ImageNet-C/P (Hendrycks & Dietterich 2019).
Resolution
Resolution
Resolution
Variation
Model
Variation
Model
Variation
Model
64
128
224
32
128
224
128
224
Resize
11.3
37.9
55.3
47.9
Resize
10.6
36.5
54.0
48.1
Resize
12.2
40.0
56.8
53.2
Fine tume
11.6
39.5
57.4
51.5
Fine
11.5
39.6
58.8
55.9
Fine tune
12.6
41.5
59.1
56.6
Gaussian
FNO
38.4
47.1
46.5
4344"
Motion
FNO
36.3
46.9
48.7 474
Flastic
FNO
38.3
49.7
52.1 51.3
Noise
UNO
42.8
54.6
56.5
52.1
Noise
UNO
41.3
50.6
49.3 47.9
Transform
UNO
45.4
52.8
50.5
48.4
A-NNO
37.8
43.1
44.0
42.8
A-ENO
37.6
44.9
42.6 41.3
A-FNO
41.3
48.2
47.8
45.4
PAC-FNO
48.0
54.8
56.5
52.3
PAC-FNO
50.4
55.3
54.1
PAC-FNO
57.3
55.8
53.5
Resize
11.3
37.3
53.8
45.4
Resize
10.3
25.7
43.0
43.6
Resize
14.5 81.0
63.8
53.2
Fine tume
11.6
39.0
56.2
49.5
Fine
10.9
26.8
46.4
48.9
Fine-
15.1 52.9
66.8 56.6
Shot
FNO
38.2
45.9
41.9
41.T
7nom
FNO
35.2
38.5
37.3 37.0
FNO
48.6
66.7
62.5 42.9
Noise
UNO
41.2
53.6
55.2
50.5
Noise
UNO
33.5
40.2
39.4 39.5
Pixelate
UNO
40.8
70.9 64.6 52.2
ANO
38.1
42.9
42.8
41.2
A-FNO
36.0
35.9
33.8 33.5
A-NNO
53.2
67.8
58.9 40.8
PAC-FNO
54.0
55.0
50.0
PAC-FNO
46.5
44.6
PAC-FNO
73.2
67.9
54.4
Resize
10.8
36.9
53.2
44.6
Resize
4.9
20.2
46.6
49.6
Resize
12.9 44.4
64.3 62.4
Fine-tune
11.3
38.5
55.5
48.5
Fine-tune
5.2
21.8
49.0
52.5
Fine-tune
13.3
45.9
66.3 65.5
Impulse
TNO
37.2
45.2
41.8
38.5
PNO
20.3
31.1
41.6
45.9"
Jpeg
FNO
45.2
62.5
65.2
63.7
Snow
Noise
UNO
53.5
48.9
44.0
43.6
Compression
62.2
ANNOO
36.5
41.3
39.3
37.2
A-FNO
18.1
27.4
37.5
40.4
A-FNO
50.9
64.8
63.9
61.1
PAC-FNO
54.1
50.2
PAC-FNO
49.4
PAC-FNO
64.8
63.0
Resize
10.0
32.2
48.3
43.2
Resize
6.5 30.8
54.9
54.1
Resize
12.1 41.2
59.2
53.5
Fine-tune
11.1
36.9
56,9
55.9
Fine-tune
6.9
32.3
56.7
57.0
Fine-tune
12.5
42.6
66.2
57.1
Defocus
FNO
35.8
44.9
47.4
47.6
ENO
27.8
42.3
49.0
50.9
Speckle Noise
FNO
41.6
52.7
3133 48.8
Frost
Noise
UNO
50.5
47.3
53.6
UNO
59.9 56.2
A-FNO
38.6
47.2
47.4
47.2
A-ENO
22.6
37.7
45.3 46.0
A-FNO
42.7
49.6
49.8 48.3
PAC-FNO
37.5
56.2
PAC-FNO
56.9
56.9
PAC-ENO
60.3
Resize
11.7
33.5
38.7
31.5
Resize
8.1 40.4
62.9 59.5
Resize
10.4
34.8
51.1 46.9
Pine-tune
12.7
36.9
44.2
38.8
Fine-tume
8.5
41.7
68.8 64.7
Fine-tune
11.5
39.1
59.0
58.4
Glass
FNO
39.8
433
549
32.4
FNO
31.8
53.3
63.3 63.5
Gaussian
FNO
37.3
47.3
30.0
50.8
Noise
UNO
43.4
33.4
30.7
Contrast
UNO
65.2 62.8
Blur
UNO
54.0
53.3
50.2
A-FNO
42.8
39.6
28.7
26.1
A-ENO
34,8
55.6
62.1
59.2
A+FNO
40.1
50.2
51.3
51.2
PAC FNO
50.9
39.5
35.1
PNO
62.3
PAC FNO
60.7
59.4
Table 21: Performance of PAC-FNO according to low and high-frequency filter. We report top-1 accuracy on low-resolution images generated from ImageNet-IK in ConvNeXt-Tiny.
ImageNet-Ik
32
64
128
224
ImageNet-C/P Fog
32
64
128
224
PAC-FNO
PAC-FNO
(low pass filter)
53.5
71.4
78.7
79.0
(low pass filter)
18.0
41.7
52.4
54.4
PAC-FNO
PAC-FNO
(high pass filter)
21.6
49.4
68.2
74.8
(high pass filter)
5.92
23.0
43.2
50.2
PAC-FNO
58.9
74.5
80.2
81.5
PAC-FNO
25.4
48.2
60.1
62.8
F.7 ADDITIONAL ABLATION STUDIES
We provide an analysis of the impact of low and high-frequency information on accu- macylgeneralization through ablation experiments. Table 21 shows thah compared to PAC-FNO, using low-pass and high-pass filters results in lower accuracy and generalization. When using a high-pass filter, it is expected to show good performance in ImageNet-C/P Fog, but since the perfor-
18

---

Published as a conference paper at ICLR 2024
1.0
Normalized Magnitude
h
h
0.8
h
h
0.6
h2
0.4
0.2
0.0
8
0 Frequency
8
Figure 7: Comparison of spectral responses according to the configuration of the AC-FNO block. We test the ConvNeXT-Tiny backbone model on ImageNet-lk Ik and visualize only the hidden vector of the first layer (m 1) for the hidden vectors h, In the case of parallel (solid line), there four hidden vectors (h), n 2,, 4)), and the case of serial (dashed line), there one hidden vector (h) n (1)).
mance even on clean images is not good, does not show good performance terms of generaliza- tion. Therefore, only PAC-FNO, which uses both low and high-frequency components, shows good performance in terms of accuracyfgeneraalizatin.
F.8 EFFECTIVENESS OF PARALLEL ARCHITECTURE
In this section, we show the efficacy of the parallel configuration of the AC-FNO block by visualizing which frequencies captured. For fair comparison, visualize the first layer output, which contains the most information an original input sample, for the following two settings: AC-FNO in our proposed parallel configuration and AC-FNO in serial configuration. Figure7Jshows spectral responses according to the configuration of the AC-FNO block. The farther it is from the center, the higher its frequency is.
In Figure 7. We show that the parallel configuration, each hidden vector not only captures the low-frequency components but also captures the high-frequency components. Moreover, their fre- quencies are sometimes complementary to each other. In particular, h has large normalized mag- nitudes at high frequency ranges, which means that h3 captures high-frequency components well. On the other hand, the hidden vector of the serial configuration (dashed line) is concentrated low-frequency (center).
Additionally, since the parallel configuration of the AC-FNO block captures both high and low- frequency components, it also shows good performance for image degradations as shown in Figure Figures 3d and [e are visualizations the frequency domain. In other words, PAC-FNO consider high-frequency information well in those cases.
19

---

Published as a conference paper at ICLR 2024
(a) Original
(b) Original with Fog
(c) Original with Gaussian Blur
50
50
100
100
125
125
150
175
175
50
100
150
200
50
100
150
200
(d) Frequency domain of fog
(e) Frequency domain of Gaussian blur
Figure 8: Visualization of image degradation. (d) and (e) visualizations of degradation without clean images e.g., (b)-(a) and (c)-(a) in the frequency domain.
F.9 FLOPS AND RUNTIME
We report the FLOPS and runtime data different resolutions
Table 22: Results of FLOPs and runtime ImageNet- Ik in ConvNeXt-Tine.
Resolution
ImageNet-Il lk
Method
Metrics
56
112
224
GFLOPs
8.96
8.96
8.96
8.96
Resize
Runtimes (s)
0.006
0.006
0.006
0.006
GFLOPs
8.96
8.96
8.96
8.96
Fine-tune
Runtimes (s)
0.006
0.006
0.006
0.006
GFLOPs
180.66
412.05
1200.50
8.96
DRLN
Runtimes (s)
0.378
0.498
0.913
0.007
GFLOPs
1220.42
576.94
387,88
8.96
DRPN
Runtimes (s)
0.1532
0.164
0.171
0.007
ConvNeXt-Tins
GFLOPs
9.78
9.78
9.78
9.78
FNO
Runtimes (s)
0.016
0.016
0.016
0.016
GFLOPs
9.10
9.10
9.10
9.10
UNO
Runtimes (s)
0.018
0.018
0.018
0.018
GFLOPs
8.96
8.96
8.96
8.96
AFNO
Runtimes (s)
0.010
0.010
0.010
0.010
GFLOPs
8.98
8.98
8.98
8.98
PAC-FNO
Runtimes (s)
0.013
0.013
0.013
0.013
20


Reviews:
[{"review_id": "1Zfm29TuCw", "reviewer": "Reviewer_nwGj", "summary": "This work has developed a neural network architecture for image recognition that is designed to address the influence of complex degradation factors. It aims to capture both low-frequency and high-frequency components to balance accuracy and generalization. The authors first propose to discard the low-pass filters in the existing FNO structure to retain all frequency components. Subsequently, a parallel structure is introduced to further enhance the utilization of frequency domain information. Finally, the authors design a two-stage training strategy to ensure performance stability.", "questions": "Please refer to the Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The overall paper has a clear logical structure, and the explanation of the methodology and the presentation of the constructed mechanisms are intuitive and easy to understand.\n2. The author provides a sufficiently detailed explanation for the motivation behind each component in PAC-FNO.\n3. The problem that this work aims to address holds a certain degree of practical application value.", "weaknesses": "1. The abandonment of the low-pass filter is one of the main innovations in this work. Although the author provides an explanation for the motivation behind this operation, it is still recommended that the author conduct ablative experiments to analyze the impact of low-frequency/high-frequency information on accuracy/generalization.\n2. As for parallel architecture, the relevant experimental results have indeed proven its effectiveness. However, the explanation of parallel architecture in the method section appears somewhat lacking. It is hoped that the author can provide further analysis of the mechanism that enables it to be effective.\n3. In terms of comparative experiments, the methods used by the author for comparison appear to be lacking in both quantity and novelty. The comprehensiveness of the complex scenarios considered by the author is commendable, but it is hoped that the author can still increase the comparison results with more advanced works to more effectively validate the superiority of the proposed method.\n4. The author mentions the advantages of this work in terms of efficiency, but it seems that no experimental analysis related to efficiency has been provided (such as FLOPs and runtime on data at different resolutions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1Ae6eUADAv", "reviewer": "Reviewer_WCy8", "summary": "This paper proposes a novel neural network model, parallel-structured and all-component Fourier neural operator (PAC-FNO) to address visual recognition under low-quality images. By operating in the frequency domain, PAC-FNO is able to learn the semantics of images in various resolutions and/or natural variations for challenging image recognition with a single model. The proposed PAC-FNO is capable of handling both low-resolution and input variations typically observed in low-quality images with a single model. It can also be attached to a downstream visual recognition model, which is beneficial for handling multiple input variations at once and minimizing the changes in the downstream model during fine-tuning. In the evaluation with four visual recognition models and seven datasets, the proposed PAC-FNO achieves excellent performance.", "questions": "Please see the weaknesses above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is organized well.\n2. Extensive experimental results are provided to illustrate the effectiveness of the proposed method.", "weaknesses": "1. Is there a more advanced choice for the SR baseline model used for comparison in your experimental setup? This will affect the fairness of the performance of your experiment?\n2. It can be found that in ViT-B16, PAC-FNO shows not very good results at all low resolutions compared to other methods. What caused this phenomenon to occur? Is your method also unfriendly to other Transformer methods?\n3. The ideal low-pass filter in the FNO block removes detailed image signals that play an important role in classification in the fine-grained dataset. Is this conclusion applicable to Transformer based image classification methods? More quantitative results should be provided to confirm the universality of the proposed method.\n4. The ablation experiments about the results of the zero-padding operation and the exclusion of the low pass filter need to be completed to explain the design of the AC-FNO block.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a96vmp8Zcb", "reviewer": "Reviewer_pDdt", "summary": "The paper presents a novel solution named PAC-FNO for image recognition, demonstrating the ability to simultaneously handle images of varying resolutions and resist the impact of various types of input-induced natural variations within a singular model in recognition tasks. The proposed parallel-structured and all-component Fourier neural operator (PAC-FNO), building on the resolution invariance of FNOs in the frequency domain, notably eliminates the ideal low-pass filter found in vanilla FNOs. Additionally, it transforms the traditional serial architectures into a parallel structure, thereby considering a broader range of frequency components, retaining high-frequency details, and notably enhancing performance, especially in fine-grained datasets. The proposed approach introduces a two-stage training method that fine-tunes pre-trained image recognition models in conjunction with PAC-FNO, allowing the acquisition of commonalities among various input resolutions with minimal modifications to the backbone classification network. Through conducted experiments, the authors effectively showcase the performance of PAC-FNO, significantly improving accuracy in comparison to existing baseline models. The manuscript is well-written, and the experiments conducted are comprehensive and convincingly articulated.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper exhibits a high level of innovation. Although neural network operators based on the Fourier domain transformation for learning, due to their excellent characteristics in resolution invariance, have been recognized and applied in various areas, especially for enhancing pre-processing operators in variable-resolution input networks. However, the authors, motivated by the rational desire to retain high-frequency image details, proposed for the first time to eliminate the inherent low-pass filters in the model. Additionally, they introduced a popular parallel structure similar to Multi-head Self-Attention, further enhancing the network's performance while expanding the design philosophy of relevant operators.\n\n2. The presentation of this paper is professional and fluent. It has almost no expression errors and clearly elucidates the authors' contributions.\n\n3. The paper conducted extensive and meticulous experiments, utilizing seven image recognition benchmark datasets and applying the operators to four different backbone networks. The authors closely follow the cutting-edge developments in the field, employing more advanced VIT and ConvNeXt for experimentation, which makes the results highly persuasive.", "weaknesses": "1. Although the author compared super-resolution (SR) models for variable resolution inputs, the compared SR models are outdated and lack representation across various upscaling factors for super-resolution reconstruction. The field of super-resolution has seen significant advancements recently; thus, it is recommended to select more appropriate comparative algorithms.\n\n2. The primary advantage of Fourier Neural Operators (FNOs) lies in their use of frequency domain processing for resolution invariance. As a learnable enhancement operator, it's expected to exhibit some resilience to input natural variations. However, the author hasn't provided a detailed and explanatory analysis of the mechanisms where the operator shows robustness against natural variations. Moreover, the chosen input variations in the experiments, like fog, brightness, spatter, and saturate, represent basic degradation scenarios that can be addressed without deep learning methods. Therefore, regarding resilience to input natural variations, this might not be sufficiently emphasized as a highlight of the paper. The paper suggests exploring degradation in real-world scenarios in future work, indicating that the authors are aware of the limitations in terms of experimental performance or the algorithm proposed. However, such scenarios represent fundamental problems studied in the field of Image Recognition (IR) and hold significant practical application implications. Actually, certain degradation processes might affect high or low frequency details in the image's frequency domain. For instance, blur involves the loss of high-frequency details, prompting the author to conduct a mechanistic analysis combining frequency domain and degradation processes to enhance this aspect's interpretability.\n\n3. The original intent behind the existence of ideal low-pass filters was to reduce the number of parameters and computational complexity. While the author's innovative design to remove the inherent low-pass filter is intuitively comprehensible, the associated trade-offs are not discussed in the manuscript. It would be beneficial to provide supplementary explanations to demonstrate the worthiness of such a modification.\n\n4. The experiments thoroughly prove the advantages of parallel architectures and claim that this approach encapsulates more frequency components. However, they lack further detailed explanations and justifications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)