SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Classifier. In order to enforce scale-consistency, we need a classifier that outputs a prediction per scale. This motivated the following proposed architecture. Let  $c$  be a classifier with  $M$  classes where  $\hat{\pmb{y}} = c\circ g(\mathbf{x})\in \mathbb{R}^{|\mathcal{R}(\mathbf{x})|\times M}$ .  $\mathcal{R}(\mathbf{x})$  is defined as the set of resolutions smaller than the input resolution in the considered scales  $\mathcal{R}$ . i.e.,  $\mathcal{R}(\mathbf{x}) = \{k:k\leq \dim (\mathbf{x})$  and  $k\in \mathcal{R}\}$ . Here,  $g$  is a scale-equivariant deep-net that extracts features  $\phi = g(\mathbf{x})$  with corresponding DFT of  $\Phi$ . Our proposed classifier has the form:
$$
\dot {\boldsymbol {y}} [ k ] = \operatorname {M L P} \circ \operatorname {P o o l} \left(\mathcal {F} ^ {- 1} \left(\operatorname {P a d} _ {\mathrm {N}} \left(\Phi \left[ - \frac {| k |}{2}: \frac {| k |}{2} \right]\right)\right) \forall k \in \mathcal {R} (x) \right. \tag {19}
$$
where  $\mathrm{Pad}_{\mathbb{N}}$  is a Fourier padding operation that symmetrically pads zero to either side of the DFT to a fixed size N, Pool is a spatial pooling operation and MLP maps the pooled feature to the predicted logits  $\hat{\pmb{y}}[k]$  for each scale; Note the MLP is shared across all scales. As we are sharing the MLP, we need to ensure that the input sizes are identical. Hence, we padded the features  $\Phi$  to a fixed size. Finally, at test-time, we use the output from  $\hat{\pmb{y}}[\dim(\mathbf{x})]$  to make a prediction.
Training. Given a dataset  $\mathcal{T} = \{(\mathbf{x},y)\}$ , we train our model using the sum of two losses. The first term is a standard sum of cross entropy loss  $\mathcal{L}$  over the scales:
$$
\sum_ {k \in \mathcal {R} (\mathbf {x})} \mathcal {L} (\hat {\boldsymbol {y}} [ k ], y). \tag {20}
$$
The second term is a consistency loss to encourage the performance of high-resolution to be better than the low-resolution:
$$
\sum_ {k \in \mathcal {R} (\mathbf {x})} \max  \left(\mathcal {L} \left(\hat {\boldsymbol {y}} [ k ], y\right) - \mathcal {L} \left(\hat {\boldsymbol {y}} [ k - 1 ], y\right), 0\right). \tag {21}
$$
This is a hinge loss that penalizes the model when the cross entropy loss  $\mathcal{L}$  on high-resolution features (larger  $k$ ) is greater than that of the low-resolution features (smaller  $k$ ).
To study the effectiveness of our model, we conduct experiments on two benchmark datasets, MNIST-scale [40] and STL10 [4], following our theoretical setup using ideal downsampling. In this case, the theory exactly matches practice, and our approach achieves perfect scale-equivariance. We also conduct experiments comparing the models' generalization to unseen scales and data efficiency. Finally, we conduct experiments using a non-ideal anti-aliasing filter in down-scaling. Under this setting, our model no longer achieves zero scale equivariance-error. However, we are interested in how the models behave under this mismatch in theory and practice.
Evaluation metrics. To evaluate task performance, we report classification accuracy. Next, we introduce a metric to measure the scale-consistency. Given a sample from the test set, we check whether the cross entropy loss is less than or equal to the classification loss of its down-scaled version. We compute this as a percentage over the dataset and report the scale-consistent rate defined as:
$$
\text {S c a l e - C o n .} = \frac {1}{| \mathcal {T} |} \sum_ {(\mathbf {x}, y) \in \mathcal {T}} \mathbb {E} _ {r} \left(\mathbf {1} \left[ \mathcal {L} (\mathbf {x}, y) \leq \mathcal {L} \left(\mathcal {D} _ {r} (\mathbf {x}), y\right) \right]\right), \tag {22}
$$
where  $r$  is uniformly sampled over the set of scales for which we want to achieve equivariance and  $\mathbf{1}$  denotes the indicator function.
Finally, we quantify the equivariance-error over the final feature map given by a fully trained model on the dataset. The equivariance-error (Equi-Err.) is defined as
$$
\text {E q u i - E r r .} = \frac {1}{| \mathcal {T} | | \mathcal {S} |} \sum_ {\mathbf {x} \in \mathcal {T}} \sum_ {r \in \mathcal {R}} \frac {\left\| g \left(\mathcal {D} _ {r} (\mathbf {x})\right) - \mathcal {D} _ {r} (g (\mathbf {x})) \right\| _ {2} ^ {2}}{\left\| g \left(\mathcal {D} _ {r} (\mathbf {x})\right) \right\| _ {2} ^ {2}}. \tag {23}
$$
Table 1. Accuracy of different models on MNIST-scale (ideal downsampling) with all scales.
Table 2. Accuracy of different models on MNIST-scale (ideal downsampling) with missing scales.
Models	Acc.↑	Scale-Con.↑	Equi-Err.↓	Models	Acc.↑	Scale-Con.↑	Equi-Err.↓
CNN	0.9737	0.6621	-	CNN	0.9842	0.7617	-
Per Res. CNN	0.9388	0.0527	-	Per Res. CNN	0.9763	0.3594	-
SESN	0.9791	0.6640	-	SESN	0.9892	0.8339	-
DSS	0.9731	0.6503	-	DSS	0.9884	0.8105	-
SI-CovNet	0.9797	0.6425	-	SI-CovNet	0.9878	0.6621	-
SS-CNN	0.9613	0.3105	-	SS-CNN	0.9870	0.3593	-
DISCO	0.9856	0.5585	0.44	DISCO	0.9914	0.5371	0.35
Fourier CNN	0.9713	0.2421	0.28	Fourier CNN	0.9820	0.1250	0.23
Ours	0.9889	0.9716	0.00	Ours	0.9888	0.9366	0.00
Here,  $\mathcal{R}$  is the set of all scales over which we enforce equivariance. We report the average equivariance error over the samples of the test set  $\mathcal{T}$ . We note that this equivariance-error differs from the one reported by Sosnovik et al. [42] where they measured the error for the "scale-convolution with weights initialized randomly." Contrarily, we measure the equivariance error from end-to-end over trained models, which more closely matches how the models are used in practice.
Baselines. Following prior works in scale-equivariant neural networks [41, 42] we compare to baselines: DISCO [42], SI-ConvNet [13], SS-CNN [8], DSS [46], and SESN [41]. For the baseline, we follow the architecture and training scheme provided by Sosnovik et al. [42]. We also prepared three additional baseline models: (a) standard CNN, (b) Per Res, CNN where we train a separate CNN for each resolution in the training set, and (c) Fourier CNN [18] which utilizes Fourier layers.
5.1 MNIST-scale (Ideal downsampling)
Experiment setup. We create the MNIST-scale dataset following the procedure in prior works [13, 42]. Each image in the original MNIST dataset is randomly downsampled with a factor of  $\sim [\frac{1}{0.3} - 1]$ , such that every resolution from  $8 \times 8$  to  $28 \times 28$  contains an equal number of samples. As the baseline models (except the Fourier CNN) can not handle images of different resolutions, following prior works, lower-resolution images are zero-padded to the original resolution. We do not need to pad the input for our model and Fourier CNN. We used  $10\mathrm{k}$ ,  $2\mathrm{k}$ , and  $50\mathrm{k}$  for training, validation, and test set samples. For this experiment, we enforce equivariance over scales that correspond to the discrete resolutions of  $\mathcal{R} = \{8,\dots,28\}$ .
Implementation details. For the baselines and CNN, we follow the implementation, hyperparameters, and architecture provided in prior works [41, 42]. For Per Res. CNN, we train a separate CNN for each resolution. Each of these CNNs uses the architecture of baseline CNN. For Fourier CNN, we use the Fourier block introduced in the Fourier Neural operator [18]. Inspired by their design, we use  $1 \times 1$  complex convolution in the Fourier domain along with the scale-equivariant convolution. We follow the baseline for all training hyper-parameters, except we included a weight decay of 0.01.
Results. In Tab. 1, we report the accuracy of the MNIST-scale dataset. We observe that our approach achieved zero equivariance error and the highest accuracy. While all models achieve similar accuracy, there is a more notable difference in the scale consistency rate. This means that our model properly captures the additional information that comes with increased resolution.
Generalization to unseen scales. We study the generalization capabilities of the scale-equivariant modes to unseen scales; we train them on a dataset with 10k full resolution  $(28 \times 28)$  MNIST images and test on 50k samples of MNIST-scale, i.e., containing different scales. For the baselines, we added random scaling argumentation during training. In Tab. 2, we observe that our model can guarantee zero equivariance error even for the unseen scales and achieves comparable performance to baselines trained with data augmentation.
Data efficiency. We also conduct experiments studying the data efficiency of the different models. Following the same setup as MNIST-scale, we train the models on limited training examples, 5k, 2.5k, and 1k, of different resolutions and test on 50k samples across all resolutions. In Tab. 3, we observe that our model is more data efficient than the baselines. DISCO achieves the second-best
Table 3. MNIST-scale accuracy with different numbers of training samples.
Models / # Samples	5000	2500	1000
CNN	0.9432	0.9389	0.8577
Per Res. CNN	0.9118	0.8392	0.5815
DISCO	0.9794	0.9665	0.9457
SESN	0.9638	0.9402	0.9207
SI-CovNet	0.9641	0.9437	0.9280
SS-CNN	0.9477	0.9259	0.9176
DSS	0.9654	0.9401	0.9281
Fourier CNN	0.9567	0.9419	0.8910
Ours	0.9835	0.9767	0.9606
Table 4. The classification accuracy of different models on STL10-scale dataset.
Models	Acc.↑	Scale-Con.↑	Equi-Err.↓
Wide ResNet	0.5596	0.2916	0.16
SESN	0.5525	0.4166	0.04
DSS	0.5347	0.1979	0.02
SI-CovNet	0.5588	0.2187	0.03
SS-CNN	0.4788	0.1979	1.82
DISCO	0.4768	0.3541	0.06
Fourier CNN	0.5844	0.2812	0.19
Ours	0.7332	0.6770	0.00
Table 5. Ablation on consistency loss.
# Samples	w/ consistency	w/o consistency
Acc.↑	Scale-Con.↑	Acc.↑	Scale-Con.↑
5000	0.9835	0.9296	0.9831	0.9150
2500	0.9767	0.8906	0.9755	0.8633
1000	0.9606	0.8183	0.9599	0.8144
performance. We also see that Per Res. CNN suffers the most when trained with fewer data points, as it trains a separate CNN for each scale and does not share parameters across different scales.
Ablation. We perform an ablation on the consistency loss in Eq. (21) over different training set sizes. From Tab. 5, we can observe that the consistency loss improves the accuracy of our model as well as the scale-consistency. This result validates the effectiveness of the proposed consistency loss.
5.2 STL10-scale (Ideal downsampling)
Experiment setup. Following the same procedure as the MNIST-scale dataset, we create the STL10-scale dataset. Each image of the dataset is randomly scaled with a randomly chosen downsampling factor between  $[1 - 2]$  such that every resolution from 48 to 97 contains an equal number of samples. We use 7k, 1k, and 5k samples in our training, validation, and test set. For the baseline models, we again zero-pad the downsampled images to the original size.
Implementation details. For the baseline models, we use the Wide ResNet as the CNN baseline following prior work [41, 42]. For Fourier CNN, we use six Fourier blocks followed by a two-layered MLP. For our model, we use six scale-equivariant Fourier blocks followed by a two-layer MLP. All of the models are trained for 250 epochs with Adam optimizer with an initial learning rate of 0.01. The learning rate is reduced by a factor of 0.1 after every 100 epoch. For scalability, we consider achieving equivariance over scales that correspond to the discrete resolutions in the set  $\mathcal{R} = \{48\leq 48 + i\times 8\leq 97\}\forall i\in \{0,1,2,\ldots \}$ .
Results. In Tab. 4, we observe that our model achieves zero equivariance error with higher accuracy and scale consistency over the baselines. As the baseline models accept a fixed-sized input, the downsampled images are zero-padded following prior work's preprocessing on MNIST-scale. Note, MNIST images have a uniform black background, and zero-padding does not create artifacts. However, for colored images with diverse backgrounds, such as STL-10, any padding scheme to resize the image will cause artifacts. We believe this artifact hurts the performance of baseline models on the STL10-scale dataset. However, it is unclear whether there is a more suitable padding strategy.
5.3 MNIST-scale (Non-ideal downsampling)
Ideal interpolation suffers from artifacts known as the ringing effect caused by Gibbs phenomenon [23]; see the down-scaled image in Fig. 3a. In practice, a non-ideal low-pass filter will be used instead. Taking this into consideration, we conduct the experiments using a more commonly used anti-aliasing scheme with a Gaussian blur instead of the ideal low-pass filter.
(a) Learned feature from ideal downsampling.
(b) Learned feature for non-ideal downsampling.
Figure 3. Feature visualization for ideal and non-ideal downsampling settings. In both settings, our model seems to learn spatially local features such as digit contour and edges.
Table 6. The accuracy of different models on MNIST-scale (non-ideal downsampling).
Models	Acc.↑	Scale Con.↑	Equi. Err.↓
CNN	0.9642	0.1033	-
Per Res. CNN	0.9450	0.0742	-
SESN	0.9710	0.6666	-
DSS	0.9772	0.5716	-
SI-CovNet	0.9694	0.4453	-
SS-CNN	0.9670	0.3144	-
DISCO	0.9830	0.4500	0.63
Fourier CNN	0.9745	0.1716	0.29
Ours	0.9880	0.9760	0.05
Experiment details. We follow the same experimental setup and training scheme as in MNIST-scale with the ideal downsampling experiment. The only difference is that we use a Gaussian kernel to perform anti-aliasing.
Results. From Tab. 6, we observe that our model achieves higher classification accuracy and Scale consistency. Importantly, our model achieves lower equivariance error than the baseline despite the gap in the theory of non-ideal downsampling.
We propose a family of scale-equivariant deep nets that achieve zero equivariance error measured from end to end. We formulate down-scaling in the discrete domain with proper consideration of anti-aliasing. To achieve scale-equivariance, we design novel modules based on Fourier layers, enforcing that the lower frequency content of output does not depend on the higher frequency content of the input. Furthermore, we motivated the scale-consistency property that the performance of higher-resolution input should be better than that of the lower resolution and designed a suitable classifier architecture. Empirically, our approach achieves competitive accuracy on image classification tasks, with improved scale consistency and lower equivariance-error compared to baselines. Similar to other equivariant methodologies, defining consistent scales or group actions to achieve equivalence before constructing the model is crucial. Moreover, a common challenge all equivariant and invariant techniques face is the significant demands on memory and computational resources. In our upcoming research, we plan to enhance our approach by applying it to high-resolution image datasets and dense prediction tasks, such as instance segmentation.
[1] E. H. Adelson, C. H. Anderson, J. R. Bergen, P. J. Burt, and J. M. Ogden. Pyramid methods in image processing. RCA engineer, 1984. 1, 2
[2] E. J. Bekkers. B-spline CNNs on Lie groups. In Proc. ICLR, 2020. 1, 2
[3] M. M. Bronstein, J. Bruna, Y. LeCun, A. Szlam, and P. Vandergheynst. Geometric deep learning: going beyond euclidean data. IEEE SPM, 2017. 2
[4] A. Coates, A. Ng, and H. Lee. An analysis of single-layer networks in unsupervised feature learning. In Proc. AISTATS, 2011. 2, 7
[5] T. Cohen and M. Welling. Group equivariant convolutional networks. In Proc. ICML, 2016. 1, 2
[6] P. de Haan, M. Weiler, T. Cohen, and M. Welling. Gauge equivariant mesh CNNs: Anisotropic convolutions on geometric graphs. In Proc. ICLR, 2021. 3
[7] M. Defferrard, X. Bresson, and P. Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Proc. NeurIPS, 2016. 3
[8] R. Ghosh and A. K. Gupta. Scale steerable filters for locally scale-invariant convolutional neural networks. arXiv preprint arXiv:1906.03861, 2019. 1, 2, 8
[9] K. Grauman and T. Darrell. The pyramid match kernel: discriminative classification with sets of image features. In Proc. ICCV, 2005. 1, 2
[10] J. Hartford, D. Graham, K. Leyton-Brown, and S. Ravanbakhsh. Deep models of interactions across sets. In Proc. ICML, 2018. 3
[11] K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE TPAMI, 2015. 1
[12] L. He, Y. Chen, Y. Dong, Y. Wang, Z. Lin, et al. Efficient equivariant network. In Proc. NeurIPS, 2021. 6
[13] A. Kanazawa, A. Sharma, and D. Jacobs. Locally scale-invariant convolutional neural networks. arXiv preprint arXiv:1412.5104, 2014. 1, 2, 8
[14] T. Karras, M. Aittala, S. Laine, E. Härkönen, J. Hellsten, J. Lehtinen, and T. Aila. Alias-free generative adversarial networks. In Proc. NeurIPS, 2021. 3
[15] T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. In Proc. ICLR, 2017. 3
[16] R. Kondor, Z. Lin, and S. Trivedi. Clebsch-Gordan Nets: a fully Fourier space spherical convolutional neural network. In Proc. NeurIPS, 2018. 3
[17] S. Lazebnik, C. Schmid, and J. Ponce. Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In Proc. CVPR, 2006. 1, 2
[18] Z. Li, N. B. Kovachki, K. Azizzadenesheli, B. liu, K. Bhattacharya, A. Stuart, and A. Anandkumar. Fourier neural operator for parametric partial differential equations. In Proc. ICLR, 2021. 3, 8
[19] I.-J. Liu, R. A. Yeh, and A. G. Schwing. Pic: permutation invariant critic for multi-agent deep reinforcement learning. In Proc. CORL, 2020. 3
[20] I.-J. Liu, Z. Ren, R. A. Yeh, and A. G. Schwing. Semantic tracklets: An object-centric representation for visual multi-agent reinforcement learning. In Proc. IROS, 2021. 3
[21] D. G. Lowe. Object recognition from local scale-invariant features. In Proc. ICCV, 1999. 2
[22] D. G. Lowe. Distinctive image features from scale-invariant keypoints. IJCV, 2004. 2
[23] D. G. Manolakis and V. K. Ingle. Applied digital signal processing: theory and practice. Cambridge university press, 2011. 1, 2, 9
[24] X. Mao, Y. Liu, F. Liu, Q. Li, W. Shen, and Y. Wang. Intriguing findings of frequency selection for image deblurring. In Proc. AAAI, 2023. 3
[25] H. Maron, H. Ben-Hamu, N. Shamir, and Y. Lipman. Invariant and equivariant graph networks. In Proc. ICLR, 2019. 3
[26] H. Maron, O. Litany, G. Chechik, and E. Fetaya. On learning sets of symmetric elements. In Proc. ICML, 2020. 3
[27] M. Mathieu, M. Henaff, and Y. LeCun. Fast training of convolutional networks through FFTs. arXiv preprint arXiv:1312.5851, 2013. 3
[28] C. Morris, G. Rattan, S. Kiefer, and S. Ravanbakhsh. SpeqNets: Sparsity-aware permutation-equivariant graph networks. In Proc. ICML, 2022. 3
[29] E. Nguyen, K. Goel, A. Gu, G. Downs, P. Shah, T. Dao, S. Baccus, and C. Ré. S4nd: Modeling images and videos as multidimensional signals with state spaces. In Proc. NeurIPS, 2022. 3
[30] H. Nyquist. Certain topics in telegraph transmission theory. Transactions of the American Institute of Electrical Engineers, 1928. 1
[31] H. Pratt, B. Williams, F. Coenen, and Y. Zheng. FCNN: Fourier convolutional neural networks. In ECML PKDD, 2017. 3
[32] C. R. Qi, H. Su, K. Mo, and L. J. Guibas. PointNet: Deep learning on point sets for 3D classification and segmentation. In Proc. CVPR, 2017. 3
[33] S. Ravanbakhsh, J. Schneider, and B. Póczos. Equivalence through parameter-sharing. In Proc. ICML, 2017. 2
[34] S. Ravanbakhsh, J. Schneider, and B. Poczos. Deep learning with sets and point clouds. In Proc. ICLR workshop, 2017. 3
[35] R. A. Rojas-Gomez, T.-Y. Lim, A. Schwing, M. Do, and R. A. Yeh. Learnable polyphase sampling for shift invariant and equivariant convolutional networks. In Proc. NeurIPS, 2022. 3
[36] R. A. Rojas-Gomez, T.-Y. Lim, M. N. Do, and R. A. Yeh. Making vision transformers truly shift-equivariant. arXiv preprint arXiv:2305.16316, 2023. 2
[37] D. Romero, E. Bekkers, J. Tomczak, and M. Hoogendoorn. Attentive group equivariant convolutional networks. In Proc. ICML, 2020.
[38] M. Shakerinava and S. Ravanbakhsh. Equivariant networks for pixelized spheres. In Proc. ICML, 2021. 2
[39] D. I. Shuman, S. K. Narang, P. Frossard, A. Ortega, and P. Vandergheynst. The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains. IEEE SPM, 2013. 3
[40] K. Sohn and H. Lee. Learning invariant representations with local transformations. In Proc. ICML, 2012. 2, 7
[41] I. Sosnovik, M. Szmaja, and A. Smeulders. Scale-equivariant steerable networks. In Proc. ICLR, 2020. 1, 2, 8, 9
[42] I. Sosnovik, A. Moskalev, and A. Smeulders. DISCO: accurate discrete scale convolutions. In Proc. BMVC, 2021. 1, 2, 8, 9
[43] R. Suvorov, E. Logacheva, A. Mashikhin, A. Remizova, A. Ashukha, A. Silvestrov, N. Kong, H. Goka, K. Park, and V. Lempitsky. Resolution-robust large mask inpainting with Fourier convolutions. In Proc. WACV, 2022. 3
[44] S. R. Venkataraman, S. Balasubramanian, and R. R. Sarma. Building deep equivariant capsule networks. In Proc. ICLR, 2020. 2
[45] M. Weiler and G. Cesa. General E(2)-equivariant steerable CNNs. In Proc. NeurIPS, 2019. 2
[46] D. Worrall and M. Welling. Deep scale-spaces: Equivariance over scale. In Proc. NeurIPS, 2019. 1, 2, 8
[47] J. Xu, H. Kim, T. Rainforth, and Y. Teh. Group equivariant subsampling. In Proc. NeurIPS, 2021. 3
[48] R. A. Yeh, Y.-T. Hu, and A. Schwing. Chirality nets for human pose regression. In Proc. NeurIPS, 2019. 3
[49] R. A. Yeh, A. G. Schwing, J. Huang, and K. Murphy. Diverse generation for multi-agent sports games. In Proc. CVPR, 2019. 3
[50] R. A. Yeh, Y.-T. Hu, M. Hasegawa-Johnson, and A. Schwing. Equivariance discovery by learned parameter-sharing. In Proc. AISTATS, 2022. 2
[51] M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Poczos, R. R. Salakhutdinov, and A. J. Smola. Deep sets. In Proc. NeurIPS, 2017. 3
[52] R. Zhang. Making convolutional networks shift-invariant again. In Proc. ICML, 2019. 3
[53] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia. Pyramid scene parsing network. In Proc. CVPR, 2017. 1
[54] W. Zhu, Q. Qiu, R. Calderbank, G. Sapiro, and X. Cheng. Scaling-translation-equivariant networks with decomposed convolutional filters. JMLR, 2022. 2