SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Initial Attention Layer vs Scatter Layer vs Initial Convolutional: This table compares SVT transformer where initial scatter layers and later attention layers, SVT-Inverse where initial attention layers and later scatter layers, and SVT with initial convolutional layers. Also, we show an alternative spectral layer and attention layer. This shows that the Initial scatter layer works better compared to the rest.
Model	Params(M)	FLOPS(G)	Top-1(%)	Top-5(%)
SVT-H-S	22.0M	3.9	84.2	96.9
SVT-H-S-Init-CNN	21.7M	4.1	84.0	95.7
SVT-H-S-Inverse	21.8M	3.9	83.1	94.6
SVT-H-S-Alternate	22.4M	4.6	83.4	95.0
Table 3: This table shows the ablation analysis of various spectral layers in SVT architecture such as FN, FFC, WGN, and FNO. We conduct this ablation study on the small-size networks in stage architecture. This indicates that SVT performs better than other kinds of networks.
Model	Params (M)	FLOPS (G)	Top-1 (%)	Top-5 (%)	Invertible loss(↓)
FFC	21.53	4.5	83.1	95.23	-
FN	21.17	3.9	84.02	96.77	-
FNO	21.33	3.9	84.09	96.86	3.27e-05
WGN	21.59	3.9	83.70	96.56	8.90e-05
SVT	22.22	3.9	84.20	96.93	6.64e-06
Table 4: Results on transfer learning datasets. We report the top-1 accuracy on the four datasets.
Model	CIFAR 10	CIFAR 100	Flowers 102	Cars 196
ResNet50 [21]	-	-	96.2	90.0
ViT-B/16 [13]	98.1	87.1	89.5	-
ViT-L/16 [13]	97.9	86.4	89.7	-
DeiT-B/16 [56]	99.1	90.8	98.4	92.1
ResMLP-24 [55]	98.7	89.5	97.9	89.5
GFNet-XS [47]	98.6	89.1	98.1	92.8
GFNet-H-B [47]	99.0	90.3	98.8	93.2
SVT-H-B	99.22	91.2	98.9	93.6
Table 5: The performances of various vision backbones on COCO val2017 dataset for the downstream instance segmentation task such as Mask R-CNN 1x [20] method. We adopt Mask R-CNN as the base model, and the bounding box & mask Average Precision (i.e.,  $AP^b$  &  $AP^m$ ) are reported for evaluation
Backbone	\( {AP}^{b} \)	\( {AP}_{50}^{b} \)	\( {AP}_{75}^{b} \)	\( {AP}^{m} \)	\( {AP}_{50}^{m} \)	\( {AP}_{75}^{m} \)
ResNet50 [21]	38.0	58.6	41.4	34.4	55.1	36.7
Swin-T [39]	42.2	64.6	46.2	39.1	61.6	42.0
Twins-SVT-S [9]	43.4	66.0	47.3	40.3	63.2	43.4
LITv2-S [42]	44.9	-	-	40.8	-	-
RegionViT-S [5]	44.2	-	-	40.8	-	-
PVTv2-B2 [62]	45.3	67.1	49.6	41.2	64.2	44.4
SVT-H-S	46.0	68.1	50.4	41.9	65.0	45.1
achieving a top-1 accuracy of  $84.2\%$ . Similarly, SVT-H-B surpasses all the transformers in the base category, boasting a top-1 accuracy of  $85.2\%$ . Lastly, SVT-H-L outperforms other large transformers with a top-1 accuracy of  $85.7\%$  when tested on the ImageNet dataset with an image size of 224x224.
When comparing different architectural approaches, such as Convolutional Neural Networks (CNNs), Transformer architectures (attention-based models), MLP Mixers, and Spectral architectures, SVT consistently outperforms its counterparts. For instance, SVT achieves better top-1 accuracy and parameter efficiency compared to CNN architectures like ResNet 152 [21], ResNeXt [67], and ResNeSt in terms of top-1 accuracy and number of parameters. Among attention-based architectures, MaxViT [58] has been recognized as the best performer, surpassing models like DeiT [56], CrossViT [6], DeepViT [78], T2T [72] etc. with a top-1 accuracy of 85.0. However, SVT achieves an even higher top-1 accuracy of 85.7 with less than half the number of parameters. In the realm of MLP Mixer-based architectures, DynaMixer [63] emerges as the top-performing model, surpassing MLPmixer[54], gMLP [37], CycleMLP [7], Hire-MLP[18], AS-MLP [36], WaveMLP[53], PoolFormer[71] and DynaMixer-L [63] with a top-1 accuracy of  $84.3\%$ . In comparison, SVT-H-L outperforms DynaMixer with a top-1 accuracy of  $85.7\%$  while requiring fewer parameters and computations. Hierarchical architectures, which include models like PVT [61], Swin [39] transformer, CSwin [12] transformer, Twin [9] transformer, and VOLO [73] are also considered. Among this category, VOLO achieves the highest top-1 accuracy of  $85.4\%$ . However, it's important to note that SVT outperforms VOLO with a top-1 accuracy of  $85.7\%$  for SVT-H-L. Lastly, in the spectral architecture category, models like GFNet[47], iFormer [51], LiTv2 [42], HorNet [46], Wave-ViT [70], etc. are examined. Wave-ViT was previously the state-of-the-art method with a top-1 accuracy of  $85.5\%$ . Nevertheless, SVT-H-L surpasses Wave-ViT in terms of top-1 accuracy, network size (number of parameters), and computational complexity (FLOPS), as indicated in Table 1.
3.3 What Matters: Does Initial Spectral or Initial Attention or Initial Convolution Layers?
The ablation study was conducted to show that initial scatter layers followed by attention in deeper layers are more beneficial than having later scatter and initial attention layers (SVT-H-S-Inverse). We also compare transformer models based on an alternative to the attention and scatter layer (SVT-H-S-Alternate) as shown in Table- 2. From all these combinations we observe that initial scatter layers followed by attention in deeper layers are more beneficial than others. We compare the performance of SVT when the architecture changes from all attention (PVTv2[62]) to all spectral layers (GFNet[47])
Table 6: SVT model comprises low-frequency component and High-frequency component with the help of scattering net using Dual tree complex wavelet transform. Each frequency component is controlled by a parameterized weight matrix using Patch mixing and/or Channel Mixing. This table shows details about all combinations and  $SVT_{TTEE}$  is the best performing among them.
Backbone	Low Frequency	High Frequency	Params (M)	FLOPS (G)	Top-1 (%)	Top-5 (%)
Token	Channel	Token	Channel
SVTTTT	T	T	T	T	25.18	4.4	83.97	96.86
SVTEETT	E	E	T	T	21.90	4.1	83.87	96.67
SVTEEE	E	E	E	E	21.87	3.7	83.70	96.56
SVTTEE	T	T	E	E	22.01	3.9	84.20	96.82
SVTTEX	T	T	E	X	21.99	4.0	84.06	96.76
SVTTXE	T	T	X	E	22.25	4.1	84.12	96.91
as well as a few spectral and remaining attention layers(SVT ours). We observe that combining spectral and attention boost the performance compared to all attention and all spectral layer-based transformer as shown in Table- 2. We have conducted an experiment where the initial layers of a ViT are convolutional networks and later layers are attention layers to compare the performance of SVT. The results are captured in Table- 1, where we compare SVT with transformers having initial convolutional layers such as CVT [64], CMT [17], and HorNet [46]. Initial convolutional layers in a transformer are not performing well compared to the initial scatter layer. Initial scatter layer-based transformers have better performance and less computation cost compared to initial convolutional layer-based transformers which is shown in Table- 2.
3.4 Ablation analysis
SVT uses a scattering network to decompose the signal into low-frequency and high-frequency components. We use a gating operator to get effective learnable features for spectral decomposition. The gating operator is a multiplication of the weight parameter in both high and low frequencies. We have conducted experiments that use tensor and Einstein mixing. Tensor mixing is a simple multiplication operator, while Einstein mixing uses an Einstein matrix multiplication operator [48]. We observe that in low-frequency components, Tensor mixing performs better as compared to Einstein mixing. As shown in Table- 6, we start with  $SVT_{TTTT}$ , which uses tensor mixing in both high and low-frequency components. We see that it may not perform optimally. Then we reverse it and use Einstein mixing in both low and high-frequency components - this also does not perform optimally. Then, we came up with the alternative method  $SVT_{TTEE}$ , which uses tensor mixing in low frequency and Einstein mixing in high frequency. The high-frequency further decomposes into token and channel mixing, whereas in low-frequency we simply tensor multiplication as it is an energy or amplitude component.
In the second ablation analysis, we compare various spectral architectures, including the Fourier Network (FN), Fourier Neural Operator (FNO), Wavelet Gating Network (WGN), and Fast Fourier Convolution (FFC). When we contrast SVT with WGN, it becomes evident that SVT exhibits superior directional selectivity and a more adept ability to manage complex transformations. Furthermore, in comparison to FN and FNO, SVT excels in decomposing frequencies into low and high-frequency components. It's worth noting that SVT surpasses other spectral architectures primarily due to its utilization of the Directional Dual-Tree Complex Wavelet Transform (DTCWT), which offers directional orientation and enhanced invertibility, as demonstrated in Table 3. For a more comprehensive analysis, please refer to the Supplementary section.
3.5 Transfer Learning and Task Learning
We train SVT on ImageNet1K data and fine-tune it on various datasets such as CIFAR10, CIFAR100, Oxford Flower, and Stanford Car for image recognition tasks. We compare SVT-H-B performance with various transformers such as DeiT [56], ViT [13], and GFNet [47] as well as with CNN architectures such as ResNet50 and MLP mixer architectures such as ResMLP. This comparison is shown in Table-4. It can be observed that SVT-H-B outperforms state-of-art on CIFAR10 with a top-1 accuracy of  $99.1\%$ , CIFAR100 with a top-1 accuracy of  $91.3\%$ , Flowers with a top-1 accuracy of  $98.9\%$  and Cars with top-1 accuracy of  $93.7\%$ . We observe that SVT has more representative features and has an inbuilt discriminative nature which helps in classifying images into various categories. We use a
Table 7: Latency(Speed test): This table shows the Latency (mili sec) of SVT compared with Conv type network, attention type transformer, POOI type, MLP type, and Spectral type transformer. We report latency per sample on A100 GPU. We adopt the latency table from EfficientFormer [34].
Model	Type	Params (M)	GMAC (G)	Top-1 (%)	Latency (ms)
ResNet50[21]	Convolution	25.5	4.1	78.5	9.0
DeiT-S[56]	Attention	22.5	4.5	81.2	15.5
PVT-S[62]	Attention	24.5	3.8	79.8	23.8
T2T-14[]	Attention	21.5	4.8	81.5	21.0
Swin-T[38]	Attention	29.0	4.5	81.3	22.0
CSwin-T[12]	Attention	23.0	4.3	82.7	28.7
PoolFormer[71]	Pool	31.0	5.2	81.4	41.2
ResMLP-S[55]	MLP	30.0	6.0	79.4	17.4
EfficientFormer [34]	MetaBlock	31.3	3.9	82.4	13.9
GFNet-H-S[47]	Spectral	32.0	4.6	81.5	14.3
SVT-H-S	Spectral	22.0	3.9	84.2	14.7
Table 8: Invertibility: This table shows the invertibility of SVT(DTCWT) compared with Fourier and DWT. We also compare different directional orientations and show the reconstruction loss (MSE) in an image.
Model	MSE loss(↓)	PSNR (db)(↑)
Fourier (FFT)	3.27e-05	11.18
DWT-M1	8.90e-05	76.33
DWT-M2	3.19e-05	84.67
DWT-M3	1.08e-05	91.94
DTCWT-M1	6.64e-06	137.97
DTCWT-M2	2.01e-06	138.87
DTCWT-M3	1.23e-07	142.14
pre-trained SVT model for the downstream instance segmentation task and obtain good results on the MS-COCO dataset as shown in Table- 5.
3.6 Latency Analysis
It's important to highlight that Fourier Transforms, as mentioned in the GFNet[47], are not inherently capable of performing low-pass and high-pass separations. In contrast, GFNet consistently employs tensor multiplication, a method that, while effective, may be less efficient compared to Einstein multiplication. The latter approach is known for reducing the number of parameters and computational complexity. As a result, SVT does not lag behind in terms of performance or computational complexity; rather, it gains enhanced representational power. This is exemplified in Table 7, which provides a comparison of latency, FLOPS (Floating-Point Operations per Second), and the number of parameters. Table 7 specifically demonstrates the latency (measured in milliseconds) of SVT in relation to various network types, including convolution-based networks, Attention-based Transformer networks, Pool-based Transformer networks, MLP-based Transformer networks, and Spectral-based Transformer networks. The reported latency values are on a per-sample basis, measured on an A100 GPU.
3.7 Invertibility Versus Redundancy trade-off Analysis
We conducted an experiment to illustrate that invertibility not only enhances performance but also contributes to image comprehension. To do this, we passed an image through the raw DTC-WT and performed an inverse DTC-WT operation to calculate the reconstruction loss. The experiment was executed across various values of "M" corresponding to the level of decomposition and orientations in SVT. We observed that the reconstruction loss decreased as the value of "M" increased, indicating that SVT's ability to comprehend the image improved. These orientations effectively captured higher-order image properties, enhancing SVT's performance. We further compared different spectral transforms, including the Fourier Transform, Discrete Wavelet Transform (DWT), and DTC-WT. Our findings demonstrated that the reconstruction loss was lower for DTC-WT compared to other spectral transforms, as depicted in Table 8 below. In Table 8, we quantified the mean squared error (MSE) for FFT, DWT at stages 1, 2, and 3, and DTCWT at stages 1, 2, and 3. The MSE decreased as we increased the level of decomposition (M) and the degree of selectivity. DTCWT consistently exhibited lower MSE compared to DWT. Furthermore, the peak signal-to-noise ratio (PSNR) of DTCWT surpassed that of DWT and the Fourier Transform. PSNR gauges the quality of the reconstructed image, expressing the ratio between the maximum possible power of an image and the power of noise affecting its representation, measured in decibels (dB). A higher PSNR indicates superior image quality. A high-quality reconstructed image is characterized by low MSE and high PSNR values. For further details on redundancy, please consult the supplementary table. Additionally, we have visualized the filter coefficients for all six orientations in the supplementary materials.
3.8 Limitations
SVT currently uses six directional orientations to capture an image's fine-grained semantic information. It is possible to go for the second degree, which gives thirty-six orientations, while the third degree gives 216 orientations. The more orientations, the more semantic information could be captured, but
this leads to higher computational complexity. The decomposition parameter 'M' is currently set to 1 to get single low-pass and high-pass components. Higher values of 'M' give more components in both frequencies but lead to higher complexity.
The Vision Transformer (ViT) [13] was the first transformer-based attempt to classify images into pre-defined categories and use NLP advances in vision. Following this, several transformer based approaches like DeiT[56], Tokens-to-token ViT [72], Transformer iN Transformer (TNT) [19], CrossViT [6], Class attention image Transformer(CaiT) [57] Uniformer [32], Beit. [3], SViT[45], RegionViT [5], MaxViT [58] etc. have all been proposed to improve the accuracy using multi-headed self-attention (MSA). PVT [61], SwinT [39], CSwin[12] and Twin [39] use hierarchical architecture to improve the performance of the vision transformer on various tasks. The complexity of MSA is  $\mathrm{O}(n^2)$ . For high-resolution images, the complexity increases quadratically with token length. PoolFormer [71] is a method that uses a pooling operation over a small patch which has to obtain a down-sampled version of the image to reduce computational complexity. The main problem with PoolFormer is that it uses a MaxPooling operation which is not invertible. Another approach to reducing the complexity is the spectral transformers such as FNet [31], GFNet [47], AFNO [16], WaveMix [24], WaveViT [70], SpectFormer [44], FourierFormer [40], etc. FNet [31] does not use inverse Fourier transforms, leading to an invertibility issue. GFNet [47] solves this by using inverse Fourier transforms with a gating network. AFNO [16] uses the adaptive nature of a Fourier neural operator similar to GFNet. SpectFormer [44] introduces a novel transformer architecture that combines both spectral and attention networks for vision tasks. GFNet, SpectFormer, and AFNO do not have proper separation of low-frequency and high-frequency components and may struggle to handle the semantic content of images. In contrast, SVT has a clear separation of frequency components and uses directional orientations to capture semantic information. FourierIntegral [40] is similar to GFNet and may have similar issues in separating frequency components.
WaveMLP [53] is a recent effort that dynamically aggregates tokens as a wave function with two parts, amplitude and phase to capture the original features and semantic content of the images respectively. SVT uses a scattered network to provide low-frequency and high-frequency components. The high-frequency component has six or more directional orientations to capture semantic information in images. We use Einstein multiplication in token and channel mixing of high-frequency components leading to lower computational complexity and network size. In Wave-ViT [70], the author has discussed the quadratic complexity of the self-attention network using a wavelet transform to perform lossless down-sampling using wavelet transform over keys and values. However, WaveViT still has the same complexity as it uses attention instead of spectral layers. SVT uses the scatter network which is more invertible compared to WaveViT.
One of the challenges in MSA is its inability to characterize different frequencies in the input image. Hilo attention (LiTv2) [42] helps to find high-frequency and low-frequency components by using a novel variant of MSA. But it does not solve the complexity issue of MSA. Another parallel effort named Inception Transformer came up [51], which uses an Inception mixer to capture high and low-frequency information in visual data. iFormer still has the same complexity as it uses attention as the low-frequency mixer. SVT in comparison, uses a spectral neural operator to capture low and high frequency components using the DTCWT. This removes the  $\mathrm{O}(n^2)$  complexity as it uses spectral mixing instead of attention. iFormer [51] uses a non-invertible max pooling and convolutional layer to capture high-frequency components, whereas, in contrast, SVT's mixer is completely invertible. SVT uses a scatter network to get a better directional orientation to capture fine-grained information such as lines and edges, compared to Hilo attention and iFormer.
5 Conclusions and Future Research Directions
We have proposed SVT, which helps in separating low-frequency and high-frequency components of an image, while simultaneously reducing computational complexity by using Einstein multiplication-based technique for efficient channel and token mixing. SVT has been evaluated on standard benchmarks and shown to achieve state-of-the-art performance on standard benchmark datasets on both image classification tasks and instance segmentation tasks. It also achieves comparable performance on object detection tasks. We shall experiment with SVT in other domains such as speech and NLP as we believe that it offers significant value in these domains as well.
[76] Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi-scale vision longformer: A new vision transformer for high-resolution image encoding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2998-3008, 2021.
[77] Wenqiang Zhang, Zilong Huang, Guozhong Luo, Tao Chen, Xinggang Wang, Wenyu Liu, Gang Yu, and Chunhua Shen. Topformer: Token pyramid transformer for mobile semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12083-12093, 2022.
[78] Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Zihang Jiang, Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2103.11886, 2021.