Scattering Vision Transformer: Spectral Mixing Matters
Authors: Badri Narayana Patro
Vijay Srinivas Agneeswaran
Venue: NeurIPS 2023 poster (2023)
Keywords: Vision Transformer
Scatter Network
Spectral Transformer
Token Mixing
Channel Mixing
and Einstein Blending Method

Abstract: Vision transformers have gained significant attention and achieved state-of-the-art performance in various computer vision tasks, including image classification, instance segmentation, and object detection. However, challenges remain in addressing attention complexity and effectively capturing fine-grained information within images. Existing solutions often resort to down-sampling operations, such as pooling, to reduce computational cost. Unfortunately, such operations are non-invertible and can result in information loss. In this paper, we present a novel approach called Scattering Vision Transformer (SVT) to tackle these challenges. SVT incorporates a spectrally scattering network that enables the capture of intricate image details. SVT overcomes the invertibility issue associated with down-sampling operations by separating low-frequency and high-frequency components. Furthermore, SVT introduces a unique spectral gating network utilizing Einstein multiplication for token and channel mixing, effectively reducing complexity. We show that SVT achieves state-of-the-art performance on the ImageNet dataset with a significant reduction in a number of parameters and FLOPS. SVT shows 2\% improvement over LiTv2 and iFormer. SVT-H-S reaches 84.2\% top-1 accuracy, while SVT-H-B reaches 85.2\% (state-of-art for base versions) and SVT-H-L reaches 85.7\% (again state-of-art for large versions). SVT also shows comparable results in other vision tasks such as instance segmentation. SVT also outperforms other transformers in transfer learning on standard datasets such as CIFAR10, CIFAR100, Oxford Flower, and Stanford Car datasets.  The project page is available on this webpage.\url{https://badripatro.github.io/svt/}.

Scattering Vision Transformer: Spectral Mixing Matters
Badri Narayana Patro Microsoft badripatrofmiccrooff..com
Vijay Srinivas Agneeswaran Microsoft vaguesswarartmmicroofff.cco
Abstract
Vision transformers have gained significant attention and achieved state-of-the- art performance in various computer vision tasks, including image classification, instance segmentation, and object detection. However, challenges remain in address- ing attention complexity and effectively capturing fine-grained information within images. Existing solutions often resort to down-sampling operations, such as pool- ing, to reduce computational cost. Unfortumately, such operations and can result in information loss. In this paper, we present a novel approach called Scattering Vision Transformer (SVT) tackle these challenges. SVT incorporates a spectrally scattering network that enables the capture intricate image details. SVT overcomes the invertibility issue associated with down- wown-sampling operations by separating low-frequency and high-frequency components. Furthermore, SVT introduces unique spectral gating network utilizing Einstein multiplication for token and channel mixing, effectively reducing complexity. We show that SVT achieves state-of of-th-aft performance the ImageNet dataset with significant reduction a number parameters and FL.OPS. SVT shows 2% improvement over LiTv2 and iFormer. SVT-H-S reaches 84.2% top-1 accuracy, while SVT-H-B reaches 85.2% (state-of-art for base versions) and SVT-H-L reaches 85.7% (again state-of-art for large versions). SVT also shows comparable results in other vision tasks such as instance segmentation. SVT also outperforms other transformers in transfer learning on standard datasets such as CIFARIO, CIFARI00, Oxford Flower, and Stanford Car datasets. The project page is available on this webpage (https://badripattl Saaarro.gtttte github. io/svt/).
non-invertible
Introduction
In recent years, there has been a remarkable surge in the interest and adoption of Large Language Models (LLMs), driven by the release and success of prominent models such as GPT-3, ChatGPT [1]. and Palm 88] These LLMs have achieved significant breakthroughs in the field Natural Language Processing (NLP). Building upon their successes, subsequent research endeavors have extended the language transformer paradigm to diverse domains including computer vision, speech recognition, video processing, and climate and weather prediction. In this paper, we specifically focus exploring the potential of L.MMs for vision-related tasks. By leveraging the power of these language models, aim push the boundaries of vision applications and investigate their capabilities in addressing complex vision challenges.
Several adaptations of transformers have been introduced in the field of computer vision for various tasks. For image classification, notable vision transformers include ViT [33,, DelT [56). PVT [61], Swin (39], Twin [9) and CSWin transformers [12]. The different vision transformers improved the performance of image classification tasks significantly compared to Convolutional Neural Networks (CNNs) such as ResNets and RegNets, discussed in efficient vision transformer research work [4] This treakthrough in computer vision has led to state-of-the-ar results in various vision tasks, including image segmentation such SegFormer (66), TopFormer [77) and SegViT [74] and object
37th Conference on Neural Information Processing Systems (NeurTPS 2023).

---

detection, models like DETR 4 and Yolo[[5], However, one challenge faced by vision transformers the increasing computational complexity of the self self-attention module as the sequence length or image resolution grows. Additionally, model size and the number of floating-poing operations per second (FLOPS) also increase with image resolution or sequence length. These factors need to be carefully considered and addressed to ensure efficient and scalable deployment of vision transformers in real-world applications.
One way to address the computational complexity of attention-based Transformers is to replace the attention mechanism with a Multi-Layer Perceptron (MLP) based mixer layer [54 55 18 However, it is difficult to capture spatial information in the MLP mixers. This was addressed by the paper MetaFormer (7T. MetaFormer uses pooling operation to replace the attention layer. This however has the disad vantdvantage that the pooling operation is not invertible and could possibly lose information. Fourier based Transformers such as FourierFormer(E3 minimizes the loss of information by using Fourier Transform. But it has an inherent problem of separating the low and high-frequency components. The ability to separate low-frequency and high- frequency components of an image is important. Recently, transformers such as LiTv2 42 and iFormer [5I] have been proposed to address this problem. However, both LiTv2 and iFormer have the same O(r²) complexity as they use full-fledged self-attention networks, similar to ViT [13] and DelT [56]. which are weak in capturing fine- -eeeieadidd information of images. Vit, DeIT, LiTv2 and iFormer also have limitation with respect to network size or number of parameters. We propose a Scattering Vision Transformer (SVT) which uses a spectral scattering network to address the attention complexity and Dual-Tree Complex Wavelet Transform (DTCWT) to capture the fine- tne-grained information using spectral decomposition into low-frequency and high-frequency components of an image.
[11 GFNet 47 and AFNO [T6]
SVT uses the scattering network as the initial layer of the transformer, which captures fine-grained information (lines and edges, for instance) along with the energy component. SVT also uses attention nets in the deeper layers to capture long-range dependencies. The fine-grained information is captured by the high-frequency component of the scattering network by using the DTCWT transform while the low-frequency component is the energy component. SVT uses a Spectral Gating Network (SGN) to capture the effective features in both frequency components. Generally, the high- gegh-freuunccy component has extra directional information which makes it computationally complex while performing the gating operation. SVT addresses this complexity with a novel token and channel mixing technique using the Einstein Blending Method (EBM) in high-frequency component. SVT also uses Tensor Blending Method (TBM) in the low- www-frequency component. We also observe that the DTCWT is more invertible compared to other spectral transformations in the literature which are based on Fourier transforms and discrete wavelet transforms 12. 288. We quantify the invertibility in terms of reconstruction loss in the performance study section of this paper. The use of TBM for low-frequency components and EBM for high-frequency components is our contribution. It must be noted that low-frequency components contain the energy component of the signal which requires all the frequency components to provide energy compaction, while figh-frequency components can be represented by only few components, which can be achieved using Einstein multiplication. SVT is a generic recipe for componentizing the transformer architecture and efficiently implementing transformers with lesser parameters and computational complexity with the help of Einstein multiplication. So, this can be viewed as simple and efficient learning transformer architecture with minimal inductive bias.
Our contributions are as follows:
We introduce a novel invertible scattering network based on DTCWT transformation into vision transformers to decompose image features into low-frequency and high-frequency features.
We proposed novel SGN, which uses TBM to mix www-frequency representations and EBM to mix high-frequency representations. We use an efficient way of mixing high-frequency components using channel and token mixing with the help of Einstein multiplication.
Detailed performance analysis shows that SVT outperforms all transformers including LiT v2 and iFormer on ImageNet data, with a significantly lesser number of parameters. In addition, SVT also has comparable performance on other transfer learning datasets.
We show that SVT is efficient not only performance-wii but also in terms of number of parameters (memory size) as well as in terms of computational complexity (measured in Gigaflops). We also show that SVT is efficient for inferencing, by measuring its latency and comparing it with other state-of-an art transformers.

---

Elepbast
Altencar Layer
Clacifferl
Dhwelo
MIP
MLP
Atteation Layer
Laser Narn
Narr Nore
Aknt Tume
Allcation
Scattering Tranddrrantoon
MInSA
Atication
Tayn2
Santt Laser
Lays Nem
Aneation Layer
SSarn Laner
Tumeor Blendling
Senter Layer
Scater Layer
Layer
Se Layer
W.
HP
Postion bodding
Einsteia Bleading Methed
Seattering
Projection
Laye:
Wo 0aaaaal
C- C
Sepual Inaas
Iapal
Vanilla SVT
Scatter Layer
Hicrarchical SVT
Figure 1: This figure illustrates the architectural details of the SVT model with Scatter and Attention Layer structure. The Scatter Layer comprises a Scattering Transformation that processes Low- Frequency (LF) and High-Frequency (HF) components. Subsequently, we apply the Tensor and Einstein Blending Method to obtain Low-Frequency Representation (LFR) and High-Frequency Representation (HFR), as depicted in the figure, Finally, apply the Inverse Scattering transformation using LFR and HFR.
Method
2.1 Background: Overview of DTCWT and Decoupling of Low & High Frequencies
Discrete Wavelet Transform (DWT) replaces the infinite oscillating sinusoidal functions with a set of locally oscillating basis functions. which are known as wavelets (50, (7). Wavelet is combination of low-pass scaling function 66t) and shifted version of a band-pas wavelet function known as (r).. It can be represented mathematically as given below:
x(1)= Σ c(n)((t -)
Σ Σ 1=-
w2'11 n).
where c(n) is the scaling coefficients and d(,,) n)) is the wavelet coefficients. These coefficients are computed by the inner product of the scaling function()() and wavelet function (r)) with input x(r).
c(n)= = x(1)((1- n)dt, d(.n) = 2//2 /
(2)
DWT suffers from the following issues oscillations, shift variance, aliasing, and lack directionality. One of the solutions to solve the above problems is the Complex Wavelet Transform (CWT) with complex-valle scaling and wavelet function. The Dual-Tree Complex Wavelet Transform (DT- comes very close mirroring the attractive properties of the Fourier Transform, including a smooth, nonoscillating magnitude, a nearly shift-invariant magnitude with simple near-linear phase encoding of signal shifts, substantially reduced aliasing; and better directional selectivity wavelets in higher dimensions. This makes it easier detect edges and orientational features images. The six orientations of the wavelet transformation are given by 15 45, 75", 105., 135" and 165 The dual-tree CWT employs two real DWTs, the first DWT gives the real part of the transform while the second DWT gives the imaginary part. The two real DWTs use two different sets of filters, which are jointly designed to give an approximation of the overall complex wavelet transform and satisfy the perfect reconstruction (PR) conditions.
CWT) addresses the issues of the CWT. The DT-CWT 28
26 27
Let (n),), h (n) denote the low-pass and high-pass filter pair in the upper band, while (n),, 81 (n) denote the same for the lower band. Two real wavelets associated with each of the real wavelet transforms as (t) and w, (t). The complex wavelet (1) := (t) + W. (1) can be approximated using Half-Sample Delay (9 condition,iee (x,()) is approximately the Hilbert transform of v. (f) like
(n) z hn - 0.5) = (t(() R ((y,()) = V2 Σ h (1(n))n())) (1(1) = V2 Σ (n(n)
3

---

Similarly, we can define w. (f), Ø (1), and &1 (n). Since the filters are real, no complex arithmetic is required to implement DTČWT. It is just two times more expansive ID because the total output data rate is exactly twice the input data rate. It is also easy to invert, as the two separate DWTs can be inverted. Compare DTCWT with the Fourier Transform, which is difficult to obtain low pass and high pass components of an image and it is less invertible (Loss is high when we do Fourier and inverse Fourier transform) compared to DTCWT. Also, It cannot speak about time and frequency simultaneously.
2.2 Scattering Visual Transformer (SVT) Method
Given input image I E R3X222X224 we split the image into the patch of size R 16x116 and obtain embedding of each patch token using position encoder and token embedding network. X = (1) + F FI) where Fr. Fp refer to token and position encoding network. The detailed distinct components of the SVT architecture are illustrated in Figure Scattering Visual Transformer consists of three components such as a) Scattering Transformation, b) Spectral Gating Network, c) Spectral Channel and Token Mixing.
A. Scattering Transformation:
The input image is firstly patchified into feature tensor X E RC whose spatial resolution H x W and the number of channels is C. To extract the features of an image, we feed X into a series of transformer layers. We use a novel spectral transform based on an invertibility scattering network instead of the standard self-attention network. This allows us to capture both the fine-grain and the global information in the image. The fine- aneig information consists of texture, patterns, and small features that are encoded by the high-frequency components of the spectral transform. The global information consists of the overall brightness, contrast, edges, and contours that are encoded by the fow-frequency components of the spectral transform. Given feature X E CCHHX we use scattering transform using DTCWW [50] discussed in section 2. to obtain the corresponding frequency representations XF by X F Scatter(X), The transformation in frequency domain X provides two components, one low-frequency component i.e. scaling component X3. and one high- frequency component i.e. wavelet component X The simplified formulation for the real component of F pr is:
H-1 W-1
M-IH-1N Σ
(x(u, x (u,
(x,,)) Σ Σ R= 4=0
C & Mhu
Σ ==0
Σ
Σ
d m.hate m.har
whth
A= ==0
A=
M refers to resolution/level of decomposition and refers to directional selectivity. Similarly, we compute transformation for the imaginary component of PDrewr()).
B. Spectral Gating Network:
We introduce a novel method, Spectral Gating Network (SGN), to extract spectral features from both low and high-frequency components of the scattering transform. F shows the architecture of our method. We use learnable weight parameters blend each frequency component, but we use different blending methods for low and high frequencies. For the low-frequency component X E R CCHxW we use the Tensor Blending Method (TBM), which is a new technique. TBM blends X & with & using elementwise tensor multiplication, also known as Hadamard tensor product.
M₁ = X O Wll.
where (X,, w/) E R CXHW and M E CXHWW.
(4)
We having same dimension as in Xz. Mg is the low-frequency representation of the image and it captures global information of the image. One of the biggest challenges to getting effective features in the gighffrequency components X R RxxCxHXW which are complex-valuee and have 'k' times more dimensions than the low- wwwfrrquency components (X₂). Therefore, using the same Tensor Blending Method for the high-frequency components Xy, would increase the number of parameters by 2k times and also the computational cost (GFL.OP), where & refers to directional selectivity, a factor of 22' indicating complex value comprising real and imaginary. To address this issue, we propose new technique, the Einstein Blending Method (EBM), to blend the high-frequency components X with the learnable weight parameters Wi, efficiently and effectively in the Spectral Gating Network that we propose in this paper. By using EBM, we can capture the fine-grain information in the image, such as texture, patterns, and small features.

---

To perform EBM. we first reshape tensor A from R xC B.XWXC,CC, where
W
C x C. and b >> d. We then define a weight matrix of size E RCxxC,CC,, We then perform Einstein multiplication between A and W along the last two dimensions, resulting in a blended feature tensor Y R HxXWxX,KK as shown in the Figure The formula for EBM is:
If
C. Gx
C
6
Ce
AE BEROORGG
We W'
Y= AW e RRMis
Y <XXXXXXXC A HXXXXCxXG
WC
Figure 2: Einstein Blending Method
C. Spectral Channel and Token Mixing:
We perform EBM in the channel dimension the high-frequency component we call Spectral Channel Mixing and following that we perform EBM in the token dimension of the high-frequency component, which we call Spectral Token Mixing. To perform EBM in the channel dimension, we first reshape the -requeryy component Xy from IR XXXXXXX to R BRxxHxXXC,xXC where C = C, Ca. and >> d. We then define a weight matrix of size W w. multiplication between X and Wwe along the last two dimensions, resulting blended feature tensor S ve E R2xXxHXWXX,XXC The formula for EBM in Channel mixing is:
R We then perform Einstein
yy
2XXXHWWXXXG - XXXXHXWXX,XG
CxXCxXCa by.
(5)
To perform EBM in the Token dimension, we first reshape the high- fregefrry component Sw. from X2 R XX where Height II) Width(W). We then define a weight matrix of size w, E R W We then perform Einstein multiplication between X and Wi, along the last two dimensions, resulting in blended feature tensor S E R 2X The formula for EBM Token mixing is:
W Xx 32xxxCXW>F V.
WxHxH
bw.
(6)
W/
Where X represents an Einstein multiplication, the bias terms b. E R by, E R HxH Now the total number of weight parameters in the high-frequency gating network is (C) instead of (C H x x k 2) where C >>> H and bias (C, x C2) + (H x W). This reduces the number of parameters and multiplication while performing high- figh-frequency gating operations in an image. We use a standard torch package (48) to perform Einstin multiplication. Finally, we perform inverse scattering transform using low-frequency representation( 4 and high-frequency representation (6) to bring back the spectral domain to the physical domain. Our SVT architecture consists of L layers, comprising a scatter layers and (L a) attention layers (59]. where L denotes the network's 's depth. The seatter layers, being invertible, adeptly capture both the global and the fine fine-grain information in the image effectively via low-pass and nigh-pass filters, while attention layers focus extracting semantic features and addressing long-range dependencies present in the image.
3 Experiment and Performance Studies
We evaluated SVT through various mainstream computer vision tasks including image recognition, object detection, and instance segmentation. To compare the quality of SVT transformer features, we conducted the following evaluations on standard datasets: a) We trained and evaluated Ima- geNet1K 10) from scratch for image recognition task. b) We performed transfer learning on CIFAR- 10 30, CIFAR-100 300, Stanford Car [29], and Oxford Flower- 102 [41] for Image recognition task. c) We conducted ablation studies to analyze variants of SVT transformers using scatter net with the help of various spectral mixing techniques. We also compare similar decomposition architecture. d) We fine-tune SVT for downstream instance segmentation tasks. We also perform in-depth analysis of the SVT, by conducting layer-wise analysis as well as invertibility analysis as well as latency analysis, and comparison.
results with transformers having
3.1 Comparison with Similar architectures
We compare SVT with LiTv2 (Hilo) which decomposes attention to find low and high-frequency components. We show that LiTv2 has top-1 accuracy of 83.3%, while SVT has top-1 accuracy of
5

---

Table 1: The table shows the performance of various vision backbones the ImageNet 10 dataset for image recognition tasks. * indicates additionally trained with the Token Labeling objective using Mix Token (25 and convolutional stem (conv-stem) [60] for patch encoding. This table provides results for imput image size 224 x 224. We have grouped the vision models into three categories based their GFL.Oss (Small, Base, and Large). The GFLOP ranges: Small (GFLOPs<<)) Base
(6=GFLOPs<<0), and Large
Method
Params
GFLOPS
Top-1
Top
Method
Params
GELOPS
Top
Top-5
Small
Large
ResNet-50 21
25.5M
4.1
78.3
94.3
ResNet-152
60.2M
11.6
81.3
95.5
BoTNet.
20.8M
80.4
95.0
ResNeXt101
67
83.5M
15.6
81.5
Cios-ViT-S
26.7M
5.6
81.0
gMLP.B
73.0M
15.8
81.6
Swin-T 39
0M
95.5
DefT-B
86.6M
17.6
95.6
ConViT-S 14
27. XM
5.4
81.3
95.7
SE-ResNe-1151
23
66.8M
11.6
82.2
95.9
T2T
Ciws-ViT-I
RegionVi1--T+
5
14.3M
81.5
ResNeSt-101
75
48.3M
10.2
82.3
SE.CoTNedDD.S
23.1M
4.1
81.6
95.8
ConViTT-B 14
86.5M
16.8
82.4
95.9
Twins.SVT-S
24.1M
95.6
PoolFormer
73.0M
11.8
82.5
CoaT-Lite-S
20.0M
4.0
81.9
95.5
T2T-VITI-24
64. IM
15.0
82.6
95.9
PVTv2-B2 60
25.4M
TNT-B 19
65.6M
LITv2-8 42
28.0M
3.7
82.0
CycleMLP-B4
52.UM
10.1
83.0
VTT-
Wave-ViI-S
19.8M
82.7
96.2
RegionViI-B
72.7M
13.0
83.2
96.1
CSwin-T 12
23.0M
4.3
82.7
CycleMIPP.BS
76.0M
12.3
83.2
DaViT-Ti
|
28. 3M
82.8
ViP-Targe/7
88,0M
24.4
83.2
SVT-H-S
21.7M
3.9
83.1
96.3
CaTT-S36 [
68.4M
13.9
83.3
iFormer-S
56
CMT-S 17
25.1M
4.0
83.5
BoTNet $1 S1-128
52
75.1M
19.3
83.5
96.55
31.0M
96.5
Wive-ViT-S*
70
22.7M
83.9
96.6
Wave-MLPP M.P.
53
63.0M
10.2
83.6
SVT-H-- (Ours)
22.0M
3,9
84,2
96.9
LITv2.B +2
87.0M
13.2
83.6
Base
PVTV2-B4
62.6M
83.6
96.7
ResNet-101 21
44.6M
7.9
80.0
95.0
ViL-Base 76
55.7M
13.4
83.7
T2T-ViT-19 72
39 2M
8.5
81.9
95.7
Hire MI.P
96.0M
13.4
83.8
CvT-21
Region
73.8M
13.6
GENei
54,0M
82.9
96.2
Focal Base 69
89.8M
16.0
83.8
96.5
Swin-S 30
50.0M
8.7
83.2
96.2
PVTV2-B5
62
82.0M
11.8
83.8
96.6
Twins-SVT-B
83.2
ColNeiD-1
55.8M
17.0
97.0
Co1NeiD-101 35
40.9M
8.5
83.2
96.5
DAT-B 63
88.UM
13.8
84.0
55.8M
LJTv2.M E0c
49.0M
7.5
83.3
CSwin B
78.0M
15.0
84.2
RegionViT-M+
HorNet- 8&F
15.5
MVSTv2-S
35
35.0M
83.6
DynaMixes1
+
97.0M
27.4
84.3
CSwin-S
35.00 OM
6.9
83.6
MVTV2-B
$2.0M
10.2
84.4
DaViT-S
DaViT-B 11]
15.5
VOLO-D1*
78
26. 6M
6.8
84.2
CMT-L
17
74.7M
19.5
84.8
MaxViT
MoxViT-S
58
69.0M
11.7
84.5
VOLO-D2*
58, 7M
14.1
85.2
iFormer-B
[1
97.0
VOLO-D3*
86.3M
20.6
Wave-ViT-B*
70
33.5M
84.8
97.1
Wave-ViT-L-
20
57.5M
14.8
85.5
97.3
SVT-H-B" (Ours)
32.8M
6.3
85.2
97.3
SVT-H-L-" (Ours)
54.0M
12.7
85.7
97.5
85.2% with fewer number of parameters. We also compare SVT with iFormer [5]] which captures low and figh-frequency information from visual data, whereas SVT uses an invertible spectral method, namely the scattering network, to get the low-frequency and high figh-frequnncy components and tensor and Einstein mixing respectively to capture effective spectral features from visual data. top I accuracy is 85.2, which better than iFormer-B, which 84.6 with lesser number of parameters and FLOPS. We compare SVT with WaveMLP [53] which an MI.P mixer-based technique that amplitude and phase information represent the semantic content of image. SVT uses low-frequency component as an amplitude of the original feature, while high-frreuenee component captures complex semantic changes in the input image. Our studies have shown, as depicted in Table- that SVT outperforms WaveMLP by about 1.8%.
3.2 Comparison with State of the art methods
We divide the transformer architecture into three parts based on the computation requirements (FLOP counts) small (less than GFLOPS), base (6-10 GFLOPS), and large (10-30 GFLOPS). We use similar categorization as WaveViT 770) Notable recent works falling into the small category include C-Swin Transformers [12), LiTv2 MaxVIT (58, iFormer(51], CMT transformer, PVTv2 and ViT It's worth mentioning that WiT relies on extra annotations to achieve its best results. In this context, SVT-H-S stands out as the state-of-the oftthe-rttttttt model in the small category,

---

Table 2: Initial Attention Layer vs Scatter Layer vs Initial Convolutional: This table com- pares SVT transformer where initial scatter layers and later attention layers, SVT-Inverse where ini- tial attention layers and later scatter layers, and SVT with initial convolutional layers. Also, we show an alternative spectral layer and attention layer. This shows that the Initial scatter layer works better compared to the rest.
Table 4: Results on transfer learning datasets.
We report the top-1 accuracy on the four datasets.
Model
CIFAR
CIFAR
Flowers
Cars
10
100
102
196
ReaNe150
21
96.2
90.0
13
98.1
87.1
89.5
VII-L/16 13
97.9
86.4
89.7
Deit Deit-B/16 B/16
56
99.1
90.8
98.4
92.1
ResMI [51]
98.7
89.5
97.9
89.5
GFNet-XS [+]
98.6
89.1
98.1
92.8
Model
Params(M) FLOPS(G)
Top-I(%)
Top-5(%)
G-N 47
93.2
SVT-H.S
22.0M
3.9
84.2
96.9
SVT-I-B
99.22
91.2
98.9
93.6
SVT-H---Ini--CNN
21.7M
4.1
95.7
SVT-H-S-Inwerse
21.8M
3.9
83.1
94.6
Table 5: The performances of various vision back-
SVT-H-S-llerrate
22.4M
4.6
83.4
95.0
bones on COCO val2017 dataset for the down- stream instance segmentation task such as Mask R-CNN 1x (20) method. We adopt Mask R-CNN as the base model, and the bounding box & mask Average Precision (i.e., AP & AP") are reported for evaluation
Table 3:
This table shows the ablation analysis of various spectral layers SVT architecture such as FN, FFC, WGN, and FNO, We conduct this ablation study on the small-size networks in stage architecture. This indicates that SVT performs better than other kinds networks.
Backbone
AP
AP
AP,
AP
AP
FLOPS
Top-1
Top.5
Invertible loss(1)
ResNer50
58.6
41.4
34.4
55.1
36.7
Model
(M)
(G)
(%)
(%)
Swin-T (39
42.2
64.6
46.2
39.1
61.6
42.0
FFC
21.53
4.5
83.1
95.23
Twins-VVT-S
43.4
66.0
47.3
40.3
63.2
43.4
FN
21.17
3.9
84.02
96.77
LITv2 S
+
44.9
40.8
ENO
21.33
3.9
84.09
96.86
3.27c-05
RegionVil-S
3
44.2
40.8
21.59
3.9
83,70
96.56
0590e
B2
67.1
49.6
41.2
64.2
44.4
SVT
22.22
3.9
84,20
96.93
6.64e-06
SVI-H-S
46.0
68.1
50.4
41,9
65.0
45.1
achieving top- I accuracy of 84.2%. Similarly, SVT-H-B surpasses all the transformers in the base category, boasting a top-1 accuracy of 85.2%. Lastly, SVT-H-L outperforms other large transformers with top-1 accuracy of 85.7% when tested on the ImageNet dataset with image size of 224x224.
When comparing different architectural approaches, such as Convolutional Neural Networks (CNNs), Transformer architectures (attention-base) models), MLP Mixers, and Spectral architectures, SVT consistently outperforms its counterparts. For instance, SVT achieves better top-1 accuracy and parameter efficiency compared to CNN architectures like ResNet 152 [21], ResNeXt [67], and ResNeSt terms of top-1 accuracy and number of parameters. Among attention- -based architectures, Mix 58) has been recognized as the best performer, surpassing models like DeiT (56). Cross- ViT (6), ViT (78). T2T [72] etc. with a top-1 accuracy of 85.0. However, SVT achieves an even higher top-1 I accuracy of 85.7 with less than half the number of parameters. In the realm of MLP Mixer-based architectures, DynaMixer (63] emerges as the oo-performing model, surpassing MLP. mixer[54), gMLP (37) CycleMLP 77. Hire Hire-MLP(18] AS-MLP (36). WaveMLP [53] PoolFormer [71 and DynaMixer-L 663 with a top-1 I accuracy of 84.3%. In comparison, SVT-H-L outperforms DynaMixer with top-1 accuracy of 85.7% while requiring fewer parameters and computations. Hierarchical architectures, which include models like PVT [61. Swin (39 transformer, CSwin [12] transformer, Twin 9 transformer, and VOLO 73 also considered. Among this category. VOLO achieves the highest top-1 accuracy of 85.4% However, it's important to note that SVT outperforms VOLO with top-1 accuracy of 85.7% % for SVT-H-I.. Lastly, in the spectral architecture category, models like GFNet[77, iFormer [51] LiTv2 (42). HorNet 46). Wave-ViT [70), etc. are examined. Wave-ViT was previously the state-of-the-ar method with top-1 accuracy of 85.5%. Nevertheless, SVT-H-L surpasses Wave-ViT in terms of top-1 I accuracy, network size (number of parameters), and computational complexity (FLOPS), as indicated in Table
3.3
What Matters: Does Initial Spectral Initial Attention Initial Convolution Layers?
The ablation study was conducted show that initial scatter layers followed by attention in deeper layers are more beneficial than having later scatter and initial attention layers ( SVT-H-S-Inverse) We also compare transformer models based on an alternative to the attention and scatter layer (SVT-H-S- Alternate) as shown in Table 2 From all these combinations we observe that initial scatter layers followed by attention in deeper layers are more beneficial than others. We compare the performance of SVT when the architecture changes from all attention (PVTvv 62 to all spectral layers (GFNet
7

---

Table 6: SVT model comprises low low-frequnncy component and High-frequency component with the help of scattering net using Dual tree complex wavelet transform. Each frequency component is controlled by a parameterized weight matrix using Patch mixing and/or Channel Mixing. this table shows details about all combinations and SVTTEEE is the best performing among them.
Params (M)
FLOPS
Top-1 (%)
Top-5 (%)
Low Frequency
High Frequency
Backbone
(G)
Token
Channel
Token
Channel
SVT TITT SVT FETT
T
T
T
T
25.18
4.4
83.97
96.86
E
E
T
T
21.90
4.1
83.87
96.67
BEEE
E
E
E
E
21.87
3.7
83.70
96.56
SVTTTEE T
T
E
E
22.01
3.9
84.20
96.82
SVTTTEX T
T
E
X
21.99
4.0
84.06
96.76
SVTTIXE
T
X
E
22.25
4.1
84.12
96.91
well as few spectral and remaining attention laVers(SVT ours). We observe that combining spectral and attention boost the performance compared to all attention and all spectral layer-based transformer as shown in Table- 2] We have conducted experiment where the initial layers of VIT are convolutional networks and later layers are attention layers to compare the performance of
SVT. The results are captured in Table- [ where we compare SVT with transformers having initial convolutional layers such as CVT [64], CMT [17]. and HorNet [46). Initial convolutional layers in transformer not performing well compared to the initial scatter layer. Initial scatter layer-based
transformers have better performance and less computation cost compared to initial convolutional
layer-based transformers which shown in Table- 2
3.4 Ablation analysis
SVT uses scattering network to decompose the signal into low-frequency and high-frequency components. We use a gating operator to get effective learnable features for spectral decomposition. The gating operator is a multiplication of the weight parameter in both high and low frequencies. We have conducted experiments that use tensor and Einstein mixing. Tensor mixing is a simple multiplication operator, while Einstein mixing uses an Einstein matrix multiplication operator [48). We observe that in low-frequency components, Tensor mixing performs better as compared to Einstein mixing. As shown in Table- 6 we start with SVTTTTT. which uses tensor mixing in both high and low-frequency components. We see that it may not perform optimally. Then we reverse it and use Einstein mixing in both low and high- sigh-frequency components this also does not perform optimally. Then, we came up with the alternative method SVTTTEE which uses tensor mixing in low frequency and Einstein mixing in high frequency. The high-frequency further decomposes into token and channel mixing, whereas in low-frequency we simply tensor multiplication as it is an energy or amplitude component.
In the second ablation analysis, we compare various spectral architectures, including the Fourier Network (FN), Fourier Neural Operator (FNO), Wavelet Gating Network (WGN), and Fast Fourier Convolution (FFC). When we contrast SVT with WGN, it becomes evident that SVT exhibits superior directional selectivity and more adept ability to manage complex transformations. Furthermore, in comparison to FN and FNO, SVT excels in decomposing frequencies into low and fegeffrry components. It's worth noting that SVT surpasses other spectral architectures primarily due to its utilization of the Directional Dual-Tree Complex Wavelet Transform (DTCWT), which offers direc- tional orientation and enhanced invertibility, demonstrated in Table 3 analysis, please refer to the Supplementary section.
For more comprehensive
3.5 Transfer Learning and Task Learning
We train SVT on ImageNet K data and fine-tune it on various datasets such as CIFARIO. CIFARI00, Oxford Flower, and Stanford Car for image recognition tasks. We compare SVT-H-B performance with various transformers such as Deit (56), ViT [13,, and GFNet (7) as well as with CNN architectures such as ResNet50 and MLP mixer architectures such as ResMLP. This comparison is shown in Table- [ can observed that SVT-H-B outperforms ofate-of-art on CIFAR with top-1 accuracy of 99.19 99.1%, CIFAR 100 with top-1 accuracy of 91.3% Flowers with a top-1 accuracy of 98.9% and Cars with top-1 accuracy of 93.7%. We observe that SVT has more representative features and has an inbuilt discriminative nature which helps in classifying images into various categories. We use
8

---

Table 7: Latency(Speed test): This table shows the Latency (mili sec) of SVT compared with Conv type network, attention type transformer, POOÍ type, MLP type, and Spectral type transformer. We report latency per sample on A100 GPU. We adopt the latency table from EfficientFormer
Table 8: Invertibility: This table shows the invertibility of SVT(DTCWT) compared with Fourier and DWT. We also compare different direc- tional orientations and show the reconstruction loss (MSE)
34
Type
Params
Top-1
Latency (ms)
(M)
(G)
(%)
ResNet50 21
Convolation
25.5
4.1
78.5
9.0
Deil-S
22.5
81.2
in an image.
PVT-S[62
Atlention
24.5
3.8
79.8
23.8
Model
MSE
PSNR (dbxf)
T2T-141
Atteation
21.5
4.8
81.5
21.0
Swin-T 88
Atlention
29.0
4.5
81.3
22.0
Fourier (PFT)
3.27e-05
11.18
CSwin [22
Attention
23.0
4.3
82.7
28.7
DWT-MI
8.90g-05
76.33
DWT-M2
3.19
84.67
Pool
31.0
ResMLPP-S(55
MLP
30,0
6.0
79.4
17.4
DWTMS
1,08c-05
91.94
FfficienFormer
MetaBlock
31.3
3.9
DICW T-MI
6,64c-06
137.97
GFNot-H33447
Spectral
32.0
81.5
14.3
DICWT-M2
2.01e-06
138.87
SVT-H-S
Spectral
22.0
3.9
84.2
14.7
1.23e.07
142.14
pre-trained SVT model for the downstream instance segmentation task and obtain good results on the MS-COCO dataset as shown Table-
3.6 Latency Analysis
It's important to highlight that Fourier Transforms, as mentioned in the GFNet 47 are not inherently capable of performing low-pass and high-pass separations. In contrast, GFNet consistently employs tensor multiplication, a method that, while effective, may be less efficient compared to Einstein multiplication. The latter approach is known for reducing the number of parameters and computational complexity. As a result, SVT does not lag behind in terms performance or computational complexity; rather, gains enhanced representational power. This is exemplified in Table which provides comparison of latency, FLOPS (Floating- Point Operations per Second), and the number of parameters. Table 7 specifically demonstrates the latency (measured in milliseconds) of SVT in relation to various network types, including convolution-baas based networks, Attention bosseddo--bbeed Transformer networks, Pool- based Transformer networks, MLP- Transformer networks, and snectral-basee Transformer networks. The reported latency values are on a per-sample basis, measured on an A A100 GPU.
3.7 Invertibility Versus Redundancy trade-off Analysis
We conducted an experiment to illustrate that invertibility not only enhances performance but also contributes to image comprehension. To do this, we passed an image through the raw DTC-WT and performed an inverse DTC-WT operation to calculate the reconstruction loss. The experiment was executed across various values of "M" corresponding to the level of decomposition and orientations in SVT. We observed that the reconstruction loss decreased as the value of "M" increased, indicating that SVT's ability to comprehend the image improved. These orientations effectively captured higher- order image properties, enhancing SVT's performance. We further compared different spectral transforms, including the Fourier Transform, Discrete Wavelet Transform (DWT), I), and DTCCWT.. Our findings demonstrated that the reconstruction loss lower for DTC-WT compared to other spectral transforms, as depicted in for FFT, DWT stages 1, 2, and 3, and DTCWT at stages 1, and 3. The MSE decreased as increased the level of decomposition (M) and the degree of selectivity. DTCWT consistently exhibited lower MSE compared to DWT. Furthermore, the peak signal-to-noise ratio (PSNR) of DTCWT surpassed that of DWT and the Fourier Transform. PSNR gauges the quality of the reconstructed image, expressing the ratio between the maximum possible power of an image and the power of noise affecting its representation, measured in decibels (dB). A higher PSNR indicates superior image quality. high-quality reconstructed image characterized by low MSE and high PSNR values. For further details on redundancy, please consult the supplementary table. Additionally, we have visualized the filter coefficients for all six orientations in the supplementary materials.
Table g we quantified the mean squared error (MSE)
3.8 Limitations
SVT currently uses six directional orientations to capture an image's fine-grained semantic information. is possible to go for the second degree, which gives thirty-six orientations, while the third degree gives 216 orientations. The more orientations, the more semantic information could be captured, but

---

this leads to higher computational complexity. The decomposition parameter 'M' is currently set to 1 to get single low-pat now-pass and high-pass components. Higher values of 'M' give more components in both frequencies but lead to higher complexity.
4 Related Work
The Vision Transformer (ViT) 13 was the first transformer-based attempt classify images into pre-defined categories and use NLP advances in vision. Following this, several transformer based approaches like DeiT [56). Tokens-to-token ViT [72]. Transformer iN Transformer (TNT) [19], Cross-
ViT [6]. Class attention image Transformer(CaiT [57] Uniformer [32) Beit. [3) SViT 45 RegionViT [5). ViX (58) etc. have all been proposed to improve the accuracy using multi-headed self-attention (MSA). PVT [61, Swin'T [39]. CSwin 12] and Twin (3) use hierarchical architecture to improve the performance of the vision transformer on various tasks. The complexity of MSA is O(r2). For high-resolution images, the complexity increases quadratically with token length. PoolFormer [7]] method that uses a pooling operation over small patch which has obtain down-sampled version of the image to reduce computational complexity. The main problem with PoolFormer that it uses a MaxPooling operation which is not invertible. Another approach to reducing the complexity is the spectral transformers such as FNet [31] GFNet 47, AFNO [16]. WaveMix (24). Wave VIT (70), SpectFormer [44,, FourierFormer
does not use inverse Fourier transforms, leading to an invertibility issue. GFNet 47 solves this by using inverse Fourier transforms with a gating network. AFNO [6 uses the adaptive nature of a Fourier neural operator similar to GFNet. SpectFormer 44 introduces novel transformer architecture that combines both spectral and attention networks for vision tasks. GFNet, SpectFormer, and AFNO do not have proper separation of cwwffreueeccy and of images. In contrast, SVT has a clear separation of frequency components and uses directional orientations capture semantic information. FourierIntegral 44 is similar to GFNet and may have similar issues in separating frequency components.
40. etc. FNet
components and may struggle to handle the semantic content
WaveMLP [53] is a recent effort that dynamically aggregates tokens wave function with two parts, amplitude and phase to capture the original features and semantic content of the images respectively, SVT uses scattered network to provide low-frequency and high-frequency components. The high- frequency component has six or more directional orientations to capture semantic information in images. We use Einstein multiplication in token and channel mixing of high-frequency components leading to lower computational complexity and network size. In Wave-ViT [70]. the author has discussed the quadratic complexity of the self-attention network using wavelet transform to perform lossless down-sampling using wavelet transform over keys and values. However, WiT still has the same complexity it uses attention instead of spectral layers. SVT uses the scatter network which is more invertible compared to Wave
One of the challenges in MSA is its inability to characterize different frequencies in the input image. Hilo attention (LiTv2) 42 helps to find high-frequency and low-frequency components by using a novel variant of MSA. But it does not solve the complexity issue of MSA. Another parallel effort named Inception Transformer came up (5T), which uses an Inception mixer to capture high and low-frequency information in visual data. iFormer still has the same complexity as it uses attention as the low- freefreyyency mixer. SVT in comparison, uses a spectral neural operator to capture low and high frequency components using the DTCWT. This removes the O(r²) complexity as it uses spectral mixing instead of attention. iFormer [51 uses a non-invertible max pooling and convolutional layer to capture high-frequency components, whereas, in contrast, SVT's mixer is completely invertible. SVT uses a scatter network to get a better directional orientation to capture fine-grained information such lines and edges, compared Hilo attention and iFormer.
Conclusions and Future Research Directions
We have proposed SVT, which helps in separating low-frequency and high-frequency components of an image, while simultaneously reducing computational complexity by using Einstein multiplication- based technique for efficient channel and token mixing. SVT has been evaluated on standard bench- marks and shown to achieve state-of-the-art performance on standard benchmark datasets on both image classification tasks and instance segmentation tasks. It also achieves comparable performance on object detection tasks. We shall experiment with SVT in other domains such as speech and NL.P as we believe that it offers significant value in these domains well.
10

---

References
[1] https://ppenal..m///////oodaa
2022.
[2] Hezam Albaqami, G Hassan, and Amitava Datta. Comparison of wpd, dwt and dtewt for multi-class seizure type classification. In 2021 IEEE Signal Processing in Medicine and Biology Symposium (SPMB), pages 1-7. IEEE, 2021.
[3] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In International Conference on Learning Representations, 2021.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UKK August 23-28, 2020, Proceedings, Purt I 16, pages 213-229. Springer, 2020.
[5] Chun-Fu Chen, Rameswar Panda, and Quanfu Fan. Regionvit: Regional-to-Iocal attention for vision transformers. In International Conference on Learning Representations, 2022.
[6] Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scal vision transformer for image classification. Proceedings the vision, pages 357-366, 2021.
the IEEE/CVF international conference on computer
[7] Shoufa Chen, Enze Xie, GE Chongjian, Runjian Chen, Ding Liang, and Ping Luo. Cyclemip: A mlp like architecture for dense prediction. In International Conference Learning Representations, 2022.
[8] Aakanksha Chowdhery. Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et Palm: Scaling language modeling with pathways. arXiv preprint urXiv:2204.023: 2022.
[9] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang. Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. Advances in Neural Information Processing Systems, 34:9355-9366, 2021.
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenct: large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Icce, 2009.
[11] Mingyu Ding, Bin Xiao, Noel Codella, Ping Luo, Jingdong Wang. and Lu Yuan. Davit: Dual attention vision transformers. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIV, pages 74 Springer, 2022.
[12] Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12124- 12134, 2022.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. image worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
[14] Stéphane d' d'Ascoli, Hugo Touvron, Matthew L Leavitt, Ari S Morcos, Giulio Biroli, and Levent Sagun. Couvit: Improving vision transformers with soft convolutional inductive biases. In International Conference on Machine Learning, pages 2286-2296. PMLR, 2021.
[15] Yuxin Fang, Bencheng Liao, Xinggang Wang. Jiemin Fang, Jiyang Qi, Rui Wu, Jianwei Niu, and Wenyu Liu. You only look at one sequence: Rethinking transformer in vision through object detection. Advances in Neural Information Processing Systems, 114:26883-21999 2021.
[16] John Guibas, Morteza Mardani, Zongyi Li, Andrew Tao, Anima Anandkumar, and Bryan Catanzaro. Efficient token mixing for transformers via adaptive fourier neural operators. In International Conference on Learning Representations, 2022.
[17] Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang. and Chang Xu. Cmt: Convolutional neural networks meet vision transtormers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2022.
11

---

[18] Jianyuan Guo, Yehui Tang. Kai Han. Xinghao Chen, Han Wu, Chao Xu, Chang Xu, and Yunhe Wang- Hire-mlp: Vision mlp via hierarchical rearrangement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 826-836, June 2022.
[19] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. Advances in Neural Information Processing Systems, 34:15908-15919, 2021.
[20] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-can. In Proceedings of the IEEE international conference computer vision, pages 2961-2969, 2017.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and patlern recognition, pages 770-778, 2016.
[22] Qibin Hou, Zihang Jiang, Li Yuan, Ming-Ming Cheng. Shuicheng Yan, and Jiashi Feng. Vision permutator:
A permutable mlp-like like architecture for visual recognition. IEEE Transactions on Pattern Analysis & Machine Intelligence, (01 10-1, 2022.
[23] Jie Hu, Li Shen, and Gang Sun. Squeeze-and -ed---andddeee--eeeee excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132-7141, 2018.
[24] Pranav Jeevan and Amit Sethi. Wavemix: Resource-efficien token mixing for images. arXiv preprint arXiv:220300368, 2022.
[25] Zi-Hang Jiang, Qibin Hou, Li Yuan, Daquan Zhou, Yujun Shi, Xiaojie Jin, Auran Wang, and Jiashi Feng. All tokens matter. Token labeling for training better vision transformers. Advances in Neural Information Processing Systems, 11118859-18600 2021.
[26] Nick Kingshury. Image processing with complex wavelets. Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences, 357(1760):2543 1999.
[27] Nick Kingsbury. Complex wavelets for shift invariant analysis and filtering of signals. Applied and computational harmonic analysis, 10(3)234-2533 2001.
[28] Nick G Kingsbury. The dual-tree complex wavelet transform: new technique for shift invariance and directional filters. In IEEE digital signal processing workshop. volume 86, pages 120-131. Citeseer, 1998.
[29] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained catcgorization, In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561, 2013.
[30] Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
[31] James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. Fnet: Mixing tokens with fourier transforms. arXiv preprint 2105:22005.08222 2021.
[32] Kunchang Li, Yali Wang, Junhao Zhang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unifying convolution and self-attention for visual recognition. arXiv preprint 2rXiv:2201.094 2022.
[33] Yanghao Li, Chao-Yusa Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong. Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4804-4814, 2022.
[34] Yanyu Li, Geng Yuan, Yang Wen, Ju Hu, Georgios Evangelidis, Sergey Tulyakov, Yanzhi Wang, and Jian Ren. Efficientformer: Vision transformers mobilenet speed. Advances in Neural Information Processing Systems, 12934-1112,94, 2022.
[35] Yehao Li, Ting Yao, Yingwei Pan, and Tao Mei. Contextual transformer networks for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[36] Dongze Lian, Zehao Yu, Xing Sun, and Shenghua Gao. As-mlp: An axial shifted mlp architecture for vision. In International Conference on Learning Representations, 2022.
[37] Hanxian Liu, Zihang Dai, David So, and Quoe VLe. Pay attention to mlps. Advances in Neural Information Processing Systems,
9215, 2021.
[38] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning. Yue Cao, Zheng Zhang Li Dong, et al. Swin transformer v2: Scaling capacity and resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12009-12019, 2022.
12

---

[39] Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[40] Tan Minh Nguyen, Minh Pham, Tam Minh Nguyen, Khai Nguyen, Stanley Osher, and Nhat Ho. Fouri- erformer: Transformer meets generalized fourier integral theorem. In Advances Neural Information Processing Systems, 2022.
[41] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 222.729. IEEE, 2008.
[42] Zizheng Pan, Jianfei Cai, and Bohan Zhuang. Fast vision transformers with hilo attention. In Advances in Neural Information Processing Systems, 2022.
[43] Badri N Patro and Vijay Agneeswaran. Efficiency 360: Efficient vision transformers. arXiv preprint arXiv:2302. 08374, 2023.
[44] Badri N Patro, Vinay Namboodiri, and Vijay Srinivas Agneeswaran. Spectformer: Frequency and attention what you need in a vision transformer. arXiv preprint arXiv:2304. 06446, 2023.
[45] Tianming Qiu, Ming Gui, Cheng Yan, Ziqing Zhao, and Hao Shen. Svit: Hybrid vision transformer models with scattering transform. In 2022 IEEE 32nd International Workshop on Machine Learning for Signal Processing (MLSP), pages 01-06. IEFF, 2022.
[46] Yongming Rao, Wenliang Zhao, Yansong Tang. Jie Zhou, Ser Nam Lim, and Jiwen Lu. Hornet: Effi- cient oigh-order spatial interactions with recursive gated convolutions. Advances in Neural Information Processing Systems, 35:10333-10366. 2022.
[47] Yongming Rao, Wenliang Zhao, Zheng 7hu, Jiwen Lu, and Jie Zhou. Global filter networks for image classification. Advonces in Neural Information Processing Systems, 34:980-993, 2021.
[48] Alex Rogozhnikov. Einops: Clear and reliable tensor manipulations with einstein-like notation. In International Conference on Learning Representations, 2022.
[49] Ivan W Selesnick. Hilbert transform pairs of wavelet bases. IEEE Signal Processing Letters, 876)170 2001.
[50] Ivan W Selesnick, Richard Baraniuk, and Nick Kingsbury. The dual-tree complex wavelet transform. IEEE signal processing magazine, 22(6):12 123-151,,,,1 2005.
[51] Chenyang Si, Weihao Yu, Pan Zhou, Yichen Zhou, Xinchao Wang, and Shuicheng YAN Inception transformer. In Advances in Neural Information Processing Systems, 2022.
[52] Aravind Srinivas, Tsung-Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16519 16529, 2021.
[53] Yehui Tang, Kai Han, Jianyuan Guo, Chang Xu, Yanxi Li, Chao Xu, and Yunhe Wang. An image patch is wave: Phase-aware vision mlp. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10935-10944, 2022.
[54] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung. Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. Advances in Neural Information Processing Systems, 34:24261 2021.
[55] Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, et al. Resmlp: Feedforward networks for image classification with data-efficient training. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[56] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-cfficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021.
[57] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 32 42-42,20 2021.
13

---

[58] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In Computer Vic 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIV, pages 459-479. Springer, 2022.
[59] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin, Attention all you need. Advances in neural information processing systems, 30, 2017.
[60] Pichao Wang, Xue Wang, Hao Luo, Jingkai Zhou, Zhipeng Zhou, Fan Wang. Hao Li, and Rong Jin. Scaled relu matters for training vision transformrrs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2495-2503, 2022.
[61] Wenhai Wang, Enze Xie. Xiang Li, Deng- Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 568-578, 2021.
[62] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 8(3):415 424, 2022.
[63] Ziyu Wang, Wenhao Jiang, Yiming M Zhu, Li Yuan, Yibing Song, and Wei Liu. Dynamixer: a vision mlp architecture with dynamic mixing. In International Conference on Machine Learning, pages 22691 -22701.. PMLR, 2022.
[64] Haiping Wu, Bin Xiao, Noel Codella, Mengehen I Liu, Xiyang Dai, La Yuan, and I Lei Zhang, Cvt: Introducing convolutions to vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22-31, 2021.
[65] Zhuofan Xia, Xuran Pan. Shiji Song, Li Erran Li, and Gao Huang. Vision transformer with deformable attention. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4794 4803, 2022.
[66] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 14:77777- 2021.
[67] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transforma- tions for deep neural networks. In Proceedings of the IEFE conference computer vision and pattern recognition, pages 1492-1500, 2017.
[68] Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu. Co-scale attennatentinaal image transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9981 19990, 2021.
[69] Jianwei Yang. Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal self-attention for local-global interactions vision transformers. urXiv preprint arXiv:2107.00644 2021.
[70] Ting Yao, Yingwei Pan, Yehao Li, Chong- Wah Ngo, and Tao Mei. Wave-vit: Unifying wavelet and trans- formers for visual representation learning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXV, pages 328-345. Springer, 2022.
[71] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10819- 10829, 2022.
[72] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi. Zi- Hang Jiang, Francis EH Tay. Jiashi Feng. and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch imagenet. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 558-567, 2021.
[73] Li Yuan, Qibin Hou, Zihang Jiang, Jiashi Feng, and Shuicheng Yan. Volo: Vision outlooker for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[74] Bowen Zhang, Zhi Tian, Quan Tang, Xiangxiang Chu, Xiaolin Wei, Chunhua Shen, al. Segvit: Semantic segmentation with plain vision transformers. Advances in Neural Information Processing Systems, 35:4971- 4982, 2022.
[75] Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Haibin Lin, Zhi Zhang, Yue Sun, Tong He, Jonas Mueller, R Manmatha, et al. Resnest: Split- attention networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2736 2746. 2022.
14

---

[76] Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Cao. Multi-scale vision longformer: new vision transfomer for high-resolution image encoding. Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 2998-3008, 2021.
[77] Wengiang Zhang, Zilong Huang. Guozhong Luo, Tao Chen, Xinggang Wang. Wenyu Liu, Gang Yu, and Chunhua Shen. Topformer: Token pyramid trasformer mobile semantic segmentation. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 12083-12093, 2022.
[78] Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang. Xiaochen Lian, Zihang Jiang. Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv preprint (XXv:2103.11880 2021.
15


Reviews:
[{"review_id": "pZQZ1VS6DK", "reviewer": "Reviewer_jzMY", "summary": "The paper introduces the Scattering Vision Transformer (SVT), which utilizes a spectrally scattering network to capture fine-grained information in images and addresses the invertibility issue. SVT incorporates a novel spectral mixing technique using Einstein multiplication for efficient channel and token mixing. The approach achieves state-of-the-art performance on the ImageNet dataset, significantly reducing the number of parameters and FLOPS. It also demonstrates competitive results in other vision tasks, including transfer learning on standard datasets.", "questions": "None", "limitations": "None", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "## Novelty\n\nThe use of scattering networks and Fourier-like frequency processing is novel and innovative. The paper addresses a significant problem related to the texture processing ability of vision transformers, showcasing desirable novelty and originality.\n\n## Quality & Clarity\n\nThe paper is well-organized, providing clear preliminaries, assumptions, definitions, and solutions. The experiments are detailed and concrete.\n\n## Significance\n\nSVT effectively separates low-frequency and high-frequency image components while reducing computational complexity through the Einstein multiplication-based mixing technique. It achieves state-of-the-art performance on image classification and instance segmentation tasks and shows comparable results in object detection tasks.", "weaknesses": "The computational costs and complexity limit the number of directional orientations used in SVT. Currently, SVT employs six orientations, but increasing the number of orientations would capture more semantic information at the expense of higher computational complexity. Optimization possibilities should be explored. Additionally, SVT's performance in domains such as speech and NLP remains unexplored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for his insightful comments on the paper and to guide us on the possible research directions.\n\n1. Regarding the invertibility VS redundancy trade-off, we conduct an experiment to show that invertibility helps in comprehending the image and not just contributing to the performance. We pass an image through raw DTC-WT and we put an inverse DTC-WT operation and compute the reconstruction loss.  We ran the experiment with various values of J, which represent the orientation in SVT. We observe that the reconstruction loss in the image reduces with increasing values of J, which clearly shows that as the orientations increase, SVT is able to comprehend the image better – the orientations are able to represent the high-order properties of the image which benefits SVT. We have compared different spectral transforms such as Fourier Transform, Discrete Wavelet Transform as well as DTC-WT. We observe that the reconstruction loss is lesser in DTC-WT as compared to other spectral transforms, as captured in Table 1 below. We also capture the invertibility VS redundancy trade-off in table-5.  \nIn Table 1 we measured reconstructed loss(MSE) for FFT, DWT stage -1,2,3, and DTCWT stage -1,2,3. It shows that the MSE loss is less as we increase the level of decomposition (J) and order of selectivity. DTCWT has less MSE than DWT. Similarly, the PSNR value of DTWT is gier than DWT and Fourier. The Peak Signal-to-Noise Ratio (PSNR) is a measure of the ratio between the maximum possible power of an image and the power of corrupting noise that affects its representation. It is defined via the MSE and is expressed in decibels (dB). The higher the PSNR, the better the quality of the reconstructed image. In summary, for a reconstructed image to be considered of high quality, it should have a low MSE and a high PSNR. \n\n2. Optimization possibilities – we shall explore a few optimization possibilities to reduce the redundancy in the orientations of SVT for future work and these include:\n  - The selection of relevant orientations which best capture image properties, instead of using all six orientations\n  - The use of symmetric and anti-symmetric pairs in the orientations – for instance, $15^\\circ$ and $165^\\circ$, as well as $45^\\circ$, and $135^\\circ$, are such pairs that capture similar image properties, which we can leverage to optimize the number of orientations and reduce redundancy.\n   - Use of pyramidal decomposition of orientation – the finer layer requires more orientation compared to the coarser layer.\n\n3. We shall be exploring the applicability of SVT in speech data as it is also a spectral signal. We shall also be exploring NLP datasets for SVT.\n\n4. We are adding visualization for filter characterization DT-CWT in terms of low-frequency and all 6 six directional components of high-frequency components as shown in Figure-1 and 2. Figure-1 shows the phase and magnitude of FFT and the low-frequency component of DTCWT, whereas Figure-2 shows the phase and magnitude of the high-frequency component of DTCWT. It clearly indicates that it captures six degrees such as $15^\\circ$, $45^\\circ$, $75^\\circ$, $105^\\circ$, $135^\\circ$, and $165^\\circ$. This can't be captured by FFT and DWT."}, {"review_id": "a9XyjV4Qnu", "reviewer": "Reviewer_GT1n", "summary": "The paper proposes SVT, a novel vision transformer model that addresses the challenges of attention complexity and capturing fine-grained information in images. SVT utilizes a spectral scattering network and the Dual-time Complex Wavelet Transforms (DTCWT) to decompose image features into low-frequency and high-frequency components. The paper also introduces an efficient feature mixing technique using Einstein multiplication in the high-frequency components and tensor multiplication in the low-frequency components. Experimental results demonstrate that SVT achieves state-of-the-art performance on the ImageNet dataset with reduced parameters and computational complexity.", "questions": "In the experiments, why not compare your method with more popular  CNN-based methods like Yolo  and Faster-RCNN?", "limitations": "Author has presented some limitations of the proposed methods and future plan.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.The proposed SVT model presents an innovative approach to addressing attention complexity and capturing fine-grained information in images. The use of spectral scattering and DTCWT decomposition enables efficient representation and separation of frequency components.\n2.The feature mixing technique using Einstein multiplication is a novel contribution that efficiently combines token and channel features, leading to improved performance.\n3.The experimental results on the ImageNet dataset and other vision tasks demonstrate the superiority of SVT compared to existing vision transformers, such as LiTv2 and iFormer. The significant reduction in parameters and computational complexity further enhances the practicality and scalability of SVT.\n", "weaknesses": "1. The English writing in this paper needs to be carefully reviewed as there are several grammar errors. For example, line 79, lesser should be less, line 83, \"Given\" should be in lower case \"given\". And in Figure 4's illustration, some parts are missing. There are many more errors in your paper writing.\n2. I find the approach in this paper to be interesting but overly tricky. As we have seen with LLMs, many highly technical improvements become less significant when supported by the large parameter size and data volume of such models. Moreover, in the experimental section, the performance improvement brought by SVT does not appear to be significant. From my perspective, simple and elegant techniques often yield more meaningful improvements compared to complex transformations.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for his comments.\n\n1. **Language:** Thanks for pointing out this – we shall undertake a thorough revision of the paper to address all language issues.\n\n2. SVT is a generic recipe for componentizing the transformer architecture and efficiently implementing transformers with lesser parameters and computational complexity with the help of Einstein multiplication. So, SVT can be viewed as a simple and efficient learning transformer architecture, as opposed to viewing it as a complex transformation. The essence of SVT is the tensor multiplication in low-pass and Einstein multiplication in high-pass to get efficient transformer architecture. SVT leverages DT-CWT to get the phase information or high-frequency components, which is not possible in DWT.\n\n2. We have conducted an experiment to compare SVT’s performance on object detection tasks compared to Mask-RCNN,  as well as Faster-RCNN – the results of which are tabulated in Table-6 in the attached PDF. This table provides performance results on COCO val2017 dataset for the downstream task. Here We compare  Faster-RCNN with the Mask R-CNN 1x [21] method. We have reported the bounding box (\\emph{i.e.}, $AP^b$ )  for evaluation.  The $AP^b$ scores for SVT using Mask RCNN are better than Fster-RCNN. We shall include the comparison with Yolo in the final version.\n\n3. **SVT Compared with LVM/LLM**: We wish to state the following on the comment of the reviewer about large vision models (LVM/LLM): We have observed in recent papers that certain efficient transformer models such as efficientFormer and CvT have significantly larger number of parameters, with BiT-M has 928 million parameters and achieving 85.4% accuracy on ImageNet 1K, whereas ViT-H has 632 million parameters and achieving accuracy of 85.1. Comparitively, SVT-H-L has 54 million parameters and achieves 85.7% accuracy on ImageNet 1K - nearly 10X lesser number of parameters and FLOPS but with improved accuracy, as captured in Table 3 of CvT paper reference [60] of our paper."}, {"review_id": "wquFORpbZK", "reviewer": "Reviewer_kC2p", "summary": "This paper proposes to use Dual-time Complex Wavelet Transforms to decompose images into high and low frequency components in vision transformers. With this technique, it claims to address the problem of attention complexity without the loss of information as in Fourier or DWT based Transformers. This claim is based on the invertible property of Dual-time Complex Wavelet Transforms. This paper claims to achieve state-of-art image classification performance on the ImageNet dataset with a significant reduction in a number of parameters and FLOPS, and comparable results in instance segmentation. Qualitative and quantitative results are provided.", "questions": "Overall the reviewer thinks using DTCWT in vision transformers is worth exploring, and this is why the reviewer still vote for borderline accept at this stage. If the authors cannot provide appropriate explanation/discussion about their contribution in the rebuttal period, the reviewer would downvote the score.\n\nBesides, the reviewer feel some details provided by the main paper are not interesting/exciting, e.g., Table 1. For future work, the reviewer would suggest the authors to explore more insightful things. For instance, DTCWT enjoys invertibility but has redundancy at the same time. How would the network deal with such a redundancy? Moreover, providing more quantitive or qualitative comparisons to Fourier or DWT based Transformers to analyse how the invertibility of DTCWT helps vision transformers understand visual contents, instead of just reporting the final numbers for the tasks like classification.\n\n____________________________________________________\n\n\nThe author response solves the main concerns. Under the assumption that the authors would revise the manuscript as they promised, the reviewer agreed to raise the score to weak accept.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "1. The most attractive of this paper is the usage of Dual-time Complex Wavelet Transforms. The reviewer appreciates that the authors noticed the information loss in previous Fourier or DWT based Transformers, and proposed to adopt Dual-time Complex Wavelet Transforms to solve such a problem. It is invertible (though also redundant) and hence would not lead to a loss.\n\n2. The paper provides a lot of technical details, and also the codes to reproduce the results.\n\n3. Extensive experiments are conducted in the paper.", "weaknesses": "1. Clarity of Technical Contribution\n\nThe paper's presentation raises some concerns about clarity and comprehension. The key technical contribution of the work is obscured by convoluted explanations and distracting statements.\n\nFor example, L81-L93, at the beginning of the method section, this paper detailedly talks about something like \"In Vanilla SVT, Given an image, we split it into patches of size 16*16. We use a linear  projection layer to get the embedding feature for each patch ......\" These sentences seem to depict unique contributions of the proposed method but, upon closer inspection, merely explain standard operations in the Vision Transformer (ViT). Similarly, lines 102-109 are bewildering, appearing repetitive and unclear about the authors' intended message.\n\nMoreover, the paper repeatedly emphasizes \"tensor multiplication\" and \"Einstein multiplication\" across various sections. These appear to be mere implementation details, not substantial technical contributions.\n\nThe manuscript's current style is closer a technical report rather than an academic paper. The authors are strongly encouraged to (a) explicitly state the key contribution of this work in the rebuttal and method section, and (b) condense lines 81-126 into a succinct, high-level introduction of the method, ideally within 10 lines.\n\n2. Claims need support\n\nSome of the claims in the paper, while potentially plausible, lack sufficient support/proof. For example, L52, \"the ability to separate low-frequency and high-frequency components of an image is also important\". In L100, \"SVT has improved robustness compared to most other transformers, which will also be established in the performance studies.\" These could be true, but the authors did not prove the robustness is improved.\n\n3. Minors\n\nThere are some minor typos/problems to be fixed. For example,  L130, \"Where\" to \"where\". In page 7, some table captions are bold and some are not.\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments, which shall result in significant improvements to the quality of the final camera-ready version of the paper\n\n1. One of the claims of the paper is the ability of DTCWT [46] to separate the low-frequency and high-frequency components of the image. This has now been included in the background section of the paper and is captured in **the main rebuttal section**.\n\n2. **Invertibility vs redundancy trade-off:** Regarding the invertibility vs redundancy trade-off, we conduct an experiment to show that invertibility helps in comprehending the image and not just contributing to the performance. We pass an image through raw DTC-WT and we put an inverse DTC-WT operation and compute the reconstruction loss.  We ran the experiment with various values of J, which represent the orientation in SVT. We observe that the reconstruction loss in the image reduces with increasing values of J, which clearly shows that as the orientations increase, SVT is able to comprehend the image better – the orientations are able to represent the high-order properties of the image which benefits SVT. We have compared different spectral transforms such as Fourier Transform, Discrete Wavelet Transform as well as DTC-WT. We observe that the reconstruction loss is lesser in DTC-WT as compared to other spectral transforms, as captured in Table 1 below.\nIn Table 1 we measured reconstructed loss(MSE) for FFT, DWT stage -1,2,3, and DTCWT stage -1,2,3. It shows that the MSE loss is less as we increase the level of decomposition (J) and order of selectivity. DTCWT has less MSE than DWT. Similarly, the PSNR value of DTWT is gier than DWT and Fourier. The Peak Signal-to-Noise Ratio (PSNR) is a measure of the ratio between the maximum possible power of an image and the power of corrupting noise that affects its representation. It is defined via the MSE and is expressed in decibels (dB). The higher the PSNR, the better the quality of the reconstructed image. In summary, for a reconstructed image to be considered of high quality, it should have a low MSE and a high PSNR. \n\n 3. It must be noted that Fourier Transforms cannot perform low-pass and high-pass separation, as mentioned in the GFNet paper.  GFNet always uses only tensor multiplication, which may be inefficient compared to Einstein multiplication, which is efficient and reduces the number of parameters and computational complexity. Thus, while there is offset and SVT does not lag in performance nor in computational complexity but gains better representational power. This is shown in Table 2, where we compare latency, FLOPS, number of parameters, and reconstruction loss.  The table-2 shows the Latency (mili sec) of SVT compared with the Convolution type network, attention type transformer network, POOl type transformer network, MLP type transformer network, and Spectral type transformer network. We report latency per sample on A100 GPU. We have also visualized the filter coefficients of all six orientations in Supplementary Figure 1.\n\n 4. **Robustness:** We also use an SVT pre-trained on the ImageNet-1K dataset and use the ImageNet-C dataset to measure the robustness of SVT compared with various transformers such as ViT-B, MLP-Mixer, and SVT as shown in Table 7 of the attached PDF. The table compares accuracy and mCE score on the ImageNet-C dataset.  SVT-H-B has less mCE score compared to other transformers.\n\n5. The quality of writing will be improved considerably in the final version and we shall also include a Mathematical Formulation of DTCWT.\n\n## Mathematical Formulation of DTCWT\n\n\\begin{equation*}\n     x(t)=  \\sum_{n=-\\infty}^{\\infty}c(n)\\phi(t-n)  +\\sum_{j=0}^{\\infty}\\sum_{n=-\\infty}^{\\infty}d(j, n)2^{j/2}\\psi(2^{j}t-n).     \n\\end{equation*}\nwhere $c(n)$ is the scaling coefficients and $d(j,n)$ is the wavelet coefficients.\n\\begin{equation*}\nc(n)  =\\int_{-\\infty}^{\\infty}x(t)\\phi(t-n)dt, \\quad\nd(j, n) =2^{j/2}\\int_{-\\infty}^{\\infty}x(t)\\psi(2^{j}t-n)dt.\n\\end{equation*}\n\nThe complex value wavelet is given by \n\\begin{equation*}\n\\psi_{\\rm c}(t)=\\psi_{\\rm r}(t)+{\\rm j}\\psi_{\\rm i}(t).\n\\end{equation*}\nWhere $\\psi_r(t)$  is real and even and $j\\psi_i(t)$ is imaginary and odd\n\\begin{equation*}\nd_{\\rm c}(j, n)=d_{\\rm r}(j, n)+{\\rm j}\\ d_{\\rm i}(j, n)\n\\end{equation*}\n\nWith Magnitude and phase \n\\begin{equation*}\n\\vert d_{\\rm c}(j, n)\\vert =\\sqrt{[d_{\\rm r}(j,n)]^{2}+[d_{\\rm i}(j,n)]^{2}}, \\quad\n\\angle d_{\\rm c}(j,n)= \\arctan \\left({d_{\\rm i}(j,n)\\over d_{\\rm r}(j,n)}\\right)\n\\end{equation*}\n\nLet $h_0(n), h_1(n)$ denote the low-pass and high-pass filter in the upper band, while $g_0(n), g_1(n)$ denote the same for the lower band. The wavelets corresponding to the upper band and lower band are denoted by $\\psi_h(n), \\psi_g(n)$. The filters are designed to get the complex wavelet by satisfying the  Perfect Reconstruction (PR) conditions. \nThe complex wavelet filter can be represented as $\\psi(t):= \\psi_h(t)+j\\psi_g(t)$.  \nwhere $\\psi_g(t)$  is approximately the Hilbert transform of $\\psi_h(t)$ i.e.  $\\psi(t) \\approx \\mathcal{H}{\\psi_h(t)}$ . \n\n\\begin{equation*} \n\\psi_{h}(t)=\\sqrt{2}\\sum_{n}h_{1}(n)\\phi_{h}(t), \\quad\n \\phi_{h}(t)=\\sqrt{2}\\sum_{n}h_{0}(n)\\phi_{h}(t)\n\\end{equation*}\n\n Two low-pass filters should satisfy a very simple property: one of them should be approximately a half-sample shift of the other.\n\n$g_{0}(n)\\approx h_{0}(n-0.5) \\Rightarrow \\psi_{g}(t)\\approx {\\cal H}\\{\\psi_{h}(t)\\} $\n\nSince the filters are real, no complex arithmetic is required for implementing DTCWT. It is just two times more expansive in 1-D. It is also easy to invert, as the two separate DWTs can be inverted. \n\nFinal Dual-Tree CWT can be designed using follow steps:\n- Approximate half-sample delay property\n- PR (orthogonal or biorthogonal)\n- Finite support (FIR filters)\n- Vanishing moments/good stopband\n- Linear-phase filters: Only the complex filter responses need to be linear-phase; this can be achieved by taking\n $g_0(n)=h_0(N−1−n)$\n"}, {"review_id": "MQlejRqMGH", "reviewer": "Reviewer_upE5", "summary": "This paper proposed to use DTCWT in the transformer model in early stage, with the motivation of saving computation cost without information loss. The proposed scattering module has a low-pass and a high-pass branch. Tensor multiplication & Einstein multiplication are applied to the LF & HF branch respectively. The proposed module replaces the first 2 stages in hierarchical ViT models. The proposed SVT model achieved superior performance on image classification on the ImageNet1K dataset, and the downstream transfer learning results. ", "questions": "How do we compare SVT with combining ViT with CNN, i.e. an architecture with CNN in early stages while transformer in late stages?", "limitations": "The proposed module is a highly hand engineered component. Adding manually designed components to the deep learning models may contradict with the motivation of removing inductive bias to the model and building generic learning based architectures. ", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Replacing low level transformer modules with the proposed DTCWT based module achieves better accuracy and latency trade-off. Applying DTCWT to transformer models may not be explored yet by prior work.", "weaknesses": "DTCWT has been used in the previous CNN based deep learning method, e.g. Uses of Complex Wavelets in Deep Convolutional Neural Networks. \nExisting work already shows that replacing early transformer stage with convolutional layers could give better accuracy and latency trade-off.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n1. **Comparing SVT with ViT with initial CNN layers:** We have conducted an experiment where the initial layers of a ViT are convolutional networks and later layers are attention layers to compare the performance of SVT. The results are captured in Table 4 in the attached PDF, where we compare SVT  with transformers having initial convolutional layers such as CVT, CMT, and HorNet. Initial  Convolutional layers in a transformer are not performing well compared to the initial scatter layer.    Initial scatter layer-based transformers are better performance and less computation cost compared to  Initial  Convolutional layer-based transformers which is shown in Table 4. We also measure the latency of various transformers as shown in Table 2. In this table, we compare latency, FLOPS, number of parameters, and reconstruction loss. The table-2 shows the Latency (mili sec) of SVT compared with the Convolution type network, attention type transformer network, POOl type transformer network, MLP type transformer network, and Spectral type transformer network. We report latency per sample on A100 GPU.\n\n2. SVT is a generic recipe for componentizing the transformer architecture and efficiently implementing transformers with lesser parameters and computational complexity with the help of Einstein multiplication. So, this can be viewed as a simple and efficient learning transformer architecture with minimal inductive bias. We provide the source code for SVT in the paper and the reviewer is free to verify the above claim."}, {"review_id": "mNX3IQ7PiV", "reviewer": "Reviewer_ghvx", "summary": "Assistant\nThe article is about the Scattering Vision Transformer, or SVT, which is a new adaptation of transformers for computer vision tasks. Its unique feature is the use of a spectrally scattering network, which captures fine-grained information about an image and addresses the issue of information loss caused by down-sampling operations. The SVT achieves state-of-the-art performance on the ImageNet dataset and outperforms other transformers in other vision tasks.", "questions": "1. Could you provide an explanation for why scatter layer is used in shallower layers but traditional attention layer is used in deeper layers?\n2. Is there any evidence, such as references, mathematical induction, or related research studies, to support the idea of using tensor multiplication for low pass and Einstein multiplication for high pass?", "limitations": "There appear to be no potential negative social impacts.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The use of DT-CWT, which is both shift-invariant and free from aliasing, is a robust and reliable approach.\n2. The combination of tensor multiplication in low-frequency components and Einstein multiplication in high-frequency components is a compelling and efficient technique.\n3. The experimental results demonstrate great promise for the proposed method.", "weaknesses": "1. To enhance comprehension, a brief illustration of DT-CWT through either descriptive language or illustrative figures should be provided in the method or background section.\n2. Not all layers of vision transformers are scatter layers. An explanation and study should be provided.\n3. The paper would benefit from the inclusion of mathematical theorem proofs. I found it difficult to understand the benefits of DT-CWT, particularly the decoupling of low and high frequencies. The paper's methods read more like a technical report than a compelling story of motivation, possibly due to the omission of some logistic or preliminary details.\n4. To further evaluate the performance, provide comparisions of speed and over-fitting tests on ImageNet V2. \n5. There are some typos in the paper. For example, on line 260, \"figure ??\" is not clear. Additionally, the official reference uses brackets [xx], whereas parentheses are used in the text. Please ensure consistency and accuracy in all references.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed comments, which we believe are insightful and shall help in improving the quality of the final submission.\n\n1. A background section on DT-CWT [46] shall be added to the final version of the paper to help readers understand it, before explaining SVT method details. This has now been included in **the main rebuttal**.\n\n2. We compare the performance of SVT when the architecture changes from all attention ( PVTv2[58] ) to all spectral layers ( GFNet[44] ) well as a few spectral and remaining attention layers(SVT ours). We observe that combining spectral and attention boost the performance compared to all attention and all spectral layer-based transformer as shown in Table-2 of the main paper.\n\n3. We provide a mathematical characterization of DT-CWT [46]  below and how it helps in decoupling low and high-frequency components. We will include this in the background section of the paper.\n\n4. We have conducted an experiment to measure the latency of SVT and compare the same with ViT and GFNet, the results of which are given in the Table-2 of the attached PDF.  This is shown in Table 2 below, where we compare latency, FLOPS, number of parameters, and reconstruction loss. The table-2 shows the Latency (mili sec) of SVT compared with the Convolution type network, attention type transformer network, POOl type transformer network, MLP-mixer type transformer network, and Spectral type transformer network. We report latency per sample on A100 GPU.\n\n5. We have also conducted an experiment to test robustness by using the ImageNet C dataset, the results of which are reproduced in table-7 of the attached PDF. We also use an SVT pre-trained on the ImageNet-1K dataset and use the ImageNet-P dataset to measure the robustness of SVT compared with various transformers such as ViT-B, MLP-Mixer, and SVT as shown in the table-7 of the attached PDF. The table compares accuracy and mCE score on the ImageNet-C dataset. SVT-H-B has less mCE score compared to other transformers.\n\n6. Thanks for pointing out typos etc. in the paper, we shall ensure a thorough revision of the paper and make sure to address all language issues.\n\n7. The ablation study was conducted to show that initial scatter layers followed by attention in deeper layers are more beneficial than having later scatter and initial attention layers ( SVT-H-S-Inverse). We also compare transformer models based on an alternative the attention and scatter layer (SVT-H-S-Alternate). The results are documented in the table-3 of the attached PDF. From all these combinations we observe that initial scatter layers followed by attention in deeper layers are more beneficial than others.\n\n8. The use of tensor multiplication for low pass and Einstein multiplication for high pass is our contribution. It must be noted that low-frequency components contain the energy component of the signal which requires all the frequency components to provide energy compaction, while high-frequency components can be represented by only a few components, which can be achieved using Einstein multiplication. We have evaluated the same empirically and shown in Table 7 of the main paper that SVT_{TTEE} is the best performing compared to other alternatives. We found that using Einstein multiplication in channel and token mixing makes the transformer architecture more efficient in terms of FLOPS and no. of parameters, without compromising on accuracy. We have also revised the contributions section of the camera-ready paper to reflect this.\n\n\n\n## Mathematical Formulation of DTCWT\n\n\\begin{equation*}\n     x(t)=  \\sum_{n=-\\infty}^{\\infty}c(n)\\phi(t-n)  +\\sum_{j=0}^{\\infty}\\sum_{n=-\\infty}^{\\infty}d(j, n)2^{j/2}\\psi(2^{j}t-n).     \n\\end{equation*}\nwhere $c(n)$ is the scaling coefficients and $d(j,n)$ is the wavelet coefficients.\n\\begin{equation*}\nc(n)  =\\int_{-\\infty}^{\\infty}x(t)\\phi(t-n)dt, \\quad\nd(j, n) =2^{j/2}\\int_{-\\infty}^{\\infty}x(t)\\psi(2^{j}t-n)dt.\n\\end{equation*}\n\nThe complex value wavelet is given by \n\\begin{equation*}\n\\psi_{\\rm c}(t)=\\psi_{\\rm r}(t)+{\\rm j}\\psi_{\\rm i}(t).\n\\end{equation*}\nWhere $\\psi_r(t)$  is real and even and $j\\psi_i(t)$ is imaginary and odd\n\\begin{equation*}\nd_{\\rm c}(j, n)=d_{\\rm r}(j, n)+{\\rm j}\\ d_{\\rm i}(j, n)\n\\end{equation*}\n\nWith Magnitude and phase \n\\begin{equation*}\n\\vert d_{\\rm c}(j, n)\\vert =\\sqrt{[d_{\\rm r}(j,n)]^{2}+[d_{\\rm i}(j,n)]^{2}}, \\quad\n\\angle d_{\\rm c}(j,n)= \\arctan \\left({d_{\\rm i}(j,n)\\over d_{\\rm r}(j,n)}\\right)\n\\end{equation*}\n\nLet $h_0(n), h_1(n)$ denote the low-pass and high-pass filter in the upper band, while $g_0(n), g_1(n)$ denote the same for the lower band. The wavelets corresponding to the upper band and lower band are denoted by $\\psi_h(n), \\psi_g(n)$. The filters are designed to get the complex wavelet by satisfying the  Perfect Reconstruction (PR) conditions. \nThe complex wavelet filter can be represented as $\\psi(t):= \\psi_h(t)+j\\psi_g(t)$.  \nwhere $\\psi_g(t)$  is approximately the Hilbert transform of $\\psi_h(t)$ i.e.  $\\psi(t) \\approx \\mathcal{H}{\\psi_h(t)}$ . \n\n\\begin{equation*} \n\\psi_{h}(t)=\\sqrt{2}\\sum_{n}h_{1}(n)\\phi_{h}(t), \\quad\n \\phi_{h}(t)=\\sqrt{2}\\sum_{n}h_{0}(n)\\phi_{h}(t)\n\\end{equation*}\n\n Two low-pass filters should satisfy a very simple property: one of them should be approximately a half-sample shift of the other.\n\n$g_{0}(n)\\approx h_{0}(n-0.5) \\Rightarrow \\psi_{g}(t)\\approx {\\cal H}\\{\\psi_{h}(t)\\} $\n\nSince the filters are real, no complex arithmetic is required for implementing DTCWT. It is just two times more expansive in 1-D. It is also easy to invert, as the two separate DWTs can be inverted. \n\nFinal Dual-Tree CWT can be designed using follow steps:\n- Approximate half-sample delay property\n- PR (orthogonal or biorthogonal)\n- Finite support (FIR filters)\n- Vanishing moments/good stopband\n- Linear-phase filters: Only the complex filter responses need to be linear-phase; this can be achieved by taking\n $g_0(n)=h_0(N−1−n)$\n"}]

Author rebuttal:
We thank all the reviewers for their constructive suggestions. We intend to incorporate the feedback to obtain an improved revision of our paper – we sincerely believe that the comments shall improve the quality of the paper significantly.

We provide clarifications for the points raised by the reviewers:

1. We have added a separate background section and included a detailed explanation of Dual-Tree Complex Wavelet Transform (DTCWT) as suggested by the reviewers. Here we have included a diagram in the attached PDF for your kind reference. The same shall be added to the camera-ready version of the paper.

   - The explanation also captures the ability of the DTCWT to separate low-frequency and high-frequency components of an image as well as the use of DTCWT in vision transformers.
   - We also give a mathematical characterization of DTCWT. 

2. We have added several experiments to support & validate the claims made in the paper.

    - **Initial Convolutional Layers VS Scatter Layers**: These experiments include a comparison of SVT with transformer architectures having initial convolutional layers and final attention layers. This is now substantiated with detailed experiments, where we compare SVT with Hornet, CMT, and CVT. The new table is numbered Table 5 in the attached PDF.

    - **Efficiency of Einstein Multiplication based SVT Implementation**: We have also conducted a new experiment to measure the latency of SVT and compare it with the latency of other types of transformers. The table, which is captured in Table 2 in the attached PDF, shows that SVT has lower latencies in spite of having redundancies, which proves the efficiency of the Einstein multiplication in high-frequency components. SVT also has lower FLOPS and a lesser number of parameters due to the efficient implementation.

    - **Qualitative and quantitative study of Invertibility**: We have conducted an additional experiment to compare the invertibility of DTCWT with Fourier transform and DWT – we measure the reconstruction loss of the transformer based on all three and compare the same in Table 1 of the attached PDF.  We have also visualized qualitatively the invertibility property of DTCWT, Fourier, and DWT shown in Figure-1,2. Table 1 captures the invertibility studies. 

    - **Invertibility VS Redundancy Trade-off**: We have analyzed the trade-off between invertibility and redundancy of SVT, where we were able to show that if we increase the number of orientations and number of stages, the reconstruction loss decreases, which is indicative of better invertibility at the cost of redundancy. The efficient implementation based on Einstein multiplication offsets the redundancy with improved performance with a reduced number of parameters and computational cost. We also show that SVT’s orientation better represents the higher-order properties of the image. This is captured in Table 6 of the attached PDF.

    - **Robustness**: We have also conducted an experiment to measure the robustness of SVT by pre-training on ImageNet 1K and comparing it with other architectures like ResNet, ViT, and MLP Mixer on the ImageNet C dataset. This is captured in Table 7 of the attached PDF and clearly demonstrates that SVT. SVT achieves better robustness compared to other transformer architectures.

    - **Initial Scatter VS Initial Attention**: We have conducted an additional experiment to show that SVT with initial scatter layers and deeper attention layers performs better than SVT having initial attention layers and deeper scatter layers. This is captured in Table 3, in the attached PDF.

    - **Faster RCNN comparison**: We have included the SVT VS Faster RCNN comparison in Table 4.

3. We have revised the paper significantly to take care of grammatical and language issues, as suggested by reviewers.

## Overview of DTCWT and Decoupling of Low & High Frequencies

Discrete Wavelet Transform (DWT) replaces the infinite oscillating sinusoidal functions with a set of locally oscillating basis functions, which are known as wavelets Kingsbury et al.[46]. Wavelet is a combination of low-pass scaling function $\phi(t)$ and a shifted version of a band-pass wavelet function known as $\psi(t)$. It can be represented mathematically as given below:
\begin{equation*}
     x(t)=  \sum_{n=-\infty}^{\infty}c(n)\phi(t-n)  +\sum_{j=0}^{\infty}\sum_{n=-\infty}^{\infty}d(j, n)2^{j/2}\psi(2^{j}t-n).     
\end{equation*}
where $c(n)$ is the scaling coefficients and $d(j,n)$ is the wavelet coefficients.

Kingsbury et al.[46] have identified four issues in DWT including oscillations, shift variance, aliasing, and lack of directionality. One of the solutions to solve the above problems is the Complex Wavelet Transform (CWT). CWT is inspired by Fourier representation and has a complex-valued scaling function and complex-valued wavelet function, as given below:

$\psi_{\rm c}(t)=\psi_{\rm r}(t)+{\rm j}\psi_{\rm i}(t)$

CWT is a double redundant tight frame in 1-D, able to overcome the four shortcomings mentioned above. DTCWT is a specific redundant type of CWT, which is based on two Filter Bank (FB) trees. The DTCWT uses two real DWTs, with the first one giving the real part of the transform, while the second one gives the imaginary part. The two real DWTs use two different sets of filters, which are jointly designed to give an approximation of the overall complex wavelet transform and satisfy the Perfect Reconstruction (PR) conditions.

Let $h_0(n), h_1(n)$ denote the low-pass and high-pass filter in the upper band, while $g_0(n), g_1(n)$ denote the same for the lower band. The wavelets corresponding to the upper band and lower band are denoted by $\psi_h(n), \psi_g(n)$. The filters are designed to get the complex wavelet by satisfying the PR conditions. Since the filters are real, no complex arithmetic is required for implementing DTCWT. It is just two times more expansive in 1-D. It is also easy to invert, as the two separate DWTs can be inverted. 



Meta-review: Reviewers were enthusiastic about the paper and recognized a new, innovative approach to an important problem. Reviewers some minor point in their review which have been addressed in the rebuttal. We invite the authors to take these changes into account in the camera-ready version of the paper.
Decision: Accept (poster)