SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Badri Narayana Patro
badripatro@microsoft.com
Vijay Srinivas Agneeswaran
vagneeswaran@microsoft.com
Vision transformers have gained significant attention and achieved state-of-the-art performance in various computer vision tasks, including image classification, instance segmentation, and object detection. However, challenges remain in addressing attention complexity and effectively capturing fine-grained information within images. Existing solutions often resort to down-sampling operations, such as pooling, to reduce computational cost. Unfortunately, such operations are non-invertible and can result in information loss. In this paper, we present a novel approach called Scattering Vision Transformer (SVT) to tackle these challenges. SVT incorporates a spectrally scattering network that enables the capture of intricate image details. SVT overcomes the invertibility issue associated with down-sampling operations by separating low-frequency and high-frequency components. Furthermore, SVT introduces a unique spectral gating network utilizing Einstein multiplication for token and channel mixing, effectively reducing complexity. We show that SVT achieves state-of-the-art performance on the ImageNet dataset with a significant reduction in a number of parameters and FLOPS. SVT shows  $2\%$  improvement over LiTv2 and iFormer. SVT-H-S reaches  $84.2\%$  top-1 accuracy, while SVT-H-B reaches  $85.2\%$  (state-of-art for base versions) and SVT-H-L reaches  $85.7\%$  (again state-of-art for large versions). SVT also shows comparable results in other vision tasks such as instance segmentation. SVT also outperforms other transformers in transfer learning on standard datasets such as CIFAR10, CIFAR100, Oxford Flower, and Stanford Car datasets. The project page is available on this webpage (https://badripatro.github.io/svt/).
In recent years, there has been a remarkable surge in the interest and adoption of Large Language Models (LLMs), driven by the release and success of prominent models such as GPT-3, ChatGPT [1], and Palm [8]. These LLMs have achieved significant breakthroughs in the field of Natural Language Processing (NLP). Building upon their successes, subsequent research endeavors have extended the language transformer paradigm to diverse domains including computer vision, speech recognition, video processing, and even climate and weather prediction. In this paper, we specifically focus on exploring the potential of LLMs for vision-related tasks. By leveraging the power of these language models, we aim to push the boundaries of vision applications and investigate their capabilities in addressing complex vision challenges.
Several adaptations of transformers have been introduced in the field of computer vision for various tasks. For image classification, notable vision transformers include ViT [13], DeIT [56], PVT [61], Swin [39], Twin [9], and CSWin transformers [12]. The different vision transformers improved the performance of image classification tasks significantly compared to Convolutional Neural Networks (CNNs) such as ResNets and RegNets, as discussed in efficient vision transformer research work [43]. This breakthrough in computer vision has led to state-of-the-art results in various vision tasks, including image segmentation such as SegFormer [66], TopFormer [77] and SegViT [74] and object
detection, models like DETR [4] and Yolo[15]. However, one challenge faced by vision transformers is the increasing computational complexity of the self-attention module as the sequence length or image resolution grows. Additionally, model size and the number of floating-point operations per second (FLOPS) also increase with image resolution or sequence length. These factors need to be carefully considered and addressed to ensure efficient and scalable deployment of vision transformers in real-world applications.
One way to address the computational complexity of attention-based Transformers is to replace the attention mechanism with a Multi-Layer Perceptron (MLP) based mixer layer [54, 55, 18]. However, it is difficult to capture spatial information in the MLP mixers. This was addressed by the paper MetaFormer [71]. MetaFormer uses a pooling operation to replace the attention layer. This however has the disadvantage that the pooling operation is not invertible and could possibly lose information. Fourier based Transformers such as FourierFormer[53], FNet[31], GFNet [47] and AFNO [16] minimizes the loss of information by using Fourier Transform. But it has an inherent problem of separating the low and high-frequency components. The ability to separate low-frequency and high-frequency components of an image is important. Recently, transformers such as LiTv2 [42] and iFormer [51] have been proposed to address this problem. However, both LiTv2 and iFormer have the same  $O(n^{2})$  complexity as they use full-fledged self-attention networks, similar to ViT [13] and DeIT [56], which are weak in capturing fine-grained information of images. Vit, DeIT, LiTv2 and iFormer also have a limitation with respect to network size or number of parameters. We propose a Scattering Vision Transformer (SVT) which uses a spectral scattering network to address the attention complexity and Dual-Tree Complex Wavelet Transform (DTCWT) to capture the fine-grained information using spectral decomposition into low-frequency and high-frequency components of an image.
SVT uses the scattering network as the initial layer of the transformer, which captures fine-grained information (lines and edges, for instance) along with the energy component. SVT also uses attention nets in the deeper layers to capture long-range dependencies. The fine-grained information is captured by the high-frequency component of the scattering network by using the DTCWT transform while the low-frequency component is the energy component. SVT uses a Spectral Gating Network (SGN) to capture the effective features in both frequency components. Generally, the high-frequency component has extra directional information which makes it computationally complex while performing the gating operation. SVT addresses this complexity with a novel token and channel mixing technique using the Einstein Blending Method (EBM) in high-frequency component. SVT also uses a Tensor Blending Method (TBM) in the low-frequency component. We also observe that the DTCWT is more invertible compared to other spectral transformations in the literature which are based on Fourier transforms and discrete wavelet transforms [2, 28]. We quantify the invertibility in terms of reconstruction loss in the performance study section of this paper. The use of TBM for low-frequency components and EBM for high-frequency components is our contribution. It must be noted that low-frequency components contain the energy component of the signal which requires all the frequency components to provide energy compaction, while high-frequency components can be represented by only a few components, which can be achieved using Einstein multiplication. SVT is a generic recipe for componentizing the transformer architecture and efficiently implementing transformers with lesser parameters and computational complexity with the help of Einstein multiplication. So, this can be viewed as a simple and efficient learning transformer architecture with minimal inductive bias.
- We introduce a novel invertible scattering network based on DTCWT transformation into vision transformers to decompose image features into low-frequency and high-frequency features.
- We proposed a novel SGN, which uses TBM to mix low-frequency representations and EBM to mix high-frequency representations. We use an efficient way of mixing high-frequency components using channel and token mixing with the help of Einstein multiplication.
- Detailed performance analysis shows that SVT outperforms all transformers including LiT v2 and iFormer on ImageNet data, with a significantly lesser number of parameters. In addition, SVT also has comparable performance on other transfer learning datasets.
- We show that SVT is efficient not only performance-wise but also in terms of a number of parameters (memory size) as well as in terms of computational complexity (measured in Gigaflops). We also show that SVT is efficient for inferencing, by measuring its latency and comparing it with other state-of-art transformers.
Figure 1: This figure illustrates the architectural details of the SVT model with a Scatter and Attention Layer structure. The Scatter Layer comprises a Scattering Transformation that processes Low-Frequency (LF) and High-Frequency (HF) components. Subsequently, we apply the Tensor and Einstein Blending Method to obtain Low-Frequency Representation (LFR) and High-Frequency Representation (HFR), as depicted in the figure. Finally, we apply the Inverse Scattering transformation using LFR and HFR.
2.1 Background: Overview of DTCWT and Decoupling of Low & High Frequencies
Discrete Wavelet Transform (DWT) replaces the infinite oscillating sinusoidal functions with a set of locally oscillating basis functions, which are known as wavelets [50, 27]. Wavelet is a combination of low-pass scaling function  $\phi(t)$  and a shifted version of a band-pass wavelet function known as  $\psi(t)$ . It can be represented mathematically as given below:
$$
x (t) = \sum_ {n = - \infty} ^ {\infty} c (n) \phi (t - n) + \sum_ {j = 0} ^ {\infty} \sum_ {n = - \infty} ^ {\infty} d (j, n) 2 ^ {j / 2} \psi (2 ^ {j} t - n). \tag {1}
$$
where  $c(n)$  is the scaling coefficients and  $d(j,n)$  is the wavelet coefficients. These coefficients are computed by the inner product of the scaling function  $\phi (t)$  and wavelet function  $\psi (t)$  with input  $x(t)$ .
$$
c (n) = \int_ {- \infty} ^ {\infty} x (t) \phi (t - n) d t, \quad d (j, n) = 2 ^ {j / 2} \int_ {- \infty} ^ {\infty} x (t) \psi (2 ^ {j} t - n) d t. \tag {2}
$$
DWT suffers from the following issues oscillations, shift variance, aliasing, and lack of directionality. One of the solutions to solve the above problems is the Complex Wavelet Transform (CWT) with complex-valued scaling and wavelet function. The Dual-Tree Complex Wavelet Transform (DT-CWT) addresses the issues of the CWT. The DT-CWT [28, 26, 27] comes very close to mirroring the attractive properties of the Fourier Transform, including a smooth, nonoscillating magnitude, a nearly shift-invariant magnitude with a simple near-linear phase encoding of signal shifts, substantially reduced aliasing; and better directional selectivity wavelets in higher dimensions. This makes it easier to detect edges and orientational features of images. The six orientations of the wavelet transformation are given by  $15^{\circ}$ ,  $45^{\circ}$ ,  $75^{\circ}$ ,  $105^{\circ}$ ,  $135^{\circ}$ , and  $165^{\circ}$ . The dual-tree CWT employs two real DWTs, the first DWT gives the real part of the transform while the second DWT gives the imaginary part. The two real DWTs use two different sets of filters, which are jointly designed to give an approximation of the overall complex wavelet transform and satisfy the perfect reconstruction (PR) conditions.
Let  $h_0(n), h_1(n)$  denote the low-pass and high-pass filter pair in the upper band, while  $g_0(n), g_1(n)$  denote the same for the lower band. Two real wavelets are associated with each of the two real wavelet transforms as  $\psi_h(t)$ , and  $\psi_g(t)$ . The complex wavelet  $\psi_h(t) \coloneqq \psi_h(t) + \psi_g(t)$  can be approximated using Half-Sample Delay[49] condition,i.e.  $\psi_h(t)$  is approximately the Hilbert transform of  $\psi_g(t)$  like
$$
g _ {0} (n) \approx h _ {0} (n - 0. 5) \Rightarrow \psi_ {g} (t) \approx \mathcal {H} \{\psi_ {h} (t) \} \psi_ {h} (t) = \sqrt {2} \sum_ {n} h _ {1} (n) \phi_ {h} (t), \phi_ {h} (t) = \sqrt {2} \sum_ {n} h _ {0} (n) \phi_ {h} (t)
$$
Similarly, we can define  $\psi_g(t),\phi_g(t)$  , and  $g_{1}(n)$  . Since the filters are real, no complex arithmetic is required to implement DTCWT. It is just two times more expansive in 1D because the total output data rate is exactly twice the input data rate. It is also easy to invert, as the two separate DWTs can be inverted. Compare DTCWT with the Fourier Transform, which is difficult to obtain low pass and high pass components of an image and it is less invertible (Loss is high when we do Fourier and inverse Fourier transform) compared to DTCWT. Also, It cannot speak about time and frequency simultaneously.
2.2 Scattering Visual Transformer (SVT) Method
Given input image  $\mathbf{I} \in \mathbb{R}^{3 \times 224 \times 224}$ , we split the image into the patch of size  $16 \times 16$  and obtain embedding of each patch token using position encoder and token embedding network.  $\mathbf{X} = \mathcal{F}_T(\mathbf{I}) + \mathcal{F}_P(\mathbf{I})$ , where  $\mathcal{F}_T, \mathcal{F}_P$  refer to token and position encoding network. The detailed distinct components of the SVT architecture are illustrated in Figure 1. Scattering Visual Transformer consists of three components such as a) Scattering Transformation, b) Spectral Gating Network, c) Spectral Channel and Token Mixing.
A. Scattering Transformation:
The input image  $\mathbf{I}$  is firstly patchified into a feature tensor  $\mathbf{X} \in \mathbb{R}^{C \times H \times W}$  whose spatial resolution is  $H \times W$  and the number of channels is  $C$ . To extract the features of an image, we feed  $\mathbf{X}$  into a series of transformer layers. We use a novel spectral transform based on an invertibility scattering network instead of the standard self-attention network. This allows us to capture both the fine-grain and the global information in the image. The fine-grain information consists of texture, patterns, and small features that are encoded by the high-frequency components of the spectral transform. The global information consists of the overall brightness, contrast, edges, and contours that are encoded by the low-frequency components of the spectral transform. Given feature  $\mathbf{X} \in \mathbb{R}^{C \times H \times W}$ , we use scattering transform using DTCWT [50] as discussed in section-2.1 to obtain the corresponding frequency representations  $\mathbf{X}_F$  by  $\mathbf{X}_F = \mathcal{F}_{\mathrm{scatter}}(\mathbf{X})$ . The transformation in frequency domain  $\mathbf{X}_F$  provides two components, one low-frequency component i.e. scaling component  $\mathbf{X}_{\phi}$ , and one high-frequency component i.e. wavelet component  $\mathbf{X}_{\psi}$ . The simplified formulation for the real component of  $\mathcal{F}_{DTCW\tau}(\cdot)$  is:
$$
\mathbf {X} _ {F} (u, v) = \mathbf {X} _ {\phi} (u, v) + \mathbf {X} _ {\psi} (u, v) = \sum_ {h = 0} ^ {H - 1} \sum_ {w = 0} ^ {W - 1} c _ {M, h, w} \phi_ {M, h, w} + \sum_ {m = 0} ^ {M - 1} \sum_ {h = 0} ^ {H - 1} \sum_ {w = 0} ^ {W - 1} \sum_ {k = 1} ^ {6} d _ {m, h, w} ^ {k} \psi_ {m, h, w} ^ {k} \tag {3}
$$
$M$  refers to resolution/level of decomposition and  $k$  refers to directional selectivity. Similarly, we compute transformation for the imaginary component of  $\mathcal{F}_{D\tau C\mathcal{W}\tau}(\cdot)$ .
B. Spectral Gating Network:
We introduce a novel method, Spectral Gating Network (SGN), to extract spectral features from both low and high-frequency components of the scattering transform. Figure-1 shows the architecture of our method. We use learnable weight parameters to blend each frequency component, but we use different blending methods for low and high frequencies. For the low-frequency component  $\mathbf{X}_{\phi} \in \mathcal{R}^{C \times H \times W}$ , we use the Tensor Blending Method (TBM), which is a new technique. TBM blends  $X_{\phi}$  with  $W_{\phi}$  using elementwise tensor multiplication, also known as Hadamard tensor product.
$$
\mathcal {M} _ {\phi} = \left[ \mathbf {X} _ {\phi} \odot \mathcal {W} _ {\phi} \right], \quad \text {w h e r e} \left(\mathbf {X} _ {\phi}, \mathcal {W} _ {\phi}\right) \in \mathcal {R} ^ {C \times H \times W}, \text {a n d} \mathbf {M} _ {\phi} \in \mathcal {R} ^ {C \times H \times W}, \tag {4}
$$
$\mathcal{W}_{\phi}$  having same dimension as in  $\mathcal{X}_{\phi}$ .  $\mathcal{M}_{\phi}$  is the low-frequency representation of the image and it captures global information of the image. One of the biggest challenges to getting effective features in the high-frequency components  $\mathbf{X}_{\psi} \in \mathcal{R}^{k \times C \times H \times W \times 2}$ , which are complex-valued and have 'k' times more dimensions than the low-frequency components  $(\mathbf{X}_{\phi})$ . Therefore, using the same Tensor Blending Method for the high-frequency components  $X_{\psi}$  would increase the number of parameters by 2k times and also the computational cost (GFLOPS), where  $k$  refers to directional selectivity, a factor of '2' indicating complex value comprising real and imaginary. To address this issue, we propose a new technique, the Einstein Blending Method (EBM), to blend the high-frequency components  $X_{\psi}$  with the learnable weight parameters  $W_{\psi}$  efficiently and effectively in the Spectral Gating Network that we propose in this paper. By using EBM, we can capture the fine-grain information in the image, such as texture, patterns, and small features.
To perform EBM, we first reshape a tensor  $\mathbf{A}$  from  $\mathbb{R}^{H\times W\times C}$  to  $\mathbb{R}^{H\times W\times C_b\times C_d}$ , where  $C = C_b\times C_d$ , and  $b >> d$ . We then define a weight matrix of size  $W\in \mathbb{R}^{C_b\times C_d\times C_d}$ . We then perform Einstein multiplication between  $\mathbf{A}$  and  $W$  along the last two dimensions, resulting in a blended feature tensor  $Y\in \mathbb{R}^{H\times W\times C_b\times C_d}$  as shown in the Figure-2. The formula for EBM is:
$$
\mathbf {Y} ^ {H \times W \times C _ {b} \times C _ {d}} = \mathbf {A} ^ {H \times W \times C _ {b} \times C _ {d}} \text {图} \mathbf {W} ^ {C _ {b} \times C _ {d} \times C _ {d}}
$$
Figure 2: Einstein Blending Method
C. Spectral Channel and Token Mixing:
We perform EBM in the channel dimension of the high-frequency component we call Spectral Channel Mixing and following that we perform EBM in the token dimension of the high-frequency component, which we call Spectral Token Mixing. To perform EBM in the channel dimension, we first reshape the high-frequency component  $X_{\psi}$  from  $\mathbb{R}^{2\times k\times H\times W\times C}$  to  $\mathbb{R}^{2\times k\times H\times W\times C_b\times C_d}$ , where  $C = C_b\times C_d$  and  $b >> d$ . We then define a weight matrix of size  $W_{\psi_c}\in \mathbb{R}^{C_b\times C_d\times C_d}$ . We then perform Einstein multiplication between  $X_{\psi}$  and  $W_{\psi_c}$  along the last two dimensions, resulting in a blended feature tensor  $\mathbf{S}_{\psi_c}\in \mathbb{R}^{2\times k\times H\times W\times C_b\times C_d}$ . The formula for EBM in Channel mixing is:
$$
\mathbf {S} _ {\psi_ {c}} ^ {2 \times k \times H \times W \times C _ {b} \times C _ {d}} = \mathbf {X} _ {\psi} ^ {2 \times k \times H \times W \times C _ {b} \times C _ {d}} * \mathbf {W} _ {\psi_ {c}} ^ {C _ {b} \times C _ {d} \times C _ {d}} + b _ {\psi_ {c}} \tag {5}
$$
To perform EBM in the Token dimension, we first reshape the high-frequency component  $S_{\psi_c}$  from  $\mathbb{R}^{2\times k\times H\times W\times C}$  to  $\mathbb{R}^{2\times k\times C\times W\times H}$ , where  $Height(H) = Width(W)$ . We then define a weight matrix of size  $W_{\psi_t} \in \mathbb{R}^{W\times H\times H}$ . We then perform Einstein multiplication between  $X_{\psi}$  and  $W_{\psi_t}$  along the last two dimensions, resulting in a blended feature tensor  $\mathbf{S}_{\psi_t} \in \mathbb{R}^{2\times k\times C\times W\times H}$ . The formula for EBM in Token mixing is:
$$
\mathbf {S} _ {\psi_ {t}} ^ {2 \times k \times C \times W \times H} = \mathbf {S} _ {\psi_ {c}} ^ {2 \times k \times C \times W \times H} * \mathbf {W} _ {\psi_ {\mathrm {t}}} ^ {W \times H \times H} + b _ {\psi_ {t}} \tag {6}
$$
Where  $\boxdot$  represents an Einstein multiplication, the bias terms  $b_{\psi_c} \in \mathbb{R}^{C_b \times C_d}$ ,  $b_{\psi_t} \in \mathbb{R}^{H \times H}$ . Now the total number of weight parameters in the high-frequency gating network is  $(C_b \times C_d \times C_d) + (W \times H \times H)$  instead of  $(C \times H \times W \times k \times 2)$  where  $C >> H$  and bias is  $(C_b \times C_d) + (H \times W)$ . This reduces the number of parameters and multiplication while performing high-frequency gating operations in an image. We use a standard torch package [48] to perform Einstein multiplication. Finally, we perform inverse scattering transform using low-frequency representation (4) and high-frequency representation (6) to bring back the spectral domain to the physical domain. Our SVT architecture consists of L layers, comprising  $\alpha$  scatter layers and  $(L - \alpha)$  attention layers [59], where L denotes the network's depth. The scatter layers, being invertible, adeptly capture both the global and the fine-grain information in the image effectively via low-pass and high-pass filters, while attention layers focus on extracting semantic features and addressing long-range dependencies present in the image.
3 Experiment and Performance Studies
We evaluated SVT through various mainstream computer vision tasks including image recognition, object detection, and instance segmentation. To compare the quality of SVT transformer features, we conducted the following evaluations on standard datasets: a) We trained and evaluated ImageNet1K [10] from scratch for image recognition task. b) We performed transfer learning on CIFAR-10 [30], CIFAR-100 [30], Stanford Car [29], and Oxford Flower-102 [41] for Image recognition task. c) We conducted ablation studies to analyze variants of SVT transformers using scatter net with the help of various spectral mixing techniques. We also compare our results with transformers having similar decomposition architecture. d) We fine-tune SVT for downstream instance segmentation tasks. e) We also perform an in-depth analysis of the SVT, by conducting layer-wise analysis as well as invertibility analysis as well as latency analysis, and comparison.
3.1 Comparison with Similar architectures
We compare SVT with LiTv2 (Hilo) [42] which decomposes attention to find low and high-frequency components. We show that LiTv2 has a top-1 accuracy of  $83.3\%$ , while SVT has a top-1 accuracy of
85.2% with a fewer number of parameters. We also compare SVT with iFormer [51] which captures low and high-frequency information from visual data, whereas SVT uses an invertible spectral method, namely the scattering network, to get the low-frequency and high-frequency components and uses tensor and Einstein mixing respectively to capture effective spectral features from visual data. SVT top-1 accuracy is 85.2, which is better than iFormer-B, which is at 84.6 with a lesser number of parameters and FLOPS. We compare SVT with WaveMLP [53] which is an MLP mixer-based technique that uses amplitude and phase information to represent the semantic content of an image. SVT uses a low-frequency component as an amplitude of the original feature, while a high-frequency component captures complex semantic changes in the input image. Our studies have shown, as depicted in Table- 1, that SVT outperforms WaveMLP by about  $1.8\%$ .
3.2 Comparison with State of the art methods
We divide the transformer architecture into three parts based on the computation requirements (FLOP counts) - small (less than 6 GFLOPS), base (6-10 GFLOPS), and large (10-30 GFLOPS). We use a similar categorization as WaveViT [70]. Notable recent works falling into the small category include C-Swin Transformers [12], LiTv2[42], MaxVIT[58], iFormer[51], CMT transformer, PVTv2[62], and WaveViT[70]. It's worth mentioning that WaveViT relies on extra annotations to achieve its best results. In this context, SVT-H-S stands out as the state-of-the-art model in the small category,