SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Qihang Fan  $^{1,2}$ , Huaibo Huang $^{1}$ , Xiaojiang Zhou $^{1,3}$ , Ran He $^{1,2*}$
$^{1}$ MAIS & CRIPAC, Institute of Automation, Chinese Academy of Sciences, Beijing, China  
 $^{2}$ School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
<sup>3</sup>University of Science and Technology of China, Hefei, China
fanqihang.159@gmail.com, huaibo.huang@cripac.ia.ac.cn,
xq525@mail.ustc.edu.cn，rhe@nlpr.ia.ac.cn
Recent advancements in vision backbones have significantly improved their performance by simultaneously modeling images' local and global contexts. However, the bidirectional interaction between these two contexts has not been well explored and exploited, which is important in the human visual system. This paper proposes a Fully Adaptive Self-Attention (FASA) mechanism for vision transformer to model the local and global information as well as the bidirectional interaction between them in context-aware ways. Specifically, FASA employs self-modulated convolutions to adaptively extract local representation while utilizing self-attention in down-sampled space to extract global representation. Subsequently, it conducts a bidirectional adaptation process between local and global representation to model their interaction. In addition, we introduce a fine-grained downsampling strategy to enhance the down-sampled self-attention mechanism for finer-grained global perception capability. Based on FASA, we develop a family of lightweight vision backbones, Fully Adaptive Transformer (FAT) family. Extensive experiments on multiple vision tasks demonstrate that FAT achieves impressive performance. Notably, FAT accomplishes a  $77.6\%$  accuracy on ImageNet-1K using only 4.5M parameters and 0.7G FLOPs, which surpasses the most advanced ConvNets and Transformers with similar model size and computational costs. Moreover, our model exhibits faster speed on modern GPU compared to other models.
Vision Transformers (ViTs) have recently garnered significant attention in the computer vision community due to their exceptional ability for long-range modeling and context-aware characteristics. However, because of the quadratic complexity of self-attention in ViT [13], its computational cost is extremely high. As a result, many studies have emerged to improve ViT's computational efficiency and performance in various ways. For instance, some methods restrict tokens that perform self-attention to a specific region and introduce inductive bias to ViT [34; 12; 66; 55]. Further, some methods aim to transform ViT into lightweight backbones with fewer parameters and computational requirements [38; 40; 4; 30; 37], achieving promising results but still not matching the performance of the most advanced ConvNets [51]. How to design an excellent lightweight Vision Transformer remains a challenge.
In current state-of-the-art Vision Transformers, some either excel in creating local feature extraction modules [34; 12; 66] or employing efficient global information aggregation modules [57; 58], while others incorporate both [42; 41]. For instance, LVT [66] unfolds tokens into separate windows and
*Ran He is the corresponding author.
Figure 1: Illustration of the human visual system (top) and our FASA (bottom). The human visual system can perceive both local and global contexts and model the bidirectional interaction between them. Our FASA follows this mechanism and consists of three parts: (a) local adaptive aggregation, (b) global adaptive aggregation, and (c) bidirectional adaptive interaction. Our FASA models local information, global information, and local-global bidirectional interaction in context-aware manners.
applies self-attention within the windows to extract local features, while PVT [57; 58] leverages self-attention with downsampling to extract global features and reduce computational cost. Unlike them, LITv2 [41] relies on window self-attention and spatial reduction attention to capture local and global features, respectively. In terms of the local-global fusion, most methods use simple local-global sequential structures [38; 40; 37; 26; 82], whereas others combine local and global representation with simple linear operations through local-global parallel structures [48; 41; 43]. However, few works have investigated the bidirectional interaction between local and global information. Considering the human visual system where bidirectional local-global interaction plays an important role, these simplistic mixing methods are not fully effective in uncovering the intricate relationship between local and global contexts.
In Fig. 1, we illustrate how humans observe an object and notice its body details. Using the example of a fox, we can observe two types of interaction that occur when humans focus on either the fox's nose or the entire animal. In the first type of interaction, known as Local to Global, our understanding of the local feature transforms into the "Nose of a fox." In the second type of interaction, called Global to Local, the way we comprehend the global feature changes to the "A fox with nose." It can be seen that the bidirectional interaction between local and global features plays an essential role in the human visual system. Based on this fact, we propose that a superior visual model should not only extract good local and global features but also possess adequate modeling capabilities for their interaction.
Figure 2: Top-1 accuracy v.s. FLOPs on ImageNet-1K of recent SOTA CNN and transformer models. The proposed Fully Adaptive Transformer (FAT) outperforms all the counterparts in all settings.
In this work, our objective is to model the bidirectional interaction between local and global contexts while also improving them separately. To achieve this goal, we introduce three types of Context-Aware Feature Aggregation modules. Specifically, as shown in Fig. 1, we first adaptively aggregate local and global features using context-aware manners to obtain local and global tokens, respectively. Then, we perform point-wise cross-modulation between these two types of tokens to model their
bidirectional interaction. We streamline all three processes into a simple, concise, and straightforward procedure. Since we use context-aware approaches to adaptively model all local, global, and local-global bidirectional interaction, we name our novel module the Fully Adaptive Self-Attention (FASA). In FASA, we also further utilize a fine-grained downsampling strategy to enhance the self-attention mechanism, which results in its ability to perceive global features with finer granularity. In summary, FASA introduces only a small number of additional parameters and FLOPs, yet it significantly improves the model's performance.
Building upon FASA, we introduce the Fully Adaptive Transformer (FAT) family. The FATs follow the hierarchical design [34; 57] and serve as general-purpose backbones for various computer vision tasks. Through extensive experiments, including image classification, object detection, and semantic segmentation, we validate the performance superiority of the FAT family. Without extra training data or supervision, our FAT-B0 achieves a top-1 accuracy of  $77.6\%$  on ImageNet-1K with only 4.5M parameters and 0.7G FLOPs, which is the first model surpasses the most advanced ConvNets with similar model size and computational cost as far as we know. Additionally, as shown in Fig. 2, our FAT-B1, B2, and B3 also achieve state-of-the-art results while maintaining similar model sizes and computational costs.
Vision Transformer. Since the earliest version of ViT [13] appeared, numerous works have been proposed that focus on enhancing the self-attention mechanism. Many methods restrict self-attention to a subset of tokens, sacrificing global receptive fields to reduce the computation of ViT [34; 12; 66; 61; 1]. Despite having only local perception capabilities, the context-aware nature of self-attention enables these methods to achieve excellent performance. A famous among them is the Swin-Transformer [34], which divides all tokens into windows and performs self-attention within each window, achieving highly competitive performance. In contrast, some methods downsample the  $K$  and  $V$  in self-attention to preserve the global receptive field while reducing computation by minimizing the number of token pairs involved in the calculation [57; 60; 40; 47; 68; 5]. One such method, PVT [57], leverages large-stride convolution to process  $K$  and  $V$ , effectively lowering their spatial resolution. In addition to these methods, numerous efforts have been made to introduce ViT into the design of lightweight vision backbones [26; 40; 37; 15; 38; 4]. For example, MobileViT [38] concatenates convolution and self-attention to obtain a powerful, lightweight backbone. Nevertheless, a performance gap exists between lightweight vision transformers and state-of-the-art lightweight CNNs such as NAS-based EfficientNet [51].
Local-Global Fusion. A high-performance vision backbone typically possesses exceptional capabilities for both local and global perception. The capabilities are achieved by either connecting local and global perception modules in a serial manner [58; 31; 10; 26; 40; 55; 1], as demonstrated by DaViT [10], or by simultaneously modeling local and global information within a single module [48; 41; 43], such as the inception transformer [48]. However, current approaches to fusing local and global information are overly simplistic. In serial models, the process of fusing local and global information is not adequately represented. In parallel structures, almost all methods rely on linear modules that depend entirely on trainable parameters to fuse local and global information [41; 48; 43]. These fusing approaches lack the ability to model the interaction between local and global information, which is inconsistent with the human visual system shown in Fig. 1. In contrast, our proposed FASA module models bidirectional interaction between local and global information while separately modeling each one.
Self-Attention with Downsampling. Currently, many models utilize self-attention with downsampling, a technique that was earlier used in PVT and so on to improve the computational efficiency [57; 58; 43; 41; 48; 61; 5]. PVT [57] reduces the spatial resolution of  $K$  and  $V$  using non-overlapping large stride convolutions, which decreases the number of token pairs involved in self-attention and lowers the computational cost while maintaining a global receptive field. Similarly, PVTv2 [58] uses large stride average pooling to downsample  $K$  and  $V$ . In contrast, inception transformer [48] employs large stride average pooling to downsample all three  $Q$ ,  $K$ , and  $V$ , then upsamples the tokens to their original size after self-attention is applied. However, using excessively large strides or non-overlapping downsampling as in these methods may lead to significant information loss. In FASA, we propose a fine-grained downsampling strategy to alleviate this issue.
Figure 3: Illustration of the FAT. FAT is composed of multiple FAT blocks. A FAT block consists of CPE, FASA and ConvFFN.
3.1 Overall Architecture.
The overall architecture of the Fully Adaptive Transformer (FAT) is illustrated in Fig. 3. To process an input image  $x \in \mathcal{R}^{3 \times H \times W}$ , we begin by feeding it into the convolutional stem used in [63]. This produces tokens of size  $\frac{H}{4} \times \frac{W}{4}$ . Following the hierarchical designs seen in previous works [48; 50; 46; 49], we divide FAT into four stages to obtain hierarchical representation. Then we perform average pooling on the feature map containing the richest semantic information. The obtained one-dimensional vector is subsequently classified using a linear classifier for image classification.
A FAT block comprises three key modules: Conditional Positional Encoding (CPE) [6], Fully Adaptive Self-Attention (FASA), and Convolutional Feed-Forward Network (ConvFFN). The complete FAT block is defined by the following equation (Eq. 1):
$$
X = \operatorname {C P E} \left(X _ {i n}\right) + X _ {i n},
$$
$$
Y = \operatorname {F A S A} (\ln (X)) + X, \tag {1}
$$
$$
Z = \operatorname {C o n v F F N} (\mathrm {L N} (Y)) + \operatorname {S h o r t C u t} (Y).
$$
Initially, the input tensor  $X_{in} \in \mathcal{R}^{C \times H \times W}$  passes through the CPE to introduce positional information for each token. The subsequent stage employs FASA to extract local and global representation adaptively, while a bidirectional adaptation process enables interaction between these two types of representation. Finally, ConvFFN is applied to enhance local representation further. Two kinds of ConvFFNs are employed in this process. In the in-stage FAT block, ConvFFN's convolutional stride is 1, and ShortCut = Identity. At the intersection of the two stages, the ConvFFN's convolutional stride is set to 2, and ShortCut = DownSample is achieved via a Depth-Wise Convolution (DWConv) [23] with the stride of 2 along with a  $1 \times 1$  convolution. The second type of ConvFFN accomplishes downsampling using only a small number of parameters, avoiding the necessity of patch merging modules between the two stages, thereby saving the parameters.
3.2 Fully Adaptive Self-Attention
In Fig. 1, we present the visual system of humans, which not only captures local and global information but also models their interaction explicitly. Taking inspiration from this, we aim to develop a similar module that can adaptively model local and global information and their interaction. This leads to the proposal of the Fully Adaptive Self-Attention (FASA) module. Our FASA utilizes context-aware manners to model all three types of information adaptively. It comprises three modules:
global adaptive aggregation, local adaptive aggregation, and bidirectional adaptive interaction. Given the input tokens  $X \in \mathcal{R}^{C \times H \times W}$ , each part of the FASA will be elaborated in detail.
3.2.1 Definition of Context-Aware Feature Aggregation
In order to provide readers with a clear comprehension of FASA, we will begin by defining the various methods of context-aware feature aggregation (CAFA). CAFA is a widely used feature aggregation method. Instead of solely relying on shared trainable parameters, CAFA generates token-specific weights based on the target token and its local or global context. These newly generated weights, along with the associated context of the target token, are then used to modulate the target token during feature aggregation, enabling each token to adapt to its related context. Generally, CAFA consists of two processes: aggregation  $(\mathcal{A})$  and modulation  $(\mathcal{M})$ . In the following, the target token is denoted by  $x_{i}$ , and  $\mathcal{F}$  represents a non-linear activation function. Based on the order of  $\mathcal{A}$  and  $\mathcal{M}$ , CAFA can be classified into various types. Various forms of self-attention [56; 58; 57; 45] can be expressed as Eq. 2. Aggregation over the contexts  $X$  is performed after the attention scores between query and key are computed. The attention scores are obtained by modulating the query with the keys, and then applying a Softmax to the resulting values:
$$
y _ {i} = \mathcal {A} (\mathcal {F} (\mathcal {M} (x _ {i}, X)), X), \tag {2}
$$
In contrast to the approach outlined in Eq. 2, recent state-of-the-art ConvNets [22; 17; 67] utilize a different CAFA technique. Specifically, they employ DWConv to aggregate features, which are then used to modulate the original features. This process can be succinctly described using Eq. 3.
$$
y _ {i} = \mathcal {M} \left(\mathcal {A} \left(x _ {i}, X\right), x _ {i}\right), \tag {3}
$$
In our FASA module, global adaptive aggregation improves upon the traditional self-attention method with fine-grained downsampling and can be mathematically represented by Eq. 2. Meanwhile, our local adaptive aggregation, which differs slightly from Eq. 3, can be expressed as Eq. 4:
$$
y _ {i} = \mathcal {M} \left(\mathcal {F} \left(\mathcal {A} \left(x _ {i}, X\right)\right), \mathcal {A} \left(x _ {i}, X\right)\right), \tag {4}
$$
In terms of the bidirectional adaptive interaction process, it can be formulated by Eq. 5. Compared to the previous CAFA approaches, bidirectional adaptive interaction involves two feature aggregation operators  $(\mathcal{A}_1$  and  $\mathcal{A}_2)$  that are modulated with each other:
$$
\begin{array}{l} y _ {i 2} = \mathcal {M} (\mathcal {F} (\mathcal {A} _ {1} (x _ {i}, X)), \mathcal {A} _ {2} (x _ {i}, X)), \\ y _ {i i} = \mathcal {M} (\mathcal {F} (\mathcal {A} _ {2} (x _ {i}, X)), \mathcal {A} _ {3} (x _ {i}, X)). \end{array} \tag {5}
$$
$$
y _ {i 1} = \mathcal {M} \left(\mathcal {F} \left(\mathcal {A} _ {2} \left(x _ {i}, X\right)\right), \mathcal {A} _ {1} \left(x _ {i}, X\right)\right).
$$
3.2.2 Global Adaptive Aggregation
The inherently context-aware nature and capacity to model long-distance dependencies of self-attention make it highly suitable for adaptively extracting global representation. As such, we utilize self-attention with downsampling for global representation extraction. In contrast to other models that downsample tokens [57; 58; 60; 48] by using large stride convolutions or pooling operations, we adopt a fine-grained downsampling strategy to minimize loss of global information to the greatest extent possible. In particular, our fine-grained downsampling module is composed of several basic units. Each unit utilizes a DWConv with a kernel size of  $5 \times 5$  and a stride of 2, followed by a  $1 \times 1$  convolution that subtly downsamples both  $K$  and  $V$ . After that,  $Q$ ,  $K$ , and  $V$  are processed through the Multi-Head Self-Attention (MHSA) module. Unlike regular MHSA, we omit the last linear layer. The complete procedure for the global adaptive aggregation process is illustrated in Eq. 6 and Eq. 7. Fisrt, we define our fine-grained downsample strategy and the base unit of it in Eq. 6:
$$
\operatorname {B a s e U n i t} (X) \triangleq \operatorname {C o n v} _ {1 \times 1} (\operatorname {B N} (\operatorname {D W C o n v} (X))), \tag {6}
$$
$$
\operatorname {p o o l} (X) \triangleq \operatorname {B N} (\operatorname {D W C o n v} (\operatorname {B a s e U n i t} ^ {(n)} (X)),
$$
where  $(\mathrm{n})$  represents the number of base units that are concatenated and pool denotes our fine-grained downsample operator. Then, the MHSA is conducted as Eq. 7:
$$
\begin{array}{l} Q, K, V = W _ {1} \otimes X, \\ X ^ {\prime} \quad \text {M I H S A} (Q, \operatorname {c o n v l} (K), \operatorname {c o n v l} (V)) \end{array} \tag {7}
$$
$$
X _ {g l o b a l} ^ {\prime} = \operatorname {M H S A} (Q, \operatorname {p o o l} (K), \operatorname {p o o l} (V)). \tag {7}
$$
where the mathematical symbol  $\otimes$  denotes the operation of matrix multiplication,  $W_{1}$  is a learnable matrix.
3.2.3 Local Adaptive Aggregation
In contrast to self-attention, convolution possesses an inductive bias that facilitates extracting high-frequency local information. However, convolutional feature aggregation solely relies on parameters-shared convolutional kernels, resulting in a lack of context-awareness. To address this issue, we utilize a self-modulating convolutional operator that embeds context-awareness into convolution, enabling it to extract local representation adaptively. Specifically, we generate context-aware weights through the Sigmoid and combine them with DWConv to adaptively aggregate local information. This process is summarized in Eq. 8, where  $\odot$  represents the Hadamard product, and  $Q$  is directly derived from Eq. 7 for saving parameters of linear projection:
$$
Q ^ {\prime} = \operatorname {D W C o n v} (Q),
$$
$$
X _ {\text {l o c a l}} ^ {\prime} = Q ^ {\prime} \odot \operatorname {S i g m o i d} \left(Q ^ {\prime}\right) = Q ^ {\prime} \odot \frac {1}{1 + e ^ {- Q ^ {\prime}}}. \tag {8}
$$
3.2.4 Bidirectional Adaptive Interaction
As illustrated in Fig. 1, when modeling the fox and its nose separately, two types of interactions - "Local to Global" and "Global to Local" - occur between their respective features. Inspired by the human visual system, we design a bidirectional adaptive interaction process that incorporates both types of interactions. To achieve bidirectional interaction, we adopt a method similar to the one described in Sec. 3.2.3 but utilize cross-modulation instead of self-modulation. Specifically, the equation for Local to Global interaction is given by Eq. 9:
$$
\begin{array}{l} X _ {\text {l o c a l}} = X _ {\text {l o c a l}} ^ {\prime} \odot \operatorname {S i g m o i d} \left(X _ {\text {g l o b a l}} ^ {\prime}\right) = X _ {\text {l o c a l}} ^ {\prime} \odot \frac {1}{1 + e ^ {- X _ {\text {g l o b a l}} ^ {\prime}}} \tag {9} \\ = Q ^ {\prime} \odot \frac {1}{1 + e ^ {- Q ^ {\prime}}} \odot \frac {1}{1 + e ^ {- X _ {g l o b a l} ^ {\prime}}}, \\ \end{array}
$$
Similar to Eq. 9, Global to Local interaction is achieved by Eq. 10:
$$
\begin{array}{l} X _ {\text {g l o b a l}} = X _ {\text {g l o b a l}} ^ {\prime} \odot \operatorname {S i g m o i d} \left(X _ {\text {l o c a l}} ^ {\prime}\right) = X _ {\text {g l o b a l}} ^ {\prime} \odot \frac {1}{1 + e ^ {- X _ {\text {l o c a l}} ^ {\prime}}} \tag {10} \\ = X _ {g l o b a l} ^ {\prime} \odot \frac {1}{1 + e ^ {- Q ^ {\prime} \odot \frac {1}{1 + e ^ {- Q ^ {\prime}}}}}, \\ \end{array}
$$
After completing the two interaction processes, the local and global representation contain information about each other. To merge these representation, we use point-wise multiplication and achieve intermingling among channels with a linear projection, as shown in Eq. 11:
$$
\begin{array}{l} Y = W _ {2} \otimes \left(X _ {\text {g l o b a l}} \odot X _ {\text {l o c a l}}\right) \\ = W _ {2} \otimes \left(X _ {g l o b a l} ^ {\prime} \odot Q ^ {\prime} \odot \frac {1}{1 + e ^ {- Q ^ {\prime}}} \odot \frac {1}{1 + e ^ {- X _ {g l o b a l} ^ {\prime}}} \odot \frac {1}{1 + e ^ {- Q ^ {\prime} \odot \frac {1}{1 + e ^ {- Q ^ {\prime}}}}}\right) \tag {11} \\ \end{array}
$$
It is worth noting that to get a faster and implementable model, we omit the last high-order variable in Eq. 11 and arrive at a concise expression. More details can be found in appendix. Through the combination of local adaptive aggregation, global adaptive aggregation, and bidirectional adaptive interaction, we present the comprehensive FASA module. As depicted in Fig. 3, our model, which draws inspiration from the human visual system, is both straightforward and simple. It adaptively models local, global, and the interaction between the two in context-aware manners. More discussion about bidirectional adaptive interaction can be found in the appendix.
We conducted experiments on a wide range of vision tasks, including image classification on ImageNet-1K [9], object detection and instance segmentation on COCO 2017 [33], and semantic segmentation on ADE20K [80]. In addition to them, we also make ablation studies to validate the importance of each component. More details can be found in the appendix.
Settings. We train our models on ImageNet-1K [9] from scratch. And we follow the same training strategy in DeiT [54] for a fair comparison. Different from DeiT, we employ 20 epochs of linear warm-up, while in DeiT it is 5. The maximum rates of increasing stochastic depth [24] are set to  $0.05 / 0.05 / 0.1 / 0.15$  for B0/B1/B2/B3, respectively. More details can be found in the appendix.