SI-BiViT: Binarizing Vision Transformers with Spatial Interaction

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-20 09:46:33

Abstract: Binarized Vision Transformers (BiViTs) aim to facilitate the efficient and lightweight utilization of Vision Transformers (ViTs) on devices with limited computational resources. Yet, the current approach to binarizing ViT leads to a substantial performance decrease compared to the full-precision model, posing obstacles to practical deployment. By empirical study, we reveal that spatial interaction (SI) is a critical factor that impacts performance due to lack of token-level correlation, but previous work ignores this factor. To this end, we design a ViT binarization approach dubbed SI-BiViT to incorporate spatial interaction in the binarization process. Specifically, an SI module is placed alongside the Multi-Layer Perceptron (MLP) module to formulate the dual-branch structure. This structure not only leverages knowledge from pre-trained ViTs by distilling over the original MLP, but also enhances spatial interaction via the introduced SI module. Correspondingly, we design a decoupled training strategy to train these two branches more effectively. Importantly, our SI-BiViT
is orthogonal to existing Binarized ViTs approaches and can be directly plugged.
Extensive experiments demonstrate the strong flexibility and effectiveness of SI-BiViT by plugging our method into four classic ViT backbones in supporting three downstream tasks, including classification, detection, and segmentation. In particular, SI-BiViT enhances the classification performance of binarized ViTs by an average of 10.52\% in Top-1 accuracy compared to the previous state-of-the-art. The code will be made publicly available.

Main paper: 
 related works 2.1 vision transformer: 
 The Transformers [49] is originally proposed to solve natural language processing (NLP) problems and achieves great success by its long-range interaction mechanism. Vision Transformer (ViT) [14] first introduces transformers into the field of computer vision. Images are divided into non-overlapping patches, with each patch being embedded into a token. These tokens are then processed using the same pipeline as those used in NLP. Deit [48] optimizes data augmentation and distillation tokens for ViT and further enhances the generalization ability of ViT. However, these models suffer from the drawback of high computational complexity, resulting in significantly lower efficiency problems. This is mainly due to the quadratic computational complexity of the self-attention mechanism [23, 50] and the fixed size of embeddings throughout the computation process [58] . To mitigate these issues, LGViT [54] incorporates heterogeneous exiting heads, namely, local perception head and global aggregation head, to achieve an efficiency-accuracy trade-off. Swin Transformer [34] and Nest Transformer [62] design compact and hierarchical architectures to capture global and local information more efficiently. Additionally, other studies explore quantization on ViT [13, 16, 27, 37] , pruning techniques [56, 57] and ditillation [9, 29] . Compared to these compression approaches, model binarization shows significant advantages in reducing calculation and memory.As a pioneer work in model binarization, BNN [45] quantizes both weights and activations to 1 bit, which significantly accelerates the inference of neural networks and saves memory occupancy through XNOR and popcount operator. However, binarization will lead to a severe performance drop, causing a severe performance drop. Consequently, numerous studies endeavor to enhance model binarization techniques. [38] proposes Bi-Real net, which connects the real activations to activations of the consecutive block through an identity shortcut and further strengthens the network's representation ability. [28] considers the angle alignment between the full-precision weight vector and its binarized version and proposes to rotate the weight vector to reduce the angular bias. These works primarily focus on minimizing quantization errors [3, 28, 31] , designing better Straight-Through Estimator (STE) [4, 38] , improving better distribution for BNN [12, 36] and increasing the information capacity of binarized models [44] .Binarization of transformer-based models has attracted a lot of interest since they achieve great success in a lot of downstream tasks. Unfortunately, existing binarization approaches designed for convolutional neural networks cannot migrate well to the transformers due to distinct architectural differences. BiBERT [43] proposes a Bi-Attention structure to maximize the information entropy of the self-attention module and further proposes direction-matching distillation to reduce the gradient mismatch problem in attention. Liu et al. [35] propose a two-set binarization scheme and a successively distilling mechanism to preserve more information from pre-trained models. He et al. [19] propose softmax-aware binarization to address the long-tailed distribution of softmax attention. They also introduce a cross-layer binarization scheme to decouple the binarization of the attention module and MLP module. Li et al. [26] propose a ranking-aware distillation method to alleviate attention distortion. However, most existing approaches overlook the key effect of spatial interactions, which severely hinders the performance of binarized ViTs. 
method 3.1 preliminaries: 
 Binarized Neural Networks generally use sign function to quantize weights and activations to -1 and +1 [45] . Unfortunately, the gradient of sign function is non-differentiable and thus cannot be updated via backpropagation. To overcome the non-differentiable issue, Straight-Through Estimator(STE) [2] is applied to approximate gradients in the back-propagation. The formula of forward and backward is:EQUATIONBackward:EQUATIONwhere the 𝑏 denotes the binarized values. Gradient clipping [20] is commonly used to prevent unstable weight oscillation. STE only backpropagates gradients when the absolute values are smaller than 1 and truncates them when they exceed 1.In vision transformer, images are represented by token sequences 𝐴 ∈ R 𝑛×𝑑 , where 𝑛 is the sequence length and 𝑑 is the length of embedding. We binarize them as follows:EQUATIONwhere 𝐴 and 𝐴 𝑏 denote input real-valued activation and binarized activation respectively. 𝛾 ∈ R 𝑑 is a channel-wise learnable factor introduced by [36] , which can better control the threshold of activations in binarization.Similarly, the weights 𝑊 ∈ R 𝑑 ×𝑑 ′ in the linear layer are also binarized channel-wisely. In order to reduce quantization error, a scaling factor 𝜃 𝑤 is commonly adopted [44] . The weights get binarized as follows:EQUATIONwhere 𝜇 𝑤 is the mean value of 𝑊 .When weights and activations are all binarized, the computationally heavy operations of floating-point matrix multiplication are replaced by binarized matrix multiplication, which can be accelerated by efficient bitwise XNOR operations and popcount operations [3] :EQUATIONwhere 𝐴 𝑏 and 𝑊 𝑏 are binarized activations and binarized weights.For the first time, we empirically reveal lack of token-level correlation severely limits the performance of binarized ViTs. In vision transformer, an input image is represented as a sequence of patches and each token represents the feature of the patch. Denotes these tokens as 𝑍 ∈ R 𝑛×𝑑 , where 𝑛 is the length of the sequence and each token 𝑍 𝑖 is a 𝑑-dimension vector, describing the feature of a patch.In order to analyze spatial interaction in the ViT, following the definition of representation ability [24, 38] , we introduce the concept of spatial interaction capability (SIC) to measure token-level correlation in binarized vision transformers:EQUATIONwhere Inter(𝑍 𝑖 ) represents the number of tokens 𝑍 𝑖 interacted with. 𝑅𝑎𝑛𝑔𝑒 (𝑍 𝑖 ) denotes the range of interaction strength. The multihead self-attention calculates attention score matrix 𝑎𝑡𝑡𝑛 ∈ R 𝑛×𝑛 and 𝑎𝑡𝑡𝑛 𝑖,𝑗 represents the interaction strength between token 𝑖 and token 𝑗. Thus, each token needs to interact with all other 𝑛 -1 tokens in MSA. So Inter(𝑍 𝑖 ) = 𝑛 -1 in MSA. In binarized models, the 𝑅𝑎𝑛𝑔𝑒 (𝑍 𝑖 ) of MSA is 2, because 𝑎𝑡𝑡𝑛 𝑖,𝑗 ∈ {0, 1}. 0 indicates these two tokens have no interaction and 1 indicates there is an interaction. Thus, The spatial interaction capability of MSA in binarized ViT can be represented as 2𝑛(𝑛 -1), where 𝑛 is the length of tokens.As another fundamental component in ViT, the multi-layer perception(MLP) is stacked by linear layers. The linear layer processes each token 𝑍 𝑖 using a weight matrix 𝑊 ∈ R 𝑑 ×𝑑 ′ , thereby extracting information in each token independently and expanding its dimension to 𝑑 ′ :EQUATIONwhere 𝑍 𝑖 ∈ R 1×𝑑 is a token from 𝑍 . It indicates that each 𝑍 𝑖 after linear layers has no interaction with other tokens, thus Inter(𝑍 𝑖 ) = 0 in MLP module. So the spatial interaction capability of MLP is 𝑛 ×0×2 = 0. While the SIC of the MSA module is 2𝑛(𝑛 -1), the MLP module contributes 0 spatial interaction, thus severely hindering the spatial interaction in binarized ViTs.To further uncover the relationship between the model's performance and spatial interaction, we gradually increase spatial interaction capability in four ViT backbones by adding more and more MSA modules. We reduce the MLP module accordingly to make the model size and computation roughly the same. As shown in Figure 2 , The classification accuracy has a clear upward trend with more and more spatial interaction. The curves on four ViT backbones all demonstrate that there is a positive relationship between spatial interaction and performance. This indicates that existing binarized vision transformers do lack spatial interaction and suffer severe performance drops.In this section, we present our SI-BiViT, which integrates spatial interaction into the binarization process in a plug-and-play manner. Specifically, we incorporate an SI module alongside the Multi-Layer Perceptron (MLP) to establish a dual-branch structure. This structure not only capitalizes on the knowledge from pre-trained ViTs by distilling over the original MLP but also enhances spatial interaction through the introduced SI module. Consequently, we devise a decoupled training strategy to effectively train these two branches. The entire framework is depicted in Figure 3 .There are various SI modules, such as cycle-FC [8] and multi-head self-attention [49] , which can improve token-level correlation. Here, we introduce the Token-mixing Linear [46] which is highly efficient. Denotes these tokens as 𝑍 ∈ R 𝑛×𝑑 , where 𝑛 is the length of the sequence and each token 𝑍 𝑖 is a 𝑑-dimension vector. 𝑍 ′ ∈ R 𝑑 ×𝑛 is obtained by transposing 𝑍 .Token-mixing Linear is represented as:EQUATIONwhere 𝑍 ′ 𝑖 ∈ R 1×𝑛 and 𝑊 ∈ R 𝑛×𝑛 ′ is weight parameters. Tokenmixing Linear operation captures spatial information by mixing features from different tokens. According to Equation 6, 𝐼𝑛𝑡𝑒𝑟 (𝑍 𝑖 ) = 𝑛 -1 since each token needs to interact with all other tokens. And the spatial interaction ability of Token-mixing Linear is 2𝑛 ′ (𝑛 -1). When 𝑛 ′ = 𝑛, its SIC is the same as that of MSA module. Moreover, we can adjust the spatial interaction ability by 𝑛 ′ . We find it achieves a balance between performance and accuracy when 𝑛 ′ = 𝑛 in Section 4.3.3. Compared to other SI modules [8, 49] , token-mixing Linear is highly efficient and outperforms other SI modules in both efficiency and performance.In model binarization, knowledge distillation is a prevalent and effective technique to improve performance by distilling intermediate features from full-precision models to binarized models. Moreover, distillation can significantly strengthen representation capabilities and relieve coarse gradients. [26, 40, 43] . However, direct applying distillation does not improve performance when we adopt SI module to replace the MLP module As shown in Table 1 . This is because there exist substantial differences in architecture between the SI module and MLP module, making it difficult to retain knowledge from pre-trained models. In fact, it may damage the performance after distillation, causing a decrease of 2.18%. To address this issue, we maintain the MLP module and introduce a spatial interaction module alongside it, forming a dual-branch structure. This ensures better distillation as well as improving spatial interaction. There is an obvious improvement(5.72%) in performance as shown in Table 1 . Moreover, we reduce the size of the MLP module, as we utilize knowledge distillation, and a smaller MLP can retain most information from pre-trained models without significant performance degradation. Additional details can be found in the Appendix.We further observe that our SI modules encounter challenges in optimizing due to gradient conflict problems. This refers to the situation where the gradient from the classifier does not align with that from knowledge distillation(shown in Figure 4 ), thereby hindering the optimization of our SI module. We define 𝑊 to represent the weights of the SI module, L 𝑐𝑙𝑠 representing the loss from the classifier, and L 𝑑𝑖𝑠𝑡𝑖𝑙𝑙 representing the loss from dense distillation. The distillation loss at one transformer block is formulated as:(a) (b) SI Module !ℒ !"# ! $ !ℒ %&#'&" ! $ forward backward SI Module !ℒ $%& ! ' !ℒ ()&*)% ! ' forward backwardEQUATIONwhere 𝐴 𝑠𝑡𝑢𝑑𝑒𝑛𝑡 is the output after the MLP module in binarized ViTs. 𝐴 𝑡𝑒𝑎𝑐ℎ𝑒𝑟 is the corresponding output from pre-trained models. Conflicting gradients are then described as follows:EQUATIONwhere 𝑔 + reflects the accumulation of harmonious gradients and 𝑔 - indicates the accumulation of conflicting gradients from distillation. 𝑊 𝑖 is one element from weight of SI module and 𝜕L 𝑑𝑖𝑠𝑡𝑖𝑙𝑙 /𝜕𝑊 𝑖 denotes gradient of𝑊 𝑖 from knowledge distillation and ⊙ is Hadamard product. 𝐼 𝑊 𝑖 is a indicator function defined as:EQUATION)where 𝜎 is the sign function. 𝜕L 𝑐𝑙𝑠 /𝜕𝑊 𝑖 denotes gradient of𝑊 𝑖 from knowledge distillation. As illustrated in Figure 4 , 𝑔 -(𝑊 ) reaches 1.97 and it is comparable to 𝑔 + (𝑊 ) at 2.45. This causes optimization oscillation when training the SI module and hinders its optimization.In order to decouple the training of these two branches, we propose a novel decoupling training strategy that not only leverages knowledge from pre-trained ViTs by distilling over the original MLP, but also enhances spatial interaction via the SI module. in the first stage, we train a model with binarized activations and full-precision weights. At this stage, our dual branch consists only of MLP branch, and we distill the intermediate features after MLP, which preserve most knowledge from the pre-trained model. The SI module is abandoned in the first stage. In the second stage, both the Stage-1: Update W MLP and W SI by L 𝑐𝑙𝑠 .19:End For 20:Return optimized W MLP and W SI .weights and activations are binarized. We do not distill intermediate activations at this stage and inject our spatial interaction module to enhance spatial interaction. Then the conflicting gradient is calibrated and the SI module is only optimized by gradients from the classifier.To better balance MLP module and SI module, we further introduce a channel-wise self-balancing factor 𝛼 ∈ R 𝑑 in the second stage. It can dynamically adjust the magnitude of different channels and be updated in an end-to-end manner with negligible costs. Experiments in Section 4.3.2 show that these factors vary a lot across different channels and do learn unique features.Overall, the training procedure is illustrated in Algorithm 1. 
experiments 4.1 implementation details: 
 4.1.1 Datasets and architectures. We conduct image classification experiments on two standard benchmarks: TinyImageNet [53] and ImageNet-1k [11] . We also conduct object detection and segmentation tasks on MS-COCO dataset [30] . For classification tasks, data augmentation is adjusted according to DeiT [47] , which is common in ViT. To demonstrate the versatility of our approach, we employ four prevail vision transformer backbones: ViT [14] , DeiT [48] , Swin [34] , and NesT [62] . All the layers in Transformer models are binarized except the first input patch-embedding layer and the last classifier layer, which is a common practice of BNNs. The binary operations (BOPs) and floating-point operations (FLOPs) are counted separately, and the operations (OPs) are calculated by OPs = BOPs/64 + FLOPs [36, 45] . For object detection and segmentation tasks, we evaluate the Swin transformer backbone with two classic object detection frameworks Mask R-CNN [18] and Cascade Mask R-CNN [5] . [42] and Timm [52] library. For classification tasks, we employ AdamW [39] optimizers with weight decay of 2e-5 and train models for 300 epochs using a cosine annealing schedule with 5 epochs of warm-up. We optimize each stage of the decoupled training strategy with 150 epochs. For object detection and segmentation tasks, we follow the official training pipeline in Swin [34] and replace the backbone with our SI-BiViT. We implement it by mmdetection [7] . 4.1.3 Baseline model. We first design a binarized vision transformer by adopting several existing techniques. Specifically, these techniques include:• Binarization method. We use Equation 3 to binarize activation and Equation 4 to binarize weights. • Activation function. We replace the GeLU activation function with RPRelu introduced by [36] . RPRelu can make activation function adaptively learn the parameters for distributional reshaping. • Two-stage training. Following [4] , in the first stage, parameters are randomly initialized and we only binarize weights.In the second stage, parameters are initialized from stage one and both weights and activation are binarized. • Normalization. IR-Net [44] finds that BatchNorm [21] can help preserve information and accelerate training. We replace all LayerNorm [1] with BatchNorm [21] .4.2.1 Results on Tiny-ImageNet. As depicted in Table 2 , we compare our method with state-of-the-art binarization techniques, such as BiViT [19] and BiBERT [43] , across various vision transformer backbones. We provide performance and efficiency cost metrics for classification tasks on Tiny-ImageNet datasets. Overall, our SI-BiViT achieves superior performance compared to the previous state-of-the-art (SOTA) on all ViT backbones, with an average increase of 10.51%. Specifically, on vanilla ViT backbones, SI-BiViT outperforms the BiViT method by a significant margin of 17.34% in ViT-Tiny and 15.54% in ViT-Small without incurring computation overhead. Additionally, on hierarchical ViT backbones such as the Swin Transformer and NesT Transformer, our SI-BiViT demonstrates an improvement of 3.03% and 4.38%, respectively, showcasing its flexibility and effectiveness. Furthermore, our SI-BiViT effectively narrows the performance gap between binarized vision transformers and their full-precision counterparts. Specifically, we reduce the gap to 19.80% in ViT-Small and 20.68% in Swin-Tiny, thereby enhancing the practicality of binarized ViT models.ImageNet. Moreover, we compare our method with state-of-the-art binarization techniques on ImageNet datasets. We exclude BiBERT [43] from the comparison due to its poor performance, as BiViT [19] consistently outperforms it on all tasks in Tiny-ImageNet dataset. We also utilize larger models such as ViT-base and Deit-small on ImageNet compared to Tiny-ImageNet dataset. We achieve an accuracy of 56.43% in NesT-Tiny and 63.90% in Swin-Tiny, surpassing BiViT by 6.54% and 4.29%, respectively. Notably, our SI-BiViT outperforms BiViT by more than 35% and 21% on ViT-B and Deit-S, nearly doubling the accuracy of BiViT. This highlights the adaptability of our SI-BiViT to more complex classification tasks and larger ViT backbones.BiViT by 1.6% in object detection and 1.3% in instance segmentation using Mask R-CNN. Moreover, in the Cascade Mask R-CNN setting, the performance of our SI-BiViT is even more impressive, achieving 30.5% average precision (AP) in object detection and 28.5% AP in instance segmentation. These results further underscore the effectiveness and adaptability of SI-BiViT across various downstream tasks. The dual branch allows us to retain more information from pre-trained models, resulting in performance benefits. As depicted in Table 5 , the incorporation of a dual-branch structure contributes to an improvement of 8.35%. To optimize the dual branch more effectively, particularly the SI Module, we introduce the decoupled training strategy, which further enhances performance by a significant margin of 11.7%.Factor. To enhance spatial interaction in binarized vision transformers, we utilize a channelwise balancing factor to integrate the SI module alongside the MLP module. This channel-wise balancing factor aims to strike a balance between these two branches. As shown in Table 6 , there is a noticeable improvement after applying these factors, with an increase of 1.83% in ViT-Small and 1.43% in Swin-Tiny. Subsequently, we visualize the channel-wise balancing factor in Figure 5 and observe significant variations across different channels.To assess the effectiveness of the token-mixing linear, we compare it with other SI modules, including multi-head self-attention [49] and Cycle-FC [8] . Multihead self-attention (MSA) enhances spatial interaction through the attention mechanism, necessitating the computation of 𝑄, 𝐾, and 𝑉 , which results in increased computational complexity. On the other We randomly pick them from block 1 and block 9. The red mark highlights the maximum and minimum elements.hand, Cycle-FC augments spatial interaction and performs feature extraction within a token. However, the efficiency of Cycle-FC is not as effective as token-mixing linear in terms of spatial interaction.As demonstrated in Table 7 , token-mixing linear outperforms both MSA and Cycle-FC in terms of efficiency and performance. We can further control the Spatial Interaction Capability (SIC) in the Token-mixing Linear by adjusting the parameter 𝑛 ′ in Equation 8. We compare three settings: token-Linear-a, token-Linear-b, and token-Linear-c, which increase spatial interaction by 2 times, 3 times, and 5 times, respectively. As illustrated in Table 8 , token-Linear-c achieves the best performance, with a top-1 accuracy of 58.52%. However, this improvement in performance comes at the expense of much more computational cost. On the other hand, token-Linear-b strikes a balance between performance and efficiency, achieving a top-1 accuracy of 58.45%.Combination with existing Binarized ViT methods. Our SI module is orthogonal to existing Binarized ViT approaches, allowing it to be seamlessly integrated into existing binarization methods to further enhance performance. We apply the SI module to four prevalent binarization methods: BiT [35] , BiViT [19] , BiBERT [43] , and the baseline model. As shown in Table 9 , we observe an improvement of 8.13% in BiViT, with an average increase of 6.59% across all methods. Our SI module significantly enhances the performance of all four methods, underscoring the strong flexibility and effectiveness of SI-BiViT. Experiments on Tiny-ImageNet can be found in Appendix.As our SI-BiViT introduces a new dual branch and modifies the structure of ViT, it is necessary to explore the impact of this modification on the fullprecision model to ensure whether the improvement is introduced by improved full-precision models. We train the full-precision ViTs with the proposed dual branch and the results are shown in Table 10. In full-precision models, there are negligible performance improvements while there is a significant performance improvement in the binarized model, with 5.69% on ViT and 6.14% on Swin. It demonstrates that the huge performance benefit is brought the increased spatial interaction.To measure real-time latency on hardware, we conducted experiments on our binarized model and fullprecision model using an Nvidia-3090 GPU. The full-precision matrix operation was implemented using cuDNN [10] , while the binarized matrix operation was implemented using TC-BNN [25] . As presented in Table 11 , our method achieves a 2.28x reduction in latency compared to its full-precision counterpart. We anticipate that further acceleration could be achieved with the development of hardware specifically designed for binarization acceleration. 
conclusion: 
 In this paper, we reveal that the lack of token-level correlation greatly affects the performance of binarized vision transformers.To measure token-level correlation in different modules, we define spatial interaction capability (SIC) and observe that existing MLP modules only extract features in each token independently, which severely hinder spatial interaction in binarized ViTs. This motivates us to propose SI-BiViT. Our proposed method consists of an SI module, which is placed alongside the MLP module to formulate a dual-branch structure. This structure not only leverages knowledge from pre-trained models but also increases spatial interaction through the SI module. To avoid gradient conflicting problems, we also design a decoupled training strategy to train these two branches more effectively. Importantly, our SI-BiViT is orthogonal to existing Binarized ViT approaches and can be seamlessly integrated to further improve their performance. We conduct extensive experiments that demonstrate the flexibility and effectiveness of SI-BiViT across four classic ViT backbones when applied to several downstream tasks.  
: 
 techniques such as distillation [22, 47] , pruning [41, 63] and quantization [13, 32, 33, 59 ] have been extensively explored. Among them, Binarized Neural Network (BNN) [36, 44, 45, 55, 61] aggressively quantizes weights and activations to 1-bit and utilizes efficient XNOR and popcount bit-wise operations to accelerate inference speed as well as reduce energy consumption significantly. Binarized neural networks can typically achieve up to 32 times memory saving and 58 times speedup [45] .Although model binarization yields efficiency gains, it severely compromises performance due to substantial information loss. Several studies have aimed to bridge the performance gap between binarized models and their full-precision counterparts. Qin et al. [43] and He et al. [19] investigate the optimal threshold of attention score to preserve more information after binarization. Some studies [26, 35, 43] focus on optimization in binarized vision transformers by utilizing distillation and preserving more knowledge from pre-trained models. While existing approaches have demonstrated some effectiveness, a substantial performance gap persists between binarized ViTs and their full-precision counterparts, as evidenced by an average of 30.17% in ViT in Figure 1 .For the first time, we empirically demonstrate that the lack of token-level correlation severely limits the performance of binarized ViTs. We introduce the concept of Spatial Interaction Capability (SIC) to quantify the contribution of different modules to spatial interaction. Our analysis reveals that the MLP module solely influences feature extraction in each token independently, resulting in a spatial interaction capability of zero, which significantly hinders spatial interaction in binarized ViTs. To further investigate the relationship between the model's performance and spatial interaction, we progressively increase spatial interaction capability across four ViT backbones. As illustrated in Figure 2 , the performance exhibits a notable increase with more spatial interaction, indicating a positive correlation between spatial interaction and performance. These findings indicate that existing binarized vision transformers indeed lack spatial interaction and suffer considerable performance degradation.In this paper, we introduce a ViT binarization approach named SI-BiViT, which integrates spatial interaction into the binarization process. Specifically, we propose a dual-branch structure by incorporating a Spatial Interaction Module(SIM) alongside the existing MLP module. This design not only utilizes knowledge from pre-trained ViTs through distillation over the original MLP but also enhances spatial interaction through the SI module. To effectively train these two branches, we devise a two-stage decoupled training strategy. In the first stage, we optimize the MLP branch by distilling from pre-trained models, enabling the MLP to retain more information through binarization. However, due to the significant differences between the SI module and the MLP module, incorporating the SI module at this stage may lead to conflicting gradient problems. Hence, we defer the addition of our SI module to the second stage, where we optimize the model solely by supervision of classification loss without distillation. In the second stage, we introduce our SI module to form the dual-branch structure. To mitigate the increase in computational and memory costs resulting from the dual-branch structure, we utilize a compact token-mixing linear as our SI module. Furthermore, we introduce a channel-wise balancing factor to dynamically adjust the magnitude of the SI module.It is important to note that SI-BiViT is plug-and-play in two aspects. Firstly, it can directly replace the vanilla MLP modules and generalize to various ViT backbones, including Deit [48] , Swin [34] , Nest [62] . Secondly, SI-BiViT is orthogonal to existing Binarized ViTs approaches [19, 43] and can be further combined to improve their performance. Extensive experiments on various ViT backbones and tasks demonstrate the strong flexibility and effectiveness of SI-BiViT. As illustrated in Figure 1 , SI-BiViT increases the classification performance of binarized ViTs by an average of 10.52% in Top-1, and increases detection and segmentation tasks by an average of 1.1% in AP.In summary, our contributions are three-fold:• We are the first to study spatial interaction in binarized vision transformers. Our empirical findings demonstrate that existing binarized VITs lack spatial interaction due to a lack of token-level correlation, which is a critical factor that affects performance. • We propose SI-BiViT to incorporate spatial interaction in the binarized ViTs. Concretely, we introduce a plug-and-play dual branch that not only leverages knowledge from pretrained ViTs by distilling over the original MLP module, but also enhances spatial interaction via the introduced SI module. Additionally, we design a decoupled training strategy to effectively optimize the dual branch.• Extensive experiments on various ViT backbones and downstream tasks demonstrate the strong flexibility and effectiveness of SI-BiViT. 
 
 Appendix:  


Decision: Overall, the reviewers all express opinions of the acceptance of this work. The paper is deemed moderately to definitely suitable for the multimedia/multimodal processing community, with all reviewers confident in their assessments. The final decision tends towards acceptance, influenced by the author's rebuttal and additional clarifications provided during the review process. After collecting all the comments and discussion, AC decided to accept this paper as a Poster. 

Pls further polish the paper according to the rebuttal and comments. Look forward to seeing the final version.

Reviews: review: paper_summary: paper_summary: This paper aims to reduce the performance gap between binarized and full-precision ViT models. This paper first empirically demonstrates the lack of token-level correlation severely limits the performance of binarized ViTs. So this paper proposes SI-BiViT, which integrates spatial interaction into the binarization process. Comprehensive experiments show the effectiveness and flexibility of the proposed model.
strength_weakness: strength_weakness: 1. The authors first employ an empirical study of spatial interaction in binarized ViTs, showing that the lack of token-level correlation results in a zero spatial interaction capability and further limits the binarized ViT performance. 

2. This paper proposes a dual-branch structure by incorporating a Spatial Interaction Module alongside the existing MLP module to integrate spatial interaction into the binarization process. This paper also designs a decoupled training strategy to effectively optimize the dual branch.

3. The proposed SI-BiViT is effective on various ViT backbones and downstream tasks. Also, it is flexible to plug into ViT backbones.
limitations: limitations: 1. The authors introduce the concept of SIC to measure token-level correlation in binarized ViTs. The authors do not clarify whether there are existing measurements for this. If yes, what is the difference between the proposed one and the existing one? If not, what are the motivations for proposing SIC, and why this can correctly measure the correlation?
paper_summary: paper_summary: The authors introduce a method termed SI-BiViT, which integrates Spatial Interaction into the binarization procedure. Specifically, an SI module is incorporated alongside the MLP module, establishing a dual-branch architecture. This design not only harnesses knowledge from pretrained ViTs by distillation through the original MLP but also bolsters spatial interaction via the inclusion of the SI module. Accompanying this, a decoupled training strategy is devised to facilitate more effective training of these dual branches. Crucially, the proposed SI-BiViT approach is orthogonal to existing binarization techniques for ViTs and can be seamlessly integrated.
strength_weakness: strength_weakness: 1. Well-written and easy to understand in most parts.
2. The experiments show promising results across multiple datasets. 
3. For the first time, it proposes and quantifies the lack of "Spatial Interaction Capability" in Binarized Vision Transformers (BiViTs), highlighting its significance for performance.
limitations: limitations: 1. Perhaps some demonstration of visual effects could be shown.
2. In Tables 2 and 3, there appears to be an error in the application of bold formatting. Specifically, numbers that do not surpass the SOTA values have also been incorrectly bolded.
3. The decline in performance when using full precision with SI deserves further explanation and more comprehensive experimentation.
paper_summary: paper_summary: This paper reveals that token-level correlation is crucial for the performance of binarized vision transformers. The authors propose SI-BiViT to incorporate spatial interaction into the binarization process. They conduct extensive experiments on various downstream tasks to demonstrate the flexibility and effectiveness of their proposed method.
strength_weakness: strength_weakness: 1. The paper is well-written and easy to understand.

2. They reveal the importance of token-level correlation in binarized ViTs.

3. The experiments are extensive, and the proposed method achieves promising performance in image classification, object detection, and segmentation tasks.
limitations: limitations: It seems the proposed method shows a more significant improvement in classification performance compared to detection and segmentation. Providing a reasoned explanation for why it is more effective in classification would be beneficial and enhance understanding.