Transformer Doctor: Diagnosing and Treating Vision Transformers

Authors: jiacong  hu
hao  chen
kejia  chen
yang  gao
jingwen  ye
xingen  wang
mingli  song
zunlei  feng
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-07 17:38:35

Abstract: Due to its powerful representational capabilities, Transformers have gradually become the mainstream model in the field of machine vision. However, the vast and complex parameters of Transformers impede researchers from gaining a deep understanding of their internal mechanisms, especially error mechanisms. Existing methods for interpreting Transformers mainly focus on understanding them from the perspectives of the importance of input tokens or internal modules, as well as the formation and meaning of features. In contrast, inspired by research on information integration mechanisms and conjunctive errors in the biological visual system, this paper conducts an in-depth exploration of the internal error mechanisms of Transformers. We first propose an information integration hypothesis for Transformers in the machine vision domain and provide substantial experimental evidence to support this hypothesis. This includes the dynamic integration of information among tokens and the static integration of information within tokens in Transformers, as well as the presence of conjunctive errors therein. Addressing these errors, we further propose heuristic dynamic integration constraint methods and rule-based static integration constraint methods to rectify errors and ultimately improve model performance. The entire methodology framework is termed as Transformer Doctor, designed for diagnosing and treating internal errors within transformers. Through a plethora of quantitative and qualitative experiments, it has been demonstrated that Transformer Doctor can effectively address internal errors in transformers, thereby enhancing model performance.

Main paper: 
 introduction: 
 In the field of machine vision, models based on Transformers [1, 2] have gradually replaced convolutional neural networks as the mainstream approach. Particularly in recent years, various visual architectures improved upon Transformers have emerged incessantly [3] [4] [5] [6] , continuously pushing the performance boundaries of visual tasks. However, the vast and complex parameters of Transformers hinder researchers from gaining a deep understanding of their internal mechanisms [7] , thereby increasing the risks of applying them in sensitive domains [8] . This has spurred a considerable amount of work aimed at investigating the interpretability of Transformers to enhance their transparency [9] [10] [11] .Existing research on the interpretability of Transformers in machine vision primarily focuses on aspects such as the importance of input tokens [7, 9, 10, [12] [13] [14] , the significance of internal modules [15] , the the evolution and formation of features [16, 17] , and the meanings of intermediate represnetations [18] [19] [20] . While these studies have somewhat improved the transparency of Transformers, the internal decision-making processes, such as mechanisms leading to errors, still warrant more systematic investigation. This is crucial for enhancing the transparency of Transformers and further improving their performance.In fact, unlike in machine vision, theoretical studies on error mechanisms in biological vision have become quite mature [21] [22] [23] [24] [25] [26] . Specifically, in the perceptual process of the biological visual system, visual information such as the spatial position, shape, size, color, and texture of objects is processed and refined in the primary visual cortex [27] [28] [29] . Subsequently, these different visual cues are integrated at higher stages for final recognition [30, 31] . Numerous studies have demonstrated that errors in object recognition may arise not only from failures in feature extraction but also from incorrect integration of correctly extracted features at higher stages [21] [22] [23] [24] [25] . Errors resulting from the improper integration of features are termed conjunction errors [26] . Furthermore, some research indicates that providing effective stimuli or cues during the integration process can enhance the correctness of information integration [22] . Inspired by this, we are interested in investigating whether Transformers exhibit similar mechanisms of feature integration and conjunction errors as those observed in biological vision during recognition. If such errors exist, can they be corrected akin to the mechanisms observed in biological vision?To address these questions, we first proposed the information integration hypothesis, inspired by the biological visual. This hypothesis posits that Transformers continuously process and refines various mixed information at the primary stage and integrate them at the higher stage. When incorrect information is integrated, i.e., conjunction error occur, it results in erroneous recognition outcomes. To validate this hypothesis, we conducted extensive experimental analyses of the computational process of Transformers and found empirical evidence supporting the hypothesis. Specifically, we discovered dynamic information integration among tokens in Transformer's Multi-Head Self-Attention (MHSA) component and static information integration within tokens in the Feed-Forward Network (FFN) component, along with the presence of conjunction errors. Furthermore, we elucidated the reasons behind both dynamic and static integration. Building upon this, we proposed heuristic dynamic integration constraints for inter-token information integration and rule-based static integration constraints for intra-token information integration, enabling the rectification of conjunction errors in Transformers. We coined the entire approach as "Transformer Doctor", where the process of identifying errors based on the information integration hypothesis is referred to as diagnosing the Transformer, and the process of applying the hypothesis to rectify errors is referred to as treating the Transformer. Finally, we conducted extensive quantitative and qualitative experiments on mainstream Vision Transformer architectures, thoroughly validating the effectiveness and applicability of Transformer Doctor.The contributions of this paper can be summarized as follows:• We propose Transformer Doctor, the first framework for diagnosing and treating Vision Transformers. This framework validates and utilizes the proposed Information Integration Hypothesis, which posits that Transformers process and encode various mixed information at primary stages and integrate it at higher stages. When information is not correctly integrated, i.e., conjunction error occur, it results in prediction failures.• In diagnosing Transformers, we identify the mechanisms of inter-token information dynamic integration and intra-token information static integration within Transformers, along with the occurrence of conjunctive errors. This provides a novel perspective for understanding the internal mechanisms of Vision Transformers.• In treating Transformers, we propose heuristic dynamic constraints for inter-token information integration and rule-driven static constraints for intra-token information integration. These constraints offer an interpretable solution for optimizing Vision Transformers without introducing additional parameters or computational overhead during inference.• Extensive qualitative and quantitative experiments are conducted on mainstream Vision Transformers, validating the effectiveness and applicability of the Transformer Doctor.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Please refer to Section 1.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related works: 
 In methods aimed at interpreting or understanding Transformers, whether in the field of machine vision or natural language processing, the primary approaches focus on the importance of input tokens [7, 12, 13, 9, 14, 10] , the significance of internal modules [15] , the the evolution and formation of features [16, 17] , and the meanings of intermediate features [18] [19] [20] to understand the internal mechanisms of Transformers. Specific methodologies can be categorized as feature-based, attentionbased, gradient-based, propagation-based, perturbation-based, projection-based, or a combination of these approaches. For instance, in feature-based methods, the primary focus is on analyzing or statistically evaluating the intermediate features within Transformers [16] to understand the internal representation structure and feature distribution [17] . Attention-based methods mainly utilize the raw attention weights [7, 12, 13] or linear combinations of multi-layer attention weights [9] to compute the relative importance of input tokens. Gradient-based methods focus on computing gradients of attention weights [32, 33] , intermediate features [11, 34] , or inputs [14] to understand the differences in token importance. Propagation-based methods employ techniques like Layerwise Relevance Propagation (LRP) [35, 36] for attribution analysis of input tokens [10, [37] [38] [39] or investigate the importance of heads in Transformers [15] . Perturbation-based methods involve perturbing inputs or features and measuring the impact on model performance [40] [41] [42] [43] or Shapley value [14, 44] . Projection-based methods, such as linear probes [17, 45, 46] , project intermediate representations into human-understandable spaces to comprehend the mechanism and significance of feature transformations in Transformers [18] [19] [20] . In contrast to the aforementioned research, this paper is inspired by studies on biological visual error mechanisms, aiming to explore whether Transformers exhibit similar mechanisms of information integration and connection errors as in biological vision, and how to rectify errors within Transformers.In methods for improving Transformers, most approaches involve modifying the model architecture by introducing learnable parameters [3] [4] [5] [6] 47] or enhancing data and features [48] [49] [50] [51] [52] . These improvement methods are mostly non-interpretable and are pre-defined before training, rather than targeting further enhancement of model performance from the perspective of diagnosing and treating internal error mechanisms based on existing models. Additionally, there are methods for improving models that do not require pre-definition before training but focus on non-Transformer or non-vision tasks, such as debugging and analyzing models in traditional machine learning [53] [54] [55] [56] , optimizing deep models [57, 58] , and editing facts in natural language processing [59] [60] [61] [62] . However, due to significant differences in architecture and tasks, these methods are not suitable for analyzing and correcting error mechanisms in Transformers to improve model performance.In summary, this paper is the first work to investigate whether Transformers exhibit information integration and connection error mechanisms similar to biological vision, and the first work to explore how to further enhance model performance by diagnosing and treating internal errors in existing models.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Please refer to Section 7.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
information integration hypothesis: 
 In this section, we firstly propose the Information Integration Hypothesis for Transformers. Subsequently, we review the MHSA and FFN modules within the Transformer architecture, followed by an analysis of potential locations where the Information Integration Hypothesis may apply.Information Integration Hypothesis: Similar to biological vision, in machine vision, the Transformer continually processes and refines various mixed information in the primary stage, and integrates it in the advanced stage. When erroneous information is integrated, i.e., conjunction errors occur, it leads to incorrect predictions.When utilized for visual recognition tasks, a Transformer typically comprises L blocks, each consisting of an MHSA module and an FFN module. The input to the block can be represented as X ∈ R N ×D , where N and D denote the number and dimensionality of tokens, respectively. The output X ′ ∈ R N ×D of the MHSA module within this block can be computed as follows:! ∈ ℝ !×# $ ∈ ℝ #×$ % ∈ ℝ !×! & ∈ ℝ !×$ FFN MHSA = × diagnosing → ⊙ → ! !"# $ ! %"# $ ! " ! # " % &′ ℒ !"! treating ! " ! " ! ! ! ↑ 1 ! 1 ! = × diagnosing ! $ Y treating $ " $ " ! ! $ ↑ information static integration % $ rule-based heuristic + # ⊙ → ! !,'"#,# $,( ℒ !$! & ! $ information dynamic integrationEQUATIONwhereQ = XW (Q) , K = XW (K) , V = XW (V ) . W (Q) ∈ R D×D , W (K) ∈ R D×D, and W (V ) ∈ R D×D represent the parameter matrices for query, key, and value, respectively. Upon decomposing Eqn.(1), it can be observed that a certain tokenX ′ i ∈ R D in X ′ ∈ R N ×D (e.g., the i-th token) is an integration weighted sum of all tokens in V ∈ R N ×D :EQUATIONwhere a ∈ R N ×N represents the attention weights, which can also be referred to as integration weights within MHSA. It can be observed that each token X ′ i possesses its own integration weight a i ∈ R N . For simplicity, we have omitted the skip connections and bias terms in the above equation, and have considered only single-headed self-attention. Therefore, the concatenation and projection of multiple heads in MHSA are also omitted.Suppose the input to the FFN is X ′ ∈ R N ×D , then the output Y ∈ R N ×D of the FFN can be represented as:EQUATIONwhere W (1) ∈ R D×M and W (2) ∈ R M ×D represent the parameter matrices for the first and second linear layers, respectively. Similarly, for simplicity, we have omitted the skip connections and bias terms in the above equation. Comparing Eqn. (1) and Eqn.(3), it can be observed that apart from the activation function gelu in FFN and the softmax function in MHSA, as well as the constant 1 √ D , FFN also employs the query-key-value mechanism similar to MHSA. Motivated by this, we further decompose Eqn. (3) into a form similar to Eqn. (2) , revealing that a certain token Y i ∈ R D in Y ∈ R N ×D (e.g., the i-th token) is an integration weighted sum of all dimensions in W (2) ∈ R M ×D :EQUATIONwhere z ∈ R N ×M can be referred to as integration weights within the FFN, and it can be observed that each token Y i also possesses its own integration weights z i ∈ R M . In summary, both matrices a and z represent potential locations where the assumption of information integration may hold. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: Please refer to Section A. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
information integration hypothesis based transformer diagnosis: 
 In this section, we delve into the potential existence of information integration hypothesis mentioned in Section 3 within the MHSA and FFN. Through extensive experimental analysis, we have gathered empirical evidence supporting the hypothesis, namely dynamic integration of information among tokens and static integration of information within tokens.To explore the potential existence of the information integration hypothesis within the MHSA, we analyzed the integration weights a in Eqn. (2) . Fig. 2 (a) illustrates the magnitudes of integration weights a in blocks of different depths within the model. It can be observed that for high-confidence samples, the integration weights a in shallower blocks exhibit a diagonal pattern. However, as the depth of the block increases, the integration weights a display a vertical pattern, consistent with observations from prior studies [63] [64] [65] [66] . Additionally, we observed that for different high-confidence samples, the positions of the vertical lines in the integration weights a within the deeper blocks vary, as shown in Fig. 8 in the Appendix. This indicates that in the initial stages, MHSA primarily mixes and processes information between adjacent tokens. In the later stages, MHSA dynamically and selectively integrates specific information among tokens.Continuing, we extracted an arbitrary row a i from the integration weights a within the deeper blocks and removed the [CLS] token. After reshaping and resizing, we overlaid it onto the original image to generate a heatmap, as shown in Fig. 2(b ). From the heatmap, it is evident that the positions of the vertical lines mainly concentrate on the foreground of the input image. This suggests that in the advanced stages of the model, MHSA primarily integrates specific information among tokens containing foreground elements. However, for low-confidence samples, as depicted in Fig. 1(c ) and (d), the deeper layers of the MHSA erroneously integrate information corresponding to the background tokens. Additional quantitative and qualitative analyses of the integration weights a are presented in Appendices C and B. In summary, we have identified the first evidence of the existence of the information integration hypothesis in Transformer models:Evidence 1: In the initial stages of the Transformer, MHSA primarily mixes and processes adjacent patch information among tokens. However, in the advanced stages of the Transformer, MHSA dynamically and selectively integrates specific patch information among tokens. When integrating incorrect information among tokens, termed conjunction errors, it leads to model mispredictions. We refer to this integration as inter-token information dynamic integration. The term "dynamic" arises from the fact that the V to be integrated, as specified in equation (2), varies with each sample.To explore the potential existence of the information integration hypothesis within the FFN, we conducted visual analysis of the integration weights z in shallow and deep blocks, as depicted in Fig. 3 . Fig. 3 (b) illustrates the patterns of integration weights z in shallow and deep blocks for high-confidence samples from different classes. It can be observed that the patterns of integration weights differ between shallow and deep blocks for samples from different classes. However, in Fig. 3 (a), for high-confidence samples from the same category, the patterns of integration weights z in shallow blocks differ, while the patterns in deep blocks show less variability and remain relatively consistent. This indicates that in the initial stages of the model, FFN mixes and processes various category information within tokens. However, in the advanced stages, FFN statically selectively integrates specific category information within tokens.Furthermore, as depicted in Fig. 3(d) , even for samples from the same class, the patterns of integration weights z in deep blocks differ for low-confidence images. This suggests that when the deep FFN fails to correctly integrate specific category information within tokens, it adversely affects the model's predictions. Additional quantitative and qualitative analyses of the integration weights z are presented in Appendices E and D. In summary, we have identified the second evidence of the information integration hypothesis in Transformers:Evidence 2: In the initial stages of the Transformer, the FFN primarily mixes and processes various low-level information within tokens. However, in the advanced stages of the Transformer, the FFN statically and selectively integrates specific category information within tokens. When integrating incorrect information within tokens, termed conjunction errors, it leads to model mispredictions. The term "static" arises from the fact that the W (2) to be integrated, as specified in Eqn. (4), is fixed relative to each sample, and for samples of the same class, the integration weights z are also fixed. 
information integration hypothesis based transformer treatments: 
 In this section, we propose heuristic dynamic integration constraints and rule-based static integration constraints to correct conjunction errors in information integration, aiming to enhance model performance.In order to alleviate conjunctive errors in dynamic integration of information among tokens, we heuristically constrain the integration weights a by highlighting the foreground of images with low confidence scores. Furthermore, unlike the single-head integration weights a in Eqn. (2), we have improved the calculation of integration weights for multi-head scenarios using gradients.Specifically, for a Transformer classification model with K(K ≥ 2) classes, let k ∈ {1, 2, ..., K} represent the true label for input, and p ∈ R K denote the predicted probabilities of all classes by the Transformer. We incorporate gradients related to the true class to discern the importance of each head out of H heads in self-attention, thereby obtaining the integration weights â ∈ R N ×N in the multi-head scenario:EQUATIONwhere max(.) denotes setting negative values in the derivative to zero, thereby considering only the positive impact on the predicted probability of the true class. The introduction of gradients in Eqn. ( 5) not only helps discern which head is important but also establishes a connection between integration weights and specific classes, thereby making the weights reflecting the integration of information among tokens more accurate. Details of the comparison experiments on the introduction of gradients can be found in Section 6.2. Next, for foreground annotation t ∈ R N with low confidence, we constrain the integration of background information using the loss function L IDI :EQUATIONwhere t ∈ R N is a binary annotation, with 1 and 0 representing the presence and absence of foreground within the token, respectively.To correct conjunctive errors in static integration of information within tokens, we first establish integration rules within tokens when the model prediction is correct, and then constrain the integration weights z based on these rules. Additionally, we have also improved the integration weights z using gradients, establishing a connection with the true class k, resulting in the new integration weights ẑ ∈ R N ×M as follows:EQUATIONNext, we select S high-confidence samples for each class to calculate the average integration weights z ∈ R N ×M , and then establish binary integration rules r ∈ R N ×M for each class using a threshold τ :EQUATION)The values in r are 1 or 0, indicating the integration and non-integration of information in the corresponding dimension, respectively. Finally, based on the integration rules r for a certain class, we enforce that erroneous information within tokens is not integrated using the loss function L ISI :EQUATIONDuring actual enforcement, each training sample needs to be constrained using the integration rules corresponding to its true class.The loss functions L IDI and L ISI can be individually combined with the original loss function or used jointly:EQUATIONwhere α and β are used to balance the magnitudes of the loss functions. In our practical experiments, we found that using the loss functions L IDI and L ISI sequentially yielded the most effective results. It is important to note that the therapy model only rectifies conjunctive errors in the Transformer without altering its architecture or operational procedure. Thus, during inference, it does not incur any additional computational overhead. 
experimental settings: 
 Datasets and Transformer Architectures. To validate the effectiveness of Transformer Doctor, we conducted experiments on five mainstream datasets: CIFAR-10 [67], CIFAR-100 [67] , ImageNet-10 [68], ImageNet-50 [69] , and ImageNet-1k [68] . Furthermore, we performed experiments on various Transformer architectures used for visual classification tasks, including DeiT [48] , CaiT [3] , TNT [4] , PVT [5] , Eva [6] , and BeiT [49] , in addition to ViT [2] . It is important to note that Transformer Doctor diagnoses and treats already trained Transformer models. More experimental settings and results can be found in Appendix.Parameter Settings.During all training stage, each dataset was trained for 300 epochs using the AdamW [70] optimizer, with an initial learning rate of 0.01. The learning rate decayed according to a cosine annealing schedule, with T_max set to 300 epochs. Additionally, α and β were set to default values of 10 and 100, respectively, to balance each loss function. The default value of τ was 0.15, and the constrained loss function was applied by default to the last block.Baseline Models. In addition to using the original pre-trained Transformer models as baselines, as shown in Table 1 , we also established a blank control group, which involves no method introduction but continues training for the same epochs, as presented in Table 3 . Furthermore, we compared different ways of integrating multi-head integration weights in Eqn. 5 and not introducing gradients in Eqn. 7 as simple baselines against the proposed final method, as illustrated in Table 2 . Due to differences in computational resources, certain training configurations, such as batch size, differ from those in the original work, leading to slight variations in the baseline. However, all comparative experiments were conducted under fair conditions. For detailed experimental results, please refer to Table 3 , and for further setup information, see Appendix F.The Performance of Transformer Doctor on SOTA Models. We evaluated the performance of Transformer Doctor across five major datasets and seven mainstream architectures. As shown in Table 1 , it is evident that Transformer Doctor effectively enhances model performance on both small and large-scale datasets. Specifically, after treatment, the accuracy of CaiT improved by 1.36% on CIFAR-10 and 1.84% on ImageNet-50. The BeiT model, after treatment, saw an accuracy increase of 1.45% on CIFAR-100 and 2.39% on ImageNet-50. Additionally, we observed that generally, the larger the dataset, the more significant the performance improvement by Transformer Doctor. For instance, the accuracy improvement of ViT on ImageNet-1K was 2.09% higher than on ImageNet-10.The reason for this could be that, similar to biological vision, Transformer misrecognition can be due to both conjunction errors and feature extraction failures. Large-scale datasets effectively help the model learn a vast array of features, thus better extracting the input image's features. On this basis, treating conjunction errors can maximize model performance. However, for small-scale datasets, the dominance of not extracting effective features may outweigh conjunction errors. Therefore, focusing solely on treating conjunction errors is less effective on small-scale datasets compared to large-scale ones. More detailed experimental results can be found in Table 3 in the appendix.The Performance of Transformer Doctor with Various Computational Forms. We compared the performance of Transformer Doctor under different computational forms represented by Eqn. (5) and Eqn. (7) , as shown in Table2. It can be observed that Transformer Doctor performs better under the computational forms of Eqn. (5) and Eqn. (7) . Specifically, under dynamic integration constraints, 'mean' denotes directly averaging integration weights across all heads, 'min' and 'max' respectively represent taking the minimum and maximum integration weights within each head. Each row corresponds to ViT-Tiny and PVT-Tiny architectures, with ImageNet-10 dataset used for evaluation. (1) (2) (3) (4) (1) (2) (3) (4)Figure 4 : Comparison of inter-token integration weights before and after introducing Transformer Doctor. ( 1) and ( 3) depict integration weights before and after treatment, respectively. ( 2) and ( 4) show the corresponding heat map effects of integration weights overlaid onto the original image before and after treatment.(1) ((3) (4)Figure 5 : Comparison of intra-token integration weights before and after introducing Transformer Doctor. ( 1) and ( 2) represent the intra-token integration rules for correct predictions. ( 3) and ( 4) depict the intra-token integration weights before and after treatment, respectively.taking the minimum, maximum, or averaging fusion of integration weights for each head does not significantly improve model performance. For instance, when taking the maximum value, the accuracy of PVT-Tiny decreases by 0.08% compared to the base. However, under the computational form of Eqn. ( 5), the accuracy of PVT-Tiny improves by 2%. Under static integration constraints, the accuracy of the model under the original integration weight calculation form is 79.00%, while the accuracy under the computational form of Eqn. ( 7) is 80.40%.Comparison of the Intra-token Integration Weights a.We also compared the static integration weights within tokens before and after treatment, as shown in Fig. 5 . From panels (1) and (3), it is apparent that before treatment, the static integration weights for misclassified samples do not adhere to the correct prediction rules. However, panels (2) and (4) in Fig. 5 demonstrate that after treatment, the static integration weights align with the correct prediction rules, integrating the correct class information, which results in accurate predictions.Comparison of the Inter-token Integration Weights z. In Fig. 5 , we compared the static integration weights within tokens before and after treatment. As shown in Fig. 5 (1), before treatment, the static integration weights of misclassified samples do not adhere to the correct integration rules for prediction, i.e., integration of incorrect category information within tokens. However, as depicted in Fig. 5 (2), after treatment, the static integration weights conform to the correct integration rules for prediction, integrating the correct category information within tokens. 
conclusion: 
 This paper introduces the first framework, Transformer Doctor, designed for diagnosing and treating internal errors within Transformers simultaneously. Specifically, distinct from existing post-hoc interpretability methods for Transformers, this work draws inspiration from information integration and conjunctive errors in the biological visual system, proposing and validating the hypothesis of information integration for Transformers. Furthermore, addressing conjunctive errors within information integration, this paper presents corresponding error treatment methods. Extensive qualitative and quantitative analyses conducted on mainstream datasets and Transformer architectures demonstrate the effectiveness and applicability of Transformer Doctor.Limitations and Future Work. It is undeniable that we have only validated Transformer Doctor on mainstream visual recognition tasks, leaving more complex machine vision tasks for further exploration. Furthermore, investigating whether the information integration hypothesis holds true in the field of natural language processing or multimodal domains is also a worthwhile research endeavor. Additionally, exploring more error mechanisms and developing a more automated and intelligent Transformer Doctor framework are the focal points of our future work. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: [NA] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All assets involved in the research have been referenced accordingly.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: In the appendix, we offer additional evidence concerning the information integration mechanism and conjunction errors in the Transformer. Furthermore, more quantitative and qualitative analysis experiments on Transformer Doctor are provided. Additionally, the algorithm code for the Transformer Doctor is included in the uploaded source_codes.zip file. This paper contributes to the research field of understanding and optimizing Transformers. Specifically, drawing inspiration from biological vision, the exploration of error mechanisms in Transformers for machine vision tasks provides a novel perspective for understanding the internal workings of Transformers. Additionally, the interpretable treatment solutions proposed based on errors in Transformers offer a new pathway for optimizing Transformer models.As illustrated in Fig. 7 , we provide further visual comparisons of integration weights a with high and low confidences to demonstrate the dynamic information integration mechanism among tokens in the MHSA. The observed phenomena align with our conclusions in the main text. Specifically, in the early stages of the Transformer, as shown in Fig. 7 (a), the MHSA predominantly blends and processes information among adjacent tokens, whereas in the advanced stages, the MHSA dynamically and selectively integrates specific information among tokens. Simultaneously, for lowconfidence samples, as depicted in Fig. 7 (b), the advanced-stage MHSA erroneously integrates token information corresponding to the background.In addition, we provide more visualizations of integration weights a for high-confidence images in the final block, as shown in Fig. 8 . It can be observed that the integration weights a vary with different input samples. This indicates that in the advanced stage, the integration weights a dynamically integrate specific information for the final prediction.To verify whether incorrect predictions are indeed caused by the Transformer integrating erroneous information among tokens in the advanced stage, we selected low-confidence samples from the test set and evaluated the model's performance after replacing the original integration weights with ideal ones, as shown in Fig. 6 (a). It can be observed that forcibly integrating the correct information among(1) (2) (1) (2) (1) (2) (1) (2)Figure 8 : Visualization comparison of integration weights a in the last block of MHSA. ( 1) and ( 2) depict the visualization of a and the visualization of reshaped and resized rows of a superimposed onto the original image.tokens in the model's early stages does not improve performance. This is because the model in its early stages is merely blending and processing various information, rather than integrating specific information. However, when correct information is forcibly integrated among tokens in the advanced stages, the model's performance improves significantly.As illustrated in Fig. 9 , we provide further visual comparisons of integration weights z with high and low confidences to demonstrate the static information integration mechanism within tokens in the FFN. The observed phenomena align with our conclusions in the main text. Specifically, in the early stages of the Transformer, as shown in Fig. 9 (a), the FFN mixes and processes various categories of information within tokens. However, in the advanced stages, the FFN statically and selectively integrates specific category information within tokens. Additionally, for low-confidence samples, as depicted in Fig. 9 (b), the FFN in the advanced stages integrates incorrect category information within tokens.To verify whether incorrect predictions are indeed caused by the FFN integrating erroneous information within tokens in the advanced stage of the Transformer, we selected low-confidence samples from the test set and evaluated the model's performance after replacing the original integration weights with ideal ones, as shown in Fig. 6(b ). The conclusions are similar to those observed in the dynamic information integration among tokens: forcibly integrating correct information within tokens in the model's early stages does not improve performance. This is because, in the early stages, the model is merely mixing and processing various types of information rather than integrating specific information. However, when correct information is forcibly integrated within tokens in the advanced stages, the model's performance improves significantly. As shown in Fig. 10 , we compare additional dynamic integration weights a before and after treatment. The results align with our observations in the main text. Specifically, from Fig. 4 (1, 2), it can be seen that before treatment, the dynamic integration weights of misclassified samples exhibit irregular '+IDI', '+ISI', and '+IDI, ISI' refer to models with individually introduced dynamic integration constraint, individually introduced static integration constraint, and jointly introduced dynamic and static integration constraints, respectively. As depicted in Fig. 11 , we present additional comparisons of integration weights z before and after treatment. The results are consistent with our observations in the main text. Specifically, from Fig. 11 (1, 2), it can be observed that before treatment, the static integration weights of misclassified samples do not adhere to the correct integration rules, indicating the integration of incorrect category information within tokens. However, in Fig. 11(3, 4 ), it can be seen that after treatment, the static integration weights adhere to the correct integration rules, integrating fewer incorrect information, thus improving the model performance.We also visualized the attribution maps of model predictions using Rollout feature attribution techniques, as shown in Fig. 12 . From the figure, it can be observed that without introducing any constraints, the model tends to focus on unnecessary background information. However, when dynamic integration constraints or static integration constraints are introduced, the model starts to focus on foreground information relevant to the categories. When both dynamic and static integration constraints are simultaneously introduced, the key information relied upon by the model for prediction aligns more closely with human understanding, i.e., it focuses more on the object itself.K Ablation Study on Threshold τ in Eqn. (8) We conducted ablation experiments on the threshold τ that determines the intra-token integration rule in equation ( 8), as shown in Table 4 . Using the ViT model as an example, we observed that when the threshold τ increases from 0.0 to 0.2, the model accuracy gradually increases from 78.8% to 80.4%, exceeding the baseline. This improvement occurs because, with a very low threshold, such as 0.0, the integration rule includes redundant intra-token information weights, meaning that unnecessary intra-token information is integrated, which does not significantly enhance model performance.Conversely, when the threshold τ increases from 0.3 to 1.0, the model accuracy gradually decreases from 80.4% back to 78.0%, falling below the baseline (note that the table presents the highest accuracy on the test set, not the final accuracy after model training). This decline is due to the overly high threshold, such as 0.8, causing the integration rule to miss certain necessary intra-token information weights, meaning that essential intra-token information is not integrated, resulting in reduced model performance.L Ablation Study on Block Depth The figure demonstrates that the model's performance generally improves with increasing block depth when applying inter-token dynamic integration constraints. When these constraints are applied before the 7th block, the model's performance does not show significant improvement and can even fall below the baseline. However, after the 7th block, the model's performance noticeably exceeds the baseline. This observation reaffirms that in the early stages, the Transformer mixes and processes adjacent token information, while in the advanced stages, it dynamically and selectively integrates specific information. Thus, inter-token dynamic integration constraints are most effective at deeper blocks and can help enhance model performance.Similarly, the model accuracy surpasses the baseline only when the intra-token static integration constraints are applied after the 11th block. This is because the Transformer selectively and statically integrates specific intra-token information only at the advanced stages. Furthermore, the introduction of combined dynamic and static constraints results in higher model accuracy at deeper blocks compared to both the baseline and the introduction of individual integration constraints.NeurIPS Paper Checklist 


Decision: The reviewers unanimously support the paper. A couple of reviewers are concerns with DeiT evaluation settings and fairness in comparison. This concern still remain partly unaddressed.

Overall, the paper looks valid and it has its strength, despite the remaining concern.

Reviews: review: paper_summary: paper_summary: This paper presents Transformer Doctor, which diagnoses the issues with the Transformer attention mechanism and resolves them via several information integration hypotheses. The primary motive of the paper is to identify the source of incorrect information aggregation, which leads to erroneous predictions, and the inspiration comes from biological groundings. The paper proposed Inter-token Information Static Integration, Intra-token Information Static Integration, Heuristic Information Dynamic Integration Therapy, Rule-based Information Static Integration Therapy, Joint Therapy of Dynamic and Static Integration, and extensive evaluation. The paper improves the performance of the Transformer mechanism significantly.
strength_weakness: strength_weakness: 1. It appears from sec6.1 that the proposed approach requires a pre-trained model. Please correct me if not because I could not find explicit training settings in the paper. If it is trained for longer epochs, it increases the training time. 
2. Since the model is pre-trained, even though attention maps are improving, it is unclear whether the gains came from further fine-tuning or information integration. In other words, how much the integration mechanism contributed to the gains?
3. In Figure 4, the regions in red always lie on the object, i.e. without and with transformer doctor applied. How can it be inferred that the attended region after using the doctor mechanism is the sole reason for improved performance? Similarly Fig12.
4. Selecting the gradient based on the actual class label is possible with supervised learning. Hence, how can this method be applied to DINO-like self-supervised methods, given that DINO also uses a Transformer?

Note: I am open to adjusting ratings if my concerns are resolved, especially 1,2,3.
questions: questions: See weakness.
limitations: limitations: Overall, the main limitations are the use of pre-trained models as a baseline, unclear hypothesis verification on the improved attention maps, and usability across self-supervised models.
paper_summary: paper_summary: This study introduced a framework, namely Transformer Doctor, to reduce internal errors, e.g. conjunction errors, in a general vision transformer model. Building upon the information integration hypothesis, the proposed method performs several constraints, including heuristic dynamic constraints and rule-driven static constraints, to enhance information integration at higher layers. The experiential results are conducted on five classification datasets using seven small-scale vision transformer models.
strength_weakness: strength_weakness: -- This is a pure empirical paper with no theoretical results. However, the experimental results are insufficient to fully convince the reviewer.
* The vanilla transformer results reported in Table 1 do not align with their original paper. e.g. the DeiT-Tiny is reported at 72.2% on ImageNet, which is even higher than the improved results (66.8% $\to$ 70.6%).
* This study is only evaluated on small-scale vision transformers, with no results presented for large-scale versions. 
* The evaluation is only conducted on 10 samples per class (Appendix F). The sample size seems insufficient.
* Table 1 can be further improved by presenting results with both the vanilla and blank training.

-- The reviewer is unclear about the potential mechanism/logic/reason of the proposed method on the following questions
* why the proposed method can reduce the conjunction error
* To what extent (fully/partially) can the proposed reduce the error
* Besides the information integration hypothesis, do you rely on other assumptions?
* Is the information integration hypothesis valid in vision transformers, especially for large-scale versions?

The above questions might be potentially solved by adding theoretical analysis or more detailed explanations. 

-- This paper has many typos, even in the result presentation (e.g. Table 1 BeiT/Imagenet-1K cell).
questions: questions: See above weakness.
limitations: limitations: As I can see, this paper has no potential negative societal impact.
paper_summary: paper_summary: Inspired by the information integration mechanisms and conjunction errors in the biological visual system, this paper investigates the error mechanisms within Transformers. Through a comprehensive analysis and experimental validation of the computational processes of the two core modules of Transformers, MHSA and FFN, the authors introduce the Transformer Doctor framework. The principal components of this framework are as follows:
1. Diagnosis: The paper identifies dynamic information integration among tokens within MHSA and static information integration within tokens in FFN, as well as conjunction errors occurring during the integration process.
2. Treatment: To mitigate the conjunction errors identified in MHSA and FFN, the paper proposes heuristic dynamic information integration constraints for MHSA and rule-based static information integration constraints for FFN.

To validate the efficacy of Transformer Doctor, the authors performed extensive quantitative and qualitative analyses on various Vision Transformer architectures. The findings indicate that Transformer Doctor effectively reduces conjunction errors in Transformers and enhances overall performance.
strength_weakness: strength_weakness: 1. The visualization from vertical lines to diagonals in the attention matrices presented in Fig. 2(a,c) lacks sufficient clarity. This visualization is essential for validating the authors' information integration hypothesis in MHSA. It is recommended that the authors refer to similar visualization results in studies such as [1].
2. In Table 1, the proposed method does not demonstrate significant improvements on certain datasets, particularly on the CIFAR-10 dataset. Regarding this phenomenon, the authors need to provide a detailed analysis.
3. In the treatment of dynamic information integration in MHSA (Eqn. (6)), it is noted that not all datasets contain annotated foregrounds. The necessity for manual foreground annotation presents a significant challenge to the practical implementation of this method.
4. minor issues: \
   a.  This paper lacks a detailed introduction to Transformers. The authors should provide a conceptual overview and offer clearer explanations of certain variables in the equations. For instance, the meanings of $(a_{ij})$ in Eqn. (2) and $(z_{im})$ in Eqn. (4) should be explicitly stated.

   b. If Eqn. (4) is correct, the symbol labeled as $(X)$ in Fig. 1 should be $(Y)$.
5. In Eqn. (8), the authors constrain the static information integration in FFN by specifying aggregation rules. However, it is worth considering whether the number of samples used to establish these rules could also impact the final results.
6. Lack of explanation why the performance improvements under other computational forms are not as significant in Table 2.


[1] Trockman, Asher, and J. Zico Kolter. "Mimetic initialization of self-attention layers." International Conference on Machine Learning. PMLR, 2023.
questions: questions: See weakness
limitations: limitations: The paper has already discussed its limitations and potential impacts.
paper_summary: paper_summary: This paper proposes a vision transformer diagnosing and treating framework, namely Transformer Doctor, to reveal the problems that bring about negative impact on the network performance and fix them. The paper firstly proposes the information integrating hypothesis, which argues that transformers do information refining and integrating at the lower and higher layers respectively. Based on this hypothesis, the inter-token information dynamic integration and intra-token information static integration mechaisms are designed to show the inner mechanisms of ViTs and help to treat them.
strength_weakness: strength_weakness: 1. The evidence of the correctness of the information integration hypothesis is not strong enough, it is better to consider to prove the hypothesis in mathematic way
questions: questions: N.A.
limitations: limitations: The authors consider the limitations of the method in the part of Conclusion.

Comments: title: Response to Authors
Rebuttal by Authors [Q5-Q9]
comment: Thank you for recognizing the value of our work. We also greatly appreciate your thorough review and insightful comments.
Thank you for your valuable review and suggestions. We are pleased to hear that your concerns have been addressed.
Thank you for your response, I would like to keep my original score unchange.
Thank you for your reply, my concerns have been resolved. So I vote for acceptance.
Thank you for your positive feedback and recognition of our work. We are pleased that our response addressed your concerns. We sincerely appreciate your valuable comments, which have helped improve this work.
The authors have addressed some of my concerns about experimental results. I have increased my rating.
Thank you very much for your thoughtful review and the positive feedback. We sincerely appreciate your acknowledgment of our efforts to address your concerns and your recognition of the improvements made in the paper.

We understand your concern about the reported results for DieT, as highlighted by reviewer BM6N. We agree that ensuring fair comparisons is critical. To address this, we will include a detailed training settings section in the supplementary materials and provide a footnote or caption in the relevant tables to explain the accuracy differences for DieT. We will also add the results from the original paper for completeness.

Once again, we sincerely thank you for your valuable feedback and for recommending our work for acceptance. Your insights have been instrumental in improving the quality of our paper.
I thank the authors for the detailed responses to my comments.

Overall, I am convinced about my question on the source of the accuracy gains i.e. higher training epochs or information integration.

I have read comments from other reviewers and responses to them and they are in line with my understanding of the paper.

I have a particular concern about the reported results for DieT commented on by reviewer BM6N. I agree with the reviewer that resource constraints lead to smaller batch sizes for training and thus unfair comparison. However, I also applaud the authors for training the baseline with the same baseline settings. At the same time, I suggest authors to match the exact settings of the baselines because the results presented in this paper would become a baseline for comparison in future. Hence to avoid confusion among readers, it is advisable to add a detailed training setting section in the supplementary and also a small footnote or caption in the table stating the reasons for the accuracy difference of DieT while also adding the results from the original paper.

Based on the reviews, I vote for acceptance. I have increased my rating.
> Q5: *Why the proposed method can reduce the conjunction error*
> 

Thanks. Taking the conjunction error in MHSA as an example, in the advanced stages of Transformer’s MHSA, the integration weights $\mathbf{a}$ dynamically integrate specific foreground information between tokens for high-confidence samples, while they incorrectly integrate background-related information for low-confidence samples (lines 175-182 of the paper). During the treatment phase, our heuristic dynamic information integration method constrains the integration weights $\mathbf{a}$ through the loss function to encourage integration of foreground token information (lines 231-234 of the paper).

Importantly, we introduced gradients to differentiate the importance of each head in MHSA, obtaining the integration weights $\hat{\mathbf{a}}$ in the multi-head scenario and constraining them (lines 223-227 of the paper). We then update the MHSA parameters through backpropagation of the loss function’s gradient, which helps MHSA produce more accurate integration weights $\mathbf{a}$ and reduces the occurrence of erroneous background information integration.

Similarly, for the conjunction errors in the inter-token information static integration within FFN, we apply constraints to the integration weights $\mathbf{z}$ using the loss function, and update FFN parameters through gradient backpropagation. This process helps FFN generate more accurate integration weights $\mathbf{z}$ and reduces the occurrence of conjunction errors (lines 244-247 of the paper).

We hope these explanations address your concerns.

> Q6: *To what extent (fully/partially) can the proposed reduce the error*
> 

This is an insightful question. The extent to which conjunction errors are reduced can be assessed through the decrease in the loss functions mentioned, specifically $\mathcal{L}\_{IDI}$ in Equation (6) and $\mathcal{L}\_{ISI}$ in Equation (9). In the experiments presented in the paper, both loss functions show a significant downward trend and ultimately converge, indicating that the conjunction errors are reduced to the maximum extent possible based on these loss functions.

However, depending on the dataset size and the choice of various hyperparameters, the loss functions do not decrease to zero but rather stabilize at certain values. Therefore, while conjunction errors are reduced significantly, they are not entirely eliminated. We appreciate your question and will update this discussion in the paper to clarify this aspect further.

> Q7: *Besides the information integration hypothesis, do you rely on other assumptions?*
> 

Thank you for this insightful question. Yes, we do rely on additional assumptions. Specifically, the proposed Transformer Doctor is fundamentally based on the Information Integration Hypothesis. This hypothesis is inspired by conjunction errors observed in biological visual systems[1]. 

Studies such as [2] suggest that these conjunction errors can be mitigated through certain stimuli and cues. Similarly, during the treatment phase, we assume that such errors within the Transformer can also be improved using specific cues.

In summary, as we mentioned in Section 1, the motivation for this work stems from the error mechanisms observed in biological vision. Thus, Transformer Doctor is closely tied to insights from these related works.

[1] Treisman, Anne M., and Garry Gelade. "A feature-integration theory of attention." *Cognitive Psychology* 12.1 (1980): 97-136.

[2] Prinzmetal, William, David E. Presti, and Michael I. Posner. "Does attention affect visual feature integration?" *Journal of Experimental Psychology: Human Perception and Performance* 12.3 (1986): 361.

> Q8: *Is the information integration hypothesis valid in vision transformers, especially for large-scale versions?*
> 

Thank you for your question. The Information Integration Hypothesis is indeed valid for large-scale Vision Transformers. As addressed in response to your second comment (Q2), the depth of the Transformer, the number of heads, and the dimensionality of features do not affect the information integration process in MHSA and FFN. Quantitative experiments also confirm that the hypothesis is effective for large-scale Vision Transformers, such as ViT-Large, demonstrating similar beneficial results.

> Q9: *This paper has many typos, even in the result presentation (e.g. Table 1 BeiT/Imagenet-1K cell).*
> 

We sincerely apologize for the typographical errors. The issue you mentioned has been addressed in the revised manuscript, and we have conducted a thorough review of the entire paper to avoid similar issues.