Transformer Doctor: Diagnosing and Treating Vision Transformers
Authors: Jiacong Hu
Hao Chen
Kejia Chen
Yang Gao
Jingwen Ye
Xingen Wang
Mingli Song
Zunlei Feng
Venue: NeurIPS 2024 poster (2024)
Keywords: Transformer
Machine Vision
Biological Vision
Primary area: interpretability_and_explainability

Abstract: Due to its powerful representational capabilities, Transformers have gradually become the mainstream model in the field of machine vision. However, the vast and complex parameters of Transformers impede researchers from gaining a deep understanding of their internal mechanisms, especially error mechanisms. Existing methods for interpreting Transformers mainly focus on understanding them from the perspectives of the importance of input tokens or internal modules, as well as the formation and meaning of features. In contrast, inspired by research on information integration mechanisms and conjunctive errors in the biological visual system, this paper conducts an in-depth exploration of the internal error mechanisms of Transformers. We first propose an information integration hypothesis for Transformers in the machine vision domain and provide substantial experimental evidence to support this hypothesis. This includes the dynamic integration of information among tokens and the static integration of information within tokens in Transformers, as well as the presence of conjunctive errors therein. Addressing these errors, we further propose heuristic dynamic integration constraint methods and rule-based static integration constraint methods to rectify errors and ultimately improve model performance. The entire methodology framework is termed as Transformer Doctor, designed for diagnosing and treating internal errors within transformers. Through a plethora of quantitative and qualitative experiments, it has been demonstrated that Transformer Doctor can effectively address internal errors in transformers, thereby enhancing model performance.

Transformer Doctor: Diagnosing and Treating Vision Transformers
Jiacong Hul Hao Chen', Kejia Chen', Yang Gao Jingwen Ye', Xingen Wang 16. Mingli Song! 14.55 Zunlei Feng
CCollege of Computer Science and Technology: Zhejiang University, Schooo of Software Technology, Zhejiang University,
Electrical and Computer Engineering, National University of Singapore, State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security, *Banghenng Technology Co., Ltd.
[jiaconghu, hhahaahen____aaaaaaaaaaaaaaaeeae joyya @zju. edu. en, j jingwenyOnus. hww.://uuuuuuuuu.gee.ee.ee sg, (newroot, brooksong, zunl wwmleiifneaa. @zju. edu. cn
Abstract
Due to its powerful representational capabilities, Transformers have gradually become the mainstream model in the field of machine vision. However, the vast and complex parameters Transformers impede researchers from gaining a deep understanding of their internal mechanisms, especially error mechanisms. Existing methods for interpreting Transformers mainly focus on understanding them from the perspectives of the importance of input tokens or internal modules, well as the formation and meaning of features. In contrast, inspired by research on information integration mechanisms and conjunctive errors in the biological visual system, this paper conducts an in-depth exploration of the internal error mecha- nisms of Transformers. We first propose information integration hypothesis for Transformers in the machine vision domain and provide substantial experimental evidence to support this hypothesis. This includes the dynamic integration of information among tokens and the static integration of information within tokens in Transformers, well as the presence conjunctive errors therein. Addressing these errors, we further propose heuristic dynamic integration constraint methods and rule-based static integration constraint methods rectify errors and ultimately improve model performance. The entire methodology framework is termed as Transformer Doctor, designed for diagnosing and treating internal errors within transformers. Through plethora of quantitative and qualitative experiments, it has been demonstrated that Transformer Doctor can effectively address internal errors in transformers, thereby enhancing model performance. For more information, please visit https: http::/ttaasoooe cwwrransformer-eee doctor. .errr.gittte thub.
Introduction
In the field of machine vision, models based Transformers [1, 2] have gradually replaced con- volutional neural networks as the mainstream approach. Particularly in recent years, various visual architectures improved upon Transformers have emerged incessantly [3-6], continuously pushing the performance boundaries visual tasks. However, the vast and complex parameters of Trans- formers hinder researchers from gaining deep understanding of their internal mechanisms [7].
CCoresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

thereby increasing the risks of applying them in sensitive domains [8]. This has spurred a consid- erable amount of work aimed at investigating the interpretability of Transformers to enhance their transparency [9-11].
Existing research on the interpretability of Transformers in machine vision primarily focuses on aspects such as the importance of input tokens [7, 9, 10, 12-14], the significance of internal mod- ules [15], the the evolution and formation of features [16, 17], and the meanings of intermediate represnetations [18-20]. While these studies have somewhat improved the transparency of Trans- formers, the internal decision- mariiinnn-miinngggg processes, such as mechanisms leading to errors, still warrant more systematic investigation. This is crucial for enhancing the transparency of Transformers and further improving their performance.
In fact, unlike in machine vision, theoretical studies on error mechanisms in biological vision have become quite mature [21-26]. Specifically, in the perceptual process of the biological visual system, visual information such as the spatial position, shape, size, color, and texture of objects is processed and refined in the primary visual cortex [27-29]. Subsequently, these different visual cues are integrated at higher stages for final recognition [30, 31j. Numerous studies have demonstrated that errors in object recognition may arise not only from failures in feature extraction but also from incorrect integration of correctly extracted features at higher stages [2 [21-..]. Errors resulting from the improper integration of features are termed conjunction errors [26]. Furthermore, some research indicates that providing effective stimuli or cues during the integration process can enhance the correctness of information integration [22]. Inspired by this, we are interested in investigating whether Transformers exhibit similar mechanisms of feature integration and conjunction errors as those observed in biological vision during recognition. such errors exist, can they be corrected akin to the mechanisms observed in biological vision?
To address these questions, we first proposed the information integration hypothesis, inspired by the biological visual. This hypothesis posits that Transformers continuously process and refines various mixed information at the primary stage and integrate them at the higher stage. When incorrect infor- mation is integrated, i.e., conjunction error occur, it results in erroneous recognition outcomes. To validate this hypothesis, we conducted extensive experimental analyses of the computational process of Transformers and found empirical evidence supporting the hypothesis. Specifically, we discovered dynamic information integration among tokens in Transformer's 's Multi-Head Self-Attention (MHSA) component and static information integration within tokens in the Feed- -ordard Network (FFN) com- ponent, along with the presence of conjunction errors. Furthermore, we elucidated the reasons behind both dynamic and static integration. Building upon this, we proposed heuristic dynamic integration constraints for inter-token information integration and rule-based static integration constraints for intra-token information integration, enabling the rectification of conjunction errors in Transformers. We coined the entire approach as "Transformer Doctor", where the process of identifying errors based on the information integration hypothesis is referred to diagnosing the Transformer, and the process of applying the hypothesis to rectify errors is referred to as treating the Transformer. Finally, we conducted extensive quantitative and qualitative experiments on mainstream Vision Transformer architectures, thoroughly validating the effectiveness and applicability of Transformer Doctor.
The contributions of this paper can be summarized as follows:
We propose Transformer Doctor, the first framework for diagnosing and treating Vision Transformers. This framework validates and utilizes the proposed Information Integration Hypothesis, which posits that Transformers process and encode various mixed information at primary stages and integrate it at higher stages. When information is not correctly integrated, i.e., conjunction error occur, it results in prediction failures.
In diagnosing Transformers, we identify the mechanisms of inter-token information dynamic integration and intra-token information static integration within Transformers, along with the occurrence of conjunctive errors. This provides a novel perspective for understanding the internal mechanisms of Vision Transformers.
In treating Transformers, we propose heuristic dynamic constraints for inter-token infor- mation integration and rule-driven static constraints for intra-token information integration. These constraints offer an interpretable solution for optimizing Vision Transformers without introducing additional parameters computational overhead during inference.

---

Extensive qualitative and quantitative experiments are conducted on mainstream Vision Transformers, validating the effectiveness and applicability of the Transformer Doctor.
2 Related Works
In methods aimed at interpreting or understanding Transformers, whether in the field of machine vision natural language processing, the primary approaches focus the importance of input tokens [7, 12, 13, 9, 14, 10], the significance internal modules [15], the the evolution and formation of features [16, 17]. and the meanings of intermediate features [18-20] to understand the internal mechanisms of Transformers. Specific methodologies can be categorized as feature-based, attention- based, gradient-based, propagation-base of these approaches. For instance, in feature-based methods, the primary focus is on analyzing or statistically evaluating the intermediate features within Transformers [16] to understand the internal representation structure and feature distribution [17]. Attention-base methods mainly utilize the raw attention weights [7, 12, 13] or linear combinations of multi-layer attention weights [9] to compute the relative importance of input tokens. Gradient- based methods focus on computing gradients of attention weights [32, 33], intermediate features [11, 34], or inputs [14] to understand the differences in token importance. Propagation- based methods employ techniques like Layer- wise Relevance Propagation (LRP) 36] for attribution analysis of input tokens[10, 37-39] or investigate the importance of heads in Transformers [15]. Berturbation-baser methods involve perturbing inputs or features and measuring the impact on model performance [40-43] or Shapley value [14, 44]. Projection-baset methods, such as linear probes [17, 45, 46], project intermediate representations into human-undee understandable spaces to comprehend the mechanism and significance of feature transformations in Transformers [18-20]. In contrast to the aforementioned research, this paper is inspired by studies on biological visual error mechanisms, aiming to explore whether Transformers exhibit similar mechanisms of information integration and connection errors in biological vision, and how to rectify errors within Transformers.
perturbation-based projection-based
combination
In methods for improving Transformers, most approaches involve modifying the model architecture by introducing learnable parameters [3-6, 47] or enhancing data and features [48-52]. These improvement methods are mostly non-interpretable and are pre-defined before training, rather than targeting further enhancement of model performance from the perspective of diagnosing and treating internal eπror mechanisms based existing models. Additionally, there are methods for improving models that do not require pre-definition before training but focus on non- Transformer or non-vision tasks, such debugging and analyzing models in traditional machine learning [53-56], optimizing deep models [57, 58], and editing facts in natural language processing [59-62]. However, due to significant differences in architecture and tasks, these methods are not suitable for analyzing and correcting error mechanisms in Transformers to improve model performance.
In summary, this paper is the first work to investigate whether Transformers exhibit information integration and connection error mechanisms similar to biological vision, and the first work to explore how to further enhance model performance by diagnosing and treating internal errors in existing models.
3 Information Integration Hypothesis
In this section, we firstly propose the Information Integration Hypothesis for Transformers. Subse- quently, we review the MHSA and FFN modules within the Transformer architecture, followed by an analysis of potential locations where the Information Integration Hypothesis may apply.
Information Integration Hypothesis: Similar to biological vision, in machine vision, the Transformer continually processes and refines various mixed information in the primary stage, and integrates it in the advanced stage. When erroneous information is integrated, i.e., conjunction errors occur, it leads to incorrect predictions.
3.1 Potential Information Integration in MHSA
When utilized for visual recognition tasks, a Transformer typically comprises T. blocks, each con- sisting of an MHSA module and an FFN module. The input to the block can be represented as

---

diagnosing
treating
diagnosing
treating
1=1
N
LIDI
N
2
LisT
FFN
rule-based
information dynamic integration
MHSA
information static integration
heuristic
N
a
M
W
Figure 1: The methodology framework of Transformer Doctor. It begins by analyzing the dynamic integration mnterttoken information MHSA and the static integration of intra-token information in FFN, Subsequently, conjunction errors within them are diagnosed. and finally treated to enhance model performance.
X RNXD, where N and D denote the number and dimensionality of tokens, respectively, The output X' E NxD of the MHSA module within this block can be computed as follows:
X softmax(
VD QKT)V
(1)
where Q = XW K X V = XW w(()) E RDD w(K) E RDxD, and W(V) e RDxD represent the parameter matrices for query, key, and value, respectively. Upon decomposing Eqn. (1), it can be observed that a certain token X E RD in X' RMx (e.g., the i-th token) is an integration weighted sum of all tokens in V E RNXD;
1 VD {QK),
X
anj V softmax(
where VR- represents the attention weights, which also be referred to integration weights within MHSA. can be observed that each token X possesses its own integration weight a, RN For simplicity, we have omitted the skip connections and bias terms in the above equation, and have considered only single-headed self-attention. Therefore, the concatenation and projection of multiple heads MHSA are also omitted.
3.2 Potential Information Integration in FFN
Suppose the input to the FFN is X' E NND, then the output Y E NxD of the FFN can be represented as:
Y gelu gelg(X ('
(3)
where (1)) E RDDM and (1(), E RD represent the parameter matrices for the first and second linear layers, respectively. Similarly, for simplicity, we have omitted the skip connections and bias terms in the above equation. Comparing Eqn. (1) and Eqn. (3), it can be observed that apart from the activation function gelu in FFN and the softmax function in MHSA, well as the constant VD FFN also employs the query-key-vallee mechanism similar to MHSA. Motivated by this, we further decompose Eqn. (3) into form similar to Eqn. (2), revealing that a certain token Y; RD in Y E NXD (e.g., the i-th token) an integration weighted sum of all dimensions in (2(2) E MD
M
Y
Σ
Zim
(2(2) n2
z = gelu (elu()))a))
where z E RRN can be referred to as integration weights within the FFN, and it can be observed that each token Y, also possesses its own integration weights z, E RMM In summary, both matrices and represent potential locations where the assumption of information integration may hold.

---

(a)
(b)
Figure 2: Visual comparison of integration weights a in MHSA. (a) and (b) respectively present visualizations of weights a at different depths of blocks for high-confidence images and the overlay of reshaped and resized rows of onto the original image. Similarly, (c) and (d) depict visualizations of weights a for low-confidence images and their overlay onto the original image.
Information Integration Hypothesis based Transformer Diagnosis
In this section, delve into the potential existence of information integration hypothesis mentioned in Section 3 within the MHSA and FFN. Through extensive experimental analysis, we have gathered empirical evidence supporting the hypothesis, namely dynamic integration of information among tokens and static integration of information within tokens.
4.1 Inter-token Information Dynamic Integration
To explore the potential existence of the information integration hypothesis within the MHSA, we analyzed the integration weights a in Eqn. (2). Fig. 2(a) illustrates the magnitudes of integration weights a in blocks of different depths within the model. It can be observed that for high-confidence samples, the integration weights a in shallower blocks exhibit a diagonal pattern. However, as the depth of the block increases, the integration weights a display a vertical pattern, consistent with observations from prior studies [63-66]. Additionally, we observed that for different high-confidence samples, the positions of the vertical lines in the integration weights within the deeper blocks vary, as shown in Fig. 8 in the Appendix. This indicates that in the initial stages, MHSA primarily mixes and processes information between adjacent tokens. In the later stages, MHSA dynamically and selectively integrates specific information among tokens.
Continuing, we extracted an arbitrary row from the integration weights within the deeper blocks and removed the [CLS] token. After reshaping and resizing, we overlaid onto the original image to generate heatmap, shown in Fig. 2(b). From the heatmap, it is evident that the positions of the vertical lines mainly concentrate on the foreground of the input image. This suggests that in the advanced stages of the model, MHSA primarily integrates specific information among tokens containing foreground elements. However, for low-confidence samples, as depicted in Fig. 1(c) and (d), the deeper layers of the MHSA erroncously integrate information corresponding to the background tokens. Additional guantitative and qualitative analyses the integration weights a are presented in Appendices C and B. In summary, have identified the first evidence of the existence of the information integration hypothesis in Transformer models:
Evidence I: In the initial stages of the Transformer, MHSA primarily mixes and processes adjacent patch information among tokens. However, in the advanced stages of the Transformer, MHSA dynamically and selectively integrates specific patch information among tokens. When integrating incorrect information among tokens, termed conjunction errors, it leads to model mispredictions. We refer this integration inter-token information dynamic integration. The term "dynamic" arises from the fact that the to be integrated, as specified in Eqn. (2), varies with each sample.
4.2 Intra-token Information Static Integration
To explore the potential existence of the information integration hypothesis within the FFN, we conducted visual analysis of the integration weights z in shallow and deep blocks, as depicted in Fig. 3. Fig. 3(b) illustrates the patterns of integration weights z in shallow and deep blocks for high-confidence samples from different classes. It can be observed that the patterns of integration weights differ between shallow and deep blocks for samples from different classes. However, in Fig. 3(a), for high-confidence samples from the same category, the patterns of integration weights in shallow blocks differ, while the patterns in deep blocks show less variability and remain relatively
5

---

shallow
deep
shallow
deep
(8)
(b)
1
(c)
(d)
Figure 3: Visual comparison of integration weights z in FFN. (a) and (b) respectively illustrate visualizations of weights z for high-confidence samples of same classes and the different class in shallow and deep blocks. Similarly, (c) and (d) depict visualizations of weights for low-confidence samples of different classes and the same class in shallow and deep blocks. Each row the image represents sample, and each column represents dimension.
consistent. This indicates that in the initial stages of the model, FFN mixes and processes various category information within tokens. However, in the advanced stages, FFN statically selectively integrates specific category information within tokens.
Furthermore, as depicted in Fig. 3(d), even for samples from the same class, the patterns of integration weights z in deep blocks differ for low-confidence images. This suggests that when the deep FFN fails to correctly integrate specific category information within tokens, adversely affects the model's predictions. Additional quantitative and qualitative analyses of the integration weights z are presented in Appendices E and D. In summary, have identified the second evidence of the information integration hypothesis in Transformers:
Evidence 2: In the initial stages of the Trunsformer, the FFN primarily mixes and processes various low-level information within tokens. However, in the advanced stages of the Transformer, the FFN statically and selectively integrates specific category information within tokens. When integrating incorrect information within tokens, termed conjunction errors, leads to model mispredictions. The term "static" arises from the fact that the W(2) be integrated, specified in Eqn. (4), is fixed relative to each sample, and for samples of the same class, the integration weights z are also fixed.
5 Information Integration Hypothesis based Transformer Treatments
In this section, we propose heuristic dynamic integration constraints and rule-based static integra- tion constraints to correct conjunction errors in information integration, aiming enhance model performance.
5.1 Heuristic Information Dynamic Integration Therapy
In order to alleviate conjunctive errors in dynamic integration of information among tokens, we heuristically constrain the integration weights by highlighting the foreground of images with low confidence scores. Furthermore, unlike the single-head integration weights a in Eqn. (2), we have improved the calculation of integration weights for multi-head scenarios using gradients.
Specifically, for a Transformer classification model with K(K 2 2) classes, let & E {1. K} represent the true label for input, and R denote the predicted probabilities of all classes by the Transformer. We incorporate gradients related to the true class to discern the importance of each head out of H heads in self-attentio,, thereby obtaining the integration weights E NxN in the multi-head scenario:
Opk
(max(
Jah 0) O))a
(5)
where max(.) denotes setting negative values in the derivative to zero, thereby considering only the positive impact on the predicted probability of the true class. The introduction of gradients in Eqn. (5) not only helps discer which head is important but also establishes a connection between integration weights and specific classes, thereby making the weights reflecting the integration of information among tokens more accurate. Details of the comparison experiments on the introduction of gradients

---

can be found in Section 6.2. Next, for foreground annotation R with low confidence, we constrain the integration of background information using the loss function Linr
LIDI
2n.5) (1-tt))
(6)
/=1
where E RR is a binary annotation, with and O representing the presence and absence of foreground within the token, respectively.
5.2 Rule-based Information Static Integration Therapy
To correct conjunctive errors in static integration information within tokens, we first establish integration rules within tokens when the model prediction is correct, and then constrain the integration weights z based on these rules. Additionally, we have also improved the integration weights z using gradients, establishing connection with the true class k resulting in the integration weights 2 E R x as follows:
Opk Oz
max
Next, we select S high-confidence samples for each class to calculate the average integration weights Z E R XMM, and then establish binary integration rules r E I xMM for each class using a threshold ::
S
r = 1 2 T), 2 S
Σ 2
(8)
The values in or 0, indicating the integration and non-integration of information in the corresponding dimension, respectively. Finally, based on the integration rules r for a certain class, enforce that erroneous information within tokens is not integrated using the loss function List:
N
CISI
Σ (2n.m
@ (1 Tn.m)).
mh ==1
During actual enforcement, each training sample needs to be constrained using the integration rules corresponding to its true class.
5.3 Joint Therapy of Dynamic and Static Integration
The loss functions LIDI and LIsı can be individually combined with the original loss function or used jointly:
Liotal Cori + aLIDI + 3LISI.
(10)
where a and 3 are used to balance the magnitudes of the loss functions. In our practical experiments, we found that using the loss functions LIDI and List sequentially yielded the most effective results. It is important to note that the therapy model only rectifies conjunctive errors in the Transformer without altering its architecture or operational procedure. Thus, during inference, it does not incur any additional computational overhead.
6 Experiments
6.1 Experimental Settings
Datasets and Transformer Architectures. To validate the efffcctiveness of Transformer Doctor, conducted experiments on five mainstream datasets: CIFAR- 10 [67], CIFAR-100 [67], ImageNet- 10 [68], ImageNet-50 [69], and ImageNet-Ie 1k [68]. Furthermore,
performed experiments various Transformer architectures used for visual classification tasks, including DeiT [48], CaiT [3], TNT [4], PVT [5], Eva [6], and BeiT [49], in addition to VIT [2]. It is important to note that Transformer Doctor diagnoses and treats already trained Transformer models. More experimental settings and results can be found in Appendix.
Parameter Deringgs.Drrngg all training stage, each dataset was trained for 300 epochs using the AdamW [70] optimizer, with initial learning rate of 0.01. The learning rate decayed according to
7

---

Table 1:
Performance of Transformer Doctor on various SOTA Transformers. '+Doctor' indicates the performance of model treated with Transformer Doctor (All Score are in %).
CIFAR-10
CIFAR-100
ImageNet-10
ImageNet-50
ImageNet-IK
ViT-Tiny
82.17
56.02
78.8
59.62
64.77
+Doctor
83.00 (+0.87)
58.08 (+2.06)
80.80 (+2.00)
61.02 (+1.40)
68.86 (+4.09)
DeiT-Tiny
82.71
56.97
80.20
61.47
66.83
+Doctor
83.96 (+1.25)
59.49 (+2.52)
81 (+1.00)
63.19 (+1.69)
70.75 (+3.92)
CaiT-XXS
82.64
56.10
75.80
58.32
66.28
+Doctor
84.20 (+1.36)
60.00 (+3.90)
77.80 (+2.00)
60.16 (+1.84)
70.25 (+3.97)
TNT-Small
83.31
54.67
81.60
65.45
67.25
+Doctor
84.33 (+1.02)
55.60 (+0.93)
83.00 (+1.40)
67.64 (+2.08)
69.97 (+2.72)
PVT-Tiny
83.27
51.94
82.00
71.81
67.73
+Doctor
84.82 (+1.55)
55.10 (+3.16)
84.20 (+2.20)
74.53 (+2.72)
70.94 (+3.21)
Eva-Tiny
87.56
64.15
83.80
71.23
72.51
+Doctor
88.28 (+0.72)
64.99 (+0.84)
85.80 (+2.00)
72.95 (+1.72)
75.45 (+2.94)
BeiT--iny
74.69
49.58
79.80
71.59
70.46
+Doctor
76.20 (+1.51)
51.03 (+1.45)
82.20 (+2.40)
73.57 (+1.98)
71.98 (+3.12)
cosine annealing schedule, with T_max set to 300 epochs. Additionally, a and ,3 were set default values of 10 and 100, respectively, to balance each loss function. The default value of was 0.15, and the constrained loss function was applied by default to the last block.
Baseline Models. In addition to using the original pre-trained Transformer models as baselines, as shown in Table I, we also established a blank control group. which involves no method introduction but continues training for the same epochs, presented in Table 3. Furthermore, we compared different ways of integrating multi-head integration weights in Eqn. 5 and not introducing gradients in Eqn. 7 as simple baselines against the proposed final method, as illustrated in Table 2. Due to differences in computational resources, certain training configurations, such as batch size, differ from those in the original work, leading to slight variations in the baseline. However, all comparative experiments were conducted under fair conditions. For detailed experimental results, please refer to Table 3, and for further setup information, see Appendix F.
6.2 Quantitative Analysis
The Performance of Transformer Doctor SOTA Models. We evaluated the performance of Transformer Doctor across five major datasets and seven mainstream architectures. As shown in Table 1, it is evident that Transformer Doctor effectively enhances model performance both small and large-scale datasets. Specifically, after treatment, the accuracy of CaiT improved by 1.36% on CIFAR-10 and 1.84% on ImageNet-50. The BeiT model, after treatment, saw accuracy increase of 1.45% on CIFAR-100 and 2.39% on ImageNet-50. Additionally, we observed that generally, the larger the dataset, the more significant the performance improvement by Transformer Doctor. For instance, the accuracy improvement of ViT on ImageNet-IK was 2.09% higher than on ImageNet- 10. The reason for this could be that, similar to biological vision. Transformer misrecognition can be due to both conjunction errors and feature extraction failures. Large-scale datasets effectively help the model learn vast array of features, thus better extracting the input image's features. On this basis, treating conjunction eтrors can maximize model performance. However, for small-scal datasets, the dominance of not extracting effective features may outweigh conjunction errors. Therefore, focusing solely on treating conjunction errors is less effective on small-scale datasets compared to large-scale ones. More detailed experimental results can found in Table 3 in the appendix.
The Performance Transformer Doctor with Various Computational Forms.
We compared the performance of Transformer Doctor under different computational forms represented by Eqn. (5) and Eqn. (7), as shown in Table2. can be observed that Transformer Doctor performs better under the computational forms of Eqn. (5) and Eqn. (7). Specifically, under dynamic integration constraints,

---

Table 2: Comparison of accuracy of Transformer Doctor under different computational formulations. 'mean' denotes directly averaging integration weights across all heads, 'min' and 'max' respectively represent taking the minimum and maximum integration weights within each head. Each row corresponds to ViT-Tiny and PVT-Tiny architectures, with ImageNet- 10 dataset used for evaluation.
+IDI
+II
Base
min(a"
max (a")
mean (a")
à
78.80
79.20 (+0.40)
78.60 (-0.20)
79.20 (+0.40)
80.20 (+1.40)
79.00 (+0.20)
80.40 (+1.60) 82.40 (40.40) 83.40 (+1.40)
82.00
81.20 (-0.80)
81.20 (-0.80)
80.00 (-2.00) 83.60 (+1.36)
(I)
(3)
(2)
(3)
Figure 4: Comparison of inter-token integration weights before and after introducing Transformer Doctor. (1) and (3) depict integration weights before and after treatment, respectively. (2) and (4) show the corresponding heat map effects of integration weights overlaid onto the original image before and after treatment.
(1)
(2)
(3)
Figure 5: Comparison of intra-token token integration weights before and after introducing Transformer Doctor, (1) and (2) represent the intra-token integration rules for correct predictions. (3) and (4) depict the intra-token integration weights before and after treatment, respectively.
taking the minimum, maximum, or averaging fusion of integration weights for each head does not significantly improve model performance. For instance, when taking the maximum value, the accuracy of PVT-Tiny decreases by 0.08% compared to the base. However, under the computational form of Eqn. (5), the accuracy of PVT--Tiny improves by 2%. Under static integration constraints, the accuracy of the model under the original integration weight calculation form 79.00 79.00%, while the accuracy under the computational form of Eqn. (7) is 80.40%.
6.3 Qualitative Analysis
Comparison of the Intra-token Integration Weights a.
We also compared the static integration weights within tokens before and after treatment, as shown in Fig. 5. From panels (I) and (3), is apparent that before treatment, the static integration weights for misclassified samples do not adhere the correct prediction rules. However, panels (2) and (4) in Fig. 5 demonstrate that after treatment, the static integration weights align with the correct prediction rules, integrating the correct class information, which results in accurate predictions.
Comparison of the Inter-token Integration Weights z. In Fig. 5, we compared the static integration weights within tokens before and after treatment. As shown in Fig. 5 (1), before treatment, the static integration weights of misclassified samples do not adhere to the correct integration rules for prediction, i.e., integration of incorrect category information within tokens. However, as depicted in Fig. 5 (2), after treatment, the static integration weights conform to the correct integration rules for prediction, integrating the correct category information within tokens.

---

7 Conclusion
This paper introduces the first framework, Transformer Doctor, designed for diagnosing and treating internal errors within Transformers simultaneously. Specifically, distinct from existing post-hoc interpretability methods for Transformers this work draws inspiration from information integration and conjunctive enrors in the biological visual system, proposing and validating the hypothesis of information integration for Transformers. Furthermore, addressing conjunctive errors within information integration, this paper presents corresponding error treatment methods. Extensive qualitative and quantitative analyses conducted on mainstream datasets and Transformer architectures demonstrate the effectiveness and applicability of Transformer Doctor.
Limitations and Future Work. It is undeniable that we have only validated Transformer Doctor on mainstream visual recognition tasks, leaving more complex machine vision tasks for further exploration. Furthermore, investigating whether the information integration hypothesis holds true in the field of natural language processing or multimodal domains is also a worthwhile research endeavor. Additionally, exploring more error mechanisms and developing more automated and intelligent Transformer Doctor framework are the focal points of our future work.
10

---

Acknowledgments and Disclosure of Funding
This work is supported by National Natural Science Foundation of China (62376248), Ningbo Natural Science Foundation (2022J182) and the Fundamental Research Funds for the Central Universities (226-2024-00145).
References
[1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[2] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition scale. arXiv preprint arXiv:2010. 11929, 2020.
[3] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 32-42, 2021.
[4] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. Advances in neural information processing systems, 14:15908-15919, 2021.
[5] Wenhai Wang, Enze Xie, Xiang Li, Deng- Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt PvL v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 8(3):115 424, 2022.
[6] Yuxin Fang. Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang. Tiejun Huang, Xinlong Wang. and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 19358-19369, 2023.
[7] Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert's attention. arXiv preprint arXiv: (e..199664444) 2019.
Monirah Ali Aleisa, Natalia Beloff, and Martin White. Airm: ai recruiting model for the saudi arabia labor market. In Intelligent Systems and Applications: Proceedings of the 2021 Intelligent Systems Conference (IntelliSys) Volume 3. pages 104-124. Springer, 2022.
[9] Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. arXiv preprint arXiv:2005.000228 2020.
[10] Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 782-791, 2021.
[11] Yao Qiang, Deng Pan, Chengyin Li, Xin Li, Rhongho Jang, and Dongxiao 7hu. Attcat: Explaining transformers via attentive class activation tokens. Advances in neural information processing systems, 35:5052-500 5064, 2022.
[12] Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. Revealing the dark secrets of bert. arXiv preprint arXiv: /908.00900005599 2019.
[13] Gongbo Tang, Rico Sennrich, and Joakim Nivre. An analysis of attention mechanisms: The case of word sense disambiguation neural machine translation. arXiv preprint 11X.v.1810.07595, 2018.
[14] Pepa Atanasova, diagnostic study of explainability techniques for text classification. Accountable and Explainable Methods for Complex Reasoning over Text, pages 155-187. Springer, 2024.
[15] Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self- attention: Specialized heads do the heavy lifting, the be pruned. arXiv preprint arXiv:1905.09411 2019.
[16] Amin Ghiasi, Hamid Kazemi, Eitan Borgnia, Steven Reich, Manli Shu, Micah Goldblum, Andrew Gordon Wilson, and Tom Goldstein. What do vision transformers leam? visual exploration. arXiv preprint arXiv:2212.0672 2022.

---

[17] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in neural information processing systemss 34:12116-12128, 2021.
[18] Bjöm Deiseroth, Mayukh Deb, Samuel Weinbach, Manuel Brack, Patrick Schramowski, and Kristian Kersting. Atman: Understanding transformer predictions through memory efficient attention manipulation. Advances in Neural Information Processing Systems, 36:63337-6338 63460, 2023.
[19] Martina G Vilas, Timothy Schaumlöffel, and Gemma Roig. Analyzing vision transformers for image classification in class embedding space. Advances in Neural Information Processing Systems, 36, 2024.
[20] Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer foed-forward layers build predictions by promoting concepts in the vocabulary space. arXiv preprint arXiv:2203. 14680, 2022.
[21] William Prinzmetal. Principles of feature integration in visual perception. Perception & Psychophysics. 30(4):330-340, 1981.
[22] William Prinzmetal, David E Presti, and Michael I Posner. Does attention affect visual feature integration? Journal of Experimental Psychology: Human Perception and Performance, 12(3):361, 1986.
[23] Anne M Treisman, Marilyn Sykes, and Gary Gelade. Selective attention and stimulus integration. In Attention and performance VI, pages 333-361, Routledge, 2022.
[24] George Wolford. Perturbation model for letter identification. Psychological review, 82(3):184, 1975.
[25] Anne Treisman and Hilary Schmidt, IIlusory conjunctions in the perception of ohjects. Cognitive psychology, 14(1):107-141, 1982.
[26] Anne M Treisman and Garry Gelade. A feature-integration theory of attention. Cognitive psychology, 12(1):97-133) 136, 1980.
[27] David C Van Essen and Edgar A Deyoe. Concurrent processing in the primate visual cortex. 1995.
[28] Margaret Livingstone and David H Hubel. Psychophysical evidence for separate channels for the perception of form, color, movement, and depth. Journal of Neuroscience, 7(1 ((1))3316-3468, 1987.
[29] Rufin Vogels. Mechanisms of visual perceptual learning in macaque visual cortex. Topics in cognitive science, 2(2):239-250 2010.
[30] Anne M Treisman. Selective attention in man. British medical bulletin, 1964.
[31] Philip T Quinlan. Visual feature integration theory: past, present, and future. Psychological bulletin, (5)65) 2003.
[32] Yaru Hao, Dong, Furu Wei, and Ke Xu. Self-attention attribution: Interpreting information interactions inside transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 12963-12971, 2021.
[33] Oren Barkan, Edan Hawon, Avi Caciularu, Ori Katz, Itzik Malkiel, Omri Armstrong, and Noam Koenigstein. Grad-sam: Explaining transformers via gradient self-attention maps. In Proceedings of the 30th ACM International Conference Information & Knowledge Management, pages 2882-2887, 2021.
[34] Eric Wallace, Jens Tuyls, Junlin Wang. Sanjay Subramanian, Matt Gardner, and Sameer Singh. Allennip interpret: framework for explaining predictions of alp models. arXiv preprint arXiv:1909.0922 09251, 2019.
[35] Sehastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klausehen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7)=001 30140, 2015.
[36] Grégoire Montavon, Sebastian Lapuschkin, Alexander Binder, Wojciech Samek, and Klaus-Robert Müller, Explaining nonlinear classification decisions with deep taylor decomposition. Pattern recognition, 65:211- 222, 2017.
[37] Ameen Ali, Thomas Schnake, Oliver Eberle, Grégoire Montavon, Klaus- Robert Müller, and Lior Wolf. Xai for transformers: Better explanations through conservative propagation. In International Conference Machine Learning, pages 435 451. PMLR, 2022.
[38] Zhengxuan Wu and Desmond C Ong. On explaining your explanations of bert: An empirical study with sequence classification. arXiv preprint arXiv:2101.0019% 2021.
12

---

[39] Hila Chefer, Shir Gur, and Lior Wolf. Generic sltettio-moode explainability for interpreting bi-modal and encoder-decoder transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 397-406, 2021.
[40] Srinadh Bhojanapalli, Ayan Chakrabarti, Daniel Glasner, Daliang Li. Thomas Unterthiner, and Andreas Veit. Understanding robustness of transformers for image classification. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10231-10241, 2021.
[41] Muhammad Muzammal Naseer, Kanchana Ranasinghe, Salman H Khan, Munawar Hayat, Fahad Shah- baz Khan, and Ming- Ming-Hssuan Yang. Intriguing properties of vision transformers. Advances in Neural Information Processing Systems. 34:23296-233 23308, 2021.
[42] Shi Feng, Eric Wallace, Alvin Grissom II, Mohit lyyer, Pedro Rodriguez, and Jordan Boyd-Grabed Pathologies of neural models make interpretations difficult. arXiv preprint vrXiv:1804.0778. 2018.
[43] Vinodkumar Prabhakaran, Ben Hutchinson, and Margaret Mitchell. Perturbation sensitivity analysis to detect unintended model biases. arXiv preprint arXiv: (.10:0990004200, 2019.
[44] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
[45] Nora Belrose, 7ach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens, arXiv preprint arXiv:2303.0m 081 12, 2023.
[46] Alexander Yom Din, Taelin Karidi, Leshem Choshen, and Mor Geva. Jump to conclusions: Short-cutting transformers with linear transformations. arXiv preprint arXiv:2303.09433 2023.
[47] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283 17297, 2020.
[48] Hugo Touvron, Matthieu Cord, Mathijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347-10357. PMLR, 2021.
[49] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106. 08254, 2021.
[50] Lin Chen, Zhijie Jia, Lechao Cheng, Yang Gao, Jie Lei, Yijun Bei, and Zunlei Feng. Vit-calibrator: Decision stream calibration for vision transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 1147-1155, 2024.
[51] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[52] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9640 9649, 2021.
[53] Gabriel Cadamuro, Ran Gilad-Bachrach, and Xiaojin Zhu. Debugging machine learning models. In ICML Workshop on Reliable Machine Learning in the Wild, volume 103, 2016.
[54] Michael Brooks, Saleema Amershi, Bongshin Lee, Steven Drucker, Ashish Kapoor, and Patrice Simard. Featureinsight: Visual support for error-driven feature ideation in text classification. In 2015 IEEE Conference on Visual Analytics Science and Technology (VAST), pages 105-112. IEEE, 2015.
[55] Josua Krause, Adam Perer, and Kenney Ng. Interacting with predictions: Visual inspection of black-box machine learning models. In Proceedings of the 2016 CHI conference on human factors in computing systems, pages 5686-5697. 2016
[56] Todd Kulesza, Simone Stumpf, Margaret Burnett, Weng- Keen Wong, Yann Riche, Travis Moore, Ian Oberst, Amber Shinsel, and Kevin Mclntosh. Explanatory debugging: Supporting end-user debugging of lachine-leared programs. In 2010 IEEE Symposium on Visual Languages and Centric Computing. pages 41-48. IEEF, 2010.
[57] Jiawei Zhang, Yang Wang, Piero Molino, Lezhi Li, and David S Ebert. Manifold: A model- framework for interpretation and diagnosis of machine learning models. IEEE transactions on visualization and computer graphics, 25(1):364-377 2018.
13

---

[58] Zunlei Feng, Jiacong Hu, Sai Wu, Xiaotian Yu, Jie Song, and Mingli Song. Model doctor: A simple gradient aggregation strategy for diagnosing and treating ean classifiers. In Proceedings the AAAI Conference on Artificial Intelligence. volume 36, pages 616-624, 2022.
[59] Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang. and Furu Wei. Knowledge neurons in pretrained transformers. arXiv preprint arXiv:2104.0 2021.
[60] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359 17372, 2022.
[61] Kevin Meng, Arna Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in transformer. arXiv preprint urXiv:2210.07229, 2022.
[62] Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang. Jun Ma, and Jie Yu. Pmet: Precise model editing in transformer. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18564-18525 2024.
[63] Boyu Chen, Peixia Li, Baopu Li, Chuming Li, Bai, Chen Lin, Ming Sun, Junjie Yan, and Wanli Ouyang. Psvit: Better vision transformer via token pooling and attention sharing. arXiv preprint arXiv:2108.03442 2021.
[64] Shuning Chang, Pichao Wang, Ming Lin, Fan Wang. David Junhao Zhang. Rong Jin, and Mike Zheng Shou. Making vision transformers efficient from token sparsification view. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6195-6205, 2023.
[65] Asher Trockman and J Zico Kolter. Mimetic initialization of self-attention layers. In International Conference on Machine Learning, pages 14456--4468.. PMLR, 2023.
[66] Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Zihang Jiang, Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2103. 1186, 118.6 2021.
[67] Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
[68] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. lece, 2009.
[69] Shanghua Gao, Zhong-Yo Yu Li. Ming-Hsuan Yang, Ming-Ming Cheng, Junwei Han, and Philip Torr. Large- scale unsupervised semantic segmentation. 2022.
[70] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv:171..05100,200 2017.
14

---

Appendix
In the appendix, we offer additional evidence concerning the information integration mechanism and conjunction errors the Transformer. Furthermore, more quantitative and qualitative analysis experi- ments on Transformer Doctor are provided. Additionally, the algorithm code for the Transformer Doctor is included in the uploaded source pance__ddes.aip file.
42.5
37.5
mhsa
base
fth
basc
37.5
32.5
32.5
27.5
27.5
22.5
3
7
3
5
9
Figure 6: Comparison model accuracy on low-confidence samples from ImageNet-10 10 after replacing original integration weights with ideal integration weights in VIT. (a) and (b) depict the accuracy line plots the model with ideal integration weights replacing original integration weights step by step from shallow to deep blocks in MHSA and FFN, respectively. The ideal integration weights in MHSA are manually annotated for foreground regions in images, while those in FFN are statistically generated by integrating weights of high-confidence images for each class.
A Broader Impact
This paper contributes the research field of understanding and optimizing Transformers. Specifi- cally, drawing inspiration from biological vision, the exploration of error mechanisms in Transformers for machine vision tasks provides novel perspective for understanding the internal workings of Trans- formers. Additionally, the interpretable treatment solutions proposed based on errors in Transformers offer pathway for optimizing Transformer models.
B Additional Visual Evidence of Inter-token Information Dynamic Integration
As illustrated in Fig. 7, we provide further visual comparisons of integration weights a with high and low confidences to demonstrate the dynamic information integration mechanism among tokens in the MHSA. The observed phenomena align with our conclusions in the main text. Specifically, in the early stages of the Transformer, shown in Fig. 7(a). the MHSA predominantly blends and processes information among adjacent tokens, whereas the advanced stages, the MHSA dynamically and selectively integrates specific information among tokens. Simultaneously. for low- confidence samples, as depicted in Fig. 7(b), the advanced-stage MHSA erroneously integrates token information corresponding to the background.
addition, we provide more visualizations of integration weights a for high- images final block, shown in Fig. 8. It can be observed that the integration weights vary with different input samples. This indicates that in the advanced stage, the integration weights a dynamically integrate specific information for the final prediction.
C Quantitative Analysis of Conjunction Errors in Inter-token Information Dynamic Integration
To verify whether incorrect predictions are indeed caused by the Transformer integrating erroneous information among tokens in the advanced stage, selected low-confidence samples from the test set and evaluated the model's performance after replacing the original integration weights with ideal ones, as shown in Fig. 6(a). It can be observed that forcibly integrating the correct information among
15

---

(
(2)
(1)
(2)
(2)
(2)
(
Figure 7: Visualization comparison of integration weights in MHSA. (a) and (b) correspond to high-confidence and cow-confieence samples, respectively. (1) and (2) show the visualizations of integration weights in blocks from shallow to deep from left to right, as well as the visualization of reshaped and resized rows of superimposed onto the original image.
16

---

(2)
(2)
(2)
Figure 8: Visualization comparison of integration weights last block of MHSA. (1) and (2) depict the visualization of a and the visualization of reshaped and resized rows of superimposed onto the original image.
tokens in the model's early stages does not improve performance. This is because the model in its early stages merely blending and processing various information, rather than integrating specific information. However, when correct information is forcibly integrated among tokens in the advanced stages, the model's performance improves significantly.
D Additional Visual Evidence of Intra-token Information Static Integration
As illustrated in Fig. 9, we provide further visual comparisons of integration weights with high and low confidences to demonstrate the static information integration mechanism within tokens in FFN. The observed phenomena align with conclusions in the main text. Specifically, in the early stages of the Transformer, as shown in Fig. 9a), the FFN mixes and processes various categories of information within tokens. However, in the advanced stages, the FFN statically and selectively integrates specific category information within tokens. Additionally, for low-confidence samples, as depicted in Fig. 9()), the FFN in the advanced stages integrates incorrect category information within tokens.
E Quantitative Analysis of Conjunction Errors in Intra-token Information Static Integration
To verify whether incorrect predictions are indeed caused by the FFN integrating erroneous informa- tion within tokens in the advanced stage the Transformer, we selected low-confidence samples from the test set and evaluated the model's performance after replacing the original integration weights with ideal ones, as shown in Fig. 6(b). The conclusions are similar those observed the dynamic information integration among tokens: forcibly integrating correct information within tokens in model's ss early stages does not improve performance. This is because, the early stages, model is merely mixing and processing various types of information rather than integrating specific information. However, when correct information forcibly integrated within tokens the advanced stages, the model's 's performance improves significantly.
17

---

shallow
middle
decp
(2)
(3)
(b)
shallow
middle
deep
5
(1)
(2)
(3)
Figure 9: Visualization comparison of integration weights z in the FFN. (a) and (b) correspond to high-confidence and low-confidence samples, respectively. (1), (2), and (3) represent the visualization of integration weights z in blocks from shallow to deep for three classes. Each row of each image represents different samples of the class, and each column represents different dimensions of the integration weights z.
F
Additional Experimental Settings
To validate the dynamic integration constraints mentioned in Section 5.1, we manually annotated the foreground masks for the 10 lowest-confidence samples in each class of ImageNet- 10. For ImageNet- 50 and ImageNet- we utilized the segmentation annotations provided in the ImageNet-S dataset. This dataset includes segmentation masks for 10 randomly selected samples per class in ImageNet-50 and for 10 randomly selected samples per class in 919 classes of ImageNet- 1000. For CIFAR-10 10 and CIFAR-100, due to the small size of the images making it difficult annotate the foreground, we only used these datasets to validate the static integration constraints mentioned in Section 5.2. Additionally, within the static integration constraints, considering that some Transformers utilize the [CLS] token for visual recognition tasks, we constrained only the static integration within the [CLS] token. the experiments, we utilized two Linux servers, each equipped with 8 NVIDIA A6000 GPU cards, 24 CPU cores, and 500GB of memory.
G
Additional Results of Transformer Doctor on SOTA Transformers
Table presents additional experimental results of Transformer Doctor on various mainstream datasets and architectures, including blank control groups, solely introducing dynamic integration constraints, solely introducing static integration constraints, and jointly introducing dynamic and static integration constraints. It is evident that both solely and jointly introducing dynamic static integration constraints significantly enhance model performance. Specifically, when solely introducing dynamic integration constraints, the accuracy of CaiT XXX increased by 1.2% on ImageNet- 10 and 1.57% on ImageNet- I.. When solely introducing static integration constraints, TNT-Tiny saw accuracy improvements of 1. 1.02% on CIFAR- 10 and I. 1.58% on IeageN When jointly introducing dynamic and static integration constraints, the ViT-Tiny model's 's accuracy increased by 1.40% on ImageNet-50 and 4.09% on ImageNet-IK.
18

---

Table 3: Comparison of accuracy of Transformer Doctor across various SOTA models. '+Blank' refers to blank control model trained for the same number of epochs without introducing any constraints. '+IDI", '+SSI', and '+IDI, ISI" denote models with individually introduced dynamic integration constraint, individually introduced static integration constraint, and simultaneously intro- duced dynamic and static integration constraints, respectively. Due to the small size of CIFAR-II 10 and CIFAR- 100 images, which makes foreground annotation challenging, experiments involving dynamic integration constraints were not conducted. Additionally, we replaced the results of experi- ments simultaneously introducing dynamic integration and static integration constraints with those of experiments solely involving static integration constraints. (all scores are in %)
CIFAR-10
C-FAR-100
ImageNet-10
ImageNet-50
ImageNet-1l 1K
ViT-Tiny
82.17
56.02
78.8
59.62
64.77
+Blank
82.22 (+0.05)
56.05 (+0.03)
78.90 (+0.10)
59.71 (+0.09)
64.74 (-0.03)
+IDI
80.20 (+1.40)
60.45 (+0.83)
66.42 (+1.65)
+ISI
83.00 (+0.87)
58.08 (+2.06)
80.40 (+1.60)
60.89 (+1.27)
66.33 (+1.56)
+IDI, ISI
83.00 (+0.87)
58.08 (+2.06)
80.80 (+2.00)
61.02 (+1.40)
68.86 (+4.09)
DeiT-Tiny
82.71
56.97
80.20
61.47
66.83
+Blank
82.69 (-0.02)
56.89 (-0.08)
80.20 (+0.00)
61.57 (+0.10)
66.86 (+0.03)
+IDI
81.00 (+0.80)
62.60 (+1.13)
68.20 (+1.37)
+III
83.96 (+1,25)
59.49 (+2.52)
80.60 (+0.40)
62.16 (+0.69)
68.10 (+1.24)
+IDI, ISI
83.96 (+1.25)
59.49 (+2.52)
81.20 (+1.00)
63.19 (+1.69)
70.75 (+3.92)
CaiT-XXS
82.64
56,10
75.80
58.32
66.28
+Blank
82.64 (+0.00)
56.08 (-0.02)
75.70 (-0.10)
58.33 (+0.01)
66.38 (+0.10)
+IDI
77.20 (+1.40)
58.95 (+0.63)
67.43 (+1.15)
+ISI
84.20 (+1.36)
60.00 (+3.90)
77.00 (+1.20)
59.90 (+1.58)
67.85 (+1.57)
+IDI, ISI
84.20 (+1.36)
60.00 (+3.90)
77.80 (+2.00)
60.16 (+1.84)
70.25 (+3.97)
TNT-Small
83.31
54.67
81.60
65.45
67.25
Bllank
83.27 (-0.04)
54.74 (+0.07)
81.60 (+0.00)
65.45 (+0.00)
67.23 (-0.02)
+IDI
82.60 (+1.00)
67.33 (+1.88)
68.34 (+1.09)
+ISI
84.33 (+1.02)
55.60 (+0.93)
82.20 (+0.60)
67.03 (+1.58)
68.52 (+1.27)
+IDI, ISI
84.33 (+1.02)
55.60 (+0.93)
83.00 (+1.40)
67.64 (+2.08)
69.97 (42.72)
PVT-Tiny
83.27
51.94
82.00
71.81
67.73
+Blank
83.32 (+0.05)
51.99 (+0.05)
81.20 (+0.20)
71.79 (-0.02)
67.74 (+0.01)
+IDI
83.60 (+1.60)
74.06 (+2.25)
69.15 (+1.42)
+ISI
84.82 (+1.55)
55.10 (+3.16)
83.40 (+1.40)
74.18 (+2.37)
68.99 (+1.26)
+IDI, ISI
84.82 (+1.55)
55.10 (+3.16)
84.20 (+2.20)
74.53 (+2.72)
70.94 (+3.21)
Eva-Tiny
87.56
64.15
83.80
71.23
72.51
Blank
87.62 (+0.06)
64. 10 (-0.05)
83.82 (+0.00)
71.24
(+0.02)
72.58 (+0.07)
+IDI
85.20 (+1.40)
72.24 (+1.01)
74.08 (+1.57)
+ISI
88.28 (+0.72)
64.99 (+0.84)
85,20 (+1.40)
72.57 (+1.34)
73.85 (+1.34)
+IDI, ISI
88.28 (+0.72)
64.99 (+0.84)
85.80 (+2.00)
72.95 (+1.72)
75.45 (+2.94)
BeiT-Tiny
74.69
49.58
79.80
71.59
70.46
+Blank
74.66 (-0.03)
49.63 (+0.05)
79.79 (-0.01)
71.58 (-0.01)
70.51 (+0.05)
+IDI
81.40 (+1.60)
72.72 (+1.13)
72.21 (+1.75)
+ISI
76.20 (+1.51)
51.03 (+1.45)
80.60 (+0.80)
72.98 (+1.39)
71.72 (+1.26)
+IDI, ISI
76.20 (+1.51)
51.03 (+1.45)
82.20 (+2.40)
73.57 (+1.98)
71.98 (+3.12)
H Additional Visual Comparisons of Integration Weight a Before and After Treating Conjunction Errors
As shown in Fig. 10, we compare additional dynamic integration weights a before and after treatment. The results align with our observations in the main text. Specifically, from Fig. 4(1, 2), it can be seen that before treatment, the dynamic integration weights of misclassified samples exhibit irregular
19

---

(4)
Comparison of dynamic integration weights among tokens before and after introducing Transformer Doctor. (1) and (3) show the integration weights before and after treatment, respectively, (2) and (4) illustrate the corresponding heatmap effects of these integration weights superimposed onto the original image, where higher brightness indicates larger weight values. The experiments were conducted on VIT and ImageNet-10.
Figure 10:
(1)
(2)
(3)
(4)
(3)
(4)
(1)
(3)
(1)
(3)
(4)
Comparison of static integration weights within tokens before and after Transformer Doctor diagnosis and treatment. (1) and (2) illustrate the static integration rules during correct predictions. (3) and (4) show the static integration weights before and after treatment, respectively. The experiments were conducted on VIT and ImageNet-10.
Figure 11:
distributions, indicating the integration of erroneous background information among tokens. However, after treatment, as illustrated Fig. 4(3, 4), the integration weights selectively integrate the correct foreground information.
20

---

image
base
+IDI
+ISI
+IDLISI
base
+IDI
+ISI
+IDLISI
Comparison of feature attribution maps before and after introducing Transformer Doctor. '+IDI', '+ISI', and '+IDI, ISI' refer to models with individually introduced dynamic integration constraint, individually introduced static integration constraint, and jointly introduced dynamic and static integration constraints, respectively.
Figure 12:
Table 4: Comparison of model accuracy with different threshold values for static integration constraints within tokens. Each row corresponds to ViT-Tiny and PVT-Tiny architectures, using the ImageNet- 10 dataset.
+ISI
Base
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
78.8
78.8
80.0
80.4
79.0
77.8
78.2
78.0
78.6
78.4
78.0
78.0
82.0
82.0
82.2
83.6
83.2
82.4
82.6
81.8
81.8
80.8
81.0
81.2
I Additional Visual Comparisons of Integration Weight z Before and After Treating Conjunction Errors
As depicted in Fig. 11, we present additional comparisons of integration weights z before and after treatment. The results are consistent with our observations in the main text. Specifically, from Fig. 11 2), it can be observed that before treatment, the static integration weights of misclassified samples do adhere to the correct integration rules, indicating the integration of incorrect category information within tokens. However, in Fig. 11(3, 4), it can be seen that after treatment, the static integration weights adhere to the correct integration rules, integrating fewer incorrect information, thus improving the model performance.
J Comparison of Feature Attribution Maps via Rollout
We also visualized the attribution maps of model predictions using Rollout feature attribution techniques, shown in Fig. 12. From the figure, it can be observed that without introducing any constraints, the model tends to focus on unnecessary background information. However, when dynamic integration constraints or static integration constraints are introduced, the model starts to focus on foreground information relevant to the categories. When both dynamic and static integration constraints are simultaneously introduced, the key information relied upon by the model for prediction aligns more closely with human understanding, i.c., it focuses more on the object itself.
21

---

K
Ablation Study on Threshold T in Eqn. (8)
We conducted ablation experiments on the threshold T that determines the intra-token integration rule in equation (8), as shown in Table 4. Using the ViT model as an example, we observed that when the threshold r increases from 0.0 0.2, the model aceuracy gradually increases from 78. exceeding the baseline. This improvement occurs because, with very low threshold, such as 0.0, integration rule includes redundant intra-token information weights, meaning that unnecessary intra-token information is integrated, which does not significantly enhance model performance.
80.4%,
Conversely, when the threshold increases from 0.3 to 1.0, the model accuracy gradually decreases from 80.4% back to 78.0%, falling below the baseline (note that the table presents the highest accuracy on the test set, the final accuracy after model training). This decline is due the overly high threshold, such as 0.8, causing the integration rule to miss certain necessary intra-token information weights, meaning that essential intra-token information is not integrated, resulting in reduced model performance.
L Ablation Study on Block Depth
Fig. 13 illustrates the effects introducing Transformer Doctor at different block depths. The figure demonstrates that the model's perfor- mance generally improves with increasing block depth when applying inter-token dynamic inte- gration constraints. When these constraints applied before the 7th block, the model's perfor-
base
+IDI - +IS
+IDLISI
91.0
80.5
80.0
79.5
mance does not show significant improvement
and can even fall below the baseline. However,
79.0
after the 7th block, the model's performance noticeably exceeds the baseline. This observa-
78.5
78.0
tion reaffirms that in the early stages, the Trans- former mixes and processes adjacent token infor- mation, while in the advanced stages, it dynam- ically and selectively integrates specific infor- mation. Thus, inter-token dynamic integration constraints are most effective at deeper blocks and can help enhance model performance.
77.5
77.0
3
5 7
9
Figure 13: Comparison of accuracy when intro- ducing Transformer Doctor in different blocks of the Transformer. Similarly, '+IDI', '+IS', and '+/DI, ISI' refer to models with individually in- troduced dynamic integration constraint, individ- ually introduced static integration constraint, and jointly introduced dynamic and static integration constraints, respectively.
Similarly, the model accuracy surpasses the baseline only when the intra-token static integra- tion constraints are applied after the 11th block. This is because the Transformer selectively and statically integrates specific intra-token informa- tion only at the advanced stages. Furthermore, the introduction of combined dynamic and static constraints results in higher model accuracy at deeper blocks compared to both the baseline and the introduction of individual integration constraints.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to Section 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to Section 7.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Please refer to Section 6. and the code provided the supplementary materi- als.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

reproducible, they Answer: [Yes]
Justification: We have provided the code in the supplementary materials and included references for the datasets and network architecture used. For more details, please refer Sections 6.1 and F.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.co
Seriides/CodeSubiiis
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //nips. wttpccccpppcccccccccccccccccccct wwwpc/ggusss/.ee/eoo /et////ooosuisseeeeeeeeee ssionPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Please refer Sections 6.1 and F.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: To ensure the reproducibility and accuracy of the results, each experimental result presented in the paper is the average of three more experimental runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
25

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Section F.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: /ttp///eerrp//eeeee//eee/ httpppplicc/chccssuidideee
Answer: [Yes]
Justification: Ethics.
We strictly adhere to the ethical standards outlined in the NeurIPS Code of
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer to Section A.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets involved in the research have been referenced accordingly.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We have provided executable code in the supplementary materials.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "uEa2VXnEgr", "reviewer": "Reviewer_ZXFK", "summary": "This paper presents Transformer Doctor, which diagnoses the issues with the Transformer attention mechanism and resolves them via several information integration hypotheses. The primary motive of the paper is to identify the source of incorrect information aggregation, which leads to erroneous predictions, and the inspiration comes from biological groundings. The paper proposed Inter-token Information Static Integration, Intra-token Information Static Integration, Heuristic Information Dynamic Integration Therapy, Rule-based Information Static Integration Therapy, Joint Therapy of Dynamic and Static Integration, and extensive evaluation. The paper improves the performance of the Transformer mechanism significantly.", "questions": "See weakness.", "limitations": "Overall, the main limitations are the use of pre-trained models as a baseline, unclear hypothesis verification on the improved attention maps, and usability across self-supervised models.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written.\n2. Each proposed component is well-motivated and discussed.\n3. The step-by-step introduction of the hypothesis from section 3 to section 5 provides a clear picture.\n4. The inter-token and intra-token diagnosis hypotheses are very promising for understanding the internals of MSHA.\n5. The improved attention maps show the potential of the proposed information integration mechanism.\n6. The integration equations are straightforward and easier to understand.", "weaknesses": "1. It appears from sec6.1 that the proposed approach requires a pre-trained model. Please correct me if not because I could not find explicit training settings in the paper. If it is trained for longer epochs, it increases the training time. \n2. Since the model is pre-trained, even though attention maps are improving, it is unclear whether the gains came from further fine-tuning or information integration. In other words, how much the integration mechanism contributed to the gains?\n3. In Figure 4, the regions in red always lie on the object, i.e. without and with transformer doctor applied. How can it be inferred that the attended region after using the doctor mechanism is the sole reason for improved performance? Similarly Fig12.\n4. Selecting the gradient based on the actual class label is possible with supervised learning. Hence, how can this method be applied to DINO-like self-supervised methods, given that DINO also uses a Transformer?\n\nNote: I am open to adjusting ratings if my concerns are resolved, especially 1,2,3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your diligent review and comments. We are pleased that you found the paper well-written and appreciated our step-by-step introduction of the proposed methods and hypotheses. Your recognition of our diagnostic hypotheses and the potential of the proposed information integration mechanism is highly encouraging. Below are our responses to each of your comments (each of your comments is highlighted in italics).\n\n> Q1: *It appears from sec6.1 that the proposed approach requires a pre-trained model. Please correct me if not because I could not find explicit training settings in the paper. If it is trained for longer epochs, it increases the training time.*\n> \n\nWe apologize for any confusion caused. The proposed Transformer Doctor indeed diagnoses and treats an already trained model, i.e., a model that has undergone normal training and fitting, as mentioned on lines 261-263 of the paper. Similar to most interpretability methods based on trained models, the proposed information integration hypothesis assumes that the trained model exhibits error mechanisms akin to those in biological vision (lines 126-129 of the paper). Specifically, in the diagnostic and therapeutic experimental setup, the pre-trained model was trained for 300 epochs, and continuing training does not improve performance (Table 3 in Appendix G). During the diagnosis and treatment phase, we retrain for an additional 100 epochs using the same training settings, which yields excellent results without significantly increasing the overall training time. We will emphasize this part in the paper to avoid any further confusion.\n\n> Q2: *Since the model is pre-trained, even though attention maps are improving, it is unclear whether the gains came from further fine-tuning or information integration. In other words, how much the integration mechanism contributed to the gains?*\n> \n\nThank you for your comment. The pre-trained model used in our study is a fully trained and fitted model, so further fine-tuning does not enhance its performance. However, significant improvements in accuracy are observed only after diagnosis and treatment with Transformer Doctor. This can be seen in Table 3 of Appendix G. For instance, in the case of ViT-Tiny on ImageNet-10, the baseline accuracy is 78.90%. After further fine-tuning (\"*+Blank*\"), the accuracy remains almost unchanged. However, with the application of information dynamic integration constraints (\"*+IDI*\"), information static integration constraints (\"*+ISI*\"), and both dynamic and static integration constraints (\"*+IDI, ISI*\"), the accuracy improves by 1.40%, 1.60%, and 2.00% respectively. Therefore, the performance gains are primarily due to the integration mechanism improvements rather than further fine-tuning. Similarly, the improvements in attention maps are also mainly attributable to the integration mechanism rather than further fine-tuning.\n\n> Q3: *In Figure 4, the regions in red always lie on the object, i.e. without and with transformer doctor applied. How can it be inferred that the attended region after using the doctor mechanism is the sole reason for improved performance? Similarly Fig12.*\n> \n\nThank you for your valuable comments. It is true that in Figures 4 and 12, some images show red regions on the object both before and after applying Transformer Doctor. However, after applying Transformer Doctor, the red regions on some objects become more focused (e.g., the \"orange\" in Figure 12), indicating that the model is attending to more useful features. Additionally, in some images, the red regions initially lie on the background, but after applying Transformer Doctor, they shift to the object (e.g., the \"leopard\" in Figure 4), suggesting that the model is now focusing on information most relevant to the prediction task. Furthermore, referring back to Table 3 in Appendix G, the performance improvement does not come from further fine-tuning, indicating that it is solely due to the proposed Transformer Doctor mechanism. We hope these explanations address your concerns.\n\n> Q4: *Selecting the gradient based on the actual class label is possible with supervised learning. Hence, how can this method be applied to DINO-like self-supervised methods, given that DINO also uses a Transformer?*\n> \n\nThis is an excellent question. While our method has been validated on most commonly used Transformer architectures, it is indeed challenging to directly apply the proposed method to self-supervised learning methods. As you correctly pointed out, the calculation of gradients in our method requires the availability of actual labels. In methods like DINO, which update parameters through contrastive learning and momentum updates for visual representation learning, it is difficult to compute gradients related to specific true labels, significantly limiting the applicability of our method.\n\nA potential approach is to utilize the intrinsic mechanisms within self-supervised learning methods to find alternatives to the gradient-based approach. For instance, in DINO, the contrastive learning signal between the teacher and student networks can be used to estimate the importance of different heads in the Transformer. This signal could serve as a proxy for the gradients in our method.\n\nThank you for raising this important question again. Exploring the integration hypothesis and diagnostic and treatment methods for more complex tasks like self-supervised learning will be a direction of our ongoing research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FYO7COoy3J", "reviewer": "Reviewer_BM6N", "summary": "This study introduced a framework, namely Transformer Doctor, to reduce internal errors, e.g. conjunction errors, in a general vision transformer model. Building upon the information integration hypothesis, the proposed method performs several constraints, including heuristic dynamic constraints and rule-driven static constraints, to enhance information integration at higher layers. The experiential results are conducted on five classification datasets using seven small-scale vision transformer models.", "questions": "See above weakness.", "limitations": "As I can see, this paper has no potential negative societal impact.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "-- This study investigates the topic of improving vision transformers, which seems interesting and of practical importance. \n\n-- The proposed method is inspired by solutions in biological vision. The solution might be reasonable.", "weaknesses": "-- This is a pure empirical paper with no theoretical results. However, the experimental results are insufficient to fully convince the reviewer.\n* The vanilla transformer results reported in Table 1 do not align with their original paper. e.g. the DeiT-Tiny is reported at 72.2% on ImageNet, which is even higher than the improved results (66.8% $\\to$ 70.6%).\n* This study is only evaluated on small-scale vision transformers, with no results presented for large-scale versions. \n* The evaluation is only conducted on 10 samples per class (Appendix F). The sample size seems insufficient.\n* Table 1 can be further improved by presenting results with both the vanilla and blank training.\n\n-- The reviewer is unclear about the potential mechanism/logic/reason of the proposed method on the following questions\n* why the proposed method can reduce the conjunction error\n* To what extent (fully/partially) can the proposed reduce the error\n* Besides the information integration hypothesis, do you rely on other assumptions?\n* Is the information integration hypothesis valid in vision transformers, especially for large-scale versions?\n\nThe above questions might be potentially solved by adding theoretical analysis or more detailed explanations. \n\n-- This paper has many typos, even in the result presentation (e.g. Table 1 BeiT/Imagenet-1K cell).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your diligent review and comments. We are pleased that you find this research on improving Vision Transformer both interesting and practically valuable. We are also glad that you consider our biologically inspired approach to be reasonable. Below are our responses to each of your comments (each of your comments is highlighted in italics).\n\n> Q1: *The vanilla transformer results reported in Table 1 do not align with their original paper. e.g. the DeiT-Tiny is reported at 72.2% on ImageNet, which is even higher than the improved results (66.8% → 70.6%).*\n> \n\nWe apologize for any confusion caused. To achieve the best possible performance for models such as ViT and TNT within our limited computational resources, we adhered to the following detailed experimental settings:\n\nWe used data augmentation methods including Auto Contrast, Equalize, Invert, Rotate, Posterize Increasing, Solarize Increasing, and Solarize Add. The AdamW optimizer was employed with a momentum setting of 0.9, weight decay of 0.05, and epsilon of 1e-8. The learning rate scheduler used was CosineLRScheduler with an initial learning rate of 1e-2, a minimum learning rate of 1e-5, no warmup steps, a cycle limit of 1, and T_max set to 300. The training batch size was 256, and the models were trained for 300 epochs.\n\nThese settings differ from the commonly used configurations for DeiT, such as a total batch size of 1024 and an initial learning rate of 1e-3. However, it is important to emphasize that this does not impact the fairness of exploring the internal error mechanisms of the models or addressing their errors. During the experimental phase, the comparison settings in the paper are sufficiently fair; except for the number of training epochs for the treated models being fewer, the experimental settings for the treated and baseline models are identical. The experimental results demonstrate that the proposed method has substantial potential for correcting model errors and enhancing performance (as shown in Table 3 in the appendix). We understand your concerns and will add a more detailed description and further experiments in the paper to clarify this matter. We sincerely hope this response addresses your concerns.\n\n> Q2: *This study is only evaluated on small-scale vision transformers, with no results presented for large-scale versions.*\n> \n\nThank you for your comment. Transformer Doctor is indeed applicable to large-scale Vision Transformers. To demonstrate this, we have additionally conducted preliminary experiments using ViT-Large on ImageNet-10. The results are as follows:\n\nBase: 61.30; +Blank: 61.50; +IDI:62.80; +ISI:62.50; +IDI, +ISI: 62.90;\n\nThese results indicate that Transformer Doctor remains effective for large-scale ViTs. The reason is that for any Vision Transformer architecture, such as ViT, the fundamental structure is similar across different sizes, with variations primarily in the number of blocks, heads, and dimensionality of hidden features. Each block consists of MHSA and FFN components. The information integration hypothesis upon which Transformer Doctor is based comprehensively covers both MHSA and FFN (Sections 3.1 and 3.2). Thus, the size of the model does not affect the fundamental efficacy of Transformer Doctor in enhancing model performance.\n\n> Q3: *The evaluation is only conducted on 10 samples per class (Appendix F). The sample size seems insufficient.*\n> \n\nThank you for your comment. In fact, the amount of data used for evaluation is more than sufficient and far exceeds 10 samples per class.  The \"10 samples per class\" mentioned refers not to the evaluation data but to the dynamic information integration constraints during training. This indicates that our proposed dynamic integration method requires only a very small amount of annotated foreground masks per class to achieve good results.\n\nOf course, having more foreground annotations during training can provide more information for dynamic integration constraints, thereby further improving model performance. However, considering practical applications, we opted to use a minimal amount of foreground annotations in our experiments to demonstrate the practicality of our proposed method.\n\n> Q4: *Table 1 can be further improved by presenting results with both the vanilla and blank training.*\n> \n\nThank you for your valuable suggestion. Due to space constraints in the main text, detailed results for both vanilla and blank training have been presented in Table 3 of Appendix G in our original manuscript. As observed, blank training shows almost no improvement over vanilla training, indicating that the performance gains are largely attributed to the proposed method. We will make an effort to include these results in Table 1 of the main text if space permits.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RGFoMV7ilL", "reviewer": "Reviewer_3nqh", "summary": "Inspired by the information integration mechanisms and conjunction errors in the biological visual system, this paper investigates the error mechanisms within Transformers. Through a comprehensive analysis and experimental validation of the computational processes of the two core modules of Transformers, MHSA and FFN, the authors introduce the Transformer Doctor framework. The principal components of this framework are as follows:\n1. Diagnosis: The paper identifies dynamic information integration among tokens within MHSA and static information integration within tokens in FFN, as well as conjunction errors occurring during the integration process.\n2. Treatment: To mitigate the conjunction errors identified in MHSA and FFN, the paper proposes heuristic dynamic information integration constraints for MHSA and rule-based static information integration constraints for FFN.\n\nTo validate the efficacy of Transformer Doctor, the authors performed extensive quantitative and qualitative analyses on various Vision Transformer architectures. The findings indicate that Transformer Doctor effectively reduces conjunction errors in Transformers and enhances overall performance.", "questions": "See weakness", "limitations": "The paper has already discussed its limitations and potential impacts.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The investigation into the error mechanisms of Vision Transformers presented in this paper is uniquely motivated. The authors provide a novel perspective by drawing an insightful connection between error mechanisms in biological vision and machine vision. The paper conducts in-depth analyses of both MHSA and FFN, uncovering intriguing phenomena related to dynamic and static information integration, respectively.\n\n2. The paper offers a comprehensive and detailed review of existing interpretability methods. The proposed approaches for diagnosing and treating Transformer errors are rigorous and persuasive. Extensive experiments, featuring abundant quantitative results and qualitative visual analyses, convincingly demonstrate that the proposed methods effectively mitigate errors and enhance model performance.\n\n3. The structure of the paper is well-organized, clearly articulated, and highly accessible. Each section is meticulously arranged, providing a systematic and comprehensive flow from background and motivation to methods and experimental results.\n\n4. This paper's comprehensive investigation into the error mechanisms of Vision Transformers addresses a relatively underexplored area in the existing literature. Drawing inspiration from error mechanisms in biological visual systems, the proposed methodologies for diagnosing and rectifying Transformer errors provide substantial insights. The elucidated integration mechanisms and identified error phenomena within Transformers are not only of considerable interest but also highly enlightening for the field.", "weaknesses": "1. The visualization from vertical lines to diagonals in the attention matrices presented in Fig. 2(a,c) lacks sufficient clarity. This visualization is essential for validating the authors' information integration hypothesis in MHSA. It is recommended that the authors refer to similar visualization results in studies such as [1].\n2. In Table 1, the proposed method does not demonstrate significant improvements on certain datasets, particularly on the CIFAR-10 dataset. Regarding this phenomenon, the authors need to provide a detailed analysis.\n3. In the treatment of dynamic information integration in MHSA (Eqn. (6)), it is noted that not all datasets contain annotated foregrounds. The necessity for manual foreground annotation presents a significant challenge to the practical implementation of this method.\n4. minor issues: \\\n   a.  This paper lacks a detailed introduction to Transformers. The authors should provide a conceptual overview and offer clearer explanations of certain variables in the equations. For instance, the meanings of $(a_{ij})$ in Eqn. (2) and $(z_{im})$ in Eqn. (4) should be explicitly stated.\n\n   b. If Eqn. (4) is correct, the symbol labeled as $(X)$ in Fig. 1 should be $(Y)$.\n5. In Eqn. (8), the authors constrain the static information integration in FFN by specifying aggregation rules. However, it is worth considering whether the number of samples used to establish these rules could also impact the final results.\n6. Lack of explanation why the performance improvements under other computational forms are not as significant in Table 2.\n\n\n[1] Trockman, Asher, and J. Zico Kolter. \"Mimetic initialization of self-attention layers.\" International Conference on Machine Learning. PMLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and comments. We are pleased that you find our work combining Transformers with biological vision error mechanisms to be novel and that you find our methods and findings insightful and inspiring. Below are our responses to each of your comments (each of your comments is highlighted in italics).\n\n> Q1: *The visualization from vertical lines to diagonals in the attention matrices presented in Fig. 2(a,c) lacks sufficient clarity. This visualization is essential for validating the authors' information integration hypothesis in MHSA. It is recommended that the authors refer to similar visualization results in studies such as [1].*\n> \n\nThank you for pointing this out. The lack of clarity in Figures 2(a) and 2(c) is due to compression artifacts in the PDF version of the paper, which diminished the visibility of the diagonal and vertical lines. As suggested, similar observations can be found in the study you referenced [1]. Additionally, similar visualizations are presented in [2], but their focus is on parameter initialization or architectural design, which differs from our study on Transformer error mechanisms.\n\n[1] Trockman, Asher, and J. Zico Kolter. \"Mimetic initialization of self-attention layers.\" International Conference on Machine Learning. PMLR, 2023.\n\n[2] Chang, Shuning, et al. \"Making vision transformers efficient from a token sparsification view.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2023.\n\n> Q2: *In Table 1, the proposed method does not demonstrate significant improvements on certain datasets, particularly on the CIFAR-10 dataset. Regarding this phenomenon, the authors need to provide a detailed analysis.*\n> \n\nIndeed. It is important to note that Transformer Doctor shows less improvement on smaller datasets such as CIFAR-10 compared to larger datasets. There are two potential reasons for this phenomenon. First, for smaller datasets, errors in Transformer recognition are not solely due to conjunction errors; they may also arise from insufficient feature extraction at early network stages (lines 283-289 of the paper). Second, during the treatment of Transformers, the small image size of CIFAR-10 limits the effectiveness of dynamic aggregation constraints. Consequently, we only employed static aggregation constraints for treatment. In contrast, larger datasets benefit from both dynamic and static aggregation constraints, resulting in better performance. More detailed results are provided in Table 3 of Appendix G.\n\n> Q3: *In the treatment of dynamic information integration in MHSA (Eqn. (6)), it is noted that not all datasets contain annotated foregrounds. The necessity for manual foreground annotation presents a significant challenge to the practical implementation of this method.*\n> \n\nWhile we acknowledge that not all datasets come with annotated foregrounds, it is important to note that the amount of foreground annotation required for dynamic information integration treatment is quite minimal. For instance, in our experiments using ImageNet-S, the number of images with foreground annotations in the training set is 9,190 out of a total of 1,200,000 images. Despite this small proportion, excellent results were achieved.\n\nMoreover, our further experiments indicate that selectively annotating low-confidence images in the training set is more beneficial compared to randomly annotating some images. Therefore, in practical applications, one can selectively annotate a very small number of low-confidence images (e.g., 5-10 per class) in the training set to perform the Transformer treatment effectively.\n\n> Q4: *This paper lacks a detailed introduction to Transformers. The authors should provide a conceptual overview and offer clearer explanations of certain variables in the equations. For instance, the meanings of ($a_{ij}$) in Eqn.(2) and ($z_{im}$) in Eqn.(4) should be explicitly stated.*\n> \n\nThank you for your suggestion. We provide an explanation of $\\mathbf{a}$ on line 129 of the paper, where $\\mathbf{a}\\_{ij}$ denotes the weight associated with the $j$-th token when integrating information to form the $i$-th token in inter-token information integration. Additionally, $\\mathbf{z}$ is explained on line 154 of the paper, where $\\mathbf{z}\\_{im}$ represents the weight corresponding to the $m$-th dimension when integrating information to form the $i$-th token in intra-token information integration. We will include these clarifications in the Methods section of the revised manuscript.\n\n> Q5: *If Eqn.(4) is correct, the symbol labeled as $X$ in Fig.1 should be $Y$.*\n> \n\nWe apologize for this error. We have corrected the label and thoroughly reviewed the entire manuscript to ensure that all symbols are accurate.\n\n> Q6: *Lack of explanation why the performance improvements under other computational forms are not as significant in Table 2.*\n> \n\nThank you for your question. In Table 2, the computational forms that show the best performance improvements all involve gradient-based methods. As described on lines 223-225 of the paper, gradients help differentiate the importance of each head in MHSA, leading to more accurate multi-head integration weights $\\hat{a}$. Other computational methods, such as using the minimum, maximum, or average across all heads, result in less accurate multi-head integration weights $a^h$ because they include weights from less important heads. Consequently, these methods do not achieve as good results when used for dynamic integration constraints.\n\nSimilarly, as noted on lines 238-240 of the paper, gradients establish a connection between integration weights $z$ and specific classes. This results in more precise integration weights $\\hat{z}$ for each class and avoids the issue of constraining redundant dimensions during static integration constraints. Thus, gradient-based methods yield better performance improvements. We will include this explanation in the revised version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pOnP7WrDDf", "reviewer": "Reviewer_KQuQ", "summary": "This paper proposes a vision transformer diagnosing and treating framework, namely Transformer Doctor, to reveal the problems that bring about negative impact on the network performance and fix them. The paper firstly proposes the information integrating hypothesis, which argues that transformers do information refining and integrating at the lower and higher layers respectively. Based on this hypothesis, the inter-token information dynamic integration and intra-token information static integration mechaisms are designed to show the inner mechanisms of ViTs and help to treat them.", "questions": "N.A.", "limitations": "The authors consider the limitations of the method in the part of Conclusion.", "rating": 9, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. This paper designs a possible way to explain why a transformer cannot work well and try to fix the potential problems to improve network performance.\n2. The information integration hypothesis is proposed, and two situations (self-attention and fully-connected layers) are analyzed. \n3. Based on the information integration hypothesis, a transformer diagnosis and treatment method is proposed. \n4. The experimental results on several databases can support the proposed method.", "weaknesses": "1. The evidence of the correctness of the information integration hypothesis is not strong enough, it is better to consider to prove the hypothesis in mathematic way", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback on this work. We are pleased that you highlighted the core of our research, which is designing a potential method to explain why Transformers may not perform well and attempting to address these issues to improve network performance.\n\nWe acknowledge that there are still some imperfections in our manuscript, but we have actively addressed these issues with the aim of improving the work. Below are our responses to your main comments (each of your comments is highlighted in italics).\n\n> Q1: *The evidence of the correctness of the information integration hypothesis is not strong enough, it is better to consider to prove the hypothesis in mathematic way*\n> \n\nThank you for your review and valuable feedback. We understand your concern about the mathematical proof of the information integration hypothesis. However, our study is primarily empirical, aiming to validate the hypothesis through experimental evidence rather than mathematical derivation.\n\nSpecifically, the motivation for this work stems from error mechanisms observed in biological vision systems. We chose an empirical approach to explore the practical effects and applications of the information integration hypothesis, which is a commonly used and accepted method in related research fields. In validating of the information integration hypothesis, we conducted extensive experiments and data analyses, including numerous qualitative analyses (as detailed in Sections 4.1, 4.2, Appendix B, and Appendix D) and thorough quantitative analyses (as shown in Appendix A, Appendix C, etc.). Additionally, in the error treatment based on the information integration hypothesis, we performed extensive qualitative analyses (in Sections 6.3, Appendix H, Appendix I, Appendix J) and substantial quantitative analyses (in Sections 6.2, Appendix G, Appendix K, Appendix L). These experiments provide substantial evidence for the effectiveness of the information integration hypothesis.\n\nNevertheless, we highly value your suggestion and will emphasize in the discussion section of the paper why we chose an empirical approach and explain its advantages and limitations relative to mathematical proof. Additionally, we will attempt to supplement the theoretical background and relevant mathematical models to further support our hypothesis. We hope this response addresses your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers ZXFK, BM6N, 3nqh, and KQuQ,

Thank you for your diligent reviews and constructive feedback. We particularly appreciate your recognition of the novelty and insightfulness of our work and are pleased that you find our approach of integrating error mechanisms from biological vision with Transformers both reasonable and inspiring.

Through our detailed responses to each of your comments, we believe the paper has significantly improved. We have addressed each comment individually and collected the following common concerns raised by the reviewers. We hope our responses address your concerns and would be grateful if you could consider raising your scores. If you have any additional questions, we are more than happy to engage in further discussion.

> Reviewer BM6N pointed out that the number of foreground annotations used for evaluation is insufficient, while Reviewer 3nqh noted that not all datasets have foreground annotations, which limits further usage.
> 

We apologize for the confusion caused to both reviewers. To address Reviewer BM6N's concern, it is important to clarify that the foreground annotations are used for training, not evaluation. Considering that not all datasets have foreground annotations, our method achieves good performance with only a small number of foreground annotations. Regarding Reviewer 3nqh's concern, we emphasize that during the dynamic information integration treatment, very few manually annotated foregrounds are required to achieve significant improvements, which does not hinder the practical applicability of our method. For more details, please refer to our responses to each of your individual comments.

> Reviewer BM6N raised concerns about training baselines and the validation of large-scale Transformers.
> 

We fully understand the reviewer's concerns. In response to these issues, we have provided detailed explanations and added relevant experiments below the reviewer's comments. It is important to emphasize that the experiments in the paper were conducted under fair experimental settings for comparison and analysis. We sincerely hope that the corresponding responses address the reviewer's concerns effectively.

> Reviewers ZXFK and 3nqh both raised issues related to gradients. One mentioned the applicability of gradients in self-supervised learning methods like DINO, while the other questioned why gradient-based methods are more effective.
> 

We apologize for any confusion caused. In the responses to the reviewers' comments, we have provided detailed explanations on the role of gradients, their applicability in DINO, and potential alternatives to gradients. We believe these responses address the reviewers' concerns.

> Reviewers BM6N and 3nqh both pointed out several typos in the paper.
> 

We appreciate the reviewers’ attention to detail in identifying these typos. We have corrected these errors and thoroughly reviewed the entire paper.

In addition to addressing the above issues, detailed responses to each reviewer’s comments can be found below their respective feedback. Thank you once again to all the reviewers for their meticulous review and valuable comments, which have significantly improved the paper.

Meta-review: The reviewers unanimously support the paper. A couple of reviewers are concerns with DeiT evaluation settings and fairness in comparison. This concern still remain partly unaddressed.

Overall, the paper looks valid and it has its strength, despite the remaining concern.
Decision: Accept (poster)