Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel approach towards improving the explainability of vision transformers. Specifically, the paper proposes updating each component in the model using the B-cos transform, which is designed to increase the alignment of the inputs and the weights. The paper builds on a previous state-of-the-art work that introduced the B-cos transform on DNN/CNNs and extends it to vision transformers. Comprehensive experiments are conducted comparing the proposed approach with several state-of-the-art explainability methods.
strength_and_weaknesses: Pros
	- The proposed method is simple and seems easy to implement with the provided details.
	- Strong benchmark performance improvements across multiple explainability metrics. Further, the attribution maps of the model are highly detailed and have very low noise.

Cons
	- This paper is a direct extension of "B-cos Networks: Alignment is All We Need for Interpretability" by Bohle et al, which proposed the B-cos transform and implemented it on MLPs and CNNs. In this paper, that transform is applied to the vision transformer. However, the main contribution of this paper is incorporating the B-cos transform in the attention layers and the positional encodings. It's incorporation in the MLP/CNN layers is directly taken from the original paper. This makes the architecture seem a bit derivative.
	- The paper proposes using max-out network in equation 12. However, that would double the number of parameters in the model. It would be helpful if some comments are added regarding the increased model size. It would also be helpful to know if removing the max-out layer will significantly impact the final results.
Most of the results compared the model with the state-of-the-art visualization models. Given this model is a significant architectural modification of the original vision transformer, it would be really helpful to compare the results of the model with those of the vision transformer on other benchmark tasks.
clarity,_quality,_novelty_and_reproducibility: The paper was very well written and was very easy to follow. The authors mentioned that the code will be released upon publication. The architecture is an extension of the model proposed in "B-cos Networks: Alignment is All We Need for Interpretability" and seemed a bit derivative.
summary_of_the_review: I don't think it's a strong accept due to the slightly derivative nature of the work, but overall, the paper extends a very novel idea from DNN/CNNs to vision transformers and supports this with strong benchmark results. Due to this, I think its marginally above the acceptance threshold.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to replace the modules of transformer with dynamic linear operation so that it is more explainable. It follows the guideline of BCos Networks but adopts transformers on the top of DenseNet features. The method is evaluated on ImageNet with localization and perturbation metric. It claims to have to be more interpretable and perform competitively to ViT.
strength_and_weaknesses: Strength:
+ The idea of improving interpretation of transformer is interesting.
+ The paper is well organized and easy to follow.
+ The method is well evaluated and the results are well presented.

Weakness:
- My first concern is the claim of Bcos-ViTs. The authors are using transformers on the top of DenseNet features which is not ViT. A ViT is trained directly on images by considering each patch as a token. The method is more like Bcos Transformer, not Bcos-ViT. 
- The whole explanation principle is exactly the same as the Bcos Networks (CVPR2022) which makes this paper more like an extension from deep network to transformer. What is the new challenge here for explaining DenseNet+transformer using Bcos idea as compared to explaining a DenseNet?
- The method itself cannot be adopted to explain existing ViT models like [Chefer et al. (2021)], but has to be re-trained to gain interpretation ability. But this paper does not show how the ViT would performance if all the transformers are replaced with the proposed block. It is very misleading to claim DenseNet+Transformer as ViT in Fig 5. How would a standard ViT perform if the proposed Bcos method is used?
clarity,_quality,_novelty_and_reproducibility: The idea of extending the Bcos Networks to transformer is new, but could be incremental. The presentation and writing are good.  The link of the code cannot be opened.
summary_of_the_review: Overall, I feel that the contribution does not meet the bar of ICLR and some claims are very misleading to readers. I would recommend weak reject for the current version.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new Transformer architecture for better holistic explainability. The architecture generates explanations that reflect all the model components instead of only the attention layers. To do so, the authors take explicit designs for each of the Transformer modules so that they can be summarized by a single linear transformation. The authors perform the main and ablative experiments on ImageNet to evaluate the effectiveness of the proposed methods.
strength_and_weaknesses: Strength:

•	The qualitative results look promising. The proposed method generates visualizations with richer details than the baseline methods shown in Fig. 7 and Fig. A1. 
•	The quantitative results look impressive. The proposed B-cos Transformer have higher localisation and perturbation scores than the baseline methods under the same model accuracy. Compared to the conventional ViT models of the same depth, B-cos Transformer has better overall performance across different depths. 
•	The paper is well-written and easy to follow.

Weakness:

•	The authors argue that B-cos Transformer performs at least as well as the baseline ViTs under the same number of backbone layers in Section 5.1. However, the experimental setting is unfair without comparing the computational complexity to the baseline methods, especially when the authors replace the conventional tokenization module with a pre-trained B-cos DenseNet-121. Though deriving more explainable models, the proposed model should be toned down if they are too heavy to deploy. This is the same for Fig. 6, which also lacks comparisons of model complexity vs. Localisation/perturbation. 

•	I am also confused about where the good performance of B-cos Transformer comes from. The authors claim that the positional information design largely improves performance. However, the effectiveness of the positional information cannot be concluded from Fig. 5, with many other components modified. To fully explain the performance gain, comparisons have to be done by attaching the B-cos positional information to the conventional ViTs.

•	I am also curious about how the third column of Fig. 7 is generated, as the visualizations do not seem to be normalized.

clarity,_quality,_novelty_and_reproducibility: The overall quality of this paper is good and the technical part is novel. The authors have not provided the code base in terms of reproducibility.
summary_of_the_review: There are many confusing points that need to be addressed.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces the B-cos Vision image Transformer (ViT), which is explainable by design. The method consists in adapting the B-cos models for ConvNet to transformers networks, especially by adapting the approach to take into account normalisation layers, attention layers, and positional encodings. Experiments have been conducted on ImageNet, and show that B-cos ViTs perform on par with ViTs, while providing a more accurate explanation.
The paper initially received one reject (3), one borderline reject (5), and one borderline accept recommendation (6). The reviewers overall found the paper clear and well written, but their concern related to the novelty of the approach, clarifications on experiments, and on the mismatch in architecture between the B-cos ViT and vanilla ViT. The rebuttal brings additional results related to the impact of maxout, multiplicative attention, and statements on paper claims and contributions. After the rebuttal, all reviewers stuck on their initial grades. 

The AC carefully reads the submission and discussions. The AC considers that the paper is clear and well presented, and that introducing an new explainable by design ViT is interesting even if it does not strictly follow the vanilla ViT architecture. However, the AC also considers that the B-cos ViT is a relatively direct adaptation of the B-cos ConvNets (Böhle et al., 2022). In addition, the provided explanation from B-cos ViT are qualitatively similar to their convolutional counterpart, and no comparison is provided to analyse if the explanation can reveal systematic differences between ConvNet and transformers models. Regarding B-cos ViT architecture, the specific impact of the multiplicative attention is very large while it is small for vanilla ViT, which calls for more investigations. 
Therefore, the AC recommends paper rejection but highly encourages the authors to resubmit their work based on the reviews' feedback. 
justification_for_why_not_higher_score: - Incremental work with respect to (Böhle et al., 2022). 
- No analysis of how the explanation can shade light on the difference between ViT vs ConvNets predictions
- Consolidated experiments
Decision: Reject