Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper proposes a loss regularization method for training Vision Transformer (ViT) through matching the attention maps of ViT with the activation map of a pre-trained CNN. The authors aggregate ViT's attention maps from different layers into the number of CNN's activation maps and then use their L2 norm as the additional loss regularization of training ViT. It is shown that the proposed method can outperform the selected baseline on ImageNet when the number of training samples is small.
strengths_and_weaknesses: Strengths
The idea of matching attention maps of ViT with the activation maps looks interesting, and the motivation behind this seems reasonable.

Weakness
1. How do the authors learn the w_c in Eq. 4, is it end-to-end trainable with Eq. 6 or learned separately? Besides, do the attention map (A_c) and activation map (B_c) in Eq. 5 have the same shape? If not, how do you address it?
2. The computation complexity of Eq. 4 is also a concern. Since w_c is posed for each pixel in A_(m,n), the complexity of w_c will grow exponentially as P or the number of layers in ViT grows. How will it perform when facing larger or deeper ViT models?
3. I have checked the manuscript, but I cannot find the initialization value of λ in Eq. 6. Please include it and justify the value choice as it is an important hyper-parameter.
4. The current description of the experimental setting and results, in my opinion, is difficult for the readers to understand. Please re-organize it - first describe the dataset and different experiment settings, and then explain the results one by one. 
5. I don't think Figure 2 is sufficient to support the two hypotheses in 4.1. Besides, the selective attention link should be discussed in more detail, e.g. why it can achieve better results than the full link.
6. Please clarify whether the baseline model in Table 4 has been trained with a teacher model or not. If not, the comparison may not be fair. Besides, it seems the performance cannot outperform the ConViT with the full train set, and therefore the generality to large-scale data is in doubt.

Overall, although the idea of this paper is interesting, the methodology is not explained or designed properly, the descriptions of the current experimental settings and results are difficult to read, and also the achieved performance is not convincing. I don't think this paper can be accepted, and I tend to vote for rejection.
questions: See weaknesses above.
limitations: I didn't check but I don't think this work will lead to a potential negative societal impact
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper introduces a novel regularisation method to train ViT. Specifically, they rely on the activation features from a convolutional neural network to transfer the knowledge and augment the training of the vision transformer. Furthermore, they discuss how to select which links to keep during training in order to make the learning more efficient. Finally, they ablate and evaluate the approach on image datasets such as ImageNet and CIFAR.
strengths_and_weaknesses: **Strengths**:

- S1. The approach is simple and effective. Authors join the advantages of CNNs and ViT with their augmentation procedure in order to make learning more efficient. I really like the simplicity of the approach, which also would make it more likely to be used by the community.

- S2. Analysing the most important links is also one extra step in terms of efficiency. The paper does a good job introducing the overall intuition and the general method and then going into link selection.

- S3. The results show how the approach can be specially useful for low data regimes.

- S4. Most of previous works using transformers specially rely on very large datasets while this paper tackles the problem of training in a low data regime.

- S5. Authors provide sufficient experimental evidence to answer the main questions in the paper. The ablation study is complete and useful for the reader.

**Weaknesses**
- W1. In my opinion the paper avoid the issue of the additional overhead of running the CNN alongside the transformer training. I think the comparison table should take that into account and report FLOPS or runtime for a complete picture.

- W2. The methodology has only been used for classification. However, it seems like other task which require for awarness of the space would be a better fit for such a paper. I wonder if authors have considered evaluating on additional task.

- W3. Authors evaluate on CIFAR and ImageNet, which are very saturated dataset at this point. Although I think the point of the paper is valid, I would appreciate additional evaluations.

- W4. I am missing some qualitative comparison on *why* the techniques improve data efficiency. What is different in the learning curves? How do the attention weights change? 

- W5. Have the authors played a bit with the CNN architecture? I wonder if they could safe computation by running a very basic model. 
questions: My concerns are listed in the weaknesses section. I would expect authors to discuss:

- Include additional metrics in the table for a fair comparison with the baselines. What is the effect of the cost of running the CNN? 

- Have the authors considered other task beyond classification? 

- Have the authors considered using other datasets?

- Have the authors considered including more qualitative analysis of the learned model compared to the baseline?
limitations: Although in the form authors claim that Section 6 discussed limitations, I do think it would be necessary to extend this discussion for the paper to be accepted. I see Section 6 as a conclusion but no specific mention to the limitation is done.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces a new method to increase data efficiency for Vision Transformers (VTs). The idea of the method is that Adaptive Attention Links (AAL) between the convolutional filters of a CNN and between the attention heads of the VT are trainable parameters which connect these feature maps between the teacher CNN and the student VT, with only a small parameter increase. This way, the locality inductive bias can be learned from the teacher model, decreasing training time. The paper confirms previous results that higher layer attention maps in a VT contain more high-level semantic information than a CNN, but that other layers roughly correspond to the same type of hierarchical information. Extensive experiments show that the method improves on previous distillation approaches between a CNN and a VT. 
strengths_and_weaknesses: S1: The idea of the paper is novel, to the best of my knowledge.
S2: The idea is clear and conceptually appealing.
S3: The paper presents extensive experimentation to support its conclusions (a number of different datasets and variants of models). 
S4: The paper promotes better data efficiency for VTs, which is a well-known issue.
S5: The paper presents ablations and one example of repeated runs which show stable results (this seems reasonable in order to save computation).
S6: the method is flexible with regards to the VT student architecture, any model containing hierarchical attention maps can be used, as well as wrt the teacher model: any CNN containing hierarchical feature maps can be used. The down-stream task of the teacher model does not matter.

W1 The main weakness of the paper, in my view, is its clarity. This can be straightforwardly fixed by doing a more thorough language check, for example by using Grammarly or similar. Many sentences are hard to parse, eg lines 320-321, 234-239. Furthermore, the captions to tables and figures should be more informative which would help the reading a lot, right now they are not possible to get the gist from without finding their reference in the text. Also, captions should not be boldfaced, but that is a detail of course. 

W2, W3: Other weaknesses are lacking mention of how the hyper parameters lambda were chosen, and a lacking discussion of the potential weaknesses of the method— this should be easily amendable.

W4: No code is released.
questions: 1. How was lambda = 0.99 and 0.98 chosen? (L288)
2. In lines 37-43, do you mean source dataset instead of target dataset?
3. L175-176, did you find an improvement when using channel-wise attention maps compared to some aggregate version of attention maps?
4. What is your interpretation of the augmented attention links, what do they represent? Is it mainly about their increased flexibility of representing the CNN maps?
5. Are the X/Y results in the tables always teacher/student?
6. What does attention maps “at a similar level” mean? At a similar layer depth?
7. L266, when you say remarkable results, which results are you referring to then? If you say a strong word such as remarkable it should be clear what you mean, apart from that the result should be strong too, of course. 
8. L327-332: do you compare self-supervised approaches to your classically supervised approach here? Or do you make some kind of SS version of your approach? This is not clear

limitations: The choice of the hyper parameter lambda could be more accounted for (did it include tuning, etc). The authors do not state any potential limitations of their work, which seems a little worrying, perhaps, in terms of transparency. There are always limitations. However, if this can be added the work is sound.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a new knowledge distillation method to improve the training efficiency of ViT. More concretely, the weighted average of the attention maps in the student ViT model is regularized to approach the activation maps of the CNN teacher model, which is the so-called attention link. Based on the empirical observations, the paper further makes the link selective by only building relations between CNN activation maps and ViT attention maps at similar levels and excluding the attention maps at the high levels. Empirically, the proposed method is compared with DeiT, ConViT and other knowledge distillation methods on ImageNet or CIFAR10. In the setting of small-sized data, the attention link is shown to improve the test accuracy with shorter training time and less computation cost.
strengths_and_weaknesses: Strengths:
1. The writing of Sections 1-3 is clear and easy to follow.
2. The idea of regularizing the global token’s attention maps with the CNN activation maps is novel.
3. In the setting of small-sized data, the improvements in accuracy and training time are impressive.

Weaknesses:
(Please refer to the Questions section for more details.)
1. Some of the tables and figures lack elaboration in the caption.
2. More empirical evidence is needed to support the superiority of the attention link.

questions: 1. In Natural Language Processing, knowledge distillation has also been widely adopted in transformer-based models to enhance training efficiency. In particular, MiniLM (Wang et al., 2020) and TinyBert (Jiao et al., 2019) also focus on the regularization of attention maps. I think it’s worth including this line of literature in Related Work.
2. In Table 1, each CNN block level is associated with different numbers of ViT layer level. And beside the ViT layers at high level, the way how others are assigned to CNN block levels is also different between the alpha-link and the beta-link. I would appreciate it if there is any ablation study or verbal explanation on such a “non-intuitive” design. 
3. In Table 2, could you please explain what the two numbers in each cell stand for? (as it’s not clearly written in the caption or the main text.)
4. In Figure 2, if my guess about the x-axis corresponding to the ViT layers is correct, I’m wondering how the weights of different heads within each layer are aggregated to produce the plot. Besides, as it’s hard to tell that the block diagonal is generally darker than the side in Figure 2(b) and (c), would you mind elaborating more on how to reach the conclusion that “the ViT attention maps are highly related to the CNN activation maps located at a similar level”? 
5. I would appreciate it if the authors could explain more about the argument in line 239 - “the inductive bias is hard to be trained without a large dataset”.
6. As stated in Question 1, there are other knowledge distillations methods. For example, they may use a transformer as the teacher model instead of a CNN and also utilize the attention map regularization. I’m wondering whether it would be better to add the empirical comparison with these methods. 
limitations: Please refer to the Weakness and Questions section for details.
review_title: null

Meta-review: metareview: Four reviewers provided detailed feedback on this paper. The authors responded to the reviews and I appreciate the authors' comments and clarifications, specifically that each question/comment is addressed in detail. Additional experiments were also performed. The authors also uploaded a revised version of the paper.

After the two discussion periods, one of the four reviewers suggest to reject the paper while three reviewers rate the paper as "weak accept", so no reviewer strongly advocates for acceptance. I considered the reviewers' and authors' comments and also tried to assess the paper directly. I believe that the paper should not be accepted to NeurIPS in its current form.

Weaknesses include:
* Readability: While at least one reviewer describes part of the paper as "clear and easy to follow" one other reviewer mentions clarity as the main weakness and one other reviewer also comments in this direction. I personally found the paper hard to read as well (even after the improvements made in the revision), and I found some of the claims to be fairly generic and partially not well supported. E.g. "resolve the issues of overfitting and lengthy training time of ViT", or "The proposed scheme preserves the original architecture of ViT, which results in its general employment regardless of the architecture of ViT."
* Experimental Results: Several questions have been raised regarding the experimental results (e.g. influence of the attention link, choice of hyperparameters). These have been addressed in the discussion, but it seems to me that they were at best partially resolved.
* Relation to distillation: The results in the low-data regime rely on learning from a teacher model. This relation to distillation is recognized but somewhat under-explored. This could be a confounding factor in the analysis of the approach. For example, in one response, the authors argue that "However, we believe our source of performance gain is due to transferring CNN's inductive bias with attention". It remained somewhat unclear, whether this transfer would also hold when the CNN is not a more powerful teacher model.

Strengths include:
* The idea of regularizing the global token’s attention maps with the CNN activation maps is novel and interesting.
* The reported experimental improvements in the low-data regime are interesting.

Despite recommending the paper for rejection in its current form, I would like to encourage the authors to continue this line of work and present it again to the community with more focused discussions, insights (and possibly experiments). This is an interesting paper and it was evaluated to be close to (but below) the acceptance threshold.
confidence: Less certain
Decision: Reject