Venue: NeurIPS 2022 Workshop SVRHM

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, the authors incorporate principles from cortical connectivity at multiple scales to develop a transformer architecture that is more robust than the baseline transformer and contains lesser attentional parameters, while maintaining uncorrupted accuracy. In more detail, the authors use CCT as the baseline transformer and adapt its backbone to form the cortical transformer. They demonstrate that variants of the cortical transformer have better performance on the most difficult corruptions than the baseline transformer and transformers obtained by naively reducing the number of attentional parameters.

This paper is well-written. The adaptation of principles from cortical connectivity in this study is very interesting. However, there are a few weaknesses.
1. The biggest concern with the paper is that the variants of the cortical block show minor performance improvement over the baseline. Moreover, the improvement over the baseline is only observed for a small subset of the corruptions. 
2. This is a minor point. The fact that the CCT-based cortical transformer shows only minor performance improvement raises a concern that this result may not generalize to other transformers. The study could have benefitted from adapting the cortical block to another transformer architecture to verify that the benefits of using cortical principles generalize.
review_title: Well written. Interesting approach.

final_score: rating: 7: Good paper, accept
review_content: The authors employ an architecture, with principles at three different levels or organisation, and evaluating how well their network performance on cifar 10 (also with degration on cifar 10). 
They show that this network specifically (and also all 3 components necessary) perform better than naive compression. 
I have trouble following quite a lot of details so am unsure of my review. It might be some of the text where to much happens without a lot of detail but it is also me. 

Reducing the number of learnable parameters would make transformers much more digestible as a proposed mechanisms for human processing. 
However, cifar 10 is two simple to make any point definative.

review_title: Potentially very relevant innovation to reduce number of learnable attention parameters in transformer networks

final_score: rating: 7: Good paper, accept
review_content: Summary: In this study, three cortical mechanisms (macro, meso, and micro) were added to a transformer model and the accuracy was investigated for uncorrupted and non-adversarial images from the CIFAR-10 dataset. Overall, the models showed a similar if not slightly better performance on corrupted images but with fewer attention parameters than a typical transformer. The reduction in accuracy on the uncorrupted dataset appears to be negligible as well.

Strong Points: 
1. The results clearly show how a variety of cortical transformers perform comparably or slightly better than a typical transformer (especially on the most difficult corrupted images) despite having fewer attention parameters.
2. The biological inspiration of the cortical mechanisms across a range of scales is well aligned with the workshop theme and showcases the potential benefits of recurrences and overlapping regions.

Suggestions:
1. Consider running more trials to have greater statistical certainty of the results (standard deviations in accuracies)
2. Figure 3 is a bit cluttered and confusing with all the different model types. I think it would be clearer, especially for colorblind individuals, if different shapes were used for the different models or a different color scheme were used that is more colorblind-friendly.
3. Figure 2 would be a lot clearer if the naïve compression model were explained in the methods section.
4. While the results on the non-adversarial network were well presented, I am curious how the cortical transformer models would perform on adversarial examples.
5. I wish there was more of a discussion about why certain models performed better than others. For example, the macro+micro performed the best but adding the meso mechanisms seemed to decrease the accuracy. Why do you think this might be the case?

review_title: Promising results showcasing the potential benefits of cortical based architectures in transformers

final_score: rating: 7: Good paper, accept
review_content: The paper presents a transformer architecture that the authors endow with certain connectivity properties at varying "scales" of computation to study robustness. 

The paper is of high quality: well-written and professionally presented. The notation on page 3 is a bit dense, but I understand the space constraints.

While my familiarity with transformers is indeed limited, the work presents an exciting path forward towards understanding many properties that have been previously studied in a purely convolutional setting. The problem is at least well-motivated. However, to the uninitiated reader, the originality of the paper is somewhat unclear ("While some connections between transformer networks..." --->could you expand here how your model is different?). The review sets up the promise over previous studies with CNNs, but quickly jumps into the scale descriptions. Perhaps, for example, you could also motivate (in more text) how CCT is fundamentally different architecture.

The main results in figures 2 and 3 have very similar scales, especially for 3 seeds. Are the results statistically significant?

Also, for question 3(a), this should be answered "no," especially if the code isn't included with the submission. The justification provided is more in line with question 3(b). 
review_title: Technically solid paper that asks interesting questions 

Decision: Accept(Poster)