Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper investigates how to use self-supervised learning for multi-stage visual transformer models. Previous works have shown that SSL can learn image correspondences and lead to performant pre-trained models, while the multi-stage models can reduce the computation cost dramatically. This work tries to merge these two trends together. The solution is a new region-based loss that can be applied to the local features. The comprehensive experiments show the advantages of the resulting models on multiple tasks.
main_review: The paper has many strengths.

- It is interesting to see that the loss on local features can work well in the self-supervised learning case. This observation may find more usage in the future works of doing self-supervised learning on transformer-based models.

- The experiments are comprehensive and convincing. The paper tries the same idea on multiple transformer structures and different tasks. The proposed method can achieve the efficiency of the multi-stage transformer models while achieving good self-supervised feature pre-training for tasks including classification, segmentation, and detection.

- The info in the appendix is helpful for result reproduction and detailed understanding.

In terms of weakness, I do hope that the paper can be more clearly written, such as reorganizing the info between main text and appendix to give more intuition of L_R.

In addition, the detail of Table 2 is too scarce to understand directly. I guess the supervised baseline is Swin-T, but it is hard to infer from the paper directly.
summary_of_the_review: Overall, the manuscript looks like a good paper. I do hope reading the paper can be easier despite the page limit.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper develops an efficient self-supervised vision transformer for learning visual representations. It introduces a multi-stage architecture with sparse attentions to reduce computation complexity and proposes a new pretraining task of region matching to capture fine-grained region dependencies. The results on the ImageNet and 18 small datasets or downstream tasks are good and compared with other state-of-the-art approaches.

main_review: Strengths:  
(1) It introduces the new pretraining task to capture fine-grained region dependencies;
(2) The experimental results are good and better than other compared approaches.

Weaknesses:
(1) The patch merging module and sparse self-attention in the multi-stage ViT are very similar to the patch merging in the paper Swin Transformer [Liu et al 2021]. The authors should clearly explain the differences between this paper and Swin Transformer.
(2)The equation (2) should add more high-level descriptions. For each local feature z_i, why finding the local feature z_j from the teacher with the highest cosine similarity can capture the fine-grained region dependency? 
summary_of_the_review: The theoretical novelties are limited, this paper mainly borrows the ideas from two papers DINO [Caron et al. 2021] and Swin transformer [Liu et al 2021]. However, the experimental results are very good in this paper, and the paper provides a lot of implementation details for others to reproduce the results.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The contribution of the paper comprises:
* the observation that the Multi-Stage Vision Transformer (MSVT), as opposed to the "monolithic" Vision Transformer (VT), does not produce discriminative patch representation, and
* a loss function for self-supervised pre-training of MSVTs, that encourages discriminative patch representation.

Extensive experimental evaluation demonstrates that the proposed loss term, when added to the standard non-contrastive, self-supervised loss, brings a modest but systematic improvement in performance of the MSVT (Tab 5) and other architectures (Tab 4).

main_review: Weaknesses:
1) I find the contribution, in the form of the loss term, incremental with respect to Swin, and the performance gain, demonstrated in most experiments, modest.
2) In my opinion, Figure 1 and Table 1 misrepresent the performance gain that stems from the contribution of the paper. The baseline architecture (Swin-B and Swin-B/W=14) already outperforms almost all methods in the comparison at a significantly lower number of parameters and much higher throughput. (The corresponding numbers are reported in Tab 5.) \
I encourage the authors to modify Figure 1 and Table 1 to represent the increase in performance of Swin resulting from the use of the proposed loss term. This can be done by introducing to both parts of Figure 1 points representing the performance of the base architectures (Swin-B, -S, and -T), trained without $\mathcal{L}_R$. The performance gain due to training them with $\mathcal{L}_R$ can be indicated in the plot with vertical arrows. Table 1 can be modified by reporting the performance of the base architectures trained without the proposed loss, in addition to their performance when trained with this loss. (Basically, by merging Tab 5 into Tab 1. The "ablation" results can still be discussed in a separate section.)
3) In the current transfer learning results, the impact of the authors' contribution on transferability of the trained networks is not clear. I would prefer the experiments in learning to classify 18 small datasets to highlight the performance gain/loss that stems from training with $\mathcal{L}_R$, in addition to the current comparison of supervised to unsupervised training. This would imply adding a third bar color to Figure 3 and one more row to both parts of Table 2, to represent the performance of Swin trained in a self-supervised regime, but without $\mathcal{L}_R$. 

Strengths:
* A. I like the story-line of the paper, where an interesting observation about the discriminative (matching) power of patch descriptors motivates the construction of the loss term.
* B. The proposed loss term is shown to improve both the performance of Swin and of other architectures.
* C. The experimental evaluation is extensive, in that it also shows limitations of the proposed loss, for example, when applied to image segmentation (Tab 2 and Tab 6).
* D. According to me, the manuscript is well written and easy to understand.

Minor questions and editorial suggestions:
* Q1: in the loss term (2), you select $j^*$ by the cosine distance, but you minimize cross entropy. Have you tried using the same score for both tasks, i.e., selecting $j^*$ with the cross-entropy and minimizing cross entropy, or selecting $j^*$ by the cosine distance and minimizing the cosine distance?
* Q2: in the loss term (2), why haven't you tried optimal 1-1 matching (the Hungarian algorithm) to match source and target patches? 
* Q3: in the paragraph "design choices of $\mathcal{L}_R$", you describe the choice of "argmax" vs "Optimal Transport" in the loss term (2). I do not understand how OT can be used to select the optimal index $j^*$. Could you clarify this?  
* ES1. page 2: "loss of this property", "to alleviate this issue": Perhaps use enumeration instead of bullets and make the statements more specific: loss of the property described in 1, alleviate the issue identified in point 2.
* ES2. page 3: "ViT yields 95\% accuracy in (...) region-to-region correspondences" - consider specifying how you compute the correspondence scores/patch distances. (cosine similarity, as in subsec 4.4 page 8 ?)
* ES3. in the caption of Tab. 1, please specify W=14 is the window size.
summary_of_the_review: I find the main idea of this paper convincing, the manuscript well written, and the evaluation thorough. The fact that the authors highlight some limitations of their loss term, in particular when applied to image segmentation, attests to their scientific scrutiny. I think the paper deserves a publication. I am willing to rate it 8/10, instead of 10/10, because I find the contribution incremental and the performance gain modest.

In this round of reviews, I lower my recommendation to 6, because I find the weakness (2), described above, acute. It misrepresents the effect of the contribution on performance in the teaser figure and in the first table with quantitative results. I am confident the authors can fix it in the next revision of the paper, which will prompt me to increase the rating to 8/10.

I encourage the authors to also address the weakness (3) in the next revision of the paper. The current transfer learning experiments (section 4.2) do not evaluate the specific contribution of the paper, but rather answer more general questions. I am confident that adding evaluation more focused on the use of $\mathcal{L}_R$ will vastly benefit this part of the paper. I nevertheless do not feel that I should request completing this task within the limited review period, and I will not lower my recommendation on this basis.

EDIT:
the authors addressed the weaknesses 2) and 3) listed in the main review and answered all my questions. I am therefore happy to endorse the paper. This is a solid work and deserves to be published.
review_title: null

Meta-review: This paper proposes two techniques for improving self-supervised learning with a vision transformer. The first improvement is using a multi-stage ViT, which is very similar to Swin transformer and authors recognized this is not a major contribution. The authors further found that using a multi-stage ViT does not produce discriminative patch representation, thus proposing the second improvement with a region level loss. While both improvements are not particularly novel by themselves, combining both leads to a strong empirical result. However, It does looks like the multi-scale vision transformer is the major improvement as removing the regional loss only leads to less than 1% decrease in performance in most cases. In general this is a good "engineering" paper with a practical approach for improving self-supervised learning with vision transformation and obtained strong results, thus it's worthy of publication.
Decision: Accept (Poster)