Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: Transformers are utilized for representation learning. Instead of using them as backbones
to be optimized, this paper proposes to improve CNN backbone attentions via transformers
in the form of network prediction supervision. In the experimental validation, the proposed
method CARE is shown to improve the original representation learning methods with only
ConvNets introduced at the 100, 200, 400 epoches.
main_review: In this paper, a new self-supervised learning approach is designed to improve CNN
backbones. It shares similarities with sota pipeline in that images are created via two views
and measured representations in different branches. The main difference resides on the
additional branches introduced where there are transformers. The visual attentions of
backbone features are enhanced via transformers. Then, these features are used to
supervise original representation learning branches. CNN encoders are thus learned to
become attentive, which is guided by the transformer.

As a representation learning approach, the proposed method provides a potential to
improve backbone networks with transformers. It differs from contemporary representation
learning methods that focus on learning transformer backbones. This is a novel direction
for the representation learning pipeline where additional network properties (i.e., visual
attention from transformer) are introduced to gain the target backbones.

The transformer is updated by a moving average, which is similar to that of predictors in
CNN branches. Why using this move average instead of gradients back-propagation?
Furthermore, the transformer is designed in parallel rather than in sequence. Any
motivation on how to make this design? A direct explanation will improve the motivation of
the proposed method.

In the linear evaluation of the experiments, the performance of using ResNet-50 is reported
at 400 epochs while other backbones are not (e.g., ResNet-101 at 100 epochs and
ResNet-152 at 100 epochs). A consistent training by using same epochs will make the
results more convincing and the comparison more thorough.

------------------------------------------------Post Rebuttal--------------------------------------------------------------------------

My questions are well-addressed. After reading the opinions and reviews from other reviewers, I will increase the final score to 7.

limitations_and_societal_impact: See above. The societal impact is shown on the last page of the manuscript.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: A self-supervised visual representation learning method is proposed in this paper. Besides the prevalent two-branch CNN architecture design, the two-branch transformer structure is introduced to improve CNN backbone features via attention enhancement. The proposed method is evaluated on the several visual recognition benchmarks and shown effective through extensive experiments.
main_review: The proposed method, named  CARE, introduces a transformer to improve the CNN backbones during SSL. This is novel and interesting. Prevalent visual SSL methods (e.g., BYOL, SimCLR) use two CNN branches to create augmented two views of one image, which is formulated as  C-stream in CARE. Differently, CARE introduces two branches containing transformers to improve CNN backbone attentions, which is named as T-stream. The improvement is fulfilled via network supervision from T-stream to C-stream during training. The related work is adequately cited.
The proposed method sounds reasonable. The learned attention within the CNN backbones is visualized to show the training effectiveness. The overall framework seems complete. In the experiments, thorough evaluations are conducted to evaluate the proposed method including image classification (linear probe and semi-supervised), object detection, and semantic segmentation.
The overall pipeline is presented well and the technical details are not difficult to follow. The figures of network structure comparison, pipeline illustration, and visualizations are helpful to understand the proposed pipeline. Both the pseudo code and the source code are provided and the proposed method is trustworthy.
The overall experimental results seem promising and clearly improves the sota performance. Using two additional branches to improve the original two branches is new and motivates potential further development. It provides new thinking of using the advantage of one type of network to improve another type of network. This is a unique experimental approach to the visual SSL field.

------------------------------------------------Post Rebuttal--------------------------------------------------------------------------

The responses have addressed the issues listed in the previous comment, so I will keep my rating. 
limitations_and_societal_impact: The proposed method uses T-stream output to supervise C-stream. The network supervision shares some similarities to knowledge distillation where the teacher network is used to supervise the student network. A discussion upon knowledge distillation is useful to understand the contribution better. The negative societal impact has been illustrated and addressed on the last paragraph and this reviewer thinks it to be adequate
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper uses transformers to improve visual representation learning. The two view augmentation and the response generation solution from existing approaches are boosted by transformers for visual attention enhancement. The main contribution lies in a structure with two transformer branches to improve CNN attention and supervise the original CNN branches. I can see that the attention mechanism obtains gains due to the parallel design. The proposed method shows superiority, compared to existing work. 
main_review: I see some merits, which include:
1. The proposed research is credible, the objectives are comprehensive, clearly specified, and of high scientific quality. The paper clearly identifies a number of highly relevant challenges and proposes an appropriate methodology to address them. In the supplementary files, the code is provided as well.
2. The proposed transformer branch looks interesting because it advances the original SSL framework, where only CNN branches are introduced. On top of it, the feature response from the CNN branches is integrated with transformer attentions. 
3. The proposed method is evaluated on several benchmark datasets for different vision tasks, for example, image classification, object detection, and semantic segmentation scenarios. I would say the achieved results are solid. 

However, some questions are:
1. The proposed T-stream includes a (momentum) transformer, a (momentum) projector, and a predictor. It looks quite complicated. Why is momentum strategy utilized in these branches? How does it work? What are the benefits of using it? All these things are not clear in the paper.
2. I wonder what would happen if you enable a sequential way, e.g., CNN+Transform? Why does (not) it work? 
3. It seems contradictory - on the one hand, the paper claims that the transformer improves CNNs' attention; on the other hand, I saw that there are convolution layers within the transformer structure. Why is that? 
limitations_and_societal_impact: They were mentioned in the paper.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: In this paper, a self-supervised visual representation learning approach is developed to improve CNN backbone encoders. The transformers are introduced to improve visual attention abilities of transformers. The evaluation on the experiments has shown the proposed approach achieves favorable performance.
main_review: The main novelty of this work is to develop a four branches paradigm for self-supervised visual representation learning. The first two branches are similar to prior SSL works where there are backbone encoders, projectors, and the predictor. The last two branches are involved with transformers. And these branches are updated via either gradient back-propagations or a momentum schedule. The last two branches are new and shown to improve the backbone encoders.

The overall presentation seems complete and the technical components are clear. The paper organization is straightforward, and the visualizations are useful to show the attentions. The implementation code is provided in the supplementary files.

Results on the downstream scenarios (image classification, semi-supervised classification, object detection, and semantic segmentation) show that there is an improvement upon existing SSL methods.
limitations_and_societal_impact: The last two branches still involve projectors and a predictor. This seems not identical for the whole pipeline. These two branches are set to improve the visual attentions of the backbone encoder output. Using transformer seems sufficient to fulfill this task. Is there any possibility to remove these modules to reduce pipeline capacity?

In the experiments, some backbone encoders are evaluated by training 100 epochs. Using more training time shall be helpful to validate the performance improvement upon downstream scenarios.

On the last paragraph, societal impact is discussed.

-----------------------------Post Rebuttal---------------------------
My concerns have been well addressed in the response. After reading reviews from other reviewers, I am sure this is a good paper and increase my score to 8.
review_title: null

Meta-review: The rebuttal addressed all of the reviewers concerns, and all reviewers recommend acceptance. The AC agrees with this recommendation.
Decision: Accept (Poster)