Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Author propose a new neuroscience-inspired self-supervised learning process through adopting processes of the brain for developing a new framework. Specifically the use the concept of efference copy to derive a new loss function in a contrastive setting that can be used to perform self-supervised learning. They demonstrated the applicability of this method on a number of different datasets on image classification. There was a modest increase in performance when compared against some of the earlier work on contrastive SSL, i.e. SimCLR, which is actually the biggest limitation of this work. Nevertheless that is an interesting piece of work that is among the 1-2 other papers that have approached this problem from a more complex cognitive perspective.
strengths_and_weaknesses: Strengths.

A) the paper is well founded and written. Although I am not a neuroscientist, I appreciate that the formulations used are inspired by how our central nervous system sends stimuli to our motor neuron system to perform an action, whilst also sending a signal to itself. That is grounded to the fact that our central nervous systems learns primarily without external supervision.
B) There is a parallel presented around how to recover SSL methods such as contrastive learning by inverse models for efference copy using first principles. 

Weaknesses.

A) Although it seems to be foundationally grounded, it mainly focuses on comparisons with “very early” (oxymoron I know) work on contrastive learning (SimCLR), which I dare say that is not that relevant anymore, and is mainly used as a baseline when comapred with other approaches. It is common to compare with SimCLR but when including BYOL, SWAV, MoCo, etc. I think the authors should have added comparisons with all these approaches, even if this methods did not outperform them. I am not taking the side at all on “SOTA” or “nothing”   -  I mean that from the point of of completeness of the experimental design and rigourous representation around how this method compares with others.
B) Figure 1 is incomplete and notations are inconsistent but that has been acknowledged in the supplementary material and the authors have committed to fix these issues in the camera-ready version.
C) Presentation could have been stronger, i.e. additional figures could have been provided on how the model trains, loss function, etc. over those 1000 epochs. There is limited information in general on the results part.
D) Conclusion is rather weak, not only because it does not reflect on the neuroscience side more than it did, but it rather repeats and points to sections to draw conclusions. It is more like a summary/overview than a conclusion
questions: A) Did you consider comparing with other methods beyond SimCLR and Relic? It would have been positive irrespective of absolute performance
B) When comparing with methods that provided results in 200 epochs (arguably very few labs can train for longer than 200/300 epochs, as usually methods that present 1000 epochs on image net for instance are large lab and companies **usually**), one should compare their own method on 200 epochs rather than comparing the 1000 vs 200.
C) More of theoretical question, if this method can barely outperform SimCLR, what further improvements adopted from neuroscience could perhaps lead to further improvements that can compare with say VicReg?
limitations: I think the results section is a clear limitation and someone reading this paper will be confused as to what the current SOTA is. Contrastive methods have seen better days and we can safely say that most work now is non-contrastive - this has not been adequately articulated in the paper and further information should have been provided on other methods such as regularisation and VicReg, SWAV, etc.

The paper would have been extremely interesting 2 years ago, but as it is a different way of thinking I do not necessarily see this as a limitation, given that advances are not linear across sub-fields. What I mean by that is that it could be that in a year from now a neuroscience-inspired methods built upon this one could outperform SOTA in SSL. I just think that all these should have been covered in the main paper, and also not use the supplementary material to add so much information that 10-15% of which would fit best in the main paper. I literally looked at the supplementary material to get some essential information whereby essential information should be covered in the main paper, and leave additional context for the supplementary material.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper states a broad framework for learning in a setting where actions lead to changes to the environment/input, via the (neuroscience-inspired) notion of an efference copy, in which the agent gets a copy of the action ("motor command") in addition to sending it off to the motor system. Using a vision setting, and by setting up two simple actions, "look at the same object or not" and "change the view (affine transformation) of the object", the paper derives a loss function which is identical to a typical contrastive learning objective, plus a term that handles the case of change of view, for which they use data augmentation. The paper shows that this framework yields representations that are better for linearly classifying in several visual tasks.
strengths_and_weaknesses: Strengths

This framework is very interesting, and raises lots of obvious questions about what would happen if one were to change the set of actions. In particular, I am curious how this could be applied to speech. Typical approaches are not merely contrastive, but predictive, i.e. they have a temporal dimension (wav2vec2, CPC), so there would perhaps be some way of modifying the current approach, or perhaps a more radical approach in the same framework could be developed that takes an analysis-by-synthesis angle.

The results obtained are somewhat better than the reference systems.

Weaknesses

I think the notion of efference copy is not really doing anything here, aside from re-framing the problem, given that we are assuming that the efference copy is always exactly accurate. As such, given that there is only one example given (the specific cashing-out of the framework for vision), and given that there are no ablation studies or error bars, it is not so clear to me what we are supposed to get out of the result. Is the notion of efference copy concretely useful to improve performance? Or is it merely the combination of contrastive learning with data augmentation? Or is it not either which is in fact improving the result over previous scores (perhaps implementation differences, random initialization, etc)? In other words, to clearly establish the value of this new framework, I would expect one of two things: more, diverse examples; or a detailed and careful analysis of one example. The discussion around Figure 3 would imply that the authors are suggesting that it is the combination of contrastive learning with data augmentation which is helping, but that is not really a property of the general framework from what I can see, but rather just of this specific example and the actions that the authors have chosen. Maybe this could be okay, if the paper were reframed as being less about the efference copy framework (which doesn't seem to have any specific value here) and more about combining contrastive SSL with augmentation, but by itself that seems like a small extension with not such big empirical improvements.
questions: Can you explain concisely why the notion of efference copies is particularly valuable for this example or for other possible examples?
limitations: The framework presented is so general that it is a bit tricky to think of ethical implications, but given that the paper rotates around one specific example, I would have suggested the authors dig a little more into the object or visual classification example to see what negative impacts they can think of of either better features, or perhaps the specific model described here (e.g. would it be possible to use this model to get better predictions of held-out views of objects or faces?)
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Inspired by the concept of Efference Copy (EC), the authors propose a new self-supervised learning (SSL) framework, called Self-supervision Through Efference Copies (S-TEC). Specifically, the authors focus on inverse models of EC, which map sensory inputs to motor outputs, and make the analogy between EC and SSL by considering data augmentation as actions. The inverse models are implemented by the identity-related inverse model to determine whether two inputs are the same object or not, and the manipulation inverse model to figure out which augmentation is applied. Through the experiments, the authors demonstrate that S-TEC outperforms the reference algorithms including SimCLR and AugSelf on the image classification tasks, and analyze why S-TEC works well in a qualitative manner.
strengths_and_weaknesses: Originality: The motivation of this paper that brings the concept of EC into SSL sounds interesting. However, the novelty is a bit limited because the authors do not consider forward models that are critical parts of EC in S-TEC and it is difficult to see the significant differences from previous studies. The loss function of the identity-related inverse model is almost identical to a conventional contrastive loss used in SSL literature and the loss function of the manipulated-related inversed model is similar to [1].

Quality: The authors' claims are not supported well. First, the comparison with [1] is not enough. Since both the proposed method and [1] are motivated by augmentation-aware SSL, the two methods should be compared intensively. However, the authors compare the two methods on only STL-10 and the experiment setups are not identical. Second, the ablation study is lacking completely. For example, it would be great to see the effect of the number of clusters for the manipulation-related inverse model. Finally, the experiments are done on only image classification tasks. Considering the purpose of SSL is to learn general features for various downstream tasks, the authors should conduct more experiments on various tasks, such as object detection. 

Clarity: The paper is generally well-written and easy to follow. However, some important details are in the supplemental material not in the main text.

Significance: At this stage, the significance of this paper is limited because it is not clear what are the contributions compared with previous work and the experimental results are not enough to support the authors' claims as I wrote above.

Reference\
[1] Lee, H., Lee, K., Lee, K., Lee, H., and Shin, J. (2021). Improving transferability of representations via augmentation-aware self-supervision. Advances in Neural Information Processing Systems, 34.
questions: - Please give more details about the egocentric representation of actions in lines 140-151.
- Please clarify the difference with [1] in the experiments. For now, the comparison is done on only STL-10 and the number of training epochs for S-TEC is larger than [1], which makes me suspicious about whether the performance improvement comes from S-TEC or not. If the main difference between S-TEC and [1] is the egocentric representation of actions, please conduct an ablation study of S-TEC by replacing the egocentric representation of actions with the allocentric one as [1].
- Please conduct ablation studies about the manipulated-related inversed model, such as varying the number of clusters.
- Please conduct additional experiments on various tasks other than image classification.
limitations: Yes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors proposed to generalize the recent self-supervised learning (SSL) methods with the Efference Copy (EC) concept in neuroscience. The proposed principles of EC lead to another implementation/variant of the SSL method which is named S-TEC, which has a more generalizable loss function and richer feedback. The new model S-TEC is benchmarked with other SSL methods, where the authors showed that it leads to improved performance overall. The authors also conducted additional latent space analysis that shows S-TEC tends to provide more separated feature space.
strengths_and_weaknesses: **Strength**:
	
1. The proposed methodology/assumption is interesting and novel.
- The authors made the theoretical connection between the EC and the SSL frameworks, which is particularly a suitable insight for the Neurips audience.
- The resulting formulation of the loss function is novel and general, which indeed provides new insights into the SSL domain.

2. The experiments seem to be nicely carried out.
- Although there are weaknesses in the experiment sections, I would like to point out that the experiments overall are nicely conducted. The method is tested on the major datasets that are used to evaluate SSL methods, ResNet-18 and ResNet-50 are both used, and the results are compared with a few major benchmark models. More could be done, but the overall experiments seem relatively convincing (under the assumption that the authors do not have an extensive amount of compute to use).
- The latent space analysis is interesting and insightful.

3. The paper is clear and nicely written.

**Weakness**:

1. Methodology: while the neuroscience connections give additional insights into the SSL framework, the actual implementation of the loss function ends up having many similarities/connections with previous works. Although the ‘The relation of our approach to prior work’ partly addressed the issue, it would be great if the authors could specifically remark how their method could or could not be reduced to other SSL frameworks.

2. Experiments could be more extensive:
- More models could be benchmarked, especially widely used ones such as MoCo and BYOL. 
- ImageNet results seem to be lower than the published performance of the SimCLR (69.3% as in their paper, with ResNet-50). The details of this should be more carefully addressed.
- The 200 epoch results of SimSiam are used to compare with the 1,000 epoch results of SimCLR and S-TEC, which is biased and should be replaced with a fair comparison.

3. Others:
- All figure captions (especially Figure 1 and 2) could be significantly improved. They are not self-explainable enough without carefully reading the main text.
- The definition of the regularization loss was put into the Appendix, while it might be important to put some definition in the main text to help the readers better understand how it would affect the performance. Also, there should be some ablation experiments to evaluate how the weights of the regularization would affect the model performance.
- Line 341-342 appeared twice (also in line 293-294): ‘Moreover, we also confirmed that the benefits of our approach transfer to other, orthogonal improvements, such as provided by ReLIC, see Table 3.’

If the above weaknesses could be addressed, especially the weaknesses regarding the experiments, I am willing to raise the score.
questions: 1. As currently there are more qualitative visualizations of the latent space, it would be really interesting if more quantitative analysis could be conducted to investigate how the latent space structure is affected (e.g. how strongly augmented images distribute compared to the others).

2. The mathematical notations in the Formalism section could be more self-explainable without referring to the appendix. For example, line 181 'φ compares the outputs of f', it is unclear which type of function is used and how the outputs are compared. 

3. The related work section could be put after the Analysis section for better readability.

4. The authors could include the performance trajectory of S-TEC along the training epochs to show if the model learns faster when compared to other models. 

5. The authors could also include ablation experiments based on different augmentations to show if the model learns better when fewer/weaker augmentations are used.
limitations: The authors addressed some limitations of the work. Based on the suggestions and questions above, more limitations could be included after the rebuttal process.
review_title: null

Meta-review: metareview: This paper introduces a neuroscience-inspired approach for self-supervised learning and makes the connections between SSL and the idea of an Efference Copy (EC) in neuroscience. 

The authors provided a number of experiments showing improvements over SimCLR. However, multiple reviewers raised the concern that there are now many non-contrastive objectives for SSL that are sota, and improvements to SimCLR that have not been considered or compared. In the rebuttal phase, the authors did a number of new experiments to compare with other non-contrastive SSL objectives like BYOL, and also ran new evaluations on speech data where they compare with CPC. Overall, many of the key concerns raised by reviewers appear to be addressed by the rebuttal and new experiments provided by the authors.

While there is some noted overlap with previous methods (AugSelf) that diminished some of the enthusiasm, the established connection to neuroscience and efference was still seen as significant even in light of some methodological similarities. 

confidence: Less certain
Decision: Accept