Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a novel technique to perform model smoothing in the context of the student-teacher training framework for tasks like self-supervised learning and semi-supervised learning. In essence, the proposed method involves randomly sampling a small portion of the student model and use the selected portions to replace its teacher counterpart. The authors of the paper demonstrate that the proposed method is also complementary to the commonly used temporal moving average method and when combined together, can yield state-of-the-art performance in multiple benchmark datasets. 
main_review: All in all, the paper is very well written and easy to follow. Although the proposed method can be quite simple, it can be of great significance to machine learning practitioners as a straightforward and convenient way to improve performance. Moreover, the authors of the paper conduct a series of experiments to demonstrate the effectiveness of the paper. All in all, while I think the novelty can be somewhat limiting because the idea is quite a simple one, this paper is a good submission in terms of the quality of the work/paper. As such, I will be a little bit conservative with my score for now and would like to hear the opinions of other reviewers. I would be happy to raise my score to a 7 if there is a consensus among the reviewers.

Questions: 

1. It is stated in the paper that a crucial factor for a teacher model to yield good performance lies in its stability/small variance, and the proposed spatial ensemble is a way to reduce variance. However, intuitively, doing so (especially when combined with TMA) can potentially harm the "accuracy" of the teacher model? Is this not an important aspect for teacher models to be effective?
limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors propose a new framework for constructing teacher models in student-teacher model training, which involves a combination of spatial smoothing (piecewise copying of students to teacher) and the usual temporal smoothing (trajectory-wise parameter averaging).   They situate spatial smoothing inside a more general framework of spatial-temporal smoothing, and recover temporal smoothing and the newly proposed spatial smoothing as special cases of spatial-temporal smoothing.  They go on to show its utility in two learning scenarios (semi-supervised learning and self-supervised learning).
main_review: This paper presents a new model-smoothing technique for student-teacher training.  The paper is written with exceptional clarity.  It was always clear from first reading what the authors were attempting, what motivated them, and what they observed. Though the mechanism of spatial smoothing is under-explored, they present compelling evidence that this smoothing method for student-teacher training offers benefits over existing smoothing.  For me, it's a clear accept.

Notes:

- In section 4.3 the authors experiment with the granularity of the spatial smoothing.  They investigate layer-wise versus channel-wise, and neuron-wise spatial transfer.  In 234-247, the authors report: "As shown in the top of Table 4, SE using NW outperforms its counterpart using LW and CW and achieves comparable performance with TMA. The result reveals that SE itself benefits from the fine-grained spatial ensemble."   I feel as if there is more the authors could say here.  It feels as if this experiment could benefit from the perspective of a model averaging argument, or from analyzing the correlation between model components that NW versus CW entail.  I encourage the authors to consider following up on this.

- For Table 2, the results are so similar that it's unclear if there is any benefit of STS smoothing over TMA.  IF possible, could this experiment be replicated several times and the distributions reported?

- The caption for Figure 3 should indicate which panel represents model parameters, and which presents generated supervision signals.  I can guess that they are in order induced by the caption, but more clarity here would be appreciated.
 

limitations_and_societal_impact: The authors do not discuss the limitations of their work directly, but do indirectly present evidence that shows the limitation of how much benefit STS offers in different scenarios.  

I would like to see a statistical analysis of the benefits of STS (see above)
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Different from the popular EMA mechanism in self-supervised and semi-supervised learning, this paper proposes a new mechanism for model smoothing in spatial ensemble perspective as complementary to the popular temporal moving average.  By combining the proposed Spatial Ensemble and the popular Temporal Ensemble, this work proposes to integrate temporal moving average and spatial ensemble into a unified model smoothing method, named as Spatial-Temporal Smoothing (STS). Experiments on self-supervised and semi-supervised tasks show that STS brings improvement to some methods.
main_review: This work rethinks current Temporal Moving Average (TMA) in the student-teacher framework, and the proposed Spatial-Temporal Smoothing (STS) is novel for the student-teacher framework, which also achieves superior results on multiple baseline methods in self-supervised and semi-supervised learning.
limitations_and_societal_impact: 1. Based on the results in Table1 and Table2, the Spatial Ensemble (SE) works worse than the TMA, and it can only work when cooperating with TMA, so basically, STS can only be viewed as an upgraded version of TMA.
2. Comparing the default layer-wise (LW) with channel-wise (CW), it seems that CW achieves much better performance in Table 4, also slightly better performance than LW in STS with nearly the same time. Why does this work choose LW as the default method rather than CW. 
3. The improvements of STS on ImageNet are incremental in most methods.
review_title: null

Meta-review: There is a consensus among the reviewers that this paper presents significant results and is of interest to the conference. 
Decision: Accept (Poster)