Spatial Ensemble: a Novel Model Smoothing Mechanism for Student-Teacher Framework
Authors: Tengteng Huang
Yifan Sun
Xun Wang
Haotian Yao
Chi Zhang
Venue: NeurIPS 2021 Poster (2021)
Keywords: model smoothing
student-teacher framework

Abstract: Model smoothing is of central importance for obtaining a reliable teacher model in the student-teacher framework, where the teacher generates surrogate supervision signals to train the student. A popular model smoothing method is the Temporal Moving Average (TMA), which continuously averages the teacher parameters with the up-to-date student parameters. In this paper, we propose ''Spatial Ensemble'', a novel model smoothing mechanism in parallel with TMA. Spatial Ensemble randomly picks up a small fragment of the student model to directly replace the corresponding fragment of the teacher model. Consequentially, it stitches different fragments of historical student models into a unity, yielding the ''Spatial Ensemble'' effect. Spatial Ensemble obtains comparable student-teacher learning performance by itself and demonstrates valuable complementarity with temporal moving average. Their integration, named Spatial-Temporal Smoothing, brings general (sometimes significant) improvement to the student-teacher learning framework on a variety of state-of-the-art methods. For example, based on the self-supervised method BYOL, it yields +0.9% top-1 accuracy improvement on ImageNet, while based on the semi-supervised approach FixMatch, it increases the top-1 accuracy by around +6% on CIFAR-10 when only few training labels are available. Codes and models are available at: https://github.com/tengteng95/Spatial_Ensemble.

Spatial Ensemble: a Novel Model Smoothing Mechanism for Student-Teare Framework
Tengteng Huang
Yifan Sun
Xun Wang
Haotian Yao"
Chi Zhang
Megyii Technology
(huangtengteng, yaohaotian, zhangchi @megvii wwagvii..oo com, sunyf150tsing org.en, bnuwangxung gma com
Abstract
Model smoothing is of central importance for obtaining a reliable teacher model in the student-teacher framework, where the teacher generates surrogate supervision signals to train the student. A popular model smoothing method the Temporal Moving Average (TMA), which continuously averages the teacher parameters with the up-to-date student parameters. In this paper, we propose "Spatial Ensemble" a novel model smoothing mechanism in parallel with TMA. Spatial Ensemble randomly picks up small fragment of the student model directly replace the corresponding fragment of the teacher model. Consequentially. it stitches different fragments of historical student models into unity, yielding the "Spatial Ensemble" effect. Spatial Ensemble obtains comparable sudent--eacche learning performance by itself and demonstrates valuable complementarity with temporal moving average. Their integration. named Spatial-Tempon Smoothing, brings general (sometimes significant) improvement to the student-teacher learning framework on variety of state-of-the-ard methods. For example, based on the self-supervised method BYOI, yields +-.9% top-1 accuracy improvement on ImageNet, while based the suprrvspe approach FixMatch, it increases the top-1 accuracy by around +6% on CIFAR-10 when only few training labels are available. Codes and models are available at: / wtwp:ttuu.....nntttttteeeeeeeeee com/ tengteng95 2ttpttteenn5S/Siiiaaa SSatiall
Ensemble
Introduction
Recent years have witnessed the success of the student-teacher framework in celf-supervised [ and semi-supervised [618 learning tasks. In this framework, the teacher responsible for generating surrogate supervision signals to guide the learning of the student unlabeled data, for example, category IDs for classification, feature patterns for contrastive learning, etc.
Many important works under this framework share common model smoothing technique, i.e., the Temporal Moving Average (TMA), to generate the teacher from the student model. For example, Mean Teacher [6] for semi-supervision averages historical student weights as the teacher. MoCo []] for self-supervision employs a slowly progressing encoder (teacher), driven by a momentum update with the query encoder (student). Another state-of-the-art method for supe-supeo BYOL 2 employs a similar strategy to update the target (teacher) network with a slow moviigg average of the online (student) network. We thus recognize that model smoothing with TMA is of central importance for obtaining a reliable teacher for these works.
In this paper, we rethink the mechanism of TMA and summarize its importance as two-fold. First, TMA enables the teacher absorb parameters from different student editions and thus benefits from
corresponning author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

historical students
teacher
historical students
leacher
Average
Replace
Iteration
7-2
T-1
Iteration
(a) Temporal Moving Average
(b) the proposed Spatial Ensemble
Figure 1: Illustration of Temporal Moving Average and Spatial Ensemble.
the model ensemble. Second, TMA constrains the variance of the teachers to be small to avoid inconsistent labels produced during two adjacent updates. An abrupt change the pseudo labels will cripple training and possibly leads to optimization divergence. This is particularly important for self-supervised learning where annotations are unavailable.
We propose a novel model smoothing mechanism named "Spatial Ensemble". Compared with TMA, Spatial Ensemble facilitates similar model smoothing benefits with different mechanism, as illustrated in Figure [ In TMA, the teacher absorbs each historical student as a whole. A temporally near student absorbed into the teacher with larger weighting factor due to the Exponential Moving Average effect. In contrast, in Spatial Ensemble, the teacher randomly picks up one or more small fragments of the student to replace the corresponding fragments of the teacher during each update. All the fragments are weighted equally in the teacher model. In this way. Spatial Ensemble stitches different fragments of previous student models into unity, yielding the "Spatial Ensemble" effect. In spite of their fundamental differences, Spatial Ensemble facilitates similar benefits of model ensemble and smoothed update. The smoothed update is maintained by replacing only a small portion of parameters at each iteration. On the challenging self-supervised learning task, which is very sensitive to the quality of surrogate supervision signals, we observe that Spatial Ensemble achieves comparable performance with temporal moving average. It shows that our Spatial Ensemble can serve basic model smoothing technique for the student-teacher framework.
An important advantage of Spatial Ensemble lays in its complementarity to Temporal Moving Average. To explore their complementary benefits, we integrate both the temporal smoothing and spatial smoothing mechanism into unified method named Spatial-Tempora Smoothing (STS). STS does not directly absorb a whole student, or simply replace fragment. Instead, it updates fragment of the teacher model with a temporal moving average of the corresponding fragment of the student. This "mix-up" model smoothing benefits from the mutual complementarity of temporal moving average and Spatial Ensemble. Experiments on both self-supervised and semi-supervised tasks show that using STS for model smoothing obtains general (sometimes significant) improvement over majority of state-of-the-ar methods.
We summarize our contribution as follows:
1. We propose a novel model smoothing mechanism named Spatial Ensemble. Compared with the popular temporal moving average, Spatial Ensemble achieves a comparable model smoothing effect with different mechanism.
2. We integrate the spatial and temporal smoothing mechanism into unified method named Spatial- Temporal Smoothing. STS benefits from the mutual complementarity of two mechanisms and consistently improves student-teacher methods on self- and semi-supervised learning tasks.
3. We notice two recognizable advantages of the model learned through STS, including strong robustness against various data corruptions, and superior transferring capability to the object detection task.
2 Related Work
In this section, we present a brief review of the development on semi- and self-supervised learning, as well as the student-teacher framework. We mainly focus on the recent deep learning literature related to our method.

---

Student
Teacher
Student
Teacher
Student
Teacher
Layer
Layer
Layer I
Layer!
Layer 1
Layer I
Layer 2
Layer 2
Layer 2
Layer 2
Layer
Layer
Layer N-1
Layer N-1
Layer N-1
Layer N-1
Layer N-1
Layer N-1
Layer N
Layer N
Layer
Layer
Layer N
Layer
(a) Temporal Moving Average
(b) Spatial Ensemble
(c) Spatial-Temporal Smoothing
Figure 2: Illustration of three different model smoothing approaches TMA, SE, and STS.
Semi-supervised learning aims at achieving comparable performance to fully supervised methods, usually with a small portion of labeled data and the rest unlabeled data. It of great importance for real scenarios where annotations are difficul//cpenniive to acquire, such as medical data. Pseudo labels
two essential techniques in many recent semi-supervised approaches. Pseudo-Labeling O generates soft pseudo labels for the unlabeled data using the logits predicted by the model from the latest epoch. Laine et al. [10] improve the pseudo label generation process by leveraging a temporal ensembling prediction of different epochs. Mean-teacher [6) further simplifies this process by introducing a teacher model, which is temporal ensembling of the student model through the exponential moving average, to generate pseudo labels directly. Besides, wean-teecchr employs a consistency regularization to encourage similar predictions from the teacher and the student model. FixMatch [7] assumes that the consistency should also hold between images applied with augmentations of different degrees, and applies both weak and a strong set of data augmentations.
9 10 and consistency regularization 66 7
Self-supervised learning largely narrows down its performance gap with supervised approaches and is receiving increasing attention in the community. Recent self-supervised methods can be roughly categorized into two genres, i.e., approaches based on pretext task and approaches based on contrastive learning. There are many kinds of insightful pretext task designs. One common method is generating surrogate supervision signals by feature clustering 12 image augmentation (usually rotation [5]), relative order of clipped image patches [16 18), etc. Another mainstream approach to design a generative pretext task, e.g., image colorization
I9 20 image inpainting [2]] and
image denoising [22). and usually employ auto-encodee or generative adversarial networks 23 to transfer images. Instead of designing a pretext task, contrastive learning approaches [
29
aim at maximizing the feature similarities with augmented views produced from the identical image
sample, while minimizing those from different image samples.
Student-teacher framework is a popular architecture that has been extensively used in many recent of methods, especially for semi- and self-supervised learning tasks. Usually, temporal moving average is employed to generate the teacher from historical student models. The optimization goal is ensure the prediction consistency between the student and the teacher. Besides pulling different views of the same image closer, MoCo I employs memory bank to store negative samples and pushes views from different images apart. BYL (2) removes negative samples and leverages an asymmetric student-teacher architecture to prevent collapsing. An extra predictor is appended to the online (student) network to regress the output of the target (teacher) network.
3 Method
Preliminaries. Given a same network architecture, T and @ are the parameters of the teacher and student model. Both O and @ contains n units of parameters, i.e,, @* = (07 ...... 0, } and @ = {0S, O5 ... go J. A unit may correspond to a layer, a neuron
channel, depending on the desired spatial granularity. To analyze the model update in consecutive iterations, we further define them as functions of the iteration t, i.e., (T) {07 (t)} and @ (t)).
3

---

3.1 Spatial Ensemble
We present a new model smoothing method named Spatial Ensemble (SE), as illustrated in Fig- ure (b). During each iteration, SE randomly picks up some units of the student model replace the corresponding units of the teacher model, leaving the remaining parts of the teacher unchanged. We formulate this process
1)},
(I)
where P: ~ Ber(p) (f.e., the Bernoulli distribution with success probability p) is a binary variable. mutually independent to each other and have the
We note that all P: Ber(B) = 1.2. .n) same success probability p. It deternines whether the i-th unit of the teacher parameters is to be preserved. Specifically, if P; = the i-th unit parameters is to be preserved. If P; the i-th unit is to be replaced. Since P. ~ Ber(p), larger results in lower replacing frequency and higher proportion of preserved units. Therefore, we name p as the "preserving probability".
For clarity, we omit the iteration indicator in all the equations in the following part of this paper and rewrite Equation T as:
OT = (P.0 + + +1-
To investigate the mechanism of SE, we quantitatively analyze the smoothing effect on both the model parameters and the generated surrogate supervision signals. Specifically, we calculate the Mlcan-quuare Errors (MSE) between teachers at two adjacent epochs, as well as the MSE between their outputs. We visualize the MSE values at the early training stages (epoch I to 20) in Fig. from which we make two observations:
Remark 1: Spatial Ensemble smooths the teacher update. It is observed that without model smoothing, the MSE of both model parameters and supervision signals maintain relatively high values, indicating strong vibration during the teacher update. In another word, the teacher model and its output signals undergo substantial instability. contrast, when using the TMA and the proposed SE to update the teacher, the MSE of both model parameters and supervision signals are much smaller (than Non-smoothing), yielding a smoothed teacher update.
Remark 2: Spatial Ensemble promotes convergence. Using TMA and SE, the MSE values
gradually decrease as the training epochs increases. Such result indicates that the up-to-date teacher model gradually become closer to its prior edition, therefore approaching convergence. Without TMA or SE, the Non-smoothing mode maintains roughly constant MSE value. We thus infer that Spatial Ensemble (as well as the TMA) promotes convergence of suudent-teacher framework.
We provide the pseudocode of SE in Algorithm [1 which is easy to implement with only few lines of code. In Section 4.3 we further analyze the effect of p on student-teacher learning performance.
Algorithm Pseudocode SE.
Algorithm 2 Pseudo code of STS.
net_s, ret_t: t: student/teecher wetwork
net_a, net_t: student/teactn network m: momentum for TMA
ma momentum for THA
probability
atudent layer
pt probabillty of student layer belonging
belonging the uapicked fragment.
unpicked fragment.
params_s net_a. pazama_t paramttrrrm
params_s = net_s. parameterss params_t for 5, t in zip (params__, params_t): if (ando).rand() < continue
for 5, t in ziplparams.....m.. perams_t);
if randon. rand () < p: cone
else: t=s
else:
t + (1-m)
3.2 Spatial-Temport Smoothing
We further integrate TMA and SE into unified model smoothing method named Spatial-Tempprra Smoothing (STS), to benefit from their mutual complementarity. STS picks up several units in

---

20
20
Mean Square Error
15
10
Mean Square
10
None
Nome
TMA
TMA
SF.
SF
5
13
17
13
17
Epoch
Epech
Figure 3: Illustration of mean square error of teacher parameters (Left) and generated supervision signals (Right) at two adjacent epochs. None represents no model smoothing is used. TMA adopts a momentum of 0.999. p is set to 0.999 for SE. The vertical coordinates arranged in Log scales.
the student model and averages them with the corresponding units in the teacher model, which is formulated as:
rT = (P.0% + - PP)mm + (1- (1)(1 m)07) 1m)mm)))))
(3)
where m is the momentum for TMA.
STS degenerates to SE or TMA, given specification of hyper- typer-parameters. When 0, STS degenerates into SE. Similarly, when we set the preserving probability p = 1, which results in P; = STS degenerates into TMA:
T mOT + (1-m))) (1-m)e0,
Algorithm illustrates the pseudocode of STS. Similar SE, STS is easy to implement and be integrated into most teacher-studed frameworks. In Section 4.3 we present detailed analysis of how p and influence the performance of STS.
Experiments
4.1 Baselines and Implementation Details
We evaluate the effectiveness of the proposed Spatial-Temppra Smoothing by applying it to the state- of-the-art self-supervised approaches (MoCo 1 and BYQ (2) and semi-supervised method (Fix- Match (7).. We use the official implementation of MoCo and re-implement BYOL and FixMatch using Pytorch [30]. All experiments are conducted on a machine with 8 RTX2080 GPUs. The masking probability is set to 0.7/0.5/0.5 5 teacher model at the end training. The self-supervised models are evaluated by following the 2
for BYOL/MoCo/ixxMaa
respectively. We dump the
4 31 while semi-supervised performance assessed with top- accuracy metric. We carefully follow both the training and evaluation settings of [ 2 in their original paper for fair comparison. More details given in the following.
common protocol of linear classification [
MoCo consists of query model and key model, which simulates the same role as the student
and the teacher model, respectively. We adopt MoCo v2 with ResNet-50 as our baseline and use memory bank of size 65536. The initial learning rate is set to 0.03 and adjusted by cosine learning rate scheduler [32]. Following the original paper, we train the model using SGD with momentum of 0.9, weight decay of 0.0001, and mini-batchsize 256. Both the query and key model adopt the same data augmentations as SimCLR [4] i.e., random crop, horizontal flip, color jitter, Gaussian blur, and gray-scale conversion.
BYOL (2) reformulates the self self-supervised learning task as a regression task which maximizes the similarity of feature representations from the teacher and the student model. We train BYOL with Synchronized batch normalization
33 and use mini-batchsize of 256. The basic learning rate is 0.1 The first 10 epochs are warmed up with factor of 0.001 and
and is decayed by cosine strategy

---

gradually increased to the basic learning rate. We adopt SGD with momentum of 0.9 as the optimizer and set weight decay to lc-4. BYOL uses the same set of data augmentations SimCLR listed above, and adopts solarization with probability of 0.2 for the teacher model.
FixMatch 7 uses identical student and teacher network in the training phase while employing the TMA teacher for evaluation. We use Wide ResNet-28-2 (34] and Wide ResNet-28-8 as the backbone networks for CIFAR-IO and CIFAR-I00, respectively. We use SGD with momentum 0.9 as the optimizer and train the model for 2 iterations following the original paper. Each mini-batch comprises of 64 labeled images and 448 unlabeled images (i.e., 1:7). The initial learning rate is 0.03 and is decayed by cosine scheduler [32]. The weight decay is set to 0.0005 and 0.001 for CIFAR-10 and CIFAR-100, respectively. For data augmentation, the student only uses random crop and random horizontal flip. Besides these two augmentations, the teacher additionally adopts random augmentation 35
Evaluation. Linear Evaluation is a common metric for measuring whether the feature representation learned by the self-supervised model good. Typically, a classifier appended the feature encoder and finetuned on the down-streaming task. During the whole process, the parameters of the feature encoder, well the BN statistics, are fixed. We train the classifier appended MoCo using the SGD optimizer for 100 epochs. The initial learning rate is 30 and is decayed at the 60th and 80th epoch with factor 0.1. For BYOL, the basic learning rate 0.2 and is adjusted according to the cosine strategy during the full 80 epochs.
4.2 The Effect of SE and STS
We first evaluate the effect of SE on the self-supervised task. We select MoCo and BYOL as the baseline models and train them for 200 epochs. As shown in Table disabling the model smoothing mechanism in MoCo and BYOL leads to failure of convergence, which is also observed in their original papers. The experimental results indicate the importance of model smoothing in these self-supervised approaches. When combined with SE. MoCo/B achieves (6.7%% top-1 accuracy, demonstrating the effectiveness of SE in model smoothing.
We further validate the effectiveness of STS which integrates both the temporal and spatial smoothing strategies. As is shown in Table compared typical TMA, our method boosts the performance of MoCo (B YOL) from 67.5% (71.8%) to 67.8% (72.7%) in terms of top- accuracy, indicating the valuable complementarity of the spatial and temporal smoothing mechanism.
Method
Smoothing
top-1
top-5
Method
Smoothing
top-1
top-5
MoCo
fail
fail
MoCo
TMA
67.5
88.0
MoCo
SE
66.7
87.0
MoCo
STS
67.8
88.1
BYOL
fail
fail
BYOL BYOL
TMA
71.8
90.6
BYOL
SE
67.8
88.3
STS
72.7
90.9
Table 1: Analysis of the effectiveness of SE as a model smoothing approach.
Table 2: Analysis of the superiority of STS over TMA.
4.3 Ablation Studies
In this section, we evaluate the effect of hyperparameters in STS and then investigate Spatial Ensemble of different granularities. BYOL is adopted as the default network for the following experiments. All the ablation experiments are conducted on the ImageNet dataset (36) and trained for 200 epochs unless noted otherwise.
Effect of hyperparameters p and m. To explore how p and m influences the student-teacher learning performance, we conduct extensive experiments by selecting m from {0, 0.9, 0.99, 0.999) and varying p 0.1 to 0.9 with an interval of 0.2. We first investigate the effect of p by setting m Under this case, STS degenerates into SE. As is shown in Figure BYOL fails to converge when p = 0 while achieves increasing performance with the increase of p. demonstrating the importance of model smoothing. Figure illustrates the results with different combinations of and m. It can be observed that STS yields better performance than typical TMA (p- = 0) most cases. We also notice an interesting fact that with the increase (decrease) of m, the model achieves its best performance

---

at relatively smaller (larger) value of For example, 0.99 is the best choice when m is set to 0.9. However, for m 0.99, a smaller p of 0.7 yields much better performance than larger p of 0.99. These results demonstrate the complementarity of spatial and temporal smoothing, as well as the important role of model smoothing for seff-super learning based on the student-teacher framework.
Granularity of Spatial Ensemble. SE may have various granularities, corresponding to a layer, a channel, or a neuron as the smallest unit. When integrating SE into STS, we set the granularity to layer-wise by default. In this section, we investigate the impacts of different granularities, i.e., layer-wise (LW), channel-wise (CW) and neuron-wise (NW). Specifically, for a student parameter with shape Can Cout. where Cin and Cout denote the number of input and output channels, respectively. For CW, we randomly pick up a small set of indexes along with the Cout dimension and replace the corresponding teacher parameters. Similarly for NW, we randomly select neurons from the whole student model and replace the corresponding neurons of the teacher model. As shown in the top of [a SE using NW outperforms its counterpart using LW and CW and achieves comparable performance with TMA. The result reveals that SE itself benefits from the fine-grained spatial ensemble. Besides, three variants of STS result in comparable performance, indicating the complementarity of temporal smoothing and spatial ensemble. When combined with temporal smoothing, LW is capable to produce a representative teacher model as good as its fine-gratde -ra-grained counterpart NW. Considering that the time cost of LM is lower than both CW and NW, yet their performances are close, we use LW as default.
4.4 Self-supervised evaluation of STS
In this part, we substitute the model smoothing mechanism in BYOL 2 with our STS and compare the resulted model to other state-of-the-art methods on self-supervised learning. Furthermore, we investigate the robustness of our method to corrupted data. Finally, we evaluate the transfer capability of our method on the object detection task.
Comparison with State-of-the-art method. Table 3 presents the linear evaluation results of our method compared with the state-of-the of-t-e--t-ht-- art supf--spe method on ImageNet (36). Using YOL as the base model, STS yields competitive top-1 accuracies of 73.0% and 74.5% under 300 and 1000 epochs, outperforming previous state-of-the-art approaches. is worth noting that our model is trained with smaller batch size of 256, while B BOL requires 4096. According to the results in the original B YOL paper, when the batch size decreases from 4096 to 256, its top-1 accuracy drops around than 0.6%.
Robust corruptions on ImageNet-C. We further evaluate the robustness of our method to data corruptions. Specifically, we select ImageNet-C (37,, which is standard benchmark dataset for evaluating the model robustness to common corruptions and perturbations, as our testbed. There are 15 different corruption types in ImageNet-C. Besides, each type has 5 corruption intensities. In the left part of Figure 5 we give some image examples of corruptions from ImageNet-C C and the corresponding top-1 accuracy improvements of STS over TMA. Combining the right part of Figure 5 we observe that STS outperforms TMA under all the five corruption intensity settings and yields an improvement of around 1.5% in terms of the mean top-1 1 accuracy of 75 corruption combinations. Besides, considering a single corruption from these 75 combinations, STS leads to an improvement of around 0.5% 5.6%. These results demonstrate the superiority of our method in terms of the robustness to corruptions and further reveal the advantage of STS in enhancing the representative capability of the teacher model.
Transfer learning in object detection. We further evaluate the generalization ability of the learned representation using our approach for the more challenging object detection task. In Table 5 we compare our method with state-of-the-art culf-super approaches. All the self-supervised models are trained for 200 epochs on the ImageNet dataset and then finetuned with the joint split of VOC2007 trainval and VOC2012 train. And their results are taken from SimSiam [8]. For fair comparison, we carefully follow the training and evaluation settings in SimSiam [88] In more detail, we use the
same detection codebase as SimSiam (38) and MoCo [[. The default detector is Faster-RCNN [39] with backbone of R50-C4. We finetune all the layers and BN statisties for 24,000 iterations with 16 images per batch. The image scales is set to [480, 800] pixels in the traning stage while 800

---

Linear Evaluation on ImageNet for SE
Linear Evaluation on ImageNet for STS
70
74
60
Top-1 Accuracy (%)
50
Top-1 Accuracy
72
40
30
20
70
10
m-0.9
11-0.99
11=0,
999
0.1
0.3
0.5
0.7
0.9
0.99
0.3
0.7
0.99
Preserving Probability p
Preserving Probability p
Figure 4: Linear evaluation results of SE (Left) and STS (Right) under different hyperparameter settings.
for inference. The initial learning rate is 0.02 and is decreased by factor of 0.1 and 0.01 at when reaching 18,000 and 22,000 iterations. We can observe from Table that the representation learned by our approach can be well transferred downstream detection task and achieves comparable or superior performance than previous approaches, especially on the AP75 metric.
Method
Epoch
top-1
top-5
Jigsaw
40
90
45.7
InstDis
200
56.5
Method
Smoothing
top-1
time
BigBiGAN 42
56.5
TMA
71.8
1.0x
Local A gg.
43
60.2
SE
LW
67.8
0.97x
CPC v2 25
200
63.8
85.3
SE
CW
70.8
1.02x
SimCLR 4
200
66.6
SE
NW
71.6
1.80x
MoCo v2
200
67.5
STS
LW
72.7
0.99
PCL 44
200
67.6
STS
CW
72.8
1.04x
InfoMin Aug 24
200
70.1
89.4
STS
NW
72.8
1.82x
Swav 3
400
70.7
Table 4: Comparison of SE and STS with different granularities.
SimSiam 38
400
70.8
BYOL
300
72.5
90.8
Ours (B YOL STS)
300
73.0
91.3
VOC07+12 detection
pretrain
PIRL 45
800
63.6
AP
AP
AP75
SimCLR
1000
69.3
89.0
supervised
81.3
53.5
58.8
InfoMin Aug 24
800
73.0
91.1
SimCLR
81.8
55.5
58.8
MoCo v2
800
71.1
MoCo v2
82.3
57.0
63.3
SimSiam 38
800
71.3
SWAV
81.5
55.4
61.4
Swav 3
800
71.8
SimSiam
82.4
57.0
63.7
BYOL
1000
74.3
91.6
BYOL
81.4
55.3
61.1
Ours (BYOL STS)
1000
74.5
91.8
Ours
82.3
57.2
64.4
Table 3: Comparison with state-of-the-art self- supervised methods on ImageNet under linear evalu- ation. All the experiments based ResNet-50 and pretrained with two 224 x 224 views without multi-crop augmentation.
Table 5: Transfer learning VOC object detection benmark.
4.5
Semi-supervised Evaluation of STS
This part is to evaluate the performance of our STS when integrated with FixMatch 7 for semi- supervised learning. Following the common settings 6 [7) use CIFAR-10 and CiFAR-100 the benchmark datasets.
CIFAR-10. We evaluate our method on CIFAR-10 under four settings, i.e., using 2, 4, 25, 400 labels per class. As shown in Ta STS achieves consistently superior performance than TMA under these four different settings, yielding improvement of 6.63%, 1.83%, 0.21% %,, and 0.17%,

---

Intensity
Intensity
Intensity
Intensity4
Intensity
0.6
Top-1 Accuracy
+1.6%
+1.5%
+2.7%
+4.5%
+5.6%
TMA
-1.9%
+2.1%
2.2%
+2.3%
2.0%
STS
2
Corruption Intensity
+1.2%
+2.1%
+2.8%
1.4%
H..5
Figure 5: Left: Examples of 4 corruption types, namely defocus blur, pixelate, fog, and spatter. The numbers attached the left-botton bittoooto corner denote the improvement of STS over TMA on the corresponding corruption type and intensity in terms of top-1 accuracy. Right: Comparison of top- accuracy between TMA and STS on ImageNet-C.
CIFAR-10
CIFAR-TOU
Method
20 labels
40 labels
250 labels
4000 labels
200 labels 400 lubels
2500 labels
10000 Jabels
[[-Model
46
45.74
85.99
42.75
62.12
Pscudo-Labeling
9
50,22
83.91
42.62
63.79
Mean Teacher
90.81
64.17
MixMatch 47
52.46
88.95
93.58
32.39
60.06
71.69
UDA 48
70.95
81.18
95.12
40.72
66.87
75.50
ReMixMatch
94,56
55.72
FixMatch
86.68
92.06
94,89
95.77
42.78
52.54
70.75
76.53
FixMatch (STS)
93.31
93.89
95.10
95.94
45.85
55.25
72.22
77.92
Table 6: Comparison with state-of-the-art semi-supervised methods on the CIFAR-10 and CIFAR-100 benchmark datasets under the top-1 accuracy metric.
respectively. Although the numerical improvement relatively small when 250/4000 labels are available, is actually non-trivial since the results have already been very close the performance upper bound (97.01%), which leverages all labels of 50,000 CIFAR-10 10 images. The comparison with 20/40 labels is more valuable since how narrow the performance gap between semi-supervised and supervised learning with limited annotations is heated research topic in the community. The remarkable improvement achieved by STS demonstrate its superiority and indicates the potential practical value in real scenarios where usually few labeled data is available.
CIFAR-100. In Table 6. further provide the results CIFAR- 1-0 benchmark dataset under four settings, iee, using 2, 4, 25, and 100 labels per class. Similar to the phenomenon observed in CIFAR-10, STS achieves noticeable improvements when only few labels (200/400 labels) given, outperforming the baseline by 3.07% and 2.71%. %. Besides, with more labels (2500/1000) labels), STS also achieves consistent improvements of 1.47% and 1.39%, demonstrating the effectiveness of our approach.
Conclusion
This paper reveals a new mechanism for model smoothing from the spatial ensemble perspective. In addition to the capacity of model smoothing, an important advantage of spatial ensemble is its complementarity the popular temporal moving average. To benefit from such complementarity. we integrate temporal moving average and spatial ensemble into unified model smoothing method Smoothing (STS). Extensive experiments on self-supervised and semi- supervised tasks show that STS brings general improvement to many state-of-the-art methods. Besides, STS achieves competitive results on the ImageNet-C dataset and the downstream object detection task, demonstrating its good generalization ability.
named

---

Acknowledgment
This paper is supported by the National Key R&D Plan of the Ministry of Science and Technology (Project No. 2020AAA0104400)
References
[1] Kaiming Hc, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsu- pervised visual representation learning. In Proc. of IEEE Intl. Conf. on Computer Vision and Pattern Recognition, 2020.
[2] Jean-Basticn Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent A new approach self- supervised learning. Proc. of Advances in Neural Information Processing Systems, 2020.
[3] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning visual features by contrasting cluster assignments. In Proc. of Advances in Neural Information Processing Systems, 2020.
[4] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey F. Hinton, simple framework for contrastive learning of visual representations. In Proc. of Intl. Conf. on Machine Learning, 2020.
[5] Jure Zbontar, Li Jing. Ishan Misra, Yann LcCun, and Stéphane Deny. Barlow twins: supervieed learning via redundancy reduction. CoRR, 2021.
[6] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Proc. of Advances in Neural Information Processing Systems, 2017.
[7] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Proc. of Advances Neural Information Processing Systems, 2020.
[8] Eric Arazo, Diego Ortego, Paul Albert, Noel O'Connor, and Kevin McGuinness. Pscudo-labeling and confirmation bias in deep semi-supervised learning. Proc. of International Joint Conference on Neural Networks, 2020.
[9] Dong-Hyun Lee et al. Pscudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Proc. of Intl. Conf. on Machine Learning Workshop, 2013.
[10] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In 5th International Conference on Learning Representations, ICLR 2017, Toulon. France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https: htttps://opeereee/tt /wwp///weeeeeeeee.e////p/...e..t net Btt.erunmndBBBoOooooo00
[11] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: A regularization method for supervised and semi-supervised learning. IEEE Trans. Pattern Anal. Mach. Intell., 41:1979- 1993, 2019.
[12] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthij Douze. Deep clustering for unsupervised learning of visual features. In Proc. of European Conference on Computer Vision, 2018.
[13] Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Unsupervised pre- pre-training of image features on non-curated data. In Porc. of IEEE Intl. Conf. on Computer Vision, 2019.
[14] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning of visual features by contrasting cluster assignments. In Proc. of Advances in Neural Information Processing Systems, 2020.
[15] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In Proc. of International Conference on Learning Representations, 2018.
[16] Mchdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In Proc. of European Conference on Computer Vision, 2016.
[17] Carl Doersch, Abhinav Gupta, and Alexii A. Efros. Unsupervised visual representation learning by context prediction. In Porc, IEEE Intl. Conf. Computer Vision, 2015.
10

---

[18] Carl Doersch and Andrew Zisserman. Multi-task self-supervised visual learning. In Pore. of IEEE Inil. Conf. Computer Vision, 2017.
[19] Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful image colorization. In Proc. of European Conference on Computer Vision, 2016.
[20] Richard Zhang, Phillip Isola, and Alexei A. Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In Proc. of IEEE Intl. Conf. on Computer Vision and Pattern Recognition, 2017.
[21] Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexci A. Efros. Context encoders: Feature learning by inpainting. In Proc. of IEEE Intl. Conf. Computer Vision and Pattern Recognition, 2016.
[22] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and PierreAAnoinee Manzagol. Extracting and composing robust features with denoising autoencoders. In Proc. of Intl. Conf. on Machine Learning, 2008
[23] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Proc. of Advances in Neural Information Processing Systems, 2014.
[24] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes good views contrastive learning? Proc. of Advances in Neural Information Processing Systems. 2020.
[25] Olivier J. Hénaff. Data-efficient image recognition with contrastive predictive coding. In Proc. of Inil. Conf. on Machine Learning, 2020.
[26] Philip Bachman, R.
Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Proc. of Advances Neural Information Processing Systems, 2019.
[27] R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon Marchildon, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In Proc. of International Conference on Learning Representations, 2019.
[28] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Proc. of European Conference on Computer Vision, 2020.
[29] Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invarian representations. In Proc. of IEEE Intl. Conf. on Computer Vision and Puttern Recognition, 2020.
[30] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison. Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style. high-eerfrmmnce deep learning library. In Proc. of Advances in Neural Information Processing Systems, 2019.
[31] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, Geoffrey E. Hinton. Big self- supervised models are strong semi-supervised learners. In Proc. of Advances in Neural Information Processing Systems, 2020.
[32] Tlya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In Proc. of International Conference Learning Representations, 2017.
[33] Chao Peng, Tete Xiao, Zeming Li, Yuning Jiang, Xiangyu Zhang, Kai Jia, Gang Yu, and Jian Sun. Megdet: large mini-batch object detector. Proc. of IEEE Intl. Conf. on Computer Vision and Pattern Recognition. 2018.
[34] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks, In Proc. of British Machine Vision Conference, 2016.
[35] Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoc Lc. Randaugment: Practical automated data augmentation with reduced scarch space. In Proc. of Advances Neural Information Processing Systems, 2020.
[36] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, pages 211 251- 2015.
11

---

[37] Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to common corrup- tions and perturbations. In Proc. International Conference Learning Representations, 2019.
[38] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. CoRR, 2020.
[39] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. In Proc. of Advances in Neural Information Processing Systems, 2015.
[40] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. Proc. European Conference Computer Vision, 2016.
[41] Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proc. IEEE Intl. Conf. on Computer Vision and Pattern Recogni- tion, 2018.
[42] Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. In Proc. of Advances in Neural Information Processing Systems, 2019.
[43] Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. Porc. of IEEE Inil. Conf. on Computer Vision, 2019.
[44] Jumnan Li, Pan Zhou, Caiming Xiong, Richard Socher, and Steven C. H. Hoi. Prototypical contrastive learning unsupervised representations. CoRR, 2020.
[45] Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations, In Proc. of IEEE Intl Conf. on Computer Vision and Puttern Recognition, 2020.
[46] Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi-supervised learning with ladder networks. Proc. of Advances in Neural Information Processing Systems, 2015.
[47] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: holistic approach to semi-supervised learning. In Proc. of Advances in Neural Information Processing Systems, 2019.
[48] Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. In Proc. of Advances in Neural Information Processing Systems, 2020.
[49] David Berthlot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring. In Proc. of International Conference on Learning Representations, 2020.
12


Reviews:
[{"review_id": "qGeIs5mXEA", "reviewer": "Reviewer_P4s6", "summary": "The authors propose a new framework for constructing teacher models in student-teacher model training, which involves a combination of spatial smoothing (piecewise copying of students to teacher) and the usual temporal smoothing (trajectory-wise parameter averaging).   They situate spatial smoothing inside a more general framework of spatial-temporal smoothing, and recover temporal smoothing and the newly proposed spatial smoothing as special cases of spatial-temporal smoothing.  They go on to show its utility in two learning scenarios (semi-supervised learning and self-supervised learning).", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents a new model-smoothing technique for student-teacher training.  The paper is written with exceptional clarity.  It was always clear from first reading what the authors were attempting, what motivated them, and what they observed. Though the mechanism of spatial smoothing is under-explored, they present compelling evidence that this smoothing method for student-teacher training offers benefits over existing smoothing.  For me, it's a clear accept.\n\nNotes:\n\n- In section 4.3 the authors experiment with the granularity of the spatial smoothing.  They investigate layer-wise versus channel-wise, and neuron-wise spatial transfer.  In 234-247, the authors report: \"As shown in the top of Table 4, SE using NW outperforms its counterpart using LW and CW and achieves comparable performance with TMA. The result reveals that SE itself benefits from the fine-grained spatial ensemble.\"   I feel as if there is more the authors could say here.  It feels as if this experiment could benefit from the perspective of a model averaging argument, or from analyzing the correlation between model components that NW versus CW entail.  I encourage the authors to consider following up on this.\n\n- For Table 2, the results are so similar that it's unclear if there is any benefit of STS smoothing over TMA.  IF possible, could this experiment be replicated several times and the distributions reported?\n\n- The caption for Figure 3 should indicate which panel represents model parameters, and which presents generated supervision signals.  I can guess that they are in order induced by the caption, but more clarity here would be appreciated.\n \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not discuss the limitations of their work directly, but do indirectly present evidence that shows the limitation of how much benefit STS offers in different scenarios.  \n\nI would like to see a statistical analysis of the benefits of STS (see above)", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns are apparent to me", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "ftSlU6C_sDY", "reviewer": "Reviewer_um8Q", "summary": "This paper proposes a novel technique to perform model smoothing in the context of the student-teacher training framework for tasks like self-supervised learning and semi-supervised learning. In essence, the proposed method involves randomly sampling a small portion of the student model and use the selected portions to replace its teacher counterpart. The authors of the paper demonstrate that the proposed method is also complementary to the commonly used temporal moving average method and when combined together, can yield state-of-the-art performance in multiple benchmark datasets. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "All in all, the paper is very well written and easy to follow. Although the proposed method can be quite simple, it can be of great significance to machine learning practitioners as a straightforward and convenient way to improve performance. Moreover, the authors of the paper conduct a series of experiments to demonstrate the effectiveness of the paper. All in all, while I think the novelty can be somewhat limiting because the idea is quite a simple one, this paper is a good submission in terms of the quality of the work/paper. As such, I will be a little bit conservative with my score for now and would like to hear the opinions of other reviewers. I would be happy to raise my score to a 7 if there is a consensus among the reviewers.\n\nQuestions: \n\n1. It is stated in the paper that a crucial factor for a teacher model to yield good performance lies in its stability/small variance, and the proposed spatial ensemble is a way to reduce variance. However, intuitively, doing so (especially when combined with TMA) can potentially harm the \"accuracy\" of the teacher model? Is this not an important aspect for teacher models to be effective?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1 hour"}, "extra_text": {}}, {"review_id": "dXZV1vk3L2N", "reviewer": "Reviewer_ytVw", "summary": "Different from the popular EMA mechanism in self-supervised and semi-supervised learning, this paper proposes a new mechanism for model smoothing in spatial ensemble perspective as complementary to the popular temporal moving average.  By combining the proposed Spatial Ensemble and the popular Temporal Ensemble, this work proposes to integrate temporal moving average and spatial ensemble into a unified model smoothing method, named as Spatial-Temporal Smoothing (STS). Experiments on self-supervised and semi-supervised tasks show that STS brings improvement to some methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work rethinks current Temporal Moving Average (TMA) in the student-teacher framework, and the proposed Spatial-Temporal Smoothing (STS) is novel for the student-teacher framework, which also achieves superior results on multiple baseline methods in self-supervised and semi-supervised learning.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. Based on the results in Table1 and Table2, the Spatial Ensemble (SE) works worse than the TMA, and it can only work when cooperating with TMA, so basically, STS can only be viewed as an upgraded version of TMA.\n2. Comparing the default layer-wise (LW) with channel-wise (CW), it seems that CW achieves much better performance in Table 4, also slightly better performance than LW in STS with nearly the same time. Why does this work choose LW as the default method rather than CW. \n3. The improvements of STS on ImageNet are incremental in most methods. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: There is a consensus among the reviewers that this paper presents significant results and is of interest to the conference. 
Decision: Accept (Poster)