Bridging the Gap from Asymmetry Tricks to Decorrelation Principles in Non-contrastive Self-supervised Learning
Authors: Kang-Jun Liu
Masanori Suganuma
Takayuki Okatani
Venue: NeurIPS 2022 Accept (2022)

Abstract: Recent non-contrastive methods for self-supervised representation learning show promising performance. While they are attractive since they do not need negative samples, it necessitates some mechanism to avoid collapsing into a trivial solution. Currently, there are two approaches to collapse prevention. One uses an asymmetric architecture on a joint embedding of input, e.g., BYOL and SimSiam, and the other imposes decorrelation criteria on the same joint embedding, e.g., Barlow-Twins and VICReg. The latter methods have theoretical support from information theory as to why they can learn good representation. However, it is not fully understood why the former performs equally well. In this paper, focusing on BYOL/SimSiam, which uses the stop-gradient and a predictor as asymmetric tricks, we present a novel interpretation of these tricks; they implicitly impose a constraint that encourages feature decorrelation similar to Barlow-Twins/VICReg. We then present a novel non-contrastive method, which replaces the stop-gradient in BYOL/SimSiam with the derived constraint; the method empirically shows comparable performance to the above SOTA methods in the standard benchmark test using ImageNet. This result builds a bridge from BYOL/SimSiam to the decorrelation-based methods, contributing to demystifying their secrets.

Bridging the Gap from Asymmetry Tricks to Decorrelation Principles in Non-contrastive Self-supervised Learning
Kang-Jun Liu' Masanori Suganuma' Takayuki Okatani
'Graduate School of Information Sciences, Tohoku University
2RKENN Center for AIP
(kjliu, suganuma, okatami]@vision,
wos.touoouacc jp.
Abstract
Recent non-contrastive methods for self-supervised representation learning show promising performance. While they are attractive since they do not need negative samples, it necessitates some mechanism to avoid collapsing into trivial solution. Currently, there two approaches collapse prevention. One uses an asymmetric architecture on a joint embedding of input, e.g., BOL and SimSiam, and the other imposes decorrelation criteria the same joint embedding, e-g., Barlow- Twins and VICReg. The latter methods have theoretical support from information theory as why they can learn good representation. However, it is fully understood why the former performs equally well. In this paper, focusing on BYOL/SimSiam, which uses the stop-gradiens gradient and predictor as asymmetric tricks, we present novel interpretation of these tricks; they implicitly impose a constraint that encourages feature decorrelation similar to Sewlowwwwis///VRReee We then present a novel non-contrastive method, which replaces the stop-gradient BYOL/Sim with the derived constraint; the method empirically shows comparable performance to the above SOTA methods in the standard benchmark test using ImageNet. This result builds a bridge from B BYOL/SimSiam to the decorrelation-busn based methods, contributing to demystifying their secrets. Source code is available at hhttps: /tte:://ttttte com cww/coo////...... httm/KJJrc/brrre htt////JJrbbbbdiinggggee the- gap
1 Introduction
Recently, many methods have been proposed for self-supervised learning of visual representation [ 6.8 12 16 They share fundamental idea: learn visual representation of images invariant to a range of image transformations maintaining their semantics. This idea is implemented as the optimization of objective that the joint embeddings of different views of an input image, e.g., different image subregions cut from a single image and subjected to additional data augmentation, should be close each other.
The methods are categorized into contrastive and non-contrastive methods. The latter needs only positive samples (i.e., different views of the same images), whereas the former also needs negative samples (i.c., different images). Therefore, although stabilizes learning, contrastive methods tend to need large memory 4 or additional measures (e.g., momentum encoder [12] to alleviate it. On the other hand, the non-contrastive methods are potentially more efficient. However, their objective has trivial solution: map all inputs into single point in the feature space. Thus, non-contrastive methods need additional measures to prevent this feature collapse.
To do so, B BYOL/SimSiam 66 10 employs asymmetric structure into the joint embeddings, i.e., stop- gradient and a predictor; see Fig. a). The predictor is subnetwork placed on top of one of the two
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

0.4
BYOL
Linear BYOL
0.3
SimSlam
Barlow Twins
0.2
VICReg
Ours
0.1
0 20
40
60
80
100
Epochs
Figure 1: Experimental results verify claim that asymmetry tricks implicitly encourage feature decorrelation. Σ is the correlation matrix of features extracted from ImageNet validation images. BYOL/SimSiam makes E approach to identity matrix I, so do feature decorrelation methods (i.e., Barlow-Twins/VICReg This is also the case with the proposed method. See Sec. 5.2 for more details.
woint-embedding pipelines; it predicts the output of the other pipeline from its input. B BYOL/SimSiam uses the loss of minimizing the prediction error, aiming at the above objective learning invariant feature representation. It back-propagatee. the gradient only to the first pipeline having the predictor while it stops the gradient flow the second pipeline. (While BYOL inherited momentum encoder for the second pipeline, later it was recognized 6 [14
that it is not indispensable.)
However, has not been well understood why the above asymmetric structure prevents collapse and further enables to learn good representation. Tian et al. theoretically tackle this question, deriving some results helping to understand the tricks [14. Nonetheless, still lack a complete understanding of the working mechanism of the asymmetry tricks, especially why they lead to the learning of good representation.
In this paper, we show that the asymmetry tricks employed by BYOL/SimSiam have an additional implicit effect, such that the features extracted from different images will be decorrelated. Specifi- cally, we extend Tian et al.Js analyses under similar assumptions. We then show that the updating dynamics of the predictor's weights and the features (i.e., the predictor's inputs) indicates that the minimization implicitly imposes constraint achieving feature decorrelation; see Fig. | Finally, we show through experiments that we can eliminate stop-gradient by using the constraint regularizer with the standardization of features. Specifically, using the regularizer and the standard invariance loss leads learning as good representation as the state-of-the. oo-thh--rttttt SSL methods, including BYOL/SimSiam.
explicit
This result provides link from B BYOL/SimSiam to the other group of non-contrastive methods, i.e. Barlow-Twell Twins [66 and VICReg [ These methods do not employ an asymmetric structure and instead incorporate explicit objective of decorrelating features of different input images. As explained [66. information theory supports the goodness feature decorrelation for representation learning, explaining why it helps learn good representation. The link between the two groups of methods implies that the same is true for B BYOL/SimSiam. In short, it implies that the asymmetry tricks implicitly achieve feature decorrelation, leading the learning of good representation without collapse.
2 Related Work
2.1 Asymmetry Tricks: BYOL/SimSiam
BYOL [00 and SimSiam ( employ asymmetric architectures; see Fig. 2 a). Let fl, 2 Rd be the outputs an identical network for two views of an identical image æ. BYOL minimizes the following loss:
p(f') ||((f))|2
(f
f2
27
CBYOL = E,
StopGrad
(1)
1f

---

L2 Joes
(144))
1.2 Joes
Proietor
Poojedot
Projector
Projector
Prjjecur
Projector
Projector
Projecter
Backsone
Backlion:
Relchone
Bacaluuee
Bacchone
Bachlone
Bacalome 1
2
x
(a) SYOL/SimSiam
(b) Barlow- -woins
(c) VICReg
(d) Ours
Figure 2: Illustrations of representative non-contrastive self-supervised methods for representation learning and our method.
where p(-) a predictor, e.g., two-layer MLP with ReL.U and batch normalization in the intermediate layer; and StopGrad indicates that the gradients are not back- proppaaoee with the second path for f². There are several differences between BYO and SimSiam; while BYOL uses momentum encoder to compute f², SimSiam does not, and SimSiam uses loss based on cosine similarity.
unclear why BYOL and SimSiam can avoid collapsing only impose augmentation invariance. In [4], incorporating some simplification of the model and assumptions on the data, the authors analyze the dynamics of how the above minimization updates the network weights. They then show that i) there is an implicit balancing effect between the projector's 's weights and the predictor's weight, which may help prevent collapse; ii) stop-gradien, indispensable for collapse prevention. They then present a method named DirectPred, which directly sets the weight of linear predictor based on the principal component analysis of the extracted features. However, these do not fully explain the working mechanism of BYOL/SimSiam, especially why they can learn good representation.
trivial solution even though they
2.2 Decorrelation-basee Barlow-TwinsVIICRes
As shown in Fig. bbb,, Barlow-Twin 16
minimizes the following loss:
CB |corr(F*, F²) /11
where Fl (E R) and (E R are matrices storing all the f.'s and ff of batch of inputs I,'s (i = 1, 1,....,). n). respectively; corr(F'. (F) Rd is the cors-coorelation between F' and F2 along the batch dimension and is the identity matrix. Note that Zbontar et al. separate the loss into the terms of diagonal and non-diagonal components of corr( For(F' FF) with different weighting constants, which is omitted in (2) for brevity.
As shown in Fig. Cc) VICReg 1 minimizes the following loss:
LVICReg
> ll
f
V
+CFF2))
2n
d
+ 11 Σ (m, C(FF) +6) + max 1 +)),
(3)
where C(F') and C(F²) are the auto-covariance matrices of f and f², respectively, and μl and // are weighting constants. As shown above, VICReg separates the diagonal and non- non-diagonal components of the matrices and assigns different weights, as in Barlow-Twins [66] and employs hinge loss for the diagonal term.
Barlow-Twin and VICReg share a similar objective, i.e., decorrelating features of different inputs, although the targets slightly different (cross-correlation vs. auto-covariance).
3 Roles of Asymmetry Tricks
In this section, we consider what role(s) the asymmetry tricks, i.e., predictor and stop-gradient, play. We extend the results of the study [14, showing new interpretation.

---

3.1 Problem Statement
We succeed the assumptions/setings employed [4 except an assumption on the data distribution. Namely, we approximate the backbone + projector (i.e., the mapping from an ipput image r to its feature f) be linear mapping. We then consider (a variant of) BYOL with linear predictor minimizing the following & loss:
1 C -= TZ II (fo
where Eg is the expectation outputs of the projector for two different views of z; Wp is the weight the linear predictor. Note that the original version of BYOL employs different loss ( and nonlinear predictor, i.e., MIP having two or more layers with ReLUs. We do not use the assumption on x's distribution employed in [14 and instead assume each to be normalized, i.e., _Tx 1.
the distribution of an input image fl and f2(E Rd) are the
In the implementation BYOL/SimSiam and others, Ex is replaced with the average over mini-batch. Using F [ft..... ful/ vn and 1 [/ 1.... / VT, where is the mini-batch size, we have F FIT = Ex [ etc. As result, (4) is rewritten as
L
W StopGrad
2 IW, StopGrad F3)
(5)
2m
BYOL/SimSiam use not only the orientation F - 12 but also the other way F - Ft; letting L' be the loss for the second way, they minimize C + L'. For brevity, we show only in what follows unless otherwise noted.
Finally, our goal is to understand how Wp are updated and how F''s behave a result when minimizing (4) or equivalently (5)
3.2 Dynamics W and F
Following [14], regard the network weights varying during loss minimization time-dependent
variables, e.g., Wp = (t)... We do this also for intermediate layer outputs, e.g., F F'(())
Theorem 3.1. When minimizing the loss (5) with weight decay n, (t( and Fl (t) satisfy (t( T Wp(t)= = (t) - =** C,
(6)
where C is a constant matrix determined by the initial weights.
To prove Theorem 3.1 need the following lemmas. Lemma 3.2. The derivatives of the loss with respect W, and Fl can be respectively given as follows:
Oc
W W.Flpt FIT - F2F1T
OW,
p
OC
W W.T.WFT W, - W p2.
(8)
OFI
Lemma 3.3. When minimizing the loss updated with the following velocity:
with weight decay 1) under the above assumptions, F² F²)
dF (t)
ac
F
nF
(9)
dt
OF1
We show the proof of these lemmas in the supplementary material. Now we prove Theorem 3.1
Proof of Theorem 3.7 When minimizing the loss (5) with gradient descent and weight decay, the velocity of W, and F during the minimization are respectively given by
dW,(1)
OL
W
dt ow, W P
(10)
dF'(t)
AC
F1
dt 0F nF
(11)

---

100
200
200
200
200
200
2001
200
200
200
200
2001
20
Epochs
Figure 3: Visualization of Wp IR x at different epochs of BYOL with linear predictor. The predictor weight Wp approaches a diagonal matrix ef with positive value c.
175 13
Eggunalass
0.0
0.0
200
2%
2%
230
350
25
-00
Rpode
Rouds
(a)
(b)
Figure 4: The predictor weight W, of BYOL with linear predictor (a) becomes symmetric and (b) its eigenvalues converge to either positive value or 0, as training epochs increase.
We then calculate W, W.W, FIFIT, to which substitute 7 and (8) yielding
(t(() dtp(t) dt + n (tT dt) F² (1) + +nF (1).
(12)
dt
Then, adding the above with its transposed version and then taking integral over t, we have
e 2nd w (t) Wt(t) 2²d (t) +C,
(13)
where C' is a constant matrix determined by the initial values (0) and (0). The multiplication with e 2rp yields
For large f, 6 reduces
(t) W Wt)()= = (1)
(14)
This means that minimizing uncentered covariance fi, fi...., fn.
makes W, Wp approach FIFIT (1/n) feft the
3.3 Relation Feature Decorrelation
Equation (14) indicates the decorrelation of features if W (t) is diagonal. We empirically found this true for BYOL with linear predictor; W, quickly converges diagonal matrix as increases, shown in Fig. This empirical observation together with [14 implies the connection between B BOLL/SimSiam and the decorrelation-be -baed methods.
We provide another empirical evidence based on the eigenvalues of W Figure 4 () shows that W
becomes symmetric with epochs. Figure shows that the majority of its eigenvalues approximately approach single constant and the others go to zero. (This agrees with theoretical result in [14] that each eigenvalue of W, will be either 0 or a positive value stable convergence/equilibiuum state.) These two observations, i.e., W. is symmetric and has an identical eigenvalue, indicates that W, approaches eT.
Deriving Method without StopGrad
4.1 Basic Idea
Aiming to validate the above implication, we consider eliminating StopGrad from (5) and instead impose (14) explicitly in training. Specifically, climinate StopGrad and incoporate |[ W.
5

---

F'F as regularization term as
C - (1W NllK W, - F Fr 11)
(15)
The underlying thought is that since the presence of the asymmetric tricks leads to (1) in the analysis of the dynamics, wish the major difference between the two categories of non-contrastive methods. We will show our empirical findings below that this approach works well when used with either the standardization of features (by adding standardization layer on the projector's 's output) or the elimination of the predictor. We will refer the former STD and the latter I-PRED (i.e., identity predictor).
if the explicit imposition of (14) can substitute StopGrad, which
4.2 Some Consideration for Implementation
BYOI AYmm/SimSiam consider the symmetric loss, i.e., the sum of the losses in the both ways (i.e., F1 ++ F2). Incorporating the proposed regularizer in each way, the total loss will be
C / [IIW - 11 + I1 1 - -F*1l()
X(IW, W, F' F' 11W W, F2 r22 [1)] (16)
Now, we consider replacing F1 and F2 in the third and fourth terms as - W Wp respectively. Assuming the first and second terms are sufficiently small, we can say that this modification will not change the objective here. We found through experiments that this modification leads to slightly better performance and less sensitivity to the choice X.
P F2 and F2 -
Going back to one way expression (i.e., Fd - F2), we write the modified loss by pairing the first and the fourth terms of (16) (and the second and the third terms in the other way) as follows:
Augmentation Invariance
C Clav 1[,
F
Goov WW W, (A((FFF'TWW I.
(17)
Covariance Maintaining
where Cnvv and Ccov are weighting constants of the two terms, and d the feature dimension size (i.c., the column size of F);; the division by d makes the effective range of Coov similar to Cinv.
We summarize the interpretation of the above loss here. The first term enforces augmentation invariance of f. and f? for any i(= 1....,n) and the second term enforces covariance maintaining, or specifically, avoiding the collapse of (uncentered) covariance of fi.
should be noted again that the loss 17 does not have a factor directly preventing feature collapse, with YOOL/iim An obvious collapsing case is when the projector always outputs zero, i.e,, f f for any i. In this case, the loss vanishes when W, O. To avoid this, we apply standardization to the output of the projector, i.e., Fl [fi..... f'l] Specifically, we compute
(18)
where = Eド and 0² - - 2 and set F1 <- AAI/v..AAII.. We perform the same for F2, Note that SimSiam (6] employs the same strategy, and interestingly, does Barlow-Twins: it is reported (I] that Barlow-Twins performs slightly worse without standardization layer. However, is also known that standardization alone is not enough: the space of f can be degenerate, as reported in [13] As will be shown later. we empirically found that the loss [17 plus the feature standardization successfully prevents W. from being rank-deficient.
4.3 (Dis)similarity with Other Methods
Our method does not use StopGrad, which is stark difference from BYOL.SimSiam/Dired Pred. this sense, our method may be more similar to decorrelation-basee methods, i.e., Barlow- Twins/VICRe,

---

Table I: Results for different combina- tions of the number of partitions, Cinv and Crov in (19
Table 2: Dependency on the size of the projec- tor.
PROJECTOR
#PART
INV
Cov
Acc
#PART
INV
Cov
AcC
4096-256
54.7
61.4
4096-256
32
10
63.7
25
60.6
2048-288 2048-0008200002 2048
1
61.4
25
1
60.1
2018-2048-200
32
10
66.6
25
25
63.4
2018-2018-2044
32
25
67.3
25
67.3
8192-8192-8192
25
69.0
25
25
66.2
32
25
1
67.3
Roughly speaking, we can say that Barlow-Twins enforces the cross-correlation between f and 12 be an identity matrix, and VICReg enforces the auto-covariance of [ (and of (²2) be an identity matrix. In the same perspective, our method deals with the auto-correlation of F (and F*); specifically, it enforces them
the same (i.e., W Wp). This does not mean that our method decorrelates features unless W p Wp is a diagonal matrix. However, similar to B BYOL as discussed in Sec. 3.3 W approaches diagonal matrix with training iteration t in our method well, although its effect weaker than BYL and others; see Fig.
When Wp = I, this effectively eliminates the predictor, resulting in our method getting closer to VICReg. The remaining differences are as follows: hinged loss vs. l norm for the correlation constraint term, a different treatment of diagonal/non-diaggnall elements of the covariance matrix vs. their equal treatment, and the use auto-covariance vs. auto-correlation.
As discussed in [6]. there is an explanation based on information theory as to why decorrelating features leads to learning good representation. However, it is not rigorous proof since the losses of Barlow-Twin and VICReg differ from the ideal objective function suggested by the theory. The same is true of our method. The implication of the goodness of feature decorrelation may be effective on our method as well.
4.4 Partitioned Correlation Constraint Term
We found through preliminary experiments that partitioning the correlation constraint term yields better results. To be specific, we partition Frl, the set of features from a single batch, into k subsets
of an equal size n' = n/k, as F = (1/vk)(F1 P2 m.. PE.. We then modify the loss 17 as
= W WW 12 d
COOv Σ I WIW,,
W 12.
(19)
We will experimentally show the effectiveness of this partitioned loss later. Its theoretical validation will be left to future research. A small remark is that the partitioned loss is suitable for distributed training of the model in a data-parallel manner. We need only compute each partitioned term separately on single computational node, e.g., GPU. Note that this method differs from those found to be effective in [IS]. Specifically, we do introduce different variances for each partition, while theirs use different variances for multiple branches of the Siamese networks. Our method synchronizes the mean and variance among all the nodes, unlike their asymmetrically synchronized "AsymBN." See the pseudo-code shown in the supplementary material for more details.
5
Experimental Results
5.1 Experimental Setting
Architecture We follow the previous studies for the network architecture. Specifically, we use Resnet-50 [1] without the last fully-connected layer (i.e., the classification layer) as a backbone. Following [6] we zero-initialize the weights of the last batch normalization layer in each residual branch (i.e. zero_ine sepp_intt__reee_ residual=True in PyTorch). We then employ projector on top of the backbone, which is a MLP with two or three layers basically having the same width, each intermediate layer

---

having BatchNorm and ReLU. Following the previous methods but BYOL, we synchronize all the batch norm layers, including the standardization layer used for F' and F², across different devices, e.g., using SyneBatchNorn of Pytorch. Our method uses a predictor as with B BYOL/SimSian Note however that ours a linear predictor, in contrast with BYOL/SimSiam whose default settings employ non-linear predictor, i.e., a two-layer MLP with ReLU at the intermediate layer.
Devices We use a machine equipped with two Intel Xcon Platinum 8360Y Processors, eight NVIDIA A100 GPU and 520 GiB DDR4 RAM.
Augmentation We follow the procedure of BYOL 10] and Barlow-Twins [6 for data augmen- tation creating two views of input images. Specifically, we build an augmentation pipeline in the following order: random cropping followed by resizing to 224 x 224, horizontal flipping, color jittering, grayscale convertion, Gaussian blurring, and solarization. While random cropping with resizing is always applied, others are probabilistically applied with randomly chosen parameters. Following the above papers, the application of blurring and solarization asymmetric, i.e., their probabilities differ between the two views. SimSiam and VICReg employ symmetric setting.
Optimization We also follow BYL and Barlow- for the optimizer and its hyperparameter settings. To be specific, LARS is used along with cosine learning rate decay with 10-epoch linear warm-up; weights in BatchNorm and biases are excluded from weight decay and LARS adaptation. We set the base learning rate 0.3 for the whole model, which is multiplied by BatchSize 22////eee/..... train our model for 100 epochs, and the employed learning rate decay to D. at the 100-th epoch. We set the batch size 2048.
/256. We
Evaluation For evaluation of methods, we follow the previous studies 16 16]. Specifically, we use ImageNet (ILSVRC2012) 7 for all the experiments. We use the proposed sulf--uper learning method to train the above backbone, along with projector and predictor, using all the images the training split, where the above augmentation and optimization procedures are employed. We then evaluate the method's performance by training and testing linear classifier using the features extracted by the backbone. We follow the standard evaluation protocol employed in the previous studies. Specifically, we train linear classifier with SGD with momentum 0.9, batch size = 256, weight decay- = 10-6, 6, and learning rate subject to cosine scheduling with the base rate = 0.3, for 100 epochs.
5.2 Feature Decorrelation
The previous sections show that the asymmetry tricks implicitly enforce feature decorrelation. To
experimentally validate this, we analyze how the auto-correlation matrix Σ of extracted features (i.e., fP's) changes during training. For input images, we use all the images of the ImageNet validation split without applying random data augmentation. Figure shows how || - I| changes during the training of a network by different methods. We can see that all the tested methods decrease ||Σ - I| with epochs, including those using the asymmetry tricks (i.e., BYOL and SimSiam) and those using different forms of explicit decorrelation constraint (i.e., Barlow-Twing VICReg, and ours). It is noteworthy that this applies the BOO with linear predictor (i.e., the assumed configuration) and also B SYOL/SimSiam with a non-linear predictor.
5.3 Performance with Different Configurations
Our method has three hyperparameters, i.c., the partition number ks the weight Cinv of the invariance term, and the weight Geov of the correlation term, in 19). We conduct experiments to examine how their choice affects the performance using the above evaluation procedure (i.c., ImageNet and linear probe evaluation). We use projector whose size is 0048-04482200 2048220888800 here. Table shows the results for several combinations of these parameters. We can observe that partitioning the correlation loss and setting the weights Cinv Ceov yield better performance.
The decorrelation-basee methods, such as Barlow- and VICReg, are reported to show better performance with a larger projector. We examine our method's dependency on the size of the projector. Table shows the results. It is observed that a larger projector leads to better performance, similar to Barlow-Twins/VICe Twins/WICiessee this tendency persists for different hyperparameter settings.
8

---

Table 3: Effects of feature standardization and an identity predictor. STD and 7-PRED indicate the use of the standardization and an identity predictor, respectively.
INV
Cov
STD
1-PRED
STOPGRAD
ACC
0
COLLAPSE
V
50.9
61.4
O
VV
COLLARSE
I
62.6
LV
50.9
58.7
O
V
55.0
5.4 Feature Standardization, Predictor, and StopGrad
As explained in Sec. 4.2 we apply standardization to the features, intending to prevent collapse. We conduct experiments to verify its effects.
As shown in Fig. [ our method shows a similar effect of feature decorrelation to existing methods, which is caused by W,*s behaviour that W, W. approaches diagonal. This raises question what if we set Wp = from the beginning. Setting W means that we do not use predictor, making our method further closer to the decorrelation methods. Thus, we additionally examine this in the experiments.
Table 3 shows the results. (We fixed & = and Cinv and set Geov = 0 or 1 for simplicity. The projector size to 2048-2048- 2048.) First, we can confirm that the feature standardization helps prevent collapse independently of the presence of the correlation term (i.e., Coov = 0 or 1).
that fixing W I, equivalently eliminating the predictor, works well at least with this specific configuration. However, our method having learnable predictor with a different configuration achieves the best performance, as will be shown later.
Second,
Third, "StopGrad" the table indicates the case of applying StopGrad similar to BYOL/SimSiam. Comparing the result with/without it under the same configuration (i.c., Glav 1 and Coov 0), can see StopGrad improves the performance, i.c., 50.9% + 55.0%. Switching StopGrad with our correlation term yields further better results, i.e., 55.0% 61.4%.
Finally, we can see that simply removing StopGrad from BYOL/SimSiam will lead to collapse. We need an additional method to prevent collapsing, i.e., either standardization layer or COV I-PRED (i.e., the correlation constraint term plus the elimination of the predictor).
5.5 Comparison with Previous Methods
the best performance of our method with other methods"] As methods' performance varies depending on the size of projectors, we compare them for a fixed projector size. With the projector size of 2048-2048-2045 our method achieves 67.3%, which is better than Barlow- Twins (63.49 (3.4%% and VICReg (65.1 1%) and slightly lower than SimSiam (68.1%) SimSiam' 's higher performance may be attributable to the use of a non-linear predictor. With 3192-8192-88 method outperforms Barlow-Twins and VICReg again, i.e., 69.0% vs. 68.6%. BYOL tends to learn slower due to the use of momentum encoder and performs worse for epochs 100.
our
6 Conclusion and Discussion
In this paper, we first theoretically showed (under some assumptions) that minimizing the loss (4) with StopGrad and predictor (i.e., BYOL/SimSiam) leads that (a) W, Wp - pit We then showed an empirical observation with BYOL (and many others including ours) that (b) W p Wr + diagonal with increasing We proposed novel method incorporating an explicit regularizer to enforce (a) while removing StopGrad. Finally, we presented experimental results
co.
We used the official code for these methods other than VICReg.

---

Table 4: Results of linear probe evaluation on ImageNet. We run all methods for 100 training epochs before the training of linear classifier. Note that only for VICReg, we copy-paste the results in the same setting from [
METHODS
PROJECTOR
PREDICTOR
ACC
BYOL
1096-256
4096-256
66.7
SIMSIAM
2018-208-2018
512-2048
68.1
BARLOW-TWINS
8192-8192-8199
68.6
2048 2048-2045
63.4
VICREG
8192-8192-8199
68.6
2048-200 2048-2048
65.1
OURS
4096-255
256
63.7
2048- 200--0448225-
2048
67.3
8192-8192-8192
8192
69.0
that the method works; it learns good representation BYOL/SimSiam even though it does not use StopGrad. We have also presented some extensions (Sec. 4.2 and 4.4) that empirically help convergence and better learning. These all point to the connection between BYOL/SimSiam and the decorrelation-bast based methods.
Acknowledgements
This work was supported by JST [Moonshot Research and Development), Grant Number (JP- MJMS2032] and by JSPS KAKENHI Grant Number 20H05952 and 19H01110.
References
[1] A. Bardes, Ponce, and LeCun. VICReg: Vatiane-iivaranee-cvaaaann regularization for self-supervised learning. In International Conference Learning Representations, 2022.
[2] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems, pages 9912 12-99224, 2020.
[3] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9650-9660 9660, 2021.
[4] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, pages 1597-1607, 2020.
[5] T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. E. Hinton. Big self-supervised models are strong supiisuper learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, pages 22245-22255. 2020.
[6] X. Chen and K. He. Exploring simple siamese representation learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1,750- 15758, 2021.
[7] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 248-255 2009.
[8] A. Ermolov, A. Siarohin, E. Sangineto, and N. Sebe. Whitening for self-supervised representa- tion learning. In Proceedings of the 38th International Conference on Machine Learning, pages 3015-3024, 2021.
[9] S. Gidaris, A. Bursuc, G. Puy, N. Komodakis, M. Cord, and P. Perez. Obow: Online bag- of-visual-words generation for self-supervised learning. In Proceedings of the IEEF/CV Conference on Computer Vision and Pattern Recognition (CVPR), pages 6830-6840, 2021.
[10] J.-B. Grill, F. Strub, Altché, C. Tallec, Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, B. Piot, K. Kavukcuoglu, R. Munos, and M. Valko.
10

---

Bootstrap your own latent A new approach to self-supervised learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances Neural Information Processing Systems, pages 21271 2020.
[11] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[12] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9729-9738, 2020.
[13] T. Hua, W. Wang, Z. Xue, S. Ren, Y. Wang, and H. Zhao. On feature decorrelation in self- supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9598-9608, 2021.
[14] Y. Tian, X. Chen, and S. Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In Proceedings of the 38th International Conference on Muchine Learning, pages 10268 10278,00882200 2021.
[15] X. Wang, H. Fan, Y. Tian, D. Kihara, and X. Chen. On the importance of asymmetry for siamese representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11570-1679,) 2022.
[16] J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny. Barlow twins: Sele-super learning via redundancy reduction. In Proceedings of the 38th International Conference on Machine Learning, pages 12310-12320, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5.4
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 3.1
(b) Did you include complete proofs of all theoretical results? [Yes] See supplementary material.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as URL)? [Yes] https
the-gap
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 5
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)? [Yes] See Section 5
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No]

---

(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "n7_uIUSN3sh", "reviewer": "Reviewer_R4L1", "summary": "This paper studies the role of asymmetric “tricks” (like predictor layer and stop-gradient) to prevent feature collapse in *non-contrastive methods* like BYOL/SimSiam. Extending prior theoretical results on a linear representation model [Tian et al., 2021], the paper argues that these tricks implicitly enforce “decorrelation” of features of different inputs, similar to other non-contrastive methods like Barlow-Twins and VICRig, thus establishing a connection between these two paradigms. Based on this analysis, the paper also proposes a non-contrastive method that uses a linear prediction head, explicitly enforces the decorrelation-like constraint that is derived, and does not use stop-gradient. Experiments suggest that the proposed method performs similarly (or better) than all of the aforementioned non-contrastive method on ImageNet linear probe evaluation.", "questions": "Other minor points\n\n- Missing references: [1, 2, 3] either theoretically or empirically study non-contrastive learning (although [3] is very recent). [4] theoretically studies non-contrastive method (SimSiam) with feature normalization and connects it to non-linear CCA, providing a justification for feature decorrelation beyond an information theoretic one.\n- Theorem 3.1 looks similar to Theorem 1 from reference [14] from the paper. It would help to compare and discuss the differences from it. The proof of Theorem 3.1, especially the jumps from eq. 11-13 can be written more clearly. Also useful to statement the assumption that F is linear in the input in the theorem and lemma statements.\n- In the setting from Theorem 3.1, what happens in the dynamics if there is no stop-gradient? Will $W$ not converge to $I$ and there will be no feature correlation?\n- L170-171: Editing issue\n\n\nReferences\n\n[1] X. Wang, X. Chen, S. Du, Y.Tian, Towards Demystifying Representation Learning with Non-contrastive Self-supervision. 2021\n\n[2] X. Wang, H. Fan, Y. Tian, D. Kihara, X. Chen. On the Importance of Asymmetry for Siamese Representation Learning. CVPR 2022\n\n[3] Z. Wen, Y. Li. The Mechanism of Prediction Head in Non-contrastive Self-supervised Learning. 2022\n\n[4] J. Lee, Q. Lei, N. Saunshi, J. Zhou. Predicting What You Already Know Helps: Provable Self-Supervised Learning. NeurIPS 2021", "limitations": "Some limitations are discussed. No societal impact discussed; I do not forsee any immediate negative impacts.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n(S1) The paper is well motivated and studies a relevant question about why non-contrastive methods perform well and avoid trivial solutions that collapsing the representations\n\n(S2) Aims to bridge two sets of ideas to avoid collapse: asymmetric tricks and feature decorrelation\n\n(S3) Paper is clearly written and easy to follow for the most part\n\n(S4) Extensive experiments are conducted and useful ablation studies are reported. The rough conclusion is that the proposed regularization term (along with the standardization) is competitive with previous non-contrastive methods and is a better alternative to stop-gradient.\n\n\n\n\n**Weaknesses**\n\nMy main concern with the paper is about the connetion to feature decorrelation in section 3.3 and the modification to the proposed regularization in eq 17, both of which are crucial points.\n\n(W1) Section 3.3 is the main (probably only) part that supports the paper’s claim about connection between asymmetric tricks and feature decorrelation. However I found this section to be “hand-wavy” and imprecise. To elaborate, the theoretical result suggests that feature correlation for stop-gradient dynamics will converge to $WW^{\\top}$ and if the predictor $W$ converges to $I$, then feature correlation will also converge to $I$ as desired. While the paper makes an interesting empirical observation that $W$ converges to $I$ in practice for many methods, the theoretical justification seems weak as currently presented. It suggests that explanation for $W \\rightarrow I$ follows from Tian et al.’s results, but it is not clear how exactly it follows.\n- L139: “each eigenvalue of $W_p$ will be either 0 or a  positive value $\\lambda^*$ at its stable convergence/equilibrium state”. Why is this true? Which result of Tian et al. suggests this?\n- L143: “If some of its eigenvalues are 0, $W_p$ is close to a diagonal matrix”. This is not necessarily true. If some eigenvalues are 0 then $W_p$ will be low rank and how close to diagonal it is will depend on the span of the non-zero eigenvectors of $W_p$. \n\nOverall I find the connection to feature decorrelation unconvincing until more explanation is provided. What helps the discussion is the empirical finding that $W$ converges to identity.\n\n\n\n\n(W2)  Modification to regularization: The modification from $|W^{\\top} W - F F^{\\top}|^2$ to $|W^{\\top} W - W F F^{\\top} W^{\\top}|^2$ seems benign, but it is not clear how benign it is. Although the objective does not change a lot with this when the first and second terms are sufficiently small, since non-contrastive is all about the dynamics and not the objective, this could potentially play a big role. Furthermore if $W$ is assumed to be symmetric, then $|W^{\\top} W - W F F^{\\top} W|^2 = |W (I - F F^{\\top}) W|^2$ which almost directly enforces feature decorrelation for a full rank $W$. It would be good to get a clarification about this point from the authors and it might also help to report experiment results when using the original regularization $|W^{\\top} W - F F^{\\top}|^2$.\n\n\n\nGiven these issues, I am currently leaning towards a **weak reject**. Some other minor issues and questions are listed below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "kdgRAiqctcK", "reviewer": "Reviewer_LRxj", "summary": "This paper studies the dynamics behind non-contrastive self-supervised learning methods. It shows that the stop-grad approach (e.g., BYOL, SimSiam) is effectively also doing feature decorrelation, similar to BarlowTwins, VICReg. The authors start with Tian et al. (arXiv:2102.06810)'s results and show that the stop-grad will satisfy W(t)^TW(t) = F(t)F(t)^T at large step t. The authors propose a novel method that explicitly uses this term as a regularizer. This can be seen as a soft version of BarlowTwins that require decorrelation but is not necessarily full rank. Experiment results (100ep ImageNet pretraining, linear probe) show that the proposed method outperforms BarlowTwins and VICReg.", "questions": "1. What's the consideration for using a different projector in BarlowTwins and VICReg? The hyperparameters in these methods are optimized for the original projector size. In fact, the accuracy could be much higher if a different set of hyperparameters were used for such a projector head. I   suggest not to claim the advantage of a proposed method by beating a weakened baseline.\n\n2. What's the consideration of \"partition\"? This is not discussed in the theory part or the motivation. How to understand this term?", "limitations": "N/A", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed method is well motivated by the theory, which is an extension of Tian's (arXiv:2102.06810) results.\n2. The ablation study is convincing. It proves the effectiveness of the proposed regularizer term. \n3. The authors provided all experimental details, including pseudo-code and hyperparameters tuning.\n\nWeaknesses\n1. The authors claim that they extend the theory. However, most of the equations are already known in the literature. For example, any linear system under SGD will satisfy eq14; see arXiv:1905.13655, arXiv:2110.09348, arXiv:2201.12680. Therefore, I don't find Section 3.2 supports the claim that \"we theoretically show that the use of stop-gradient and a projector implicitly provides a constraint leading to similar feature decorrelation.\"\n2. The primary connection between the theory and the proposed method is that W^TW becomes a diagonal matrix and will not collapse to a low rank. The authors claim that they empirically observe this happens for BYOL, as shown in supplementary material section C. However, there is no study on the dynamics of this term in the proposed method. The authors need to show that this term follows similar behavior.\n3. The authors did an extensive study on the projector design. With various projector/predictor dimension settings, the proposed method demonstrates superior performances over BarlowTwins and VICReg. However, this study is less significant considering the main claim. The projector design is hyperparameter tuning. It would be better to provide a clear comparison between optimal settings.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JqIFTu4qKlJ", "reviewer": "Reviewer_yFdG", "summary": "The paper follows the theoretical framework proposed by Tian, Y. et al in [14], and further shown that the asymmetry in contrastive learning would lead to a feature decorrelation. As a demonstration (and an empirical proposal), the author demonstrated that the stop-gradient in contrastive learning (BYOL/ SimSiam) can be eliminated by explicitly imposing the derived constraint with additional feature standardization. The empirical results majorly supports the claim and theory of the paper. Eventually a better performance is reported with the proposed method on ImageNet (contrastive learning with linear prob).", "questions": "Could the author share additional data points on the magnitude comparison between the two losses.\n\nCould the author elaborate what are the bn (or bn without learnable parameters) introduced in the network? Do we have additional bn before the standardization. Do we perform sync for the standardization.\n\n\n\n\n", "limitations": "The standardization is not introduced/ discussed in the derivative derivation process, which might lead to different conclusion on the decorrelation part. The standardization is only introduced as an empirical trick, which weak the soundness of the paper and make the theory and finding less convincing.\n\nThe behavior of partition is completely not explained in the paper. The theory part does not introduce anything would lead to partition, which further weak the soundness of the paper. \n\nThe partition behavior (2048 / 32 = 64 per batch) was originally introduced in the paper of \"On the Importance of Asymmetry for Siamese Representation Learning\", the author did not discuss or cite the paper.\n\nStop grad or mmt encoder are not the only \"asymmetric tricks\", there are other asymmetry tricks used in contrastive learning field (for example, multi crop, scalemix, Asym Aug, Mean Encoder and Sync BN, etc. It would make the paper less precise and overclaim if the terminology of \"asymmetric tricks\" were not well/ better defined.\n\n\nMinor issues\n\nIn the Pseudo-code, the function of standardization was named as bn which can be misleading.\n\nThe Pseudo-code is over complicated, it is not necessary to introduce a hyper parameter of GPU. More importantly the num of partition are not necessarily num of gpu.\n\nline 171, the writing is not unfinished. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper pinpointed that the a small set of asymmetry tricks (stop gradient, etc) employed by BYOL/SimSiam have an additional implicit effect of feature decorrelation. More importantly, one of the major trick of stop gradient can be removed and replaced with the constraint as an explicit regularizer. This finding provides a better understanding of the working mechanism of the asymmetry tricks.\n\nThe formula derivation is sound.\n\nThe ablation in the paper is compact and clear. \n\nThe paper is in generally fairly elaborated in smooth writing.\n\nThe paper reports some meaningful datapoints to supported the claim. In table 3, the ablation prove that the stop grad can be eliminated and replaced with INV + STD (or with I-PRED) to keep the network train without collapse. Consistent datapoints supports the claim that COV helps the network converges better, outperform the counterpart without COV constraint. It is also very convincing to see empirical experiments (with no-augmented image) demonstrated the theory of the correlation constraint.\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4vbsb3K_7BI", "reviewer": "Reviewer_w8kU", "summary": "This paper proposed a new non-contrastive learning algorithm, with a additional variance preserving term in the objective, to get rid of the stop-gradient operation commonly used in non-contrastive learning. They provides some theoretical intuition for their method and empirical comparisons between different ways of preventing representation collapses.", "questions": "None", "limitations": "The major limitation of this paper is in its flawed explanation of the proposed method, which loosed its validity because (1) the unanswered reason why the theorem in section 3.2 is important for the non-contrastive learning process (with stop-gradient), and (2) why the objective in equation (16) is equivalent to that in (17), or can be motivated by the same theory behind (16). ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe empirical comparison is thorough and the ablations are very informative of the effects of each components in the method. They have a (although flawed) theory to back their choice of the new regularizer.\n\nWeaknesses:\n1. The major weakness is that the theoretical derivations are not well explained, and do not provide any insights into their proposed method. The results in section 3.2, especially the later comments in equation (14) and line 132-133 only says that with iteration $t$ tends to infinity, the matrix $W_pW_p^{\\top}$ will approach $1/n\\sum_{i}f_if_i^{\\top} $, which can still be a collapsed solution (for example when all $f_i$ are the same and $W_p$ is rank-1). The technical derivations in section 3.2 do not answer why this dynamics (the convergence of $W_pW_p^{\\top}$) is needed for the network to avoid collapse, and why it can be utilized to help designing a new covariance regularizer.\n2. The transition between equation (16) and (17) is unclear and not well-explained. The transition between (16) and (17) is muddled through with unjustified analogy. This weakens the validity of the explanation in section 3.2 on their final loss form (17) and (19).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept