Learning Minimal Representations with Model Invariance
Authors: Manan Tomar
Amy Zhang
Matthew E. Taylor
Venue: ICLR 2022 Submitted (2022)
Keywords: Representation Learning
Minimal Representations
Reinforcement Learning
Self-supervised Learning

Abstract: Sparsity has been identified as an important characteristic in learning neural networks that generalize well, forming the key idea in constructing minimal representations. Minimal representations are ones that only encode information required to predict well on a task and nothing more. In this paper we present a powerful approach to learning minimal representations. Our method, called ModInv or model invariance, argues for learning using multiple predictors and a single  representation, creating a bottleneck architecture. Predictors' learning landscapes are diversified by training independently and with different learning rates. The common representation acts as a implicit invariance objective to avoid the different spurious correlations captured by individual predictors. This in turn leads to better generalization performance. ModInv is tested on both the Reinforcement Learning and the Self-supervised Learning settings, showcasing strong performance boosts in both. It is extremely simple to implement, does not lead to any delay in walk clock times while training, and can be applied across different problem settings. 

Under review as a conference paper at ICLR 2022
LEARNING MINIMAL MODEL INVARIANCE
REPRESENTATIONS
WITH
Anonymous authors Paper under double-bling review
ABSTRACT
Sparsity has been identified as an important characteristic in learning neural net- works that generalize well, forming the key idea in constructing minimal represen- tations. Minimal representations are ones that only encode information required predict well on a task and nothing more. this paper we present powerful approach to learning minimal representations. Our method, called MODINY or model invariance, argues for learning using multiple predictors and single rep- resentation, creating a bottleneck architecture. Predictors' learning landscapes are diversified by training independently and with different learning rates. The common representation acts implicit invariance objective to avoid the different spurious correlations captured by individual predictors. This in turn leads to better generalization performance. We test MODINV both reinforcement learning and vision settings, showcasing strong performance boosts in both. It is extremely simple implement, does not lead to increased wall clock time while training, and can be applied across different problem settings.
INTRODUCTION
Learning efficient representations that generalize well is long standing problem of machine learning, and particularly of deep learning (Doersch & Zisserman, 2017; Noroozi & Favaro, 2016; Oord et al., 2018). Algorithms that exploit structure in the real world through effective inductive biases are key to solving this problem. Several inductive biases have been successfully used in the past, from early work using translational invariance for developing CNNs (LeCun et al., 2004; 2010), to recent data augmentations for developing multiple self-supervised algorithms (Chen et al., 2020; Grill et al., 2020; Chen & He, 2021). key inductive bias in multiple works sparsity of concepts (Hoefler et al., 2021). Essentially, it refers to how objects in the real world tend to interact and affect the dynamies of other objects only in small neighbourhood. Consider for example picking up a pen. This simple task only changes the dynamics of a very small part of the world, namely the pen and the hand the person grasping it. Exploiting such structure allows learning representations that can generalize better in the real world.
In this paper, we develop method that exploits sparsity by learning minimal representations. The well known saying 'ncurons that fire together wire together' points to the phenomenon that that neurons that have similar output for given input also have strong weights between them (Hebb, 2005). Ideally, given the same set of neurons model certain concepts, we would want neurons that relate to particular concept to have strengthened weights while those relating to another concept have weakened weights (Sun et al., 2016). Therefore, low correlation in the outputs of the representation would lead to less redundancy in modelling concepts and thus better generalization. This is precisely the motivation behind minimality leading sparse representations and improved generalization. Minimal representations (Tishby et al., 2000; Shamir et al., 2010) refer to ones that have sufficient but minimal information w.r.t the task hand. Therefore, it must be only possible to solve the task in hand and no other task, thus leading to better generalization for the given task. Another viewpoint is that minimal representations also lead to a reduction in spurious correlations. Since there is minimal noise present in the representation to cause spurious correlations, this again results in better generalization.

---

Under review as a conference paper at ICLR 2022
Predictor Heads
h
h
h
Our approach, which we call model invariance (MoDINV), learns minimal representations by incor- porating multiple predictors over a single, common representation for given task (see Figure 1). Fach predictor head is then trained independently given sample only trains one head) and with a learning rate different to other heads. Theoretically, this can be looked at as a manifestation of the information bottle- neck principle, which balances between sufficiency and minimality. Intuitively, this can be looked at as providing robustness to spurious correlations. Since each head learns different prediction (based on its own optimization and learning on different parts of data set), each head is susceptible to different spuri- ous correlations. However, since there is only one common representation support all the predictor heads, this representation must be robust to any spu- rious correlations as training progresses.
Faster LR
Representation
Standard LR
Slower LR
Figure I: Model Invariance (MODINV) A com- mon representation fa supports multiple predictor heads, each with different learning rate. Each head is optimized based on the task objective (e.g.. SSL loss) and captures different spurious correla- tions. As diversification in learning of predictor heads increases, the common representation must be optimal to reduce the loss for all predictors. 'Red' denotes the head being trained for cur- rent sample.
We evaluate our method on two separate settings; 1) Reinforcement Learning: The DeepMind Control Suite(DMC) (Tassa al., 2018) benchmark with distractors, which involves natural video playing in the background while the agent is controlled in the foreground and 2) Vision: The CIFAR- 10 (Krizhevsky al., 2009) and STL-10 (Coates al., 201 datasets, where no labels are used for learning the representations, followed by the standard linear probing evaluation protocol to test generalization.
Contributions: We introduce MODINV, simple method motivated by incorporating sparsity representations, while showing connections to the information bottleneck principle. MODINV is conceptually simple, easy to implement, and improves representation learning performance in both reinforcement learning (from pixels) and vision settings, reinforcement Iearning. MODINV achieves stronger than state-of-the-arr performance on the DMC Suite with distractors benchmark, both with augmentations and without. In vision, MODINV leads to better performance than the baseline method of SIMSIAM when using linear predictor both CIFAR-10 1-10 and STL-10 datasets.
PRELIMINARIES
2.1
INFORMATION BOTTLENECK PRINCIPLE
The well known information bottleneck (IB) principle (Tishby et al., 2000; Shamir al., 2010) formally describes the ideas of sufficiency and minimality for learning representations, Specifically, states that representations should contain sufficient information to do well on the downstream task but also contain minimal information about the imputs, leading to better, more robust generalization. This is in contrast to minimizing the empirical risk only, which a sound technique in the infinite sample case. However, in the finite sample case, minimizing only empirical risk can lead to poor generalization. A similar observation has been noted in works that build over the invariant risk minimization principle (IRM) (Arjovsky et al., 2019).
2.2 SELF-SUPERVISED LEARNING IN RL
RL considers the agent's interaction with the environment as discrete time y-discounted Markov Decision Process (MDP) (Puterman, 2014) M = (X, P, R, 7, Po). where denotes a finite state space and A is the action space; P = P(x' [z, a) is the transition kernel; R r(x, a) is the reward function; 7 E [0, 1) is the discount factor; and Al is the initial state distribution. The objective is to find a policy π X' A,, where A is the set of probability distributions A such that the value function of a policy π at a state & E X, (x) = E[Σ (I, az) x = xr, π] is maximized. Recently a lot progress has been made in making RL over pixels as sample efficient as when learning over true states. Most approaches can be divided into two classes. The first involves using auxiliary losses over the representation so as to inject as much information about the downstream task possible. These auxiliary objectives include reconstructing observations (Hafner et al., 2020; Ha &
2

---

Under review as a conference paper at ICLR 2022
Schmidhuber, 2018), predicting next observation, reward (Schaul et al., 2015) encoded states (Gelada et al., 2019). The second class borrows ideas from self-supervised learning methods and augments the observations (Yarats et al., 2021; Laskin et al., 2020b) with techniques like random crop, cutout, color jitter etc. Since randomly augmented samples from the same observation have the implicit invariance augmentations is enforced in the representation pipeline. In
same Q values, the case of hard exploration tasks, auxiliary objectives such as estimating value functions of random cumulants has shown to be useful as well.
A common baseline architecture used in a lot of methods is an actor-critic setup where a common convolutional encoder is used to compute the encoded latent state from the raw pixel observations. For continuous control tasks, soft actor-critic (SAC, Haarnoja et al. (2018)) is used as the actor-critic algorithm and the gradients from the actor are stopped from updating the encoded state. Only the critic trains the representation network fe. This is since allowing gradients from both actor and critic leads to noisy estimates and the eventual divergence. Moreover, some methods also add a reward prediction loss from the encoded state, transition prediction loss, or pixel reconstruction loss.
2.3 SELE-SUPERVISED LEARNING IN VISION
An important property of self-supervised methods in vision is avoiding collapse in representation, or trivial solutions where the representation simply outputs a constant. These involve the use of two similar or identical networks, online and a target network. Particular approaches largely fall in two categories, that of contrastive (Chen et al., 2020) and non-contrastive methods (Grill et al., 2020; Chen & He, 2021), depending on how they avoid representation collapse. Contrastive methods create positive and negative pairs from random augmentations of data (e.g., cutout, crop, flip), and enforce an objective such that positive samples are brought closer and negative samples are pulled away in representation space. On the other hand, non-contrastive methods do away with creating positive and negative pairs and instead use the likes of assymetric architectures, stop gradient, and momentum encoders for the target network to learn non-trivial representations.
One method we use to build over later in the paper is SIMSIAM (Chen & He, 2021), which is a non-contrastive method and only incorporates the most essential components required for preventing collapse. In particular, uses two networks, an online and a target network, where both use a backbone network (usually a ResNet 18 or ResNet 50 (He et al., 2016)) and projection MLP (1 hidden layer). Morcover, only the online network has a prediction MLP attached to it, which makes the overall architecture assymetric. The target network has stop gradient applied to it and the overall objective is a cosine similarity loss between the outputs of the online and target networks. It has been noted that stop-gradien is enough to avoid collapse and that slowly updating the target network weights to match those of the online network, i.e. exponential moving average (EMA), not necessary requirement. Adding the EMA the SIMSIAM architecture results in the B YOL (Grill et al., 2020) architecture.
3 METHOD
Consider a task Y, which can correspond to a variety of objectives. For instance, it could be an auxiliary task of predicting the next state when interacting with an RL environment, or simply classification task a vision benchmark. We are interested in training a model that can do well on task Y, where the model consists of two components, a backbone representation fo and predictor/classifirr ha which is attached over the representation. In lot of tasks that involve pixel input, spurious correlations can arise due to various reasons such as lack of non-iid data, irrelevant information, confounders etc. that can result in poor generalization test time. An attractive property for better generalization has been the idea of sparse representations, in that the mutual information of any two dimensions of the representation must be low. However, in order to avoid learning trivial solutions, the representation as whole should encode enough information about the downstream task as well. A combination of both of these objectives then leads to a model with better generalization capability. In this paper, we balance the above two objectives using the idea of model invariance (MODINV). MODINV deploys multiple predictors over a single representation, while training each predictor independently, each capturing different spurious correlations. The common representation acts as a implicit invariance loss which ensures that only the optimal representation remains at convergence. Intuitively, each predictor can be looked at augmented version of the optimal
3

---

Under review as a conference paper at ICLR 2022
1i+1
Tranwition
Recoard
fa
Encoded
Stop Grodiant
SL+1
H
LT
CR
Common Stats Action depresentade rrpresentation tation
fo
* (AA de)
5t
c-11
f2
Head
Observation
2t
Og
Action
2
++++1
f
Head
Actor Loss
Crtbic Loss
1 ++++1
Head
State Prodiction
Roward Predictin
The key idea is to use multiple models of the next encoded state, each grounded with its own reward decoder. At any training iteration. only of the heads is trained for the sampled batch of observations. Each head also independently initialized and uses a diffcrent learning rate w.r.t. the other heads, Right: Natural Distractor in the background for different tasks from DMC Suite.
Figure 2: MODINV for Reinforcement Learning. Left:
model, where the augmentation over the model space and refers to particular spurious correlations arising in the model that differ it from the optimal model. Note that each predictor head is the same in architecture, and we only diversify the learning procedure of each head (through different initialization, independent training, differennt learning rates). This is so that we eventually converge to an optimal representation for a particular predictor/classifier and not for all predictor/classifier families, which is a much more hard to optimize in practise. For all experiments, routing to decide which data sample is used to train which predictor head. Furthermore, the learning rates of the different predictor heads are chosen intuitively, i.e. if the base rate is 3e-3 then we choose one slightly higher rate (5e-3) and one slightly smaller rate (Ie-3) for K=3 predictors.
use random
4 EXPERIMENTS
4.1 REINFORCEMENT LEARNING
For RL over pixels, we build over standard actor-critic setup and use MODINY as part of an auxiliary task that predicts the next latent state (see Figure 2). This auxiliary task is used to train the representation which supports the actor-critic heads. Specifically, the input is the observation the representation fa encodes into the latent state 8, We then concatenate the latent state and the action and pass it through linear MLP to generate a common state-action representation (predicting the next state 5t+1 requires information about both the state and action). Since the true latent state is not known, predicting only the next state $1+1 can result in representation collapse". Therefore, ground the state prediction loss by also predicting the reward, which does have a valid grounding and thus avoids collapse. Finally, we deploy MODINV by using multiple predictor heads for the state prediction task, with each head being coupled with a corresponding reward prediction head.
and
4.1.1 IMPLEMENTATION DETAILS
We implement SAC (Haarnoja al., 2018) as the base agent with the actor and critic sharing a common representation. Our actor-critic setup is similar to the one used in SAC-AE (Yarats al., 2019), except the reward and state prediction task. Our transition or state prediction model is deterministic in nature. involving 3 layer encoder MLP, followed by 3 layer decoder MLP. The reward prediction network is a layer MIP. We do not tune for MI.P sizes and widths at all in reporting our results. Both the reward and transition losses train the representation alongside the critic, We use three MODINV heads, each with a slightly different learning rate of 3e 3, 3e - 4, and 3e 5 respectively.
"Recently it was shown that collapse can be prevented when using cosine loss and an architecture similar BOL instead of squared error loss.
4

---

Under review as a conference paper at ICLR 2022
4.1.2 RESULTS
We test our method on the DMC Suite with distractors. Particularly, we test on the six popular domains of Walker Walk, Reacher Easy, Hopper Hop. Finger Spin, Cheetah Run and Cartpole Swingup. We compare with other methods that learn over pixels including DREAMER, CURL., DBC, and RAD. Dreamer (Hafner et al., 2020) is a model-based method that performs pixel reconstruction and reward prediction to learn model in the latent state space and then performs model-based updates SAC agent. CURL (Laskin et al., 2020a) uses a contrastive loss similar to that in CPC (Oord et al., 2018) to train the representation network, using data augmentations to generate positive and negative samples. RAD (Laskin et al., 2020b) is another method that simply augments the samples in the replay buffer without adding any other loss function to the base SAC setup. Finally, note that both RAD and DrQ (Kostrikov et al., 2020) are similar in performance as they both use data augmentations in the replay buffer samples. Since data augmentations explicitly remove the actual distractions in the background, direct comparison with data augmentations is not fair. Therefore, we also compare our method when augmentations are added alongside MODINV (denoted by MODINY+AUG). We evaluate all methods at both 100K and 500K environment steps, which is the standard number of steps in this benchmark.
base
Our results (Table 1) show that MODINV clearly outperforms all 4 methods (DREAMER, CURL, DBC) at the 500K step mark. Note that since all methods use the same number of gradient steps, MODINV is at slight disadvantage the total number of gradient updates are divided amongst the K' predictor heads.
4.2 VISION
Cossine Similarity
Cossine Similarify
For vision, adopt the base framework of SIM- SIAM, a non-contrastive self-supervised method for learning representations. SIMSIAM has three model components, the backbone network or the representaion, a projection MLP and a predic- tion MLP. We deploy MODINV in the projection MLP, with a single backbone representation and single prediction MLP (see Figure 3). At each training iteration, the same MODINV projection head is sampled for both the online and target networks and trained using the original SIM- SIAM loss function. Unlike the reinforcement learning case, where any of the modules after the representation (state and reward prediction heads) are not used actively in the algorithm, in the vision case the projector and predictor both are used to align the backbone representation (Tian et al., 2021). Therefore, ensuring that each projection head in MODINV is trained sufficiently (as in the base case) is important.
Predictor
Predictor
Projector
Projector
Backbons fo
Backoone fa
Backbone fo
Backbone fo
2/
x
27
SimSiam
SimSiam + ModInv
Figure 3: MODINV for Vision. The key idea is to use multiple projectors for both the online and target net- works. At any training iteration. only one the heads trained for the sampled batch of data. Each head is also independently initialized and uses a different learning rate w.r.t. the other heads.
4.2.1 IMPLEMENTATION DETAILS
We test our approach on the CIFAR-10 and STL-10 datasets. As data augmentations, we use random crop, color jitter, horizontal flip, random grayscale for both datasets and also use blur for STL-10. Furthermore, we deploy three MODINV heads for the projection MLP, with the learning rates of 0.03, 0.3, 0.003 for each head respectively. The learning rate of all other components in the model is set to 0.03, with a weight decay of 0.0004, momentum 0.9. We run two sets of experiments, one with a linear prediction MIP and one with the standard two layer prediction MI.P The rest of the architecture is the same as that of SIMSIAM, with the output dimension of the projector and the predictor MLP set to 2048. We use a batch size of 512 for CIFAR- 10 and 128 for STL-
Linear Ps Non-linear Predictor. For the SSL setting we deploy MODINY in two different configu- rations. The first involves a linear predictor while the second involves a standard 2-layer predictor. The 2-layer predictor has been speculated to be using lot of 'lucky' initialization that lead to alignment between the backbone representation and the prediction MLP (Tian et al., 2021). However, the linear predictor does not have this property and thus testing in this setting offers a much more
5

---

Under review as a conference paper at ICLR 2022
Table Comparisons DMC Suite Distractors. Performance of different methods the 100K and 500K mark. report mean deviation seeds. MODINV is compared with DREAMER (Hafner et al.. 2020), CURL (Laskin et al., 2020a), DBC (Zhang et al., 2020), and RAD (Laskin et al., 2020b), MODINV without augmentations uses a s stopgrad for the critic gradients while MODINV+AUG allows gradients, similar
RAD.
100K STEP SCORES
DREAMER
CURL
DBC
MODINV
RAD
+UDINV+AUG
WALKER, WALK HOPPER, HOP CHEETAH, RUN
345 +:
60 1
198
246 +
465 d:
371
0.6 + 12
0 1
0 0
0 1
0 D
101
22
48 +
145
228 106
148
REACHER, EASY
56 + 85
186 +
202 +
216 - 141
482 + 261
635 + 333
FINGER, SPIN
188
+
170 + 137
240
122
782 31
591 +
CARTPOLE, SWINGUP
133 +
254 +
311 + 74
567 = 98
845 + 14
668 = 64
S00K STEP SCORES
WALKER, WALK
691 + 174
51 + 16
572 +: 130
742 + 99
930 +: 19
870 43
HOPPER, HOP
0.9 = 0.5
1.2
25
112 + 96
65 +
CHEETAH, RUN
266 - 73
50 +
275 + m
462 = 115
425 + 33
361 + 39
REACHER, EASY
223
199 1
197
976 1
FINGER, SPIN
387 1: 128
34 12
755 1 67
780 1 189
982 1
974 1:
CARTPOLE, SWINGUP
119 +
199 + 63
50
814 11
837 23
837 +
Table 2: Comparisons STL-10 and CIFAR-10. 10. All are based ResNet-18 pre-training. Evaluation is on a single crop.. MODINV added the base architecture of SIMSIAM (Chen & He, 2021) with 2-layer projector (2048-d) and layer predictor with 512-d hidden layer. We use weight decay of 5e 4, learning rate 0.03 with a cosine decay.
STL-10
CIFAR- 10
lincar predictor
40 ep
60
80 ep
100 ep
50 ep
100
200 ep
300 ep
SIMSIAM
77.1
80.7
83.7
85.1
40.5
33.0
41.4
41.3
SIMSIAM MODINY
77.0
78.4
81.2
83.5
51.7
70.7
78.9
82.5
SIMSIAM
77.4
80.9
84.1
86.4
67.4
76.6
81.7
SIMSIAM MODINV
78.9
78.9
83.4
85.4
64.7
75.4
79.9
robust evaluation option. Unless specified otherwise, our ablations for vision in Section use the linear predictor setting for MODINV.
4.2.2 RESULTS
We see that with linear predictor, the base SIMSIAM version fails for CIFAR-10 and only achieves 41.3% accuracy while the MODINV version achieves an accuracy of 82.5% at 330 epochs, similar to that of the 2-layer predictor case. A similar gap in performance is observed for earlier training epochs as well (Table 2). A careful reader might attribute this gap to the use of different learning rates in MODINY, since relatively slower learning rate for the non-predictor layers correspond to improved performance, especially when EMA is not used in the target network (Tian et al., 2021). To test this, train the base SIMSIAM model individually with the different projector learning rates used in the MODINV version. We see that all three learning rates fail to the same performance of around 40% accuracy. This clearly shows that MODINV contributes much more to better generalization than using slower or faster learning rates.
Note that this is not complete collapse since the accuracy still increases very gradually after 200 training epachs.

---

Under review as a conference paper at ICLR 2022
5 ABLATIONS
5.1 ABLATION ON NUMBER OF HEADS
In our experiments, we observe that only = 3 heads are sufficient to provide good performance boosts compared when not using MODINV (i.e. K = 1) across the RL and SSL settings. Furthermore, with everything else the same, adding an extra head after K leads to diminishing performance gains. This can be most clearly seen for SIMSIAM MODINV, where we see increasing gains with increasing K (see Table 3). Note that although relative gains saturate after K = 3, it is possible that further diversification predictor training can lead even better results when K
Table 3: Ablation on CIFAR-10. Performance of 100 epoch pre-training for different number heads.
head (SIMSIAM)
2 heads
3 hcads
5 heads
MODINV
40.5
57.0
70.7
70.2
5.2 CORRELATION IN DIMENSIONS
As mentioned, our initial motivation stems from sparsity in concepts. This refers to low mutual information between any two dimensions in the representation fo. A useful metric to check for such a characteristic is therefore the mean correlation between any two dimensions. We plot this metric during training with and without MODINV. Ideally, we would hope to get lower mean correlation when using MODINV as opposed to when not. Our experiments show that this is indeed the case, as the mean correlation decreases much more steadily when use MODINV, Moreover, the decrease in this metric has a strong correlation with the performance of the agent.
Es 20, 20.j
- C T= - P( + A XE 574
c22
where Cis = V Eo (Ebr))
VVV Es (20.))
Based on this observation, we also without MODINV) and add the mean correlation between dimensions of fo as an auxiliary loss. Equation I describes the exact loss, where the summation is over the batch The first term the state prediction error from Section 4.1 while the second term is the exact loss from Zbontar et al. (2021). Our results show that optimizing directly for this loss leads to much better performance in the Reacher Easy task, while much worse performance in the Walker Walk (Table 4). We choose to test on these two domains since they are hard to solve for most methods that we compare to. This an interesting observation since similar losses that have been used successfully in vision recently (Bardes et al., 202 1) might offer strong performance improvements in RL as well.
experiment where take the baseline architecture (i.e.
Table 4: Ablation for Mean Corr. Loss. Performance at 500K for MODINY and the mean correlation loss version from Eq. for different À values.
MODINY
Mean Com: Loss (Eq.
= 5e
3
= 5e
= 5e
À = 5e
WALKER, WALK
742 99
collapse
392 = 181
479 + 227
688 78
REACHER, EASY
552 + 197
collapse
713
739 + 201
678 = 293
5.3 IMPORTANCE OF DIFFERENT LEARNING RATES
Table 5: Ablation for different learning rates for MODINV. SIMSIAM + MODINV with lincar predictor with different sets of learning rates for the 3 projector heads.
[3e-2, 3e-1, 3e-3)
Same Ir 50.7
[3e-2, 3. 3e-4)
MODINV
70.7
52.0

---

Under review as a conference paper at ICLR 2022
When using multiple heads in MODINV, it important to diversify the learning of heads they capture different spurious correlations. Using different learning rates is a vital component for achieving this, besides different model initialization and independent training. We test this by evaluating performance with multiple sets of learning rates for K = 3 predictor heads. We see that performance improves when the learning rates diverse as compared to when all heads the same learning rate (Table 5). Moving on to diversifying learning of predictor heads through different optimizers is left for future work (see Section 7).
5.4 ABLATION ON ONLY PREDICTING REWARD
The RL application of MODINV involves both state prediction and reward prediction. We now test how the performance might vary we only have the reward prediction task. It is worth noting that MODINV can be applied to the actor-critic heads well, however optimization in that becomes difficult as the critic learning loss does not have a stable ground truth. In Table 6 we see that performance deteriorate in general for the three tasks, especially for Cheetah Run. This may suggest that state prediction provides a slightly more stable ground truth to optimize the representation.
5.5 ABLATION WITH DATA AUGMENTATIONS
Our results show that MODINV acts complimen- tarily augmentations. In our experiments in the RL setting, we observe that adding data aug- mentations over MODINY leads to better perfor- mance than when only using MODINV. 'Crop' the only augmentation used here, which is standard for RL environments. Essentially, crop- ping removes a lot of background information which is irrelevant to the task (top-down ap- proach), thus aiding in better or more robust estimates for next state and critic losses. However, not all augmentations have similar effect (e.g. flip), and thus end up with similar performance as the baseline. On the other hand, MODINV avoids spurious correlations by enforcing minimality, resulting in bottom-up approach to robust reward and critic loss optimization.
Table 6: Ablation on MODINV for RL. Perfor- mance at 500K for only state prediction vs both state and reward prediction.
Only Reward
State and Reward
WALKER WALK
704 + 85
742 + 99
REACHER EASY
543 + 271
552 :: 197
CHEETAH RUN
235 + 83
462 115
6 RELATED WORK AND DISCUSSION
6.1 DECODABLE INFORMATION BOTTLENECK
The IB principle does not take into account the classifier/predictor family which is attached over the representation and thus can be considered too restrictive. Moreover, the minimality term is hard to approximate for practical losses. To remedy this, the decodable information bottleneck (Dubois et al., 2020) recently introduced notions of sufficiency and minimality for given predictor/classifier family. They also provided a practical method which has a similar structure to the MODINV approach. In particular, multiple predictor heads are attached to a common representation, where one head (sufficiency head) minimizes the standard empirical risk while all other heads (minimality heads) are learnt such that they cannot predict arbitrary relabellings of the same data. This is done by providing different random relabellings and then reversing the gradients for all the minimality heads. This ensures that the common representation contains information required predict only the correct labels and no other. The idea in MODINV is similar but we do not deploy reverse gradients or relabel data, which in practise can lead instability issues. Therefore, MODINV can be seen as specific instantiation of the DBC framework, following similar theoretical guarantees while being much easier implement and resulting in better performance.
6.2 DECORRELATION IN NON-CONTRASTIVE SSL
Recently, BARLOW TWINS (Zbontar et al., 2021) introduced new non-contrastive self- supervised learning method which does not use an asymmetric architecture while still avoids collapse in representation. This was achieved by enforcing the off-diagonal elements of the cross-correlation
8

---

Under review as a conference paper at ICLR 2022
matrix of the outputs of the representation be zero. Essentially, such loss ensures that each output dimension of the representation is decorrelated with each other while minimizing the standard loss of enforcing invariance to augmentations. Interestingly, the BARLOW TWINS loss can be traced back as an approximation to the information bottleneck principle again, hence connecting decorrelation in representation dimensions introduction). We showed that MODINV also leads to more decorrelation in the representation dimensions, thus supporting this observation. However, we also showed that directly optimizing for this loss might not be a good idea always, with such variant leading to worse empirical performance than MODINV in the RL setting. Nevertheless, it is important to note that the MODINV and BARLOW TWINS objectives are primarily complimentary in nature. This is since MODINV always requires sufficiency objective to start with, which could be provided by any SSL loss which avoids collapse, be it SIMSIAM with the use of predictor and stop gradient or be it B BARLOW TWINS with the use of the decorrelation loss. In similar flavor to BARLOW TWINS, VICREG (Bardes et al., 2021) also uses the decorrelation loss from BARLOW TWINS and adds variance regularization loss to prevent collapse. Finally, Hua et al. (2021) introduced a technique called decorrelated batch normalization (DBN) and also showed that collapse be avoided with predictor.
indicator for better generalization (as noted similarly in the
6.3 AUXILIARY VALUE PREDICTION IN RL
To improve representation learning in RL, a prominent idea has been that of predicting multiple value functions from a common representation. Sutton et al. (2011) use random cumulants to define the different value functions. The value improvement path (Dabney et al., 2020) idea uses mixture of value functions of past policies. Although these works have a similar looking structure to MODINV, there couple of critical differences. First, the multiple heads in MODINV same object while the above papers use different objects (different value functions). Second, the training of each head is not independent. These two differences dissociate the above works from the IB idea, thus making them complimentary to MODINV.
predictors of the
FUTURE WORK
MODINV shows promising improvements in both the RL and SSL settings, thus hinting at the benefits offered by the IB principle in practice. Nevertheless, MODINV still one instantiation of this idea. There remain quite few directions further diversification in the K predictors can be achieved is a promising idea. For example, we can introduce different optimizers (Adam, SGD) for each predictor, while also varying parameters such as momentum, weight decay etc. Another possibility could be to update a subset of predictors for given sample, instead of only updating one predictor at each training iteration. Finally, since data augmentations change the input data, each predictor could be trained on a different augmentation scheme, thus ensuring further diversification in optimizing different predictors.
explore in future. In particular, analysing how
On a separate axis, analysing how similar MODINV for vision is with other SSL methods such as VICREG is interesting. For the RL case, MODINV can also potentially be applied to the Q functions directly (each predictor is separate Q head over common representation) instead of the reward and transition modules. This could turn out to be hard to optimize for certain environments though, since Q functions can be less developed for sparse reward settings.
CONCLUSION
We introduce a general representation learning method motivated from the ideas of sparsity and minimal-sufficiet representations. Our method, MODINV, uses multiple predictors over common representation, diversifying the training of each predictor such that the common representation acts as an implicit invariance mechanism to spurious correlations. We show instances of this idea in both reinforcement learning and vision settings. Overall, MODINV leads to more robust generalization across different evaluations, leading to believe that with further analysis, in representation generalization. MODINV can be viewed as a specific instantiation of the information bottleneck principle, one that is simple in implementation and in concept.
get further gains

---

Under review as a conference paper at ICLR 2022
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint /rXiv:/907.000 2019.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variane-iivarrancc-ovvvaaaae wtgp://aawwaaaaaaaeeeerrrriiiiia for self-supervised learning- arXiv preprint 105Vv::0000049006, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp.
1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis single-laye networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 215-223. JMLR Workshop and Conference Proceedings, 2011.
Will Dabney, André Barreto, Mark Rowland, Robert Dadashi, John Quan, Mare G Bellemare, and David Silver. The improvemen path: Towards better representations for reinforcement learning. arXiv preprint arXiv:2006. 022400000022,.... 2020.
Carl Doersch and Andrew Zisserman. Multi-task sulf--upe visual learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2051-2060, 2017.
Yann Dubois, Douwe Kiela, David J Schwab, and Ramakrishna Vedantam. Learning optimal representations with the decodable information bottleneck. arXiv preprint MrXii:2009.127800 2020.
Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Marc G. Bellemare. DeepMDP: Learning continuous latent space models for representation learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2170-2179. PMLR, 09-15 Jun 2019. URL https::tt/// / rocced www.nngs................ .mw..mm.p....... www.m...rsss...mmmmmmmmmmmmmme.e /v gel ada htmml.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint 2011:20006007333. 2020.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv: 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor- actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp- 1861-1870. PMLR, 10-15 Jul 2018. URL html.
https: http:://rrceee http://pppceeeens.c./c////
mlr press v80/haarnojaala
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference Learning Representations, 2020.
URL https: /openrevvvvveeeeeeeeeee ew. net forum? i d==1 10TC4tDS.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Donald Olding Hebb. The organization of behavior: neuropsychological theory. Psychology Press. 2005.
Torsten Hoefler. Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. arXiv preprint arXiv:2102. 005544 2021.
10

---

Under review as a conference paper at ICLR 2022
Tianyu Hua, Wenxiao Wang. Zihui Xue, Yue Wang, Sucheng Ren, and Hang Zhao. On feature decorrelation in self-supervised learning. arXiv preprint arXiv:2105.00470. 2021.
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. CoRR, abs/2004. 136449 2020. URL https: aattps://arxiv. brgg/bs//0e a00a//00 2004. 13649.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. Proceedings of the 37th International Conference on Machine Learning. Vienna, Austria, PMLR 119, 2020a. utXiv:2004.04133
Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Re- inforcement learning with augmented data. In H. Larochelle, M. Ranzato, R. Hadsell, M. R. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 19884-19895. Curran Associates, Inc., 2020b. URL https:: /ttp:////////coc/pooeoeoo neurips. cc/paper/2020 http:/eeeeeeeeeeeeeeee
681 ade wew.eeeaeeeeeaaaaeeeeeeeaaaa
Yann LeCun, Fu Jie Huang, and Leon Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 2, pp. II-104. IEEE, 2004.
Yann LeCun, Koray Kavukcuoglu, and Clément Farabet. Convolutional networks and applications in vision. In Proceedings of 2010 IEEE international symposium circuits and systems, pp. 253-256. IEEE, 2010.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European conference computer vision, pp. 69-84. Springer, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 2018.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John
Wiley & Sons, 2014.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International conference machine learning, pp. 1312-1320, PMLR, 2015.
Ohad Shamir, Sivan Sabato, and Naftali Tishby. Learning and generalization with the information bottleneck. Theoretical Computer Science, 41 (292230))266622211 2010.
Yi Sun, Xiaogang Wang, and Xiaoou Tang. Sparsifying neural network connections for face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4856-4864, 2016.
Richard S Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M Pilarski, Adam White, and Doina Precup. Horde: scalable real-time architecture for learning knowledge from unsuper- vised sensorimotor interaction. In The 10th International Conference Autonomous Agents and Multiagent Systems-Volume Volume 2, pp. 761-768, 2011.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint 17Xiv:1801.00690 2018
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. arXiv preprint (0t.::2022.08///// 2021.
Naftali Tishby, Fernando Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/000-057, 2000.
Denis Yarats. Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Im- proving sample efficiency in model-free reinforcement learning from images. arXiv preprint arXiv=1910.011 2019.
11

---

Under review as a conference paper at ICLR 2022
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations. 2021. URL https: http:///////eeeeieee/e// net www.eeeeoidddirreeeee 2 6sTvGaf.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprinl arXiv:2103.0322: 2021.
Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning Invariant Representations for Reinforcement Learning without Reconstruction. arXiv, jun 2020. ISSN 23318422. URL http:
12

---

Under review as a conference paper at ICLR 2022
Appendix
HYPERPARAMETERS
Table 7: Hyperparameters for MODINY for RL.
Hyperparameter
Values
Observation shape
(84, 84, 3)
Latent dimension
50
Replay buffer size
100000
Initial steps
1000
Stacked frames
Action repeat
finger, spin; walker, walk
8 cartpole, swingup
otherwise
SAC: Hidden units (MLP)
1024
Transition Network: Hidden units (MLP)
128
Transition Network: Num Layers (MLP)
Reward Network: Hidden units (MIP)
512
Reward Network: Num Layers (MIP)
Evaluation episodes Optimizer
10
Adam
(81, (b) (Jo u.. Q6)
(.9, .999)
(8, B2) - (a) Learning rate (fo, nu. Q6)
(.5, .999)
2c-4 cheetah, run 1c-3 otherwise
Learning rate (ax
le-4
Batch Size
128
Q function EMA
0.005
Critic target update freq Convolutional layers
Number of filters
32
Non-lincarity
ReLU
Encoder EMA
0.005
Discount
.99
13


Reviews:
[{"review_id": "TZFFT2-bRui", "reviewer": "Reviewer_vnKu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* The idea in itself is interesting and can be applicable on top of most of self supervised algorithms.\n* The authors evaluate their method on two very different sets of tasks (RL and Vision).\n* The paper is well-written easy to read.\n\nMy main concern with the paper is its experimental soundness. I would strongly encourage the authors to resubmit a new version with more thorough experiments in order to back the (potentially interesting) claims that they make.\n\nWeaknesses:\n* From the elements currently provided in the paper, I don't see why the authors' algorithm would lead to minimal representation, at least in the information-theoretical sense as defined in the paper they cite: Shamir et al., 2010. I would expect either a citation or some experiments to support this claim.\n  * For instance: \"The common representation acts as a implicit invariance loss which ensures that only the optimal representation remains at convergence. \" -> I need either a citation or evidence for that.\n  * Similarly: \"since there is only one common representation to support all the predictor heads, this representation must be robust to any spurious correlations as training progresses.\". \n\nIn particular I don't see why each predictor would not learn to filter out unnecessary information in the representation. Using the paper's example of the pen: knowing the position of a distractor object is not necessary to solve the task, yet all the predictors could learn to filter out this useless information. In this case I don't see why the proposed algorithm would incentivize to remove this information from the representation.\n\n* The authors claim \"Our experiments show that [...] the mean correlation decreases much more steadily when we use MODINV.\" \n\nAre these results in the paper? If not, adding them would benefit the paper a lot. The current Table 4. only shows that encouraging decorrelation of features enables better RL. It does not show that using ModInv provides better decorrelation of representation features.\n\n* The author choose to evaluate in the specific case of a linear predictor. They claim :\n  * \"2-layer predictor has been speculated to be using a lot of 'lucky' initialization\".\n  * \"testing in this setting offers a much more robust evaluation option\".\n\nThese claims lack either citations or experiments.\n\nEvaluating with a linear predictor is problematic for me as this case is not the standard regime. SimSiam is tuned for the 2-layer predictor regime and does not work well in the linear regime.I would expect the authors to either provide improvement in the 2-layer regime or convincing experiments or citations to support this choice.\n\n* Can you clarify : \"Unlike the reinforcement learning case, where any of the modules after the representation (state and reward prediction heads) are not used actively in the algorithm, in the vision case the projector and predictor both are used to align the backbone representation (Tian et al., 2021). \"\nFor me it seems that in both cases the predictor and projector are used as tools to learn a good backbone representation. Even in the vision case the predictor and projector are thrown away when the network is actually used (for classification for instance).\n\nTypo: \"the phenomenon that that neurons that have\"", "summary_of_the_paper": "The authors propose MODINV, a new algorithm to learn minimal and compressed representations.\nMODINV uses several predictors with different initializations and learning but all built on top of the same learnt representation.\n\nAfter explaining the algorithm and its relation to minimal representation the authors evaluate it on vision and RL tasks.\n\nOn vision tasks the authors show improvement on CIFAR-10 by adapting their method on top of SimSiam in the specific case of a linear predictor. On RL they evaluate on control tasks from pixel inputs.\n\nThe authors also provide an experiment to show that adding a loss to discourage correlation within features helps to improve RL performance.", "summary_of_the_review": "I think a lot of claims are not justified either by experiments or citations. Especially the main claim that the authors' algorithm learns a minimal representation.\n\nI also found the vision experiments not convincing: on CIFAR-10, they used SimSiam in the linear prediction case instead of the usual 2-layer predictor in which SimSiam performs best.\n\nFrom the elements provided in the paper, I could not tell whether the proposed algorithm was indeed helping to learn minimal representations, or that such a representation would generalize better compared than one learnt without the proposed method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QGKvTvgR_Yy", "reviewer": "Reviewer_LtXH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n- Simple method -- just train 3 heads with different learning rates.\n- No increase in wall-clock time\n- Clear writing\n\n**Weaknesses:**\n- **No evidence for the main claims**\n    - *”Common representation acts as an implicit invariance objective to avoid the different spurious correlations captured by individual predictors”:* There is no evidence of this, i.e. that the proposed method better deals with (or avoids) certain spurious correlations.\n    - *Each head “captures different spurious correlations”:* Again, there is no evidence for this claim.\n    - *”[...] diversify the learning of heads so they capture different spurious correlations”* Again, no evidence, e.g. an analysis of what correlations are captured by each head.\n    - *”[...] This in turn leads to better generalization performance”:* There is no evidence that the proposed method achieves better generalization performance by avoiding spurious correlations. If in-distribution test-set performance is what is meant by generalization performance, then I think this needs to be made clear. Nonetheless, to justify the original statement, it would still need to be shown that this improvement is due to the avoidance of spurious correlations.\n- **Questionable claims**:\n    - *Training independently*: the predictors are given the same samples in the same order, and even share the same (backbone) representation. “Trained independently” is a bit of a stretch…\n    - *”Strong performance boosts in both [RL and vision settings]”:* It seems that the proposed method ModInv only improves performance when additional training stability is needed (perhaps unsurprising since it's an ensemble method).\n      - *RL:* With data augmentation to already stabilize training, ModInv only improves performance on 1 of 6 tasks over RAD, decreasing performance on 4 of 6 tasks (500k steps).\n      - *Vision*: ModInv only helps when using a linear predictor on CIFAR-10. When using a nonlinear predictor on CIFAR-10 (which is the standard and improves performance with or without ModInv) or any predictor on STL-10, ModInv actually hurts performance.\n\n**Other comments:**\n- **Missing results**: Section 5.2 -- where are the results mentioned that show the mean correlation decreasing more steadily with ModInv?", "summary_of_the_paper": "This paper proposes using an ensemble of predictors or heads on top of a shared representation to improve performance. The ensemble of predictors/heads are made \"diverse\" by using different learning rates. Some performance gains are shown in RL settings, while some stability gains are shown in a particular vision setting.", "summary_of_the_review": "While the method is admirably simple and the writing clear, the main claims of the paper are either questionable or without any supporting evidence. Thus, in my opinion, this paper falls well short of acceptance for a top-tier conference like ICLR. If evidence can be provided in support of the main claims, I would be happy to revise my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "LYQvQilyp6h", "reviewer": "Reviewer_Q19W", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n- The paper is well-written and easy to understand. The intuitions and connections to relevant related works have been established meaningfully.\n- The ablations are very well done, and reasonably support many of the claims.\n\n## Weaknesses\n\nI have two main concerns to be fully confident of the gains claimed.\n\n- The proposed scheme lies very close, in principle, to a family of approaches called the [Hyperparameter Ensembles](https://arxiv.org/abs/2006.13570) (HE). HE induce diversity by training using different hyperparameters. The obvious difference is the fact that the authors here only train the predictor heads in an HE manner instead of the full network. Nevertheless, approach is very closely related.\n\n  Here, the authors claim the benefits from an information bottleneck perspective, but the intuitions only remain justified empirically. In that sense, I would then believe that the gains are due to the same reason deep ensembles work, than for an information-theoretic reason of learning minimal representations.\n\n  The fact that adding more than $K=3$ heads leads to diminishing returns may also hint to qualitative similarity to deep ensembles (where an ensemble beyond $\\sim 5$ leads to diminishing returns).\n\n- Another intuition that the authors highlight is that forcing the same representation induces an information bottleneck. This is turn should nudge the network away from learning spurious correlations. This claim is hand-wavy and is not supported either empirically or theoretically (perhaps even a toy example would suffice to qualitatively guarantee that it is doing what the authors claim it to be doing). My impression here is again that it is not that the minimal representations are learned that help avoid spurious correlations and hence better generalizations. It is instead the same reason that deep ensembles work, i.e. finding a set of good minima which induce a reasonable functional diversity. Deep ensembles are not very robust to spurious correlations either (though better than a single network), and it is unlikely that a very similar training routine proposed in this paper does something very different.\n\nNevertheless, I do want to acknowledge that the setting of using RL with pixels and the achieved gains is still a potentially novel intersection. But the authors approach the problem from an information bottleneck perspective, the robustness claims from which are not entirely supported.\n\n## Minor\n\n- Highlighting best results in Table 1 would make it easier for the reader.\n- Hyperlinks do not work in the main submission.\n\n", "summary_of_the_paper": "Inspired by the information bottleneck principle, the authors propose to learn minimal representations by independently training multiple predictive heads on top of a shared representation, such that the resultant predictor is more robust to spurious correlations. To test this proposal, the authors provide extensive set of experiments in RL from pixels and vision benchmark tasks. The ablation studies help make the case too.", "summary_of_the_review": "The main theme of my concerns is that it is not that the training has learned minimal representations but just that it is the diversity of predictor heads that provides the gains. Without sufficient empirical or theoretical evidence, I think the main claim of the paper of being robust to spurious correlations due to the implicit information bottleneck is weak. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4hPKuhxJVK_", "reviewer": "Reviewer_8QAk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I will now highlight the strengths and weaknesses of the paper. The strengths and weaknesses will highlight the quality, clarity, and significance of the proposed approach.\n\n**Strengths**\n\n1. The paper is generally well-written and easy to follow. With the exception of a few points covered under weaknesses, the authors do a good of introducing the proposed approach and walking the reader through different experimental settings.\n\n2. On the DMC suite, ModInv seems to offer improvements over Dreamers, CURL, DBC in most cases (excluding RAD settings, please refer to the weaknesses). In vision settings, when using a linear predictor, ModInv seems to offer significant improvements on CIFAR-10 when compared to a recent self-supervised approach (SimSiam). \n\n**Weaknesses**\n\n1. I’m not sure I completely follow the thread of motivation introduced in the paper. ModInv is right now motivated from a sparsity/minimality point of view (in the context of how they’ve been described in the submission) and the reasoning for the particular instantiation relies on introducing invariance across multiple predictors which subsequently reduces redundancy in the learned representations. I’m not sure if this hop of reasoning works. It’s not entirely clear to me that being resilient to changes in downstream predictors will necessarily reduce redundancy in the learned representations. Can the authors comment on this? Additionally, I would suggest the authors also discuss [A], where the information bottleneck interpretation of self-supervised learning w.r.t. a downstream task has been studied, in related work and situate their proposed approach accordingly\n\n2. The paper suffers from a few clarity issues. Highlighting them here. For instance, the ModInv structure implies that each predictor head is trained independently (a given sample only trains one head), implying that there is some underlying routing of the sample to a predictor head is going on. It’s unclear from the draft right now how this routing is performed. As in, given a training sample, how do the authors decide which head to feed the corresponding representation? If this routing is random, can the authors comment if they investigated other routing strategies? In Section 5.2, the authors mention that mean correlation across feature dimensions decreases more steadily when using ModInv. However, the draft doesn’t provide any supporting evidence (a table/plot) for this. More specifically, evidence demonstrating the evolution of mean correlation when using vanilla, ModInv, and an agent trained with mean correlation loss would help solidify this point even more. This is particularly important since the bulk of the motivation for ModInv relies on this being a desirable and observed behavior.\n\n3. I think the experimental evidence in support of the proposed approach is somewhat weak. Firstly, ModInv + Aug outperforms RAD in only one setting (Reacher, Easy at 500k; Table 1). Did the authors use the same set of augmentations being used in RAD? If yes, then the utility of ModInv is heavily diminished. Particularly since ModInv involves using extra parameters. Results on computer vision benchmarks seem strong only for one setting — CIFAR-10 with a linear predictor. ModInv is competitive or worse on STL-10 and when a non-linear predictor is used, ModInv seems to consistently hurt performance in most settings. Furthermore, the experiments in Table 4 seem to suggest that ModInv outperforms Mean Correlation Loss in only one (out of two) settings. Given these observations, I am not entirely confident of the utility of the proposed approach. Additionally, I would suggest the authors appropriately adjust the claims regarding performance improvements. An ablation mentioned in Section 4.2.2 states that using individual SimSiam models with different learning rates is worse than using different predictor heads. The paper would benefit if this ablation was done in a more comprehensive manner, i.e., across both RL and vision settings at different stages. Since ModInv relies on using multiple predictor heads and using different models is a natural alternative, including such results seem necessary and would definitely improve the paper.\n\n4. [Minor Points] Can the authors comment on how the learning rates and initializations for different predictor heads in all settings were chosen? Given that diversifying predictor heads is a big part of ModInv (as indicated from the ablations in Table 5), specifying these details seem important. In Section 6.1, the authors suggest that ModInv is similar to DIB in terms of instantiation. It’s unclear to me how that’s true. Can the authors comment on this?\n\n[A] - Self-supervised Learning from a Multi-view Perspective", "summary_of_the_paper": "Motivated from the idea that minimal representations — that encode information relevant to a task and nothing more — are likely to generalize well, the paper proposes an approach, titled Model Invariance (ModInv) to learn representations. The underlying idea is to use multiple diversified predictors solving auxiliary objectives on top of a shared representation/encoder. Different predictors are diversified in terms of their training dynamics — different initializations and learning rates. The approach relies on the hope that a shared representation that can lead to optimal performance for a diverse set of predictors is likely encoding some notion of implicit invariance to changes in predictors, thereby, introducing some notion of minimality. The authors conduct experiments on reinforcement learning and self-supervised (in computer vision) settings and claim that ModInv provides strong performance improvements in both. Based on further ablations,  the authors demonstrate how sensitive ModInv is to the number of predictors and the diversity in terms of training dynamics (in terms of learning rate), and whether ModInv is complementary to recent augmentation strategies in reinforcement learning.", "summary_of_the_review": "The points highlighted under strengths and weaknesses form the basis of my rating. In particular, the weaknesses influence my current rating of the paper the most. While the paper is generally well-written, I think the issues highlighted under weaknesses in terms of motivation, clarity issues and lack of strong experimental evidence makes me less convinced of the utility of ModInv. I would encourage the authors to address the first three points under weaknesses. Addressing those will greatly help in reconsidering my rating of the paper. The minor points highlighted under 4 are addressable, I think.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Although all reviewers had many positive comments on the paper, and the authors engaged nicely in the discussion period, at the moment there is a consensus among the reviewers that the central claims of the paper (related to minimal representations / information bottleneck) are not adequately supported by the current experiments. In particular, there were concerns that performance gains could be due to diversity of predictors, rather than minimal representations, which would need to be addressed. It's suggested that the reviewers take all of these comments and discussion into account when preparing a revised version of the paper.
Decision: Reject