Question: In Section 6.1, the authors suggest that ModInv is similar to DIB in terms of instantiation. It’s unclear to me how that’s true. Can the authors comment on this?
Evidence: Learning efficient representations that generalize well is a long standing problem of machine learning, and particularly of deep learning. Algorithms that exploit structure in the real world through effective inductive biases are key to solving this problem. Several inductive biases have been successfully used in the past, from early work using translational invariance for developing, to recent data augmentations for developing multiple self-supervised algorithms. A key inductive bias in multiple works is sparsity of concepts. Essentially, it refers to how objects in the real world tend to interact and affect the dynamics of other objects only in a small neighbourhood. Consider for example picking up a pen. This simple task only changes the dynamics of a very small part of the world, namely the pen and the hand of the person grasping it. Exploiting such structure allows learning representations that can generalize better in the real world.
In this paper, we develop a method that exploits sparsity by learning minimal representations. The well known saying 'neurons that fire together wire together' points to the phenomenon that that neurons that have similar output for a given input also have strong weights between them. Ideally, given the same set of neurons to model certain concepts, we would want neurons that relate to a particular concept to have strengthened weights while those relating to another concept to have weakened weights. Therefore, low correlation in the outputs of the representation would lead to less redundancy in modelling concepts and thus better generalization. This is precisely the motivation behind minimality leading to sparse representations and improved generalization. Minimal representations refer to ones that have sufficient but minimal information w.r.t. the task at hand. Therefore, it must be only possible to solve the task in hand and no other task, thus leading to better generalization for the given task. Another viewpoint is that minimal representations also lead to a reduction in spurious correlations. Since there is minimal noise present in the representation to cause spurious correlations, this again results in better generalization.
Faster LR Standard LR Slower LR Figure: Model Invariance (MODINV) A common representation f θ supports multiple predictor heads, each with a different learning rate. Each head is optimized based on the task objective (e.g., SSL loss) and captures different spurious correlations. As diversification in learning of predictor heads increases, the common representation must be optimal to reduce the loss for all predictors. 'Red' denotes the head being trained for the current sample.
Our approach, which we call model invariance (MODINV), learns minimal representations by incorporating multiple predictors over a single, common representation for a given task (see Figure). Each predictor head is then trained independently (a given sample only trains one head) and with a learning rate different to other heads. Theoretically, this can be looked at as a manifestation of the information bottleneck principle, which balances between sufficiency and minimality. Intuitively, this can be looked at as providing robustness to spurious correlations. Since each head learns a different prediction (based on its own optimization and learning on different parts of data set), each head is susceptible to different spurious correlations. However, since there is only one common representation to support all the predictor heads, this representation must be robust to any spurious correlations as training progresses.
We evaluate our method on two separate settings; 1) Reinforcement Learning: The DeepMind Control Suite(DMC) benchmark with distractors, which involves natural video playing in the background while the agent is controlled in the foreground and 2) Vision: The CIFAR-10 ( datasets, where no labels are used for learning the representations, followed by the standard linear probing evaluation protocol to test generalization.
We introduce MODINV, a simple method motivated by incorporating sparsity in representations, while showing connections to the information bottleneck principle. MODINV is conceptually simple, easy to implement, and improves representation learning performance in both reinforcement learning (from pixels) and vision settings. In reinforcement learning, MODINV achieves stronger than state-of-the-art performance on the DMC Suite with distractors benchmark, both with augmentations and without. In vision, MODINV leads to better performance than the baseline method of SIMSIAM when using a linear predictor on both CIFAR-10 and STL-10 datasets.
Our approach, which we call model invariance (MODINV), learns minimal representations by incorporating multiple predictors over a single, common representation for a given task (see Figure 1). Each predictor head is then trained independently (a given sample only trains one head) and with a learning rate different to other heads. Theoretically, this can be looked at as a manifestation of the information bottleneck principle, which balances between sufficiency and minimality
Answer: **DIB** uses the information bottleneck principle but for a particular family of predictors only. Here, we use the same idea (same architecture for all predictors) and thus it is closely related to DIB. Note that DIB was using fake labels and reversing gradients for the K predictors so as to enforce minimality. However, that creates a min-max optimization problem which could be hard to solve in practice. We don’t face that issue but of course need to pay in terms of more effort to make the predictors as diverse as possible.