Question: The paper suffers from a few clarity issues. Highlighting them here. For instance, the ModInv structure implies that each predictor head is trained independently (a given sample only trains one head), implying that there is some underlying routing of the sample to a predictor head is going on. It’s unclear from the draft right now how this routing is performed. As in, given a training sample, how do the authors decide which head to feed the corresponding representation? If this routing is random, can the authors comment if they investigated other routing strategies?
Evidence: Consider a task Y , which can correspond to a variety of objectives. For instance, it could be an auxiliary task of predicting the next state when interacting with an RL environment, or simply a classification task on a vision benchmark. We are interested in training a model that can do well on task Y , where the model consists of two components, a backbone representation f θ and a predictor/classifier h φ which is attached over the representation. In a lot of tasks that involve pixel input, spurious correlations can arise due to various reasons such as lack of non-iid data, irrelevant information, confounders etc. that can result in poor generalization at test time. An attractive property for better generalization has been the idea of sparse representations, in that the mutual information of any two dimensions of the representation must be low. However, in order to avoid learning trivial solutions, the representation as a whole should encode enough information about the downstream task as well. A combination of both of these objectives then leads to a model with better generalization capability. In this paper, we balance the above two objectives using the idea of model invariance (MODINV). MODINV deploys multiple predictors over a single representation, while training each predictor independently, each capturing different spurious correlations. The common representation acts as a implicit invariance loss which ensures that only the optimal representation remains at convergence. Intuitively, each predictor can be looked at as an augmented version of the optimal model, where the augmentation is over the model space and refers to particular spurious correlations arising in the model that differ it from the optimal model. Note that each predictor head is the same in architecture, and we only diversify the learning procedure of each head (through different initialization, independent training, different learning rates). This is so that we eventually converge to an optimal representation for a particular predictor/classifier and not for all predictor/classifier families, which is a much more hard to optimize in practise. For all experiments, we use a random routing to decide which data sample is used to train which predictor head. Furthermore, the learning rates of the different predictor heads are chosen intuitively, i.e. if the base rate is 3e-3 then we choose one slightly higher rate (5e-3) and one slightly smaller rate (1e-3) for K=3 predictors.
For all experiments, we use a random routing to decide which data sample is used to train which predictor head.
Answer: Yes, this is done by randomly choosing a predictor head all throughout the paper. We did not experiment with more sophisticated schemes but do think it is an interesting direction to improve along.