Question: Is there any clear trait of the trade-off in Fig (2)?How about the IRM Results in Fig 2?
Evidence: Our experiment in Section 4.1 showed that linear models exhibit a very clean trade-off between complexity and generalization. However there are many subtle factors that influence effective model complexity in neural networks besides architecture: learning rate, learning rate schedule, weight decay, early stopping, etc; and the repeatably of learning is hampered by non-convexity, stochastic initialization, and stochastic mini-batches, etc. The previous evaluation study of attempted some degree of control over these factors for state of the art neural DG methods by applying a common hyperparameter search strategy that aimed to find the best tuning (cf: Fig) for each competitor. They nevertheless found existing methods to perform unreliably. Our conjecture is that the number of seed samples (3) and hyper-parameter search trials (20) used was far too few to reliably tune the deep networks evaluated, and therefore the tuning of each model was inaccurate and performance estimates unreliable. Since dense and accurate hyper-parameter tuning for neural networks is more challenging than our linear models in Section 4.1, we take a different approach. Rather than trying to control complexity directly, we take trained DomainBed methods and measure their complexity retrospectively.
Setup To ensure that complexity can be measured accurately, we work with 2-layer MLPs. Specifically, we take fixed ImageNet pre-trained ResNet-18 features, and feed them to MLPs, which are then trained using the DomainBed framework. 2-layer MLPs are sufficient to instantiate many of the state of the art DG algorithms in DomainBed. We train and compare ERM, CORAL, Mixup , MMD, RSC, SD, VRex, and IRM with two hyperparameter choices. We also report the results of an ERM model, checkpointed at a range of training iterations.
Measuring Model Complexity Because we have limited our attention to 2-layer MLPs, we can take advantage of a model capacity metric that is specialised for this class of models. We retrospectively determine the complexity of a trained network using the measure proposed by Neyshabur et al. (), who also use this measure to bound the Rademacher complexity of a 2-layer MLP hypothesis classes. More concretely, the expression used for computing complexity is
where U is the weight matrix of the first layer, U 0 is its random initialisation, and V is the weight matrix of the second layer. We use • F to denote Frobenius norm, and • 2 to indicate the spectral norm. Note that for simplicity we have omitted constant factors that depend only on the architecture and problem setting, and not the learned weights, as we use the same architecture for all methods we investigate.
The results in Figure summarise the trade-off between measured model complexity (xaxis), and held-out domain test accuracy (y-axis), averaged over all choices of held out domain for each dataset (See Supplementary for full breakdown across all held-out domains). The top plot compares several of the published neural DG methods implemented in DomainBed. The main message of the top plot is that results are consistent with the hypothesis that the resulting complexity of the models trained by various DG methods is a key factor in determining domain generalisation accuracy (compare Fig. with the clean result for linear where models we are able to intervene and control complexity directly in Fig.)-thus explaining the previously erratic behaviour observed in domain-bed.
To provide a different view of the same issue, the bottom plot reports a vanilla untuned ERM model check-pointed every 300 training iterations between (up to a total of 15,000) but expressed in the same complexity units as above. Because neural models can gain complexity with iterations (see, e.g.,;) we also see typical overfitting-underfitting trade-off curves. This shows that, as expected, proper choice of early stopping criterion is important. It also shows that over a similar dynamic range of complexity (0.5-3), ERM (below) and alternative models (above) span a similar dynamic range of accuracy (e.g., 6% for PACS). This suggests that complexity, as controlled by whatever mechanism, is a key determinant of DG performance.
How Do DG Models Control Complexity? By introducing different modifications to standard ERM, DG models explicitly or implicitly modify the bias and variance of the function class to be learned. highlight neural models as being dependent on learning rate, batch size, dropout rate, and weight decay; with other factors being choice of optimiser and weight initialiser, etc. For example, higher dropout rate and weight decay tend to reduce complexity, while some other factors influence complexity in a less transparent way. RSC introduce more sophisticated dropout-like mechanisms, which would be expected to reduce complexity. Meanwhile alignment-based methods like CORAL and MMD effectively add auxiliary losses, which will implicitly affect complexity, but are hard to explicitly link to it. Consistency based methods like IRM and VRex penalise loss variance, which also tends to reduce the generalisation gap in the single task case. The specific setting of all the corresponding hyper-parameters (e.g., regulariser strength, dropout rates) influence final model complexity, which we argue is the key determinant of performance.
Figure 2: Performance of neural networks as a function of their measured model complexity after training using DomainBed (DB). The overall results are consistent with a standard bias-variance trade-off (cf. Fig 1 for linear models): performance depends on how well each model was tuned by DB’s hyper-parameter search procedure. Top: Leave-one-domain-out cross-validation performance of various neural DG algorithms evaluated by DomainBed. Horizontal error bars correspond to the standard deviation of the model complexity measured for each cross-validation iteration. Bottom: Performance of ERM model checkpointed at different training iterations. The central tendency is obtained via fitting a support vector regression model (Shevade et al., 2000) with a 6th order polynomial kernel.
Answer: We now illustrate the tradeoff between overfitting and underfitting regimes using polynomial regression curves. In the underfitting regime (left) performance is sharply lower, in the overfitting regime (right) performance decays slowly, with best performance somewhere in the middle. With regards to IRM: Short answer: The apparent discrepancy is not due to MLP vs ResNet and we do not contradict the prior work. We now include a version of IRM that is tuned to perform closer to ERM by reducing its regularisation strength. Long answer:It’s important to note that these models are run without careful hyperparameter tuning in Fig 2. Depending on hyperparameters, models will achieve different complexities (e.g., as we showed in Fig 1). So the specific placement of any given model (such as IRM, ERM, etc) on the graph will depend on hyperparameter choice. Our point is not about the absolute position of any given model, but the fact that when complexity is allowed to vary by any means (due to regulariser strength, training iterations, choice of ERM vs IRM vs REX etc), the resulting complexity (x-axis) determines the performance (y-axis).