Question: What is meant by domain shift?
Evidence: Machine learning systems have shown exceptional performance on numerous tasks in computer vision and beyond. However performance drops rapidly when the standard assumption of i.i.d. training and testing data is violated. This domain-shift phenomenon occurs widely in many applications of machine learning, and often leads to disappointing results in practical machine learning deployments, since data 'in the wild' is almost inevitably different from reference training sets.
Given the practical significance of this issue, a large number of methods have been proposed that aim to improve models' robustness to deployment under a different distribution than used for training, a problem setting known as domain generalisation (DG). These methods span diverse approaches such as specialised neural architectures, data augmentation strategies, and regularisers. Nevertheless, the DG problem setting is difficult to model formally for principled derivation and theoretical analysis of algorithms, the target domain of interest is unobservable during training, and cannot be directly approximated by the training domains due to unknown distribution shift. Therefore the majority of these existing approaches are based on poorly understood empirical heuristics.
To make matters worse, a recent study by assessed the state of DG research with a carefully conducted comparative evaluation of algorithms on a large benchmark suite under a common platform. They found that published methods were not as effective as claimed, and in particular reported that 'no existing method reliably beats a well tuned empirical risk minimization (ERM) baseline'. We argue that this negative result highlights the need for better theory in this area in order to understand why existing algorithms have such erratic performance, and to guide the development of principled algorithms that are more effective and reliable.
To this end, our first contribution is to present an intuitive learning-theoretic bound for DG performance. Intuitively, while the held-out domain of interest is indeed unobservable during training, we can bound its performance using learning theoretic tools similar to the standard ones used to bound the performance on (unobserved) testing data given (observed) training data. In particular we show that the performance on a held out target domain is bounded by the performance on known source domains, plus two additional model complexity terms, that describe how much a model can possibly have overfitted to the training domains. This theoretical contribution leads to several insights. Firstly, our theory suggests that DG performance is governed by a trade-off between empirical risk and model complexity that is analogous to the corresponding and widely understood trade-off that explains generalisation in standard i.i.d. learning as an overfitting-underfitting trade-off. Based on this, we hypothesise that performance variability is determined by implicit or explicit regularisation. That is, the plethora of different strategies available from data-augmentation to specialised optimisers -actually affect DG performance by explicitly or implicitly choosing different fit-complexity trade-offs. We corroborate this hypothesis by evaluating a number of models in the DomainBed suite in terms of complexity, and showing that their apparently erratic performance in is actually consistent with an explanation in terms of implied complexity.
Practically, our analysis suggests that the model selection strategy) is a factor in DG performance that is at least as important as the actual mechanism of model complexity control (i.e., Tuning of regularisation strength vs specific parametric design of regulariser). In particular, regularisation should be stronger when optimizing for future DG performance than when optimizing for performance on seen domains. Unfortunately, model complexity is hard to carefully control in deep learning due to the large number of relevant factors (architecture, regularisers, implicit regularisation from optimiser, etc). attempted to address this by hyper-parameter search in the DomainBed benchmark, but are hampered by the computational infeasibility of accurate hyper-parameter search. In this paper, we use linear models and off-the-shelf self-supervised features to demonstrate much more clearly how cross-domain performance depends on complexity. Specifically, our theoretical and empirical results show that, contrary to the conclusion of, simple domain-wise cross-validation is a better objective to drive DG model selection.
In summary, based on our new generalisation bound, and associated empirical analysis, our takehome messages are: (i) Model fit vs complexity trade-off is a key determinant of DG performance, that explains existing DG algorithm performance variability. (ii) The complexity control strategy used to determine bias-variance trade-off is crucial in practice, with peak DG performance achieved when optimizing model complexity based on domain-wise validation. (iii) Regularisation required for optimal DG is greater than for conventional optimization for within-domain performance.
However performance drops rapidly when the standard assumption of i.i.d.training and testing data is violated. This domain-shift phenomenon occurs widely in many applications of machine learning (Csurka, 2017; Zhou et al., 2021; Koh et al., 2021), and often leads to disappointing results in practical machine learning deployments, since data ‘in the wild’ is almost inevitably different from reference training sets.
Answer: Most DG methods assume p(y|x) remains constant and only p(x) changes, as this is the type of distribution shift encountered in popular DG benchmarks. However, it’s worth noting that our theoretical analysis applies to both situations: change in p(x) and change in p(y|x). We have added some discussion about this in the paper.