Question: How can there be exactly k labeled samples per class, isn't this too strict?
Evidence: While deep neural networks have demonstrated remarkable ability in representation learning, their performance relies heavily on the assumption that training (source) and test (target) data distributions are the same. Real-world data collection is often resource-constrained, such that test samples may be subject to domain shift, also known as covariate shift, due to factors such as illumination, pose, style and data collection procedures. To prevent severe performance degradation when models are deployed, adaptation to the target distribution is needed.
A range of methods to address domain shift have been developed, with varying requirements on source and target domain data. Domain generalization considers generalization without seeing target data, but existing methods still have a considerable generalization gap. On the other hand, domain adaptation (DA) assumes the availability of target data during training. Most DA methods work under the vanilla unsupervised DA (UDA) setting, taking that labelled source and unlabelled target data are fully accessible for joint training. Some methods have been proposed under the few-shot DA setting, which assume that labelled source data and a few labelled target samples are available. Recent work is shifting towards a more challenging setting of source-free UDA, where a pre-trained source model is adapted with only unlabelled target data. However, these methods require access to the entire target dataset during adaptation.
Very recently, test-time adaptation methods have been proposed to continuously adapt during test time with streaming unlabelled target data, by updating batch normalization (BN) layers batch-by-batch. However, these methods face 3 main challenges: 1) estimation performance is dependent on large batch size to estimate BN parameters and statistics, 2) test samples need to be class-balanced which may not be practical in real-world deployment, 3) there is no guarantee self-supervision objectives can correct domain shift without using any target domain label information. For instance, entropy minimization can lead to undesired trivial solutions where all outputs "collapse" to one or a few classes.
Motivated by the problems of test-time adaptation methods, we propose a challenging but practical source-free DA setting by adapting a pre-trained source model using k-shot support from target do- shows existing settings in the DA literature and the proposed setting. Considerations for real-world usage motivates our proposed source-free k-shot setting to address domain shift:
• Data availability: Our setting helps to protect privacy of the source domains, and has low requirements for target data availability of only k labelled samples per class during adaptation. During testing, test batch can be of any size with no restrictions. • Inference efficiency: Model parameters are not updated at test-time.
• Accuracy: Our setting is not dependent on test-time data streaming conditions and selfsupervised objectives, and hence enables more reliable and accurate adaptation.
We propose a k-shot method to adapt batch normalization (BN) layers of deep source models to address domain shift. As far as we know, our work is the first source-free domain adaptation method with a few-shot setting. Although BN layer modulation has been explored in existing literature, reliably optimizing BN layers with extremely few support samples is a new and challenging problem. Naively optimizing high-dimensional parameters in BN layers risks a severely ill-posed problem caused by data scarcity, and can result in unreliable estimates that easily over-fit to the small support set.
In this work, we introduce a new parameterization of BN layers, and approximate the optimal highdimensional target domain BN statistics by a linear combination of spanning vectors representing both source and target domains. Specifically, we linearly combine a small set of spanning vectors obtained from source domain BN statistics and support set, and optimize the combination coefficients by supervised loss on the support set. This significantly reduces the number of parameters to adapt on BN layers. Our proposed method is inspired by the success of controlling sample stylization through BN layers, and we aim to approximate the optimal style to stylize the target domain samples to best address domain shift. We evaluate the proposed method on different image classification benchmark datasets. We provide experimental validations and comparisons with state-of-the-art methods to demonstrate that our approach compares favorably in adaptation accuracy.
Table 1 shows existing settings in the DA literature and the proposed setting. Considerations for real-world usage motivates our proposed source-free k-shot setting to address domain shift: • Data availability: Our setting helps to protect privacy of the source domains, and has low requirements for target data availability of only k labelled samples per class during adaptation. During testing, test batch can be of any size with no restrictions.
Answer: In our proposed setting, we require k labelled samples to ensure equal representation amongst all classes, a conventional requirement in few-shot learning literature. Our experiment results in Section 4 show that even k=1 can improve the source model. The framework of our proposed method also applies when there are more than k samples per class.