Question: Whether you miss any baselines?
Evidence: As far as we know, there are no existing methods designed and evaluated specifically for our sourcefree few-shot DA setting. We apply existing methods in DA and few-shot transfer learning to provide benchmark performance. We compare with AdaBN by replacing source BN statistics with those calculated on target support set, finetuning the source model on BN layers or classifier or feature extractor, finetuning entire model with L 2 or L 2 -SP or DELTA regularization, Late Fusion which averages scores from source and target classifier, and FLUTE which optimizes BN parameters with nearest-centroid classifier. FLUTE assumes the availability of multiple source datasets to train multiple sets of BN parameters for further blending to initialize the finetuning process. Since we only have access to the pre-trained source model in our setting, we reduce FLUTE to the single source dataset case and initialize FLUTE with single source BN parameters. It is conventional to update the classifier when sufficient target samples are available, hence besides implementing with source classifier (SC), we also implement our proposed method with finetuned classifier (FC) and nearestcentroid classifier (NCC). We follow learning settings in for regularized finetuning and use SGD optimizer with momentum and weight decay 0.0004, and L 2 regularization is also added to finetuning on classifier or feature extractor to prevent overfitting. For all other methods that finetune on BN layers, we use Adam optimizer following. We set learning rate 0.001, mini-batch size 32 and epochs 10 for all methods and datasets evaluated.
From Table, we observe that regularized finetuning tends to adapt well at k = 1, but performance can lag behind with larger support sets. AdaBN does not consistently improve adaptation; we show in Appendix Figure that completely replacing source BN statistics degrades performance for VisDA. Overall, finetuning LCCS with source classifier has at least comparable performance with regularized finetuning for small support sets. For larger support sets, finetuning LCCS with NCC has the best performance, with classification accuracy consistently higher than FLUTE which also uses NCC. We also compare our method to L 2 and FLUTE on Office and OfficeHome in Table 16. Though the transfer learning methods produce comparable performance on the easy dataset of Camelyon17 (two classes, two domains), LCCS outperforms on more difficult datasets, which demonstrates the effectiveness of the proposed low-dimensional finetuning strategy.
LCCS Finetuning vs BN Finetuning: We further study our proposed constrained optimization of BN layer versus the unconstrained alternative of optimizing directly on the high-dimensional BN parameters γ and β in Equation 1, with fixed source classifier. We also evaluate a combination of the two strategies by first finetuning LCCS followed by finetuning BN parameters. From Figure, we see that when the support set is small at k ≤ 3, finetuning LCCS is always better than finetuning BN parameters. The results are mixed at larger k, which is expected since the unconstrained optimization is also less prone to overfit with more samples. In addition, we observe that finetuning LCCS first provides good initialization for further finetuning of BN parameters, and consistently achieves better adaptation performance than finetuning BN parameters alone. We make similar observations when using NCC as classifier in Appendix Section C.5.  Figure: Finetuning LCCS has best performance when support set is extremely small (k ≤ 3).
With larger support set (k ≥ 5), finetuning LCCS+BN parameters attains better performance than finetuning BN parameters alone.
We additionally compare our proposed method with source-free UDA methods which adapts with entire unlabelled target dataset, including AdaBN, SHOT, SFDA and SDDA, as well as the few-shot method L 2 and FLUTE, in Table on Office and Office-Home. We observe that self-supervision over the entire unlabelled target can produce good adaptation performance. Despite using only 5 samples per class, finetuning LCCS has equal and better adaptation performance than source-free UDA methods in 5 out of 6 domain pairs on Office. SHOT outperforms in the more challenging OfficeHome dataset, which reflects the difficulty of source-free few-shot setting. However, our proposed method performs the best out of the few-shot methods evaluated, which demonstrates its effectiveness in the few-shot setting. We refer readers to Appendix Section C.4 for detailed results and comparisons with nonsource-free few-shot DA methods.
We compare the resources of the three source-free methods from different settings: SHOT, Tent, and our fewshot LCCS finetuning. From Table, the number and storage size of samples for our proposed method is much smaller than those used for SHOT. At test time, our method requires no computation for updating the model, which is at least on par with test-time adaptation methods since they can need additional backward passes for online adaptation. For instance, Tent needs at least an additional backward pass. More importantly, our setting is one way to address test-time adaptation's reliance on streaming conditions which may challenge its usefulness in practice.
We apply existing methods in DA and few-shot transfer learning to provide benchmark performance. We compare with AdaBN (Li et al., 2016) by replacing source BN statistics with those calculated on target support set, finetuning the source model on BN layers or classifier or feature extractor, finetuning entire model with L2 or L2 -SP (Li et al., 2018) or DELTA (Li et al., 2019) regularization, Late Fusion (Hoffman et al., 2013) which averages scores from source and target classifier, and FLUTE (Triantafillou et al., 2021) which optimizes BN parameters with nearest-centroid classifier.****We additionally compare our proposed method with source-free UDA methods which adapts with entire unlabelled target dataset, including AdaBN (Li et al., 2016), SHOT (Liang et al., 2020), SFDA (Kim et al., 2020) and SDDA (Kurmi et al., 2021), as well as the few-shot method L2 and FLUTE, in Table 6 on Office and Office-Home.****Additionally, we compare with state-of-the-art source-free UDA method SHOT (Liang et al., 2020), which adapts on the entire unlabelled target dataset, on OfficeHome. From Table 16, SHOT performs better than finetuning LCCS in most domain pairs.
Answer: We have added Table 5 in Section 4.2 for source-free few-shot adaptation comparison. We include AdaBN, Late Fusion, finetuning methods (finetune BN/classifier/feature extractor, regularized finetuning), cross-domain few-shot learning (FLUTE), hypothesis transfer (finetune BN/feature extractor). Note that finetuning feature extractor is equivalent to SHOT (Liang et al., 2020) with supervised instead of unsupervised loss. We have also included additional comparisons with source-free UDA which uses entire unlabelled target dataset in Table 6 and non-source-free few-shot DA in Appendix Section C.4. Overall, our proposed method achieves competitive performance against other source-free few-shot methods. Source-free UDA can achieve better performance than our proposed method in some cases, but at a cost of less data-efficiency and increased model update computation.