Question: Can you provide any details on Tent and other baseline methods?
Evidence: We compare our method with the baseline source models, as well as with test-time adaptation methods that also augment only batch normalization layers namely Test-time BN and the state-of-the-art Tent) that adapts BN parameters by entropy minimization with default hyperparameters i.e. Adam or SGD with momentum with mini-batch size 128 and learning rate 0.001. We point out that test-time adaptation methods are not designed for the same setting and are not directly comparable with our proposed method. We include them to show empirically that adapting with a few labelled samples can achieve better and more reliable classification performance than adapting with a large streaming dataset. On all datasets, we set mini-batch size as 32, and use Adam optimizer with 0.001 learning rate for finetuning LCCS. For larger support sets at k ≥ 5, we use the nearest-centroid-classifier as in instead of source classifier. We set n = k × # classes for µ LCCS for Camelyon17 and VisDA, and n = 1 otherwise.
From Table, our proposed method improves over source models even with a single example per class for all datasets evaluated, and overall is comparable to or better than Test-time BN and Tent when k ≥ 5. We observe that Tent performance is dependent on optimizer choice, with Adam outperforming SGD by approximately 2% in PACS and SGD outperforming Adam by as much as 39.2% on VisDA. Since the better performing optimizer is dataset-dependent, this makes the practical usage of Tent challenging. For PACS, our proposed method performs better than the best case of Tent when support set has 5 or 10 samples per class. For Camelyon17, all BN adaptation methods improve over source model accuracy by at least 18% in the best scenario, implying that replacing source BN statistics with target BN statistics is effective in addressing domain shift in this dataset. For the 182-class iWildCam dataset, the test dataset is imbalanced; 102 classes have at least 1 sample, and only 87 classes have at least 5 samples. Hence to prevent introducing more class imbalance in the adaptation and evaluation processes, we only evaluate the setting where k = 1, which translates to 102 support samples. Our proposed method improves slightly over source model, while all test-adaptation methods obtained lower F1-score than the original baseline performance.
Similarly for VisDA, our proposed method obtains the best target performance. In particular, all test-time adaptation methods result in worse performance than original source model. We provide standard errors of results in Table in Appendix Section C.2.
Effect of test batches: Although test-time adaptation methods can improve the source model without supervision of labelled samples, their performance relies on streaming conditions and severely degrades with smaller mini-batch size and class-imbalanced mini-batches as shown in Table, and in detail in Appendix Section C.2. We constructed long-tailed imbalanced PACS and VisDA following the procedure in, where sample sizes decay exponentially from the first to last class and α is the ratio of the largest to smallest class. The good performance of test-time methods is  dependent on having large, evenly classes-distributed mini-batches. On the contrary, our proposed method is not dependent on streaming conditions and can more reliably adapt to the target domain.
As far as we know, there are no existing methods designed and evaluated specifically for our sourcefree few-shot DA setting. We apply existing methods in DA and few-shot transfer learning to provide benchmark performance. We compare with AdaBN by replacing source BN statistics with those calculated on target support set, finetuning the source model on BN layers or classifier or feature extractor, finetuning entire model with L 2 or L 2 -SP or DELTA regularization, Late Fusion which averages scores from source and target classifier, and FLUTE which optimizes BN parameters with nearest-centroid classifier. FLUTE assumes the availability of multiple source datasets to train multiple sets of BN parameters for further blending to initialize the finetuning process. Since we only have access to the pre-trained source model in our setting, we reduce FLUTE to the single source dataset case and initialize FLUTE with single source BN parameters. It is conventional to update the classifier when sufficient target samples are available, hence besides implementing with source classifier (SC), we also implement our proposed method with finetuned classifier (FC) and nearestcentroid classifier (NCC). We follow learning settings in for regularized finetuning and use SGD optimizer with momentum and weight decay 0.0004, and L 2 regularization is also added to finetuning on classifier or feature extractor to prevent overfitting. For all other methods that finetune on BN layers, we use Adam optimizer following. We set learning rate 0.001, mini-batch size 32 and epochs 10 for all methods and datasets evaluated.
From Table, we observe that regularized finetuning tends to adapt well at k = 1, but performance can lag behind with larger support sets. AdaBN does not consistently improve adaptation; we show in Appendix Figure that completely replacing source BN statistics degrades performance for VisDA. Overall, finetuning LCCS with source classifier has at least comparable performance with regularized finetuning for small support sets. For larger support sets, finetuning LCCS with NCC has the best performance, with classification accuracy consistently higher than FLUTE which also uses NCC. We also compare our method to L 2 and FLUTE on Office and OfficeHome in Table 16. Though the transfer learning methods produce comparable performance on the easy dataset of Camelyon17 (two classes, two domains), LCCS outperforms on more difficult datasets, which demonstrates the effectiveness of the proposed low-dimensional finetuning strategy.
LCCS Finetuning vs BN Finetuning: We further study our proposed constrained optimization of BN layer versus the unconstrained alternative of optimizing directly on the high-dimensional BN parameters γ and β in Equation 1, with fixed source classifier. We also evaluate a combination of the two strategies by first finetuning LCCS followed by finetuning BN parameters. From Figure, we see that when the support set is small at k ≤ 3, finetuning LCCS is always better than finetuning BN parameters. The results are mixed at larger k, which is expected since the unconstrained optimization is also less prone to overfit with more samples. In addition, we observe that finetuning LCCS first provides good initialization for further finetuning of BN parameters, and consistently achieves better adaptation performance than finetuning BN parameters alone. We make similar observations when using NCC as classifier in Appendix Section C.5.  Figure: Finetuning LCCS has best performance when support set is extremely small (k ≤ 3).
With larger support set (k ≥ 5), finetuning LCCS+BN parameters attains better performance than finetuning BN parameters alone.
We compare our method with the baseline source models, as well as with test-time adaptation methods that also augment only batch normalization layers namely Test-time BN (Nado et al., 2020) and the state-of-the-art Tent (Wang et al., 2021a) that adapts BN parameters by entropy minimization with default hyperparameters i.e. Adam or SGD with momentum with mini-batch size 128 and learning rate 0.001. We point out that test-time adaptation methods are not designed for the same setting and are not directly comparable with our proposed method. We include them to show empirically that adapting with a few labelled samples can achieve better and more reliable classification performance than adapting with a large streaming dataset. On all datasets, we set mini-batch size as 32, and use Adam optimizer with 0.001 learning rate for finetuning LCCS. For larger support sets at k ≥ 5, we use the nearest-centroid-classifier as in (Triantafillou et al., 2021) instead of source classifier. We set n = k × # classes for µLCCS for Camelyon17 and VisDA, and n = 1 otherwise.****As far as we know, there are no existing methods designed and evaluated specifically for our sourcefree few-shot DA setting. We apply existing methods in DA and few-shot transfer learning to provide benchmark performance. We compare with AdaBN (Li et al., 2016) by replacing source BN statistics with those calculated on target support set, finetuning the source model on BN layers or classifier or feature extractor, finetuning entire model with L2 or L2 -SP (Li et al., 2018) or DELTA (Li et al., 2019) regularization, Late Fusion (Hoffman et al., 2013) which averages scores from source and target classifier, and FLUTE (Triantafillou et al., 2021) which optimizes BN parameters with nearest-centroid classifier. FLUTE assumes the availability of multiple source datasets to train multiple sets of BN parameters for further blending to initialize the finetuning process. Since we only have access to the pre-trained source model in our setting, we reduce FLUTE to the single source dataset case and initialize FLUTE with single source BN parameters. It is conventional to update the classifier when sufficient target samples are available, hence besides implementing with source classifier (SC), we also implement our proposed method with finetuned classifier (FC) and nearestcentroid classifier (NCC). We follow learning settings in (Li et al., 2019) for regularized finetuning and use SGD optimizer with momentum and weight decay 0.0004, and L2 regularization is also added to finetuning on classifier or feature extractor to prevent overfitting. For all other methods that finetune on BN layers, we use Adam optimizer following (Wang et al., 2021a). We set learning rate 0.001, mini-batch size 32 and epochs 10 for all methods and datasets evaluated.
Answer: We have added details of baseline methods at the start of Section 4.1 and 4.2. For Tent with Adam optimizer, we use default hyperparameters of mini-batch size 128 and learning rate 0.001 in (Wang et al., 2021), but we still observe performance degradation in Camelyon17, iWildCam and VisDA in Table 3. Although the rate of adaptation can be reduced by decreasing learning rate, it is difficult to reliably select hyperparameters without labelled samples, and this is a challenge of using test-time adaptation methods in practice.