Question: Perhaps proposed method has minor performance improvements over baselines such as Tent?
Evidence: We compare our method with the baseline source models, as well as with test-time adaptation methods that also augment only batch normalization layers namely Test-time BN and the state-of-the-art Tent) that adapts BN parameters by entropy minimization with default hyperparameters i.e. Adam or SGD with momentum with mini-batch size 128 and learning rate 0.001. We point out that test-time adaptation methods are not designed for the same setting and are not directly comparable with our proposed method. We include them to show empirically that adapting with a few labelled samples can achieve better and more reliable classification performance than adapting with a large streaming dataset. On all datasets, we set mini-batch size as 32, and use Adam optimizer with 0.001 learning rate for finetuning LCCS. For larger support sets at k ≥ 5, we use the nearest-centroid-classifier as in instead of source classifier. We set n = k × # classes for µ LCCS for Camelyon17 and VisDA, and n = 1 otherwise.
From Table, our proposed method improves over source models even with a single example per class for all datasets evaluated, and overall is comparable to or better than Test-time BN and Tent when k ≥ 5. We observe that Tent performance is dependent on optimizer choice, with Adam outperforming SGD by approximately 2% in PACS and SGD outperforming Adam by as much as 39.2% on VisDA. Since the better performing optimizer is dataset-dependent, this makes the practical usage of Tent challenging. For PACS, our proposed method performs better than the best case of Tent when support set has 5 or 10 samples per class. For Camelyon17, all BN adaptation methods improve over source model accuracy by at least 18% in the best scenario, implying that replacing source BN statistics with target BN statistics is effective in addressing domain shift in this dataset. For the 182-class iWildCam dataset, the test dataset is imbalanced; 102 classes have at least 1 sample, and only 87 classes have at least 5 samples. Hence to prevent introducing more class imbalance in the adaptation and evaluation processes, we only evaluate the setting where k = 1, which translates to 102 support samples. Our proposed method improves slightly over source model, while all test-adaptation methods obtained lower F1-score than the original baseline performance.
Similarly for VisDA, our proposed method obtains the best target performance. In particular, all test-time adaptation methods result in worse performance than original source model. We provide standard errors of results in Table in Appendix Section C.2.
Effect of test batches: Although test-time adaptation methods can improve the source model without supervision of labelled samples, their performance relies on streaming conditions and severely degrades with smaller mini-batch size and class-imbalanced mini-batches as shown in Table, and in detail in Appendix Section C.2. We constructed long-tailed imbalanced PACS and VisDA following the procedure in, where sample sizes decay exponentially from the first to last class and α is the ratio of the largest to smallest class. The good performance of test-time methods is  dependent on having large, evenly classes-distributed mini-batches. On the contrary, our proposed method is not dependent on streaming conditions and can more reliably adapt to the target domain.
We compare our method with the baseline source models, as well as with test-time adaptation methods that also augment only batch normalization layers namely Test-time BN and the state-of-the-art Tent) that adapts BN parameters by entropy minimization with default hyperparameters i.e. Adam or SGD with momentum with mini-batch size 128 and learning rate 0.001. We point out that test-time adaptation methods are not designed for the same setting and are not directly comparable with our proposed method. We include them to show empirically that adapting with a few labelled samples can achieve better and more reliable classification performance than adapting with a large streaming dataset. On all datasets, we set mini-batch size as 32, and use Adam optimizer with 0.001 learning rate for finetuning LCCS. For larger support sets at k ≥ 5, we use the nearest-centroid-classifier as in instead of source classifier. We set n = k × # classes for µ LCCS for Camelyon17 and VisDA, and n = 1 otherwise.
From Table, our proposed method improves over source models even with a single example per class for all datasets evaluated, and overall is comparable to or better than Test-time BN and Tent when k ≥ 5. We observe that Tent performance is dependent on optimizer choice, with Adam outperforming SGD by approximately 2% in PACS and SGD outperforming Adam by as much as 39.2% on VisDA. Since the better performing optimizer is dataset-dependent, this makes the practical usage of Tent challenging. For PACS, our proposed method performs better than the best case of Tent when support set has 5 or 10 samples per class. For Camelyon17, all BN adaptation methods improve over source model accuracy by at least 18% in the best scenario, implying that replacing source BN statistics with target BN statistics is effective in addressing domain shift in this dataset. For the 182-class iWildCam dataset, the test dataset is imbalanced; 102 classes have at least 1 sample, and only 87 classes have at least 5 samples. Hence to prevent introducing more class imbalance in the adaptation and evaluation processes, we only evaluate the setting where k = 1, which translates to 102 support samples. Our proposed method improves slightly over source model, while all test-adaptation methods obtained lower F1-score than the original baseline performance.
Similarly for VisDA, our proposed method obtains the best target performance. In particular, all test-time adaptation methods result in worse performance than original source model. We provide standard errors of results in Table in Appendix Section C.2.
Effect of test batches: Although test-time adaptation methods can improve the source model without supervision of labelled samples, their performance relies on streaming conditions and severely degrades with smaller mini-batch size and class-imbalanced mini-batches as shown in Table, and in detail in Appendix Section C.2. We constructed long-tailed imbalanced PACS and VisDA following the procedure in, where sample sizes decay exponentially from the first to last class and α is the ratio of the largest to smallest class. The good performance of test-time methods is  dependent on having large, evenly classes-distributed mini-batches. On the contrary, our proposed method is not dependent on streaming conditions and can more reliably adapt to the target domain.
For larger support sets at k ≥ 5, we use the nearest-centroid-classifier as in (Triantafillou et al., 2021) instead of source classifier. We set n = k × # classes for µLCCS for Camelyon17 and VisDA, and n = 1 otherwise.****Although test-time adaptation methods can improve the source model without supervision of labelled samples, their performance relies on streaming conditions and severely degrades with smaller mini-batch size and class-imbalanced mini-batches as shown in Table 4, and in detail in Appendix Section C.2. We constructed long-tailed imbalanced PACS and VisDA following the procedure in (Cao et al., 2019), where sample sizes decay exponentially from the first to last class and 伪 is the ratio of the largest to smallest class. The good performance of test-time methods is dependent on having large, evenly classes-distributed mini-batches. On the contrary, our proposed method is not dependent on streaming conditions and can more reliably adapt to the target domain.
Answer: In Section 4.1, we have improved the performance of our proposed method in Table 3 by using nearest-centroid-classifier (NCC) in place of source classifier when k≥ 5 and setting n>1 for Camelyon17 and VisDA. The use of NCC classifier is possible due to the availability of labelled target samples. In Table 4, we have added experiments showing performance degradation in test-time adaptation methods when test mini-batch size is small or class-imbalanced, while our proposed method is not reliant on data streaming conditions and consistently has best performances in these different conditions.