Question: In stage 1, the parameters of the feature extractor are updated, not the classifier; What would be effective if we update the classifier too?
Evidence: Datasets We evaluate our method on both domain adaptation and long-tailed recognition benchmarks, including VisDA-C, Office Home, Sketch, ImageNet-LT. Figure presents some of example images to illustrate domain shifts.
Metric We report top-1 accuracy (denoted as acc.) on the whole dataset for all datasets. On VisDA-C, we additionally report the percentage accuracy of each category and the corresponding average of all categorical accuracies (denoted as avg.), due to the imbalanced distributed label space. On Office Home, we calculate the average for all kinds of domain shifts as the summary number for each method. On long-tailed recognition benchmarks, we additionally report percentage accuracy on many-shot (more than 100 samples), medium-shot (20-100 samples), and few-shot (less than 20 samples) following the evaluation protocol from;.
Baselines We choose the most recent fully test-time adaptation method, TENT, and source-free adaptation framework, SHOT, as our "online" and "offline" adaptation baselines. TENT does not alter training, while SHOT minimally customizes the source architecture and training. Entropy minimization is the target optimization objective for both TENT and SHOT. SHOT additionally regularizes optimization by the information maximization (InfoMax) loss to augment entropy minimization on each sample with diversity maximization across samples. Following SHOT, we therefore augment TENT into TENT-IM by including this regularization. Alongside their role as baselines, these methods can serve as teacher models for our stage 1. We also compare with unsupervised domain adaptation (UDA) baselines, including DANN, DAN, ADR, CDAN+E, CDAN+BSP, CDAN+TN, SAFN, SWD, DSBN+MSTN, STAR. It is worth noting that all these UDA methods are fine-tuning from the ImageNet pretrained ResNet-101 source model, with access to both source and target data. TENT-IM and SHOT likewise initialize their representation from the source model. In contrast, our method trains ResNet-18 from scratch, and entirely on the target data, by contrastive learning for the representation and teacher-student learning for the classification.
For long-tailed recognition, we choose learnable weight scaling (LWS) as the train-time baseline. LWS first decouples the full model into representation and classification, and then only re-scales the classifier parameters with class-balanced sampling. Our on-target class distribution learning extends LWS to test-time, by re-scaling the parameters on unlabeled target data.
Architecture For comparability with state-of-the-art models, we choose 18/50/101-layer ResNet models for both main results and ablation studies. When reproducing the prior works, we keep the architecture the same, for example, the weight-normalization augmented ImageNet pretrained ResNet-101 for SHOT.
We choose the most recent fully test-time adaptation method, TENT (Wang et al., 2021), and source-free adaptation framework, SHOT (Liang et al., 2020), as our “online” and “offline” adaptation baselines. TENT does not alter training, while SHOT minimally customizes the source architecture and training. Entropy minimization is the target optimization objective for both TENT and SHOT. SHOT additionally regularizes optimization by the information maximization (InfoMax) loss (Gomes et al., 2010; Shi & Sha, 2012; Hu et al., 2017) to augment entropy minimization on each sample with diversity maximization across samples. Following SHOT, we therefore augment TENT into TENT-IM by including this regularization. Alongside their role as baselines, these methods can serve as teacher models for our stage 1. We also compare with unsupervised domain adaptation (UDA) baselines, including DANN (Ganin & Lempitsky, 2015), DAN (Long et al., 2015), ADR (Saito et al., 2018), CDAN+E (Long et al., 2018), CDAN+BSP (Chen et al., 2019), CDAN+TN (Wang et al., 2019b), SAFN (Xu et al., 2019), SWD (Lee et al., 2019), DSBN+MSTN (Chang et al., 2019), STAR (Lu et al., 2020). It is worth noting that all these UDA methods are fine-tuning from the ImageNet pretrained ResNet-101 source model, with access to both source and target data. TENT-IM and SHOT likewise initialize their representation from the source model. In contrast, our method trains ResNet-18 from scratch, and entirely on the target data, by contrastive learning for the representation and teacher-student learning for the classification.
Answer: Please see the discussion of adaptation parameters in the cited TENT and SHOT papers. Both find that it is more effective to adapt a subset of parameters than to directly optimize the classifier parameters during this stage. Note that the design choices of Stage 1, for source-free adaptation, are not the subject of our contribution. We investigate if parameters can be learned fully on target data by self-supervision on target and distillation from source predictions, where those source predictions can be improved by Stage 1. Our on-target adaptation differs in its Stage 2, with contrastive learning, and its training of the classifier parameters on target data via distillation and teacher-student updates with Stage 3.