Question: Does the novelty seem a bit limited? The paper combines some standard ideas (target adaptation, distillation, contrastive pre-train on target, fix-match), the main novelty seems to be in the system - combining these for source free adaptation.
Evidence: Deep networks achieve tremendous success on various visual tasks at the expense of massive data collection and annotation efforts. Even more data is needed when training (source) and testing (target) data differ, as the model must be adapted on the new data to maintain accuracy. To reduce the annotation effort on new data, unsupervised domain adaptation (UDA) approaches transfer knowledge from labeled source data to unlabeled target data. Standard UDA requires simultaneous optimization on the source and target data to do so. However, this requirement may not be entirely practical, in that shifted or future target data may not be available during training. Furthermore, (re-)processing source data during testing may be limited by computation, bandwidth, and privacy. Most importantly, it is the target data that ultimately matters for testing. In this work, we therefore turn our attention from source to target, and how to learn more from it.
Recent work adapts to the target data without the source data or even adapts during testing. However, these "source-free" and "test-time" approaches still rely heavily on the source parameters for finetuning. Source-free adaptation initializes from source parameters then optimizes on target data without the joint use of source data. Testtime adaptation partially updates source parameters on the target data while testing. Such approaches reduce reliance on the source data, and can even improve accuracy, but have they made full use of the target data? Many of the model parameters are fixed or regularized toward the source parameters. We investigate whether more can be learned from target, and more accuracy gained, by not transferring the source parameters.
We propose on-target adaptation to unshackle the target representation from the source representation. To do so, we (1) factorize the representation from the classifier and (2) separate the source parameters from the source predictions. By factorizing the representation from the classifier, we can train the representation entirely on the target data by self-supervision. Given this on-target representation, we can then supervise a new classifier from source predictions by distillation, without transferring the source parameters. Not transferring parameters frees our target model from the constraints of the source architecture, so that we can experiment with distinct target architectures. In this way, we can even change the model size to optimize a target-specific model that is more accurate and more efficient. In contrast to prior work on adaptation, this uniquely allows for learning 100% of the target model parameters on target data, as illustrated by Figure. Figure: Domain adaptation adjusts a model trained on source data for testing on target data. We contrast methods by their updates on source and target. Unsupervised domain adaptation (UDA) jointly learns 50/50 on source/target. Source-free adaptation transfers source parameters, then selectively learns on target. Our on-target approach learns 100% of the testing model parameters on target by neither sharing nor transferring source parameters, but instead distilling source predictions.
To realize our proposed factorization and separation, we employ contrastive learning, sourcefree adaptation, and teacher-student distillation. We initialize the target representation by selfsupervision with contrastive learning. We turn the source model into a teacher model by sourcefree adaptation, and then generate pseudo-labels to supervise distillation. We lastly train the student model on the teacher supervision, starting from the target representation and new classifier parameters, and repeat this teacher-student cycle by resetting the student classifier parameters between epochs. Contrastive learning has recently enabled self-supervised representations to compete with or even surpass supervised representations. We show it provides a sufficient target representation.
Our experiments show on-target adaptation achieves state-of-the-art accuracy and computational efficiency on common domain adaptation benchmarks. For model accuracy, our method brings ∼3% absolute improvement compared to state-of-the-art unsupervised and source-free domain adaptation methods on VisDA-C and ImageNet Sketch while reducing 50%+ of parameters. For computation, our method reduces FLOPs by 50+% and memory by 75+% for each forward pass of the target model. In the long-tailed classification setting, on-target class distribution learning equals the state-of-the-art learnable weight scaling without needing source data. Ablation experiments support the generality of on-target representation learning across architectures, contrastive learning methods, losses, and amount of optimization.
Our contribution is to investigate whether the source data should be the primary source of target model parameters, and to propose an alternative: on-target adaptation. Our insight is that the source representation can be fully decoupled from source supervision. Domain adaptation normally emphasizes the representation of source data, by either jointly optimizing on source data or transferring source parameters. On-target adaptation emphasizes the representation of target data instead, by distilling source predictions into a self-supervised target representation. We are the first to show this is feasible, as a new kind of source-free adaptation. Furthermore we show it improves accuracy and reduces computation on standard benchmarks like VisDA-C.
Deep networks achieve tremendous success on various visual tasks at the expense of massive data collection and annotation efforts. Even more data is needed when training (source) and testing (target) data differ, as the model must be adapted on the new data to maintain accuracy. To reduce the annotation effort on new data, unsupervised domain adaptation (UDA) approaches transfer knowledge from labeled source data to unlabeled target data. Standard UDA requires simultaneous optimization on the source and target data to do so. However, this requirement may not be entirely practical, in that shifted or future target data may not be available during training. Furthermore, (re-)processing source data during testing may be limited by computation, bandwidth, and privacy. Most importantly, it is the target data that ultimately matters for testing. In this work, we therefore turn our attention from source to target, and how to learn more from it.
Recent work adapts to the target data without the source data or even adapts during testing. However, these "source-free" and "test-time" approaches still rely heavily on the source parameters for finetuning. Source-free adaptation initializes from source parameters then optimizes on target data without the joint use of source data. Testtime adaptation partially updates source parameters on the target data while testing. Such approaches reduce reliance on the source data, and can even improve accuracy, but have they made full use of the target data? Many of the model parameters are fixed or regularized toward the source parameters. We investigate whether more can be learned from target, and more accuracy gained, by not transferring the source parameters.
We propose on-target adaptation to unshackle the target representation from the source representation. To do so, we (1) factorize the representation from the classifier and (2) separate the source parameters from the source predictions. By factorizing the representation from the classifier, we can train the representation entirely on the target data by self-supervision. Given this on-target representation, we can then supervise a new classifier from source predictions by distillation, without transferring the source parameters. Not transferring parameters frees our target model from the constraints of the source architecture, so that we can experiment with distinct target architectures. In this way, we can even change the model size to optimize a target-specific model that is more accurate and more efficient. In contrast to prior work on adaptation, this uniquely allows for learning 100% of the target model parameters on target data, as illustrated by Figure. Figure: Domain adaptation adjusts a model trained on source data for testing on target data. We contrast methods by their updates on source and target. Unsupervised domain adaptation (UDA) jointly learns 50/50 on source/target. Source-free adaptation transfers source parameters, then selectively learns on target. Our on-target approach learns 100% of the testing model parameters on target by neither sharing nor transferring source parameters, but instead distilling source predictions.
To realize our proposed factorization and separation, we employ contrastive learning, sourcefree adaptation, and teacher-student distillation. We initialize the target representation by selfsupervision with contrastive learning. We turn the source model into a teacher model by sourcefree adaptation, and then generate pseudo-labels to supervise distillation. We lastly train the student model on the teacher supervision, starting from the target representation and new classifier parameters, and repeat this teacher-student cycle by resetting the student classifier parameters between epochs. Contrastive learning has recently enabled self-supervised representations to compete with or even surpass supervised representations. We show it provides a sufficient target representation.
Our experiments show on-target adaptation achieves state-of-the-art accuracy and computational efficiency on common domain adaptation benchmarks. For model accuracy, our method brings ∼3% absolute improvement compared to state-of-the-art unsupervised and source-free domain adaptation methods on VisDA-C and ImageNet Sketch while reducing 50%+ of parameters. For computation, our method reduces FLOPs by 50+% and memory by 75+% for each forward pass of the target model. In the long-tailed classification setting, on-target class distribution learning equals the state-of-the-art learnable weight scaling without needing source data. Ablation experiments support the generality of on-target representation learning across architectures, contrastive learning methods, losses, and amount of optimization.
Our contribution is to investigate whether the source data should be the primary source of target model parameters, and to propose an alternative: on-target adaptation. Our insight is that the source representation can be fully decoupled from source supervision. Domain adaptation normally emphasizes the representation of source data, by either jointly optimizing on source data or transferring source parameters. On-target adaptation emphasizes the representation of target data instead, by distilling source predictions into a self-supervised target representation. We are the first to show this is feasible, as a new kind of source-free adaptation. Furthermore we show it improves accuracy and reduces computation on standard benchmarks like VisDA-C.
Deep networks achieve tremendous success on various visual tasks at the expense of massive data collection and annotation efforts. Even more data is needed when training (source) and testing (target) data differ, as the model must be adapted on the new data to maintain accuracy. To reduce the annotation effort on new data, unsupervised domain adaptation (UDA) approaches transfer knowledge from labeled source data to unlabeled target data. Standard UDA requires simultaneous optimization on the source and target data to do so. However, this requirement may not be entirely practical, in that shifted or future target data may not be available during training. Furthermore, (re-)processing source data during testing may be limited by computation, bandwidth, and privacy. Most importantly, it is the target data that ultimately matters for testing. In this work, we therefore turn our attention from source to target, and how to learn more from it.
Recent work adapts to the target data without the source data or even adapts during testing. However, these "source-free" and "test-time" approaches still rely heavily on the source parameters for finetuning. Source-free adaptation initializes from source parameters then optimizes on target data without the joint use of source data. Testtime adaptation partially updates source parameters on the target data while testing. Such approaches reduce reliance on the source data, and can even improve accuracy, but have they made full use of the target data? Many of the model parameters are fixed or regularized toward the source parameters. We investigate whether more can be learned from target, and more accuracy gained, by not transferring the source parameters.
We propose on-target adaptation to unshackle the target representation from the source representation. To do so, we (1) factorize the representation from the classifier and (2) separate the source parameters from the source predictions. By factorizing the representation from the classifier, we can train the representation entirely on the target data by self-supervision. Given this on-target representation, we can then supervise a new classifier from source predictions by distillation, without transferring the source parameters. Not transferring parameters frees our target model from the constraints of the source architecture, so that we can experiment with distinct target architectures. In this way, we can even change the model size to optimize a target-specific model that is more accurate and more efficient. In contrast to prior work on adaptation, this uniquely allows for learning 100% of the target model parameters on target data, as illustrated by Figure. Figure: Domain adaptation adjusts a model trained on source data for testing on target data. We contrast methods by their updates on source and target. Unsupervised domain adaptation (UDA) jointly learns 50/50 on source/target. Source-free adaptation transfers source parameters, then selectively learns on target. Our on-target approach learns 100% of the testing model parameters on target by neither sharing nor transferring source parameters, but instead distilling source predictions.
To realize our proposed factorization and separation, we employ contrastive learning, sourcefree adaptation, and teacher-student distillation. We initialize the target representation by selfsupervision with contrastive learning. We turn the source model into a teacher model by sourcefree adaptation, and then generate pseudo-labels to supervise distillation. We lastly train the student model on the teacher supervision, starting from the target representation and new classifier parameters, and repeat this teacher-student cycle by resetting the student classifier parameters between epochs. Contrastive learning has recently enabled self-supervised representations to compete with or even surpass supervised representations. We show it provides a sufficient target representation.
Our experiments show on-target adaptation achieves state-of-the-art accuracy and computational efficiency on common domain adaptation benchmarks. For model accuracy, our method brings ∼3% absolute improvement compared to state-of-the-art unsupervised and source-free domain adaptation methods on VisDA-C and ImageNet Sketch while reducing 50%+ of parameters. For computation, our method reduces FLOPs by 50+% and memory by 75+% for each forward pass of the target model. In the long-tailed classification setting, on-target class distribution learning equals the state-of-the-art learnable weight scaling without needing source data. Ablation experiments support the generality of on-target representation learning across architectures, contrastive learning methods, losses, and amount of optimization.
Our contribution is to investigate whether the source data should be the primary source of target model parameters, and to propose an alternative: on-target adaptation. Our insight is that the source representation can be fully decoupled from source supervision. Domain adaptation normally emphasizes the representation of source data, by either jointly optimizing on source data or transferring source parameters. On-target adaptation emphasizes the representation of target data instead, by distilling source predictions into a self-supervised target representation. We are the first to show this is feasible, as a new kind of source-free adaptation. Furthermore we show it improves accuracy and reduces computation on standard benchmarks like VisDA-C.
Furthermore, (re-)processing source data during testing may be limited by computation, bandwidth, and privacy. Most importantly, it is the target data that ultimately matters for testing. In this work, we therefore turn our attention from source to target, and how to learn more from it.****Our experiments show on-target adaptation achieves state-of-the-art accuracy and computational efficiency on common domain adaptation benchmarks.****Our on-target approach learns 100% of the testing model parameters on target by neither sharing nor transferring source parameters, but instead distilling source predictions.
Answer: We agree, and underline in our general response (Novelty) that the novelty is one part conceptual—in how a system can learn more on target—and one part empirical—in reporting results that show the system can improve both accuracy and computational efficiency by contrastive learning and altering the target architecture. While some parts of our system are standard, its overal characteristic in training 100% of the target model parameters on target data is not (as illustrated in Figure 1).