Question: In the scheme, do the authors directly use SHOT (ICML'20) for adaptation from a large source model to a large target model?
Evidence: In this paper, a source-target unified knowledge distillation (STU-KD) scheme was developed to adapt the compact model on the edge device to the target data while protecting users' privacy. In the scheme, a large source model was first loaded and adapted to the target data on the edge device, and a large target model was obtained. Such an adaptation was enabled by a memory efficient lite residual hypothesis transfer algorithm. A collaborative knowledge distillation method that unifies the target data and the source data for training the compact model was developed to transfer the knowledge of the large target model to the compact model while preventing catastrophic forgetting. Secure aggregation was also employed in STU-KD to enhance privacy preservation. Extensive experiments showed that STU-KD can achieve the highest inference accuracy over the target data while maintaining the original high inference accuracy over the source data, compared with stateof-the-art schemes. STU-KD is now evaluated on public datasets. How to incorporate STU-KD into a real-world application can be an interesting topic for future work.
To boost the inference accuracy of the compact model, a large source model with sufficient model capacity, defined as a function f T , is adopted in STU-KD. It can learn more fine-grained representations of the target data compared with the compact model. Before being deployed to the M edge devices, the large source model f T is pretrained over the source data by minimizing an objective function J T (w T ) = E (xs,ys) [L(f T (x s ; w T ), y s )], where w T denotes the parameters of the large source model, and L(•, •) is the standard cross-entropy loss. On edge device m ∈ {1, ..., M }, the large source model is then adapted to the target data. The challenges of the adaptation are twofold. First, the target data are unlabeled so that fine-grained representations of the target data cannot be learned via supervised learning. As a result, unsupervised domain adaptation (UDA) is needed. Second, edge device m does not have the source data, so many UDA methods are not feasible for the adaptation. Thus, source-free UDA methods) must be employed. However, existing source-free UDA methods, e.g., source hypothesis transfer (SHOT), require retraining of the large source model, which leads to large memory consumption on the edge device.
To tackle the above challenges of the adaptation, a memory-efficient UDA method called lite residual hypothesis transfer (LRHT) is designed by enhancing the architecture of a source-free UDA method such as SHOT. LRHT adopts the same loss function as that of SHOT (denoted as J S in Figure), so unsupervised learning can be conducted over the unlabeled target data. The details of the loss function of SHOT are provided in Appendix A. However, the feature extractor in SHOT must be replaced with a new architecture that can be trained in a memory-efficient manner. As shown in Figure, the new architecture is designed by adding lite residual (LR) modules to the feature extractor such that its outputs can be fine-tuned by the LR modules. The LR modules fine-tune the outputs of the feature extractor by changing the intermediate activations of the feature extractor. Define the i-th building block of the feature extractor as a function g i and the LR module added to the it as a function r i . Given the activations a i−1 of the (i − 1)-th building block, LR module r i changes the activations of the i-th building block as a i = g i (a i−1 ) + r i (a i−1 ), where g i (a i−1 ) denotes the original activations of the i-th building block, and r i (a i−1 ) denotes the residual activations of LR module r i . During the adaptation, the parameters of the feature extractor and that of the classifier are fixed, and only the LR modules are trained to fine-tune the outputs of the feature extractor. Compared to the case of directly training the feature extractor (like SHOT), training the LR modules generates a much smaller volume of activations and demands much lower memory footprint. Thus, by using LRHT the large source model can be adapted to the target data to obtain a large target model in a memory-efficient manner.
To boost the inference accuracy of the compact model, a large source model with sufficient model capacity, defined as a function f T , is adopted in STU-KD. It can learn more fine-grained representations of the target data compared with the compact model. Before being deployed to the M edge devices, the large source model f T is pretrained over the source data by minimizing an objective function J T (w T ) = E (xs,ys) [L(f T (x s ; w T ), y s )], where w T denotes the parameters of the large source model, and L(•, •) is the standard cross-entropy loss. On edge device m ∈ {1, ..., M }, the large source model is then adapted to the target data. The challenges of the adaptation are twofold. First, the target data are unlabeled so that fine-grained representations of the target data cannot be learned via supervised learning. As a result, unsupervised domain adaptation (UDA) is needed. Second, edge device m does not have the source data, so many UDA methods are not feasible for the adaptation. Thus, source-free UDA methods) must be employed. However, existing source-free UDA methods, e.g., source hypothesis transfer (SHOT), require retraining of the large source model, which leads to large memory consumption on the edge device.
To tackle the above challenges of the adaptation, a memory-efficient UDA method called lite residual hypothesis transfer (LRHT) is designed by enhancing the architecture of a source-free UDA method such as SHOT. LRHT adopts the same loss function as that of SHOT (denoted as J S in Figure), so unsupervised learning can be conducted over the unlabeled target data. The details of the loss function of SHOT are provided in Appendix A. However, the feature extractor in SHOT must be replaced with a new architecture that can be trained in a memory-efficient manner. As shown in Figure, the new architecture is designed by adding lite residual (LR) modules to the feature extractor such that its outputs can be fine-tuned by the LR modules. The LR modules fine-tune the outputs of the feature extractor by changing the intermediate activations of the feature extractor. Define the i-th building block of the feature extractor as a function g i and the LR module added to the it as a function r i . Given the activations a i−1 of the (i − 1)-th building block, LR module r i changes the activations of the i-th building block as a i = g i (a i−1 ) + r i (a i−1 ), where g i (a i−1 ) denotes the original activations of the i-th building block, and r i (a i−1 ) denotes the residual activations of LR module r i . During the adaptation, the parameters of the feature extractor and that of the classifier are fixed, and only the LR modules are trained to fine-tune the outputs of the feature extractor. Compared to the case of directly training the feature extractor (like SHOT), training the LR modules generates a much smaller volume of activations and demands much lower memory footprint. Thus, by using LRHT the large source model can be adapted to the target data to obtain a large target model in a memory-efficient manner.
In this paper, a source-target unified knowledge distillation (STU-KD) scheme was developed to adapt the compact model on the edge device to the target data while protecting users’ privacy. In the scheme, a large source model was first loaded and adapted to the target data on the edge device, and a large target model was obtained. Such an adaptation was enabled by a memory efficient lite residual hypothesis transfer algorithm..****As shown in Figure 2, the new architecture is designed by adding lite residual (LR) modules (Cai et al., 2020) to the feature extractor such that its outputs can be fine-tuned by the LR modules. ****During the adaptation, the parameters of the feature extractor and that of the classifier are fixed, and only the LR modules are trained to fine-tune the outputs of the feature extractor.
Answer: In our scheme, we design lite residual hypothesis transfer instead of directly using SHOT for the adaptation from a large source model to a large target model. The problem of SHOT is that it directly trains the feature extractor of the large source model during model adaptation, and thus causes large memory footprint. To reduce the memory footprint during model adaptation, we develop lite residual hypothesis transfer as follows. A new architecture for the large source model is first designed by adding several memory efficient lite residual (LR) modules to the feature extractor of the large source model. Next, during model adaptation, the parameters of the feature extractor are fixed, and only the LR modules are trained to accomplish the adaptation.