Question: In terms of performance on the target data, TO-KD method performs better in certain settings and in the cases STU-KD outperforms it is only by narrow margins. So then what is the benefit of the additional effort of training on the source data? Especially considering that it requires additional computational resources. Clearly, the performance on the source data is worse for TO-KD, but how important is that for the device?
Evidence: For both TO-KD and STU-KD, their peak memory footprints are caused by training the large model via lite residual hypothesis transfer (LRHT).****The following schemes are compared with STU-KD: the compact model without domain adaptation, source hypothesis transfer (SHOT) (Liang et al., 2020), and target-only knowledge distillation (TO-KD).****Although STU-KD-AVG is comparable to STU-KD in terms of inference accuracy on the target data, it suffers poor inference accuracy on the source data. As shown in Figure 4, STU-KD-AVG only converges to a point with a low inference accuracy on the source data.
Answer: Both TO-KD and STU-KD are developed in this paper. They adopt the same method (i.e., lite residual hypothesis transfer and knowledge distillation) to adapt the compact model to the target data. Thus, the performance of these two schemes are similar in terms of the inference accuracy on the target data. If the compact model is not trained over the source data (like the procedure in TO-KD), then it is only adapted to the target data while forgetting the knowledge of the source data. When some source data occur in the new environment, the compact model trained by TO-KD can suffer low inference accuracy on these source data. More specifically, the inference accuracy on the source data is reduced by up to 17% as shown in our experimental results. Such phenomenon is undesired for any image recognition application. Considering a mobile robot that works in a manufacturing factory with many different working zones, it needs to recognize the objects, e.g., machines, tools, and human in these working zones. Initially, the compact machine learning model on the robot is built based on the labeled images taken in some of the working zones (i.e., the source data). For the sake of clarity, these zones are defined as source zones, and the rest zones are defined as target zones. When the robot moves to a target zone, its compact model needs to be adapted to the images from these zones (i.e., the target data), so as to ensure high inference accuracy. However, if the compact model is only adapted to the target data without being trained over the source data, it can forget the knowledge of the source data. Thus, after domain adaptation, when the robot moves back to a source zone, the performance of the compact model degrades. By training the compact model over the source data during domain adaptation, the knowledge of the source data can be maintained by the compact model. As a result, the compact model trained by STU-KD achieves high inference accuracy on both the target data and the source data. More specifically, the corresponding inference accuracy on the source data is comparable to that of the compact model before domain adaptation, according to our experimental results.