Reinforced Cross-Domain Knowledge Distillation on Time Series Data
Authors: QING XU
Min Wu
Xiaoli Li
Kezhi Mao
Zhenghua Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: Knowledge distillation
Domain adaptation
Time series
Primary area: deep_learning_architectures

Abstract: Unsupervised domain adaptation methods have demonstrated superior capabilities in handling the domain shift issue which widely exists in various time series tasks. However, their prominent adaptation performances heavily rely on complex model architectures, posing an unprecedented challenge in deploying them on resource-limited devices for real-time monitoring. Existing approaches, which integrates knowledge distillation into domain adaptation frameworks to simultaneously address domain shift and model complexity, often neglect network capacity gap between teacher and student and just coarsely align their outputs over all source and target samples, resulting in poor distillation efficiency. Thus, in this paper, we propose an innovative framework named Reinforced Cross-Domain Knowledge Distillation (RCD-KD) which can effectively adapt to student's network capability via dynamically selecting suitable target domain samples for knowledge transferring. Particularly, a reinforcement learning-based module with a novel reward function is proposed to learn optimal target sample selection policy based on student's capacity. Meanwhile, a domain discriminator is designed to transfer the domain invariant knowledge. Empirical experimental results and analyses on four public time series datasets demonstrate the effectiveness of our proposed method over other state-of-the-art benchmarks.

Reinforced Cross-Domain Knowledge Distillation on Time Series Data
Qing Xu Institute for Infocomm Research A*STAR, Singapore Nanyang Technological University Xu Xu_Qing0122 a-star wws.ttrrruu.gg......
Min Wu
Institute for Infocomm Research
A*STAR, Singapore wumin@i2r awwtaarraasta.... aww.ar..ut.d............... e
Xiaoli Li
Kezhi Mao
Institute for Infocomm Research, A*STAR, Singapore S*STAR Centre for Frontier AI Research, Singapore x1li@i2r star edu.sg
Nanyang Technological University EKZMaoentu, edu, sg
Zhenghua Chen'
Institute for Infocomm Research, A STAR, Singapore A*STAR Centre for Frontier AI Research, Singapore chen08320e ntu, edu.sg
Abstract
Unsupervised domain adaptation methods have demonstrated superior capabili- ties in handling the domain shift issue which widely exists in various time series tasks. However, their prominent adaptation performances heavily rely complex model architectures, posing an unprecedented challenge deploying them on soource-limiite devices for real-time monitoring. Existing approaches, which inte- grates knowledge distillation into domain adaptation frameworks to simultaneously address domain shift and model complexity, often neglect network capacity gap between teacher and student and just coarsely align their outputs over all source and target samples, resulting in poor distillation efficiency. Thus, in this paper, we propose an innovative framework named Reinforced Moos--Domain Knowledge Distillation (RCD-KD) which can effectively adapt student's network capa- bility via dynamically selecting suitable target domain samples for knowledge transferring. Particularly, reinforcement learning- reward function is proposed to learn optimal target sample selection policy based student's 's capacity. Meanwhile, domain discriminator is designed to transfer the domain invariant knowledge. Empirical experimental results and analyses four public time series datasets demonstrate the effectiveness of our proposed method over other state-of-the-aat benchmarks. Our source code is available at https///tthbbboooouuugggggRinnn Ceceefoorre--Crs-s-eee Cross-Domains Knwwledge-Distilllee on- on-Time-Series-Dan
baasid based module with novel
Introduction
Recent years have witnessed great successes of deep neural networks (DNNs) in various time series applications [ 2 4. Nevertheless, significant drawback impeding their scalability is the limited
"Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

generalization capability on unseen data. This challenge arises when there is a distribution disparity between the data used for training and deployment. For instance, fault diagnosis model trained on certain machines may perform poorly on the data collected from other machines which have different working conditions and configurations. Collecting and annotating data for each machine would be very laborious and costly. To handle this, various unsupervised domain adaptation (UDA) methods have been extensively explored. These methods aim to transfer the domain invariant knowledge from an existing labeled data domain (i.e., source domain) to unlabeled domain (i.e., target domain) either by explicitly minimizing certain pre-defined discrepancy metrics [3.
6 or implicitly learning representations with adversarial manners 77 [). However, these UDA methods heavily rely on the complex network architectures and their adaptation performance will significantly degrade with shallower networks [9] [T]]. The over over- prramaaammttee DNNs will inevitably lead to another practical issue in industries. For many real-world time series tasks, the developed models often required to be deployed on edge devices with very limited computational resources, such as smartphones and robots, for real-time and long-term monitoring. The intolerable computational and storage burdens make the deployment of those complex DNNs on edge devices become an unprecedented challenge.
Some pioneering efforts have been made to integrate knowledge distillation (KD) techniques into UDA frameworks to transfer the cross dross-domain knowledge from a cumbersome teacher to a compact empirically find that simply integrating
student for the reduction of model complexity. However, KD with UDA frameworks like existing works will make the compact student suffer from unsatisfying adaptation performance. The rationale behind this lies in the facts that: the one hand, due to its limited network capacity, the compact student may fail to capture the same fine-grained patterns in data as the cumbersome teacher. Coarsely aligning its feature representations
outputs with will impede its learning process and result in sub-optimal performance on the target domain. On the other hand, in the cross-domain scenario, teacher's knowledge on each individual target sample may not be always reliable and instructive due to the lack of label supervision in target domain. Blindly trusting teacher's knowledge for all samples, especially on target domain, will result in negative transfer. Therefore. to achieve good adaptation performance on the target domain, we have to adaptively transfer teacher's 's knowledge based on student's 's network capability.
the teacher like [1] [2]
Motivated by above insights, we propose a novel end-to-end framework for cross dooss- knowledge distillation to simultaneously address domain shift and model complexity. To be specifie, an adversar- ial discriminator module is designed to align teacher's s and student's 's representations between source and target domains latent feature space for iomain-invariant knowledge transfer. Meanwhile, to adaptively transfer teacher's ' knowledge on the unlabeled target domain, we formulate the target sample selection problem under a reinforcement learning framework. For a specific target sample, if the student demonstrates the ability to attain the same uncertainty level as the teacher (i.e., uncertainty consistency), or can largely mimic teacher's outputs (i.e., sample transferability), we deem such a sample suitable for knowledge distillation. Based on that, we design a novel reward function according to student's learning capability. A dueling Double Deep Q-Network (DDQN) then utilized to learn the optimal target sample selection policy for mitigating the negative effects of unsuitable knowledge from teacher. Our contributions are summarized as follows:
An end-to-end framework named Reinforced Cross-Domain Knowledge Distillation (RCD- KD) is proposed to not only effectively transfer the domain-invariant knowledge but also dynamically distill the adaptive target knowledge based on student's learning capability.
We develop an innovative reinforcement learning- module learn the optimal target sample selection policy for robust knowledge distillation. A novel reward function is designed for assessing student's learning capability terms of uncertainty consistency and sample transfcrability to dynamically transfer teacher's 's target knowledge.
The extensive experimental results on four real-world time series tasks demonstrate the superior effectiveness of our approach compared to other SOTA methods.
2 Related Work
In recent years, there are some pioneering works to tackle both domain shift and model complexity simultaneously. resource efficient domain adaptation (REDA) framework with multi-exit architec- tures is proposed in [9) where the 'casier' samples are inferred via early exits and 'harder' ones

---

inferred via top exit. Meanwhile, some other researchers leverage the knowledge distillation [3 to enhance the adaptation performance of the compact student. For instance. a framework named knowledge distillation for unsupervised single target domain adaptation (KD-STDA) is proposed in
[1] Teacher's s knowledge gradually transferred via dynamically adjusting the contributions of UDA and KD loss. Similarly, multi-level distillation for Domain Adaptation (MLD-DA) strategy is proposed in [[2] to improve the distillation efficiency via a novel cross entropy loss. However, the above two methods transfer the knowledge from both source and target domains. We empirically show that the source domain-specific knowledge might have negative contribution to student's 's gener- alization. Besides, MobileDA [14] and adversarial adaptation with distillation (AAD) [15] employ the teacher trained on source-only domain to guide student's 's training, which have already been proved inefficient due to the limited and biased knowledge from teacher model by 4 Moreover, to achieve more reliable knowledge from teacher, in [16] a maximum cluster difference metric is proposed to estimate teacher's confidence on certain sample. In 4 a framework named universal and joint knowledge distillation (UN is proposed to measure teacher's 's confidence on individual sample via the output of data-domain discriminator. However, due to the compact network architecture of the domain-shared feature extractor from student, the estimated uncertainty is not reliable. In our work, we estimate teacher's s knowledge with student's capacity and then utilize the reward for the learning process of RL-based target sample selection module. The experimental results demonstrate that our proposed method can better enhance student's performance on target domain. Meanwhile, our work also relates to active learning (AL) field specifically in terms of selecting the most critical instances from unlabeled data. Note that here we only discuss the uncertainty- sampling strategies in active learning as other query strategies (e.g.. instance correlation) are beyond the scope of our paper. In AL., the uncertainty can be measured by three metrics: least confidence [7 sample margin [19], and sample entropy (20]. Particularly, the entropy metric measures the uncertainty over the whole output prediction distribution [21] 22 our method, instead of explicitly utilizing entropy-based uncertainty as AL methods, we leverage the consistency between teacher'
and student's s entropy-based uncertainty to learn the optimal sample selection policy with dueling
DDQN. See Supplementary for more comparison results.
3 Methodology
3.1 Preliminaries
Following standard UDA setup, consider data from two domains: labeled source domain 1n. Dí. = {2x) yt) )/ and an unlabeled target domain 'gt = { which shares the same label space as source domain but has different data distributions. Here, n. and n, are the number of training samples in source and target domains, respectively. A powerful teacher model with superior adaptation performance is first pre-trained on D4, U.D Agt with SOTA UDA methods. Our objective is to train a compact student model S which is not only shallower than the teacher model but also can achieve competitive performance on unlabeled target domain. To transfer the learned knowledge from teacher to student, one can just follow standard KD [3] and force the student to mimic teacher's soften logits via Eq. D. Here, X represents batch of training samples and KL refers to the Kullback-Leibler divergence. qd and p² are the softmax outputs soften by a temperature factor J from student S and teacher T, respectively. They are calculated by q.e = exp(>e/T)) E exp(2,/T), where C is the number of classes and qe represents the student's prediction probability of a certain sample belonging to the c-th class.
LKD = Ek(pTT1a') -E
ve log(( /4E).
TEA.
DEA
However, since the teacher is trained on unlabeled target data, its prediction performance on specific target sample cannot be guaranteed. The compact architecture of student also limits its ability fully accept teacher's 's knowledge. In other words, directly minimizing the distribution discrepancy between teacher's and student's predictions over all target samples might introduce inappropriate knowledge which will mislead the student's learning process. Thus, we propose to alleviate the above issue with a novel RL-based target sample selection module which can dynamically select suitable samples to assist the knowledge transferring. Fig. iillustrates the details of our proposed method.
3

---

Redure Enecder
Teacher
Mininablee
Target
Souace Dates Roo
Stalant' Lagit
Tencher's Legits
D
Pige
* Tiashar's Frirupy
26" Student's Entrope
pr
Averaged teacher's logits
Dueling DDQN
Clocather
Natey
P
Degain Discriminalor do
Epe
LuND
(V Etpe
Action
Student
pa
q.
State Sk
Raward fg
Penturn Encodlr
Clasifier
*
3
Uncartaiity Cornistenyy
Reward Module
P
Reward ng
Action
X2
R2
R3
Tranfarhilit
q
MCD-bused
Figure 1: Illustration of proposed KCD-KD, A Monte Carlo Dropout (MCD) based reward module is utilized to generate the reward for learning the optimal target sample selection policy. Specifically, the reward function consists of three parts. The first one the action which is the output of dueling DDQN. The second part is the uncertainty consistency, estimated by entropy from student's logits 9 and the averaged logits p of N teachers generated from MCD module. third part is sample transferability based q² and p" The output of reward module ru then will be utilized for the optimization of dueling DDQN for learning optimal sample selection policy. Meanwhile, a domain discriminator & is employed to transfer the Nomain-invariall knowledge.
KT. divergence hetween
RL-based Target Sample Selection
Following 23 24 we consider target sample selection task as a Markov Decision Process which can addressed by reinforcement learning. A RL-based target sample selection module is first designed enhance the distilling efficiency of teacher's knowledge on target domain. Particularly, a dueling DDQN (25) is employed to learn the optimal target sample selection policy. The dueling architecture can effectively mitigate the risk of overestimation by separately estimating the state value and advantage function, which improves the accuracy of action-valus predictions. Meanwhile, to tackle the instability issue often encountered training deep reinforcement learning models, we leverage strategies such as target network and experience replay. Specifically, the target network provides more stable targets for updating the Q-values by maintaining a separate, slowly updated network for generating target values, while experience replay enables the model to learn from diverse set of past experiences, further enhancing stability and convergence during training. In each training batch, we utilize the learned sample selection policy adaptively transfer teacher's 's target knowledge according to student's learning capability. In the following, the detailed definition of state, action, reward and the optimization of dueling DDQN will be introduced.
State. Given a batch of target domain samples {-:²² y and student's feature extractor FS the state 82 at episode E [1, defined as the feature representations from student's '' feature extractor F (atat) ). Here, ny is the batch size and K represents the maximum episode
sy = [F length. The state Sy is forwarded to the ducling DDQN, generating set of actions that decide whether to retain or discard the corresponding target samples. The student subsequently optimized with the selected samples, and the next state 8k+11 be obtained with the updated student. A terminate state will be triggered if the target sample is not selected at time step or the episode reaches the maximum episode length K.
Agt Action. For a specific target sample x; in a batch, it only has two actions a; (0, 1} which is binary. Specifically, a, = means to retain the sample and a; means to discard it. Since the output of ducling DDQN parameterized with O, is two-dimensional Q-value vector, the optimal

---

action of for sample fgt 2; at current episode & thus can be calculated by Eq. 2 Subsequently, the binary weights for all target samples are formulated as w aaj Inyl which then are utilized to calculate the distillation loss LRKD.
af = argmax Q(F (x,"", a, O₂).
(2)
Reward. The reward function is pivotal in shaping the learning process for target sample selection policy, as it offers essential feedback to DDQN regarding the value associated with selecting specific action in the current state. To achieve reliable knowledge transferred from selected target samples, we propose to utilize model uncertainty and sample transferability to design the reward function.
The first component constructing our reward function is a Boolean function R1 = (o= == 1), which indicates whether the sample z; is retained or not. The second component of our reward function is called uncertainty consistency reward. The motivate is straightforward: due to the lacks of label in target domain, we expect that the student should have the same uncertainty level the teacher for a specific sample Ty.. For the teacher model, we employ the Monte Carlo Dropout (MCD) [26] to estimate its uncertainty, which utilizes a dropout distribution to approximate the posterior distribution (See Supplementary for more details). Practically, it means to enable the dropout of teacher model and forward N times for each sample x; and the averaged prediction P
Σn p(y c/x, 0*)
can be utilized to calculate the entropy HL = - Ee pi, "o,.log(p' for measuring its uncertainty. Here, p(y = cl2z, 0*) represents the probability of sample belonging to class c and it is the softmax outputs of teacher model the n-th forward pass. For the student, the uncertainty is caleulated with H.
E. L lo,goggg Intuitively, higher value of the predictive entropy H will be obtained when all classes are predicted to have equal probabilities, which means the model is less confident about the specific data. To ensure the student has consistent uncertainty level as the teacher, we formulate the uncertainty consistency reward R₂ = (H. > 1 Σ}= 1 HP) O (HL > 1 -n H ), where @ is the exclusive-nor operation. The third component our reward function the transferability reward formulated as R3- (D; < 1 Σド= DD)) Here, D. K kT(p, ||
) is the KI. divergence between student's ' prediction and the averaged MCD teacher prediction. Apparently, the samples whose KL divergence are lower than the averaged divergence are easier ones for the compact student to learn. With the above three Boolean functions, our reward function is defined as Eq. 33) shows:
= a1 (R1 + R2 - 0.5) + 02 (R1 + R3- - 0.5),
(3)
where @ is the exclusive-or operation. For the first part of Eq. B a positive reward value will be assigned if sample is retained and student shows consistent uncertainty as the teacher, or it is discarded and student and teacher show inconsistent uncertainty about it. Otherwise, a negative reward will be assigned. Similarly, for the second part of Eq. (3), positive reward will be assigned its transferability is higher than others and being selected, or its transferability is lower than others and not selected. We utilize an and 0: to adjust the contribution of each part. We constrain the reward within the range of 11 to to offer explicit guidance to the DDQN so that it can efficiently learn to distinguish between good and bad actions.
Dueling DDQN Optimization. The dueling deep Q-network consists of two streams as shown in Fig. state-value estimation stream V(s; θg, θy) parameterized with Og and Oy and advantages estimation stream A(s, a; f A.) for each action which parameterized with θ₂ and A- Here, O is a shared encoder. Furthermore, to balance the exploitation and exploration, we adopt the NoisyNet [27 for the fully-connected layers in OF, Ov and O Besides, a replay buffer M is designed to store the historical experience (*k, age Thy 8841, d), where d E {0, } indicates whether the next step & + 1 is the terminal step (d = 0) or not (d = 1). A batch of entries in M will be randomly sampled out for DDQN optimization.
To train the dueling DDQN (i.e., the online network Q), another target Q-network Q' desired, which has identical network architecture as Q but is optimized in a different way. Specifically, the online network Q is to estimate the Q-values Qest by aggregating two steams via Eq. (4):
Qext Vs; 0y) + A(s, OF OA)
Σ a;
A(s, a, 2E AA).
(4)
5

---

The target Q-network Q' is to generate the target Q-values as Eq. (5) shows. Here. 2 { vy A O' ( by, ('A) are the parameters of and Q. respectively. y [0. is the discount factor to balance the immediate and future rewards.
argiax Q( argmax + + 7 Qtar = rk +
(5)
BL-1
The online network Q is optimized by minimizing the Huber loss between Qest and Qtar. The target network Q' updated with moving average method as shown in Eq. (6), where á a smoothing parameter determining how much historical information of the online network to be transferred to the target network.
e' - (1-
(6)
3.3 Student Optimization
With the proposed RI. module, we can efficiently transfer adaptive knowledge from the teacher model the student model by dynamically eliminating target samples which are unsuitable for student learning. Particularly, we reformulated Eq. (I) to Eq. (7). where w [ai...,, Sn is the output of online Q-network Q. By minimizing LRKD. student's generalization capability target domain can be effectively enhanced.
CRKD Σup REX,
W3 "up Log(t /4)).
(7)
LDC Elllog@@b((( (xtg()))
(8)
Meanwhile, transfer the domain-invariant knowledge between domains from teacher to student, we design an adversarial leaning-based module as depicted in Fig. L followed (88] Particularly, a domain discriminator & is employed to distinguish the source of input feature maps (i.e., whether the feature maps are generated from the teacher with source data as inputs or the student with target data as inputs). Since the dimensions of student's and teacher's feature maps are different, an adaptor layer db is employed to match their dimensions. The domain confusion loss is then formulated as Eq. (8). is worth noting that our experiments, we utilize the DANN 77 to pre-train the teacher. Although other DA methods can also be adopted in our framework, the DANN can essentially provide a pre-trained accurate domain discriminator after teacher's training. During transferring Nomain-invariann knowledge, we can re-utilize it and only optimize the student and the adaptor layer v which will significantly improve the training efficiency. Meanwhile, may utilize some other UDA methods to pre-train the teacher. In this case, the domain discriminator has be adversarially trained against the student by minimizing loss fado as Eq. (9). More experimental results in terms of utilizing other UDA methods to train the teacher can be found in Experiments section.
also possible that one
Lado Eloog (F" (Iere))) E[log(1 (b (xtte))))
The overall loss for student optimization is calculated via Eq. 100 X is hyperparameter to balance contribution of each part. Algorithm shows details of proposed RCD-KD.
LDC A+ 2= LRKD-
(10)
Experiments
4.1 Experimental Setup
Datasets. To evaluate our method, extensive experiments are conducted on four public datasets across three different tasks, namely human activity recognition, rolling bearing fault diagnosis and sleep stage classification. To be specific, the first dataset is called human activity recognition (HAR) (29] for identifying subject's activities (ie,, walk, walk upstairs, walk downstairs, stand and sit). Sensory measurements from the accelerometer and gyroscope embedded in smartphone were

---

Algorithm/ Proposed RCD-KD
Input: Teacher T, Student S. adaptation model v, domain discriminator b online and target Q-
network Q and Q", source data D arc target data D Egt and Replay buffer M
1: for every epoch do
2: for every batch eeg Dig and X E D do
3: for episode & [1, K] do
4:
Get state Sk, sample action 0g (s:)) and update next state $k+1
5: Update S and v by minimizing L as Eq. (IO)
6:
Calculate 7% via Eq. (3: Set d' = 0 if episode end, otherwise d =
7:
Store (sk, ag. rk: S.++.d) to if + is not pre-trained then
8:
9:
Fix the parameters in S and 1b and update + via minimizing Ladv in Eq. Sample random batch (Sk, 0gg Tk, $K+1, d) from M
10:
11:
Calculate Qest via Eq. (4) and Qtar via Eq. (5. update Q via Huber loss and @' via Eq. (6)
Table 1: Performance comparison with other UDA methods.
Metric-based
Adversarial-based
Datasets
Student-Ouly
HoMM [
MDDA 5
SASA
DANN 7
CoDATS [30]
AdvSKM 332
Ours
HAR
35.9428.99
83.624
84.89=6.29
83.37=323
82,421382
73.72=8.02
70.72:4.06
94,68:1622
HHAR
58.74110.79
68.02:16.59
73.26:1.35
77.13:4.09
76.03:1..7
74.6424.18
63.24:15.99
82.37=1.84
FD
66.78.14.38
74.52:16.00
81.0015.43
86.75
7..95
77.54.19455
77.8315.71
12.6310.66
SSC
50.394-7.77
59.7945.51
57.45+3.68
59.36+3.69
57.3945.51
57.2145611
57.28+4.77
1.83
collected from 30 subjects. Considering the variability among subjects, each subject is considered single domain and the adaptation is performed between two subjects. Here, follow existing works 33 and select five transfer scenarios. The second evaluation dataset Heterogeneity human activity recognition (HHAR) [31. Compared to HAR, the sensory measurements collected with various models of smartphones from different manufacturers which are positioned with various orientations on subjects. Thus, the domain gaps between different subjects are generally considered to be larger than HAR. Five transfer scenarios are selected for evaluation same as previous work 32. The third dataset is rolling bearing fault diagnosis (FD) [33 which aims to classify the health status of rolling bearing from healthy artificial damages, damages from accelerated lifetime tests. The rolling bearing are tested under various operation conditions. Same as 34), five transfer scenarios between different operational configurations are selected for fair comparison. The last evaluation dataset sleep stage classification (SSC) dataset (35), which intends recognize subject's sleep stages (i.e., wake, non-rapid eye movement NI, N2, N3 and rapid eye movement stage) with electroencephalography waveform. Five scenarios are evaluated following previous study [34]
Implementations. For the proposed RL-based sample selection module, we set y = 0.9 and
6 0.999 following [25] in Eq. (5) and (6). respectively. We set N 10 to calculate teacher's entropy for the reward function and K 5 for the episodes to generate historical experience. Note that to guarantee fair comparison, we ensure the total training steps of ours and benchmark methods are same. Furthermore, we adopted the ID-CNN the backbone of the teacher and student models following 4 34), where student is a shallow version of teacher with less filters (See Supplementary for detailed network architectures of T' S, & and dueling DDQN). For an, 02 in Eq. (3) and X, Eq. 11),, we use the grid search and set 01 = 0.2, a2 = 1.8, À 1.0, T = 2 for experiments. More sensitivity analysis regarding N, K, X, d',, 02 and can be found in Supplementary. The averaged macro Fi-score with three independent running is reported.
4.2 Benchmark with UDA methods
To demonstrate the effectiveness of our proposed RCD-KD, we first compare it with other advanced UDA methods as shown in Table T Note that all of the benchmark UDA methods are directly applied to the compact student. From Table [] some observations can be found. In most transfer scenarios, directly applying UDA methods (either the metric-based or udversarial-basele can improve the performance of compact student model target domain. However, these methods perform inconsistently across different tasks. For instance, HoMM performs best on SSC, but worst on FD compared to other UDA methods. Meanwhile, the improvement of these UDA methods is
7

---

Table 2: Marco FI Fl-scores HAR and HHHA across three independent
HAR Transfer Scenarios
HHAR Transfer Scenarios
Methods
2-
6-423
7-+13
9-118
12-+16
Avg
0-+6
1-+6
2-+7
3-48
4-+5
Avg
Teacher
100.0
100.0
99.92
93.69
81.65
95.05
64.47
94.23
57.22
98.88
97.69
82.50
Student-Only
68.51
59.57
78. 88.88
21.02
51.71
55.94
50.46
65.95
43.22
58.84
75.22
58.74
KD-STDA
11
98.31
89.55
89.28
67.41
63.13
81.54
46.15
92.19
41.69
96.51
89.79
73.27
KA-MCD
16
89.46
59.26
63.62
58.93
45.67
63.39
65.25
90.59
42.57
85.71
85.48
73.92
MLD-DA
12
100.0
99.11
92.96
82.78
64.08
87.79
61.53
94.32
47.91
91.07
92.74
77.51
REDA
99.44
93.81
92.43
74.55
55.77
83.20
32.05
93.85
36.10
90.24
95,41
69.53
AAD 15
83.74
90.89
83.05
75.96
61.67
79.06
53.25
81.22
48.35
87.00
86.36
71.24
MobileDA 14]
92.71
90.19
91.39
77.95
64,34
83.32
46.60
93.31
49.13
98.30
96.84
76.84
UNI-KD
100.0
96.33
93.20
79.77
64.91
86.84
46.66
94.89
59.20
98,45
97.42
79.32
Ours
100.0
100.0
92.87
80.87
94.68
64.47
94.24
57.59
98,45
97.11
82.37
Table Marco Fl-scores on FD and SSC across three independent runs.
FD Transler Scenarios
SSC Transfer Scenarios
Methods
0->1
0- >3
2 51
+2
2-33
Avg
0->11
12 +5
16 >1
7 7>118
9 9-4
Avg
Teacher
88,36
86.46
88.82
99.84
99.92
92.68
51.43
68.71
73.48
72.48
76.59
68.54
Student-Only
34.94
42.14
75.27
90.41
91.13
66.78
35.62
35.87
60.15
61.24
59.05
$0.39
KD-STDA
53.17
50.95
76.76
89.24
98.66
73.76
43.75
53.45
49.04
67.23
65,56
55.81
KA-MCD
57.96
65.26
61.66
81.75
91.79
71.68
50.85
56.73
51.01
64.18
65.95
57.74
MLDD-DA
12
78.16
75.49
83.34
99.86
96.83
86.74
45.36
66.17
58.37
63.87
70.71
60.90
REDA 9
86.70
81.08
88.98
92.35
88.77
87.58
44.07
52.01
60.14
60.46
64.67
56.27
AAD
15
52.50
60.00
80.86
89.84
95.99
75.84
32.71
62.92
63.34
64.46
72.15
59.12
MobileDA 14
76.19
58.77
83.74
97.56
84
82.82
41.83
57.14
59.41
64.38
61.55
56,86
UNI-KD
78.85
82,68
92.14
97.29
99,34
90.06
44.48
60.13
62.99
71.03
72.21
62.17
Ours
89,88
85.63
88.57
99.92
99.13
92.63
49.73
70.74
72.14
71.73
76.95
68.26
very marginal and large variance can be observed, indicating the challenge of performing domain adaptation with shallow networks. On the contrary, the compact student trained with our proposed RCD-KD consistently performs better than other methods.
4.3 Benchmark with KD-based DA methods
We compare our proposed method with other KD-based DA methods as shown in Table and Table 3 We applied above methods our teacher-student settings. We also report the performance of pre- trained teacher (generally considered as the upper limit) and the student trained on source domain but tested on target domain (namely, Student-Only) with bold for each scenario and the averaged performance. Note that this comparison does not include the teacher as it benefits from more complex network architecture. See Supplementary for more experimental results of additional transfer scenarios.
the lower limit. We highlight the best performance
Some observations can be found from above two tables. Firstly, compared to Student-Only, all the benchmark methods can obviously improve compact student's generalization on target domain. However, some of them (e.g., KD-STDA in HAR. FD and SSC, KA-MCD in HAR and FD) even perform worse than directly applying UDA on compact student (e.g.. MDDA and SASA in Table []] The reason is that those methods blindly trust teacher's predictions on target domain mentioned and learning with such unreliable knowledge will result in inferior performance. Secondly, the methods using source-only teachers (i.e., AAD and MobileDA) failed achieve better performance than others (e.g., UNI-KD and MIDDDAA) which employ teachers trained on both source and target domains. This observation indicates that for cross-domain KD, it is critical for the teacher to possess the knowledge of both domains. Thirdly, introducing the source domain specific knowledge to the student like KD-STDA and MLD-DA apparently cannot guarantee better generalization on target data. Intuitively, the student still needs to pay more attention on target domain or focus on domain- shared knowledge as UNI-KD suggested. Lastly, our proposed method consistently outperforms other benchmarks over all the datasets and achieves the highest score in most of transfer scenarios. Meanwhile, our RCD-KD KD can even achieve comparable performance as the teacher model some datasets (e.g., HAR, HHAR and FD) with more compact model architectures. It indicates the effectiveness of transferring the adaptive knowledge via proposed RI--based sample selection module and
knowledge via domain discriminator.
8

---

Student UNU-D
UNHAD Ours
Mooro Ft Stooooe
Outs
Maoro
HAR
HHAR
SSC
HAR
HHAR
FD
SSC
Figure 2: TCN +D--CNN.
Figure 3: Resnet-34
Resnet-18.
Table 4: Teacher with different UDA methods.
Table 5: Framework ablation for proposed method.
T-Types
HAR
HHAR
SSC
LkD
Lnc
LARD
HAR
HHAR
FD
SSC
MDDA
89.16
81.65
90.45
61.62
79.46
75.69
72.35
52.63
SASA
88.16
80.39
90.83
63.25
85.51
79.03
77.95
62.39
CoDATS
93.39
82.98
91.08
66.89
v
89.32
78.99
89.13
60.65
DANN (ours)
94.68
82.37
92.63
68.26
8
94.68
82.37
92.63
68.26
4.4
Ablation Study
Different Teacher-Student (T S) pairs. Besides transferring the knowledge from a cumbersome ID-CNN teacher to compact 1D-CNN student following [4, we also evaluate our proposed method
with other T - S pairs. Specifically, we adopted temporal convolutional network (TCN) [37] as the teacher and ID-CNN as the student, Resnet-34 (38 as the teacher and Resnet-18 18 the student as depicted in Fig. 2 and Fig. 3 We can see that our proposed method consistently outperforms other benchmark methods, further indicating its effectiveness under different T S configurations.
Different Teacher Generation. As stated in the Methodology section, our proposed method re- utilizes the domain discriminator after teacher's ' training with DANN. To further demonstrate that our framework can be generalized to other UDA methods, we evaluate our framework with teachers generated by various DA methods shown in Table Specifically, utilize two discrepancy- -hased DA methods, i..., MDDA (5) and SASA (36] and one additional adversarial-based DA method named CoDATS 8 Note that for teachers generated from MDDA and SASA, we need train the domain discriminator as shown in Algorithm [ On the contrary, for teachers generated from CoDATS and DANN (ours), the parameters of discriminator we can see that employing teachers from MDDA and SASA slightly underperforms CoDATS and DANN. The possible reason that adding additional training steps for domain discriminator will inevitably increase training difficulty in terms of model convergence. But they still perform better than other benchmarks in Tables 2 and 3 indicating that our approach not limited by the teacher training strategies long the teacher possesses the source and target domain knowledge. A well pre-trained domain discriminator is preferred as it can improve the training efficiency.
frozen during student's training. From Table 4
Framework Ablation. There are two key components in our proposed framework: the domain- invariant knowledge transferring via LD and the RL-based domain adaptive knowledge transferring via LAKD. We conduct the ablation study to evaluate their contributions. We also include the standard knowledge distillation which the student is optimized with L as Eq. [1] shows. From T we can see that coarsely aligning teacher's 's and student's 's predictions over all target samples (LxD LDC)) might lead negative transferring some datasets (e.g., HHAR and SSC), whose performance is inferior to the one only applying LDC. Our proposed method (L + LRKD) can significantly mitigate this issue via the proposed RL-based target sample selection module.
Reinforced Sample Selection Ablation. To investigate the contribution of proposed reward and RL-based model selection module, we conduct the ablation study as shown Table 3 Some observation can be found from above table. Firstly, including all the target samples to train the compact student will inevitable introduce negative transfer, resulting in unsatisfying generalization performance on target domain. Either utilizing the model uncertainty or transfcrability to explicitly

---

Table 6: Reinforced sample selection ablation. "Full samples" denotes utilizing whole target samples for KD;
'R2'. 'Rz' denote directly utilizing proposed uncertainty and transferability for sample selection; 'RL", " "R." denote utilizing RI. with R2 and Ra as reward for sample selection; (R2 + Ra) is our proposed method.
Full Samples
Partial Samples
Datasets
R₂
R.
R3
R3
R₂ Rs
(R2 + R3)
HAR
89.32
91.65
93.91
92.31
93.96
93.53
94.68
HHAR
78.99
78.30
81.73
80.33
82.29
82.37 91.85
81.04
HD
89.13
90.17
91.93
89.51
91.08
92.63
SSC
60.65
63.16
62.98
65.81
67.49
65.20
68.26
Table 7: Comparison of Computational Complexity.
Methods
KD-STDA
KA-MCD
MLD-DA
REDA
AAD
MobileDA
UNI-KD
Ours 16.42
Time (sec)
1.68
4.55
1.91
1.78
0.91
1.28
3.26
select target sample (as shown in column R2 and R3) would improve student's performance in most of datasets. Secondly, dynamically selecting target samples using our RL-base module with either of proposed rewards (as shown in column R. and R2)) will further improve student's 's performance, indicating the effectiveness of RL--based module mitigating negative transfer. Lastly, combining model uncertainty and transferabilitty as the reward to dynamically select suitable target samples based on student's capacity yields best performance.
Computational Complexity. We performed the time complexity analysis for our method and the results are shown Table 7 Specifically, we measure the training time for our proposed method and other benchmarks with NVIDIA 2080Ti GPU. The reported results are measured with one epoch on single transfer scenario on FD dataset, which has the largest training samples (about 1,800 samples per transfer scenario) among evaluated datasets. We can that our method does require more training time compared to other benchmarks, reflecting its greater complexity. The primary computational costs arise from two factors. The first part is the generation of K historical experiences at each step. This could be significantly reduced by using a smaller K. The second factor is the MCD module which conducts multiple inference processes for uncertainty estimation. This computational burden could be further decreased by adopting alternative uncertainty estimation methods. Nevertheless, although training time is longer than other benchmarks, we argue that it is still within acceptable range, especially considering the performance improvement it could bring. Meanwhile, we also evaluate the scalability of our approach to larger dataset. See Supplementary for more scalability analysis.
5 Conclusion and Limitations
In this paper, we propose a framework for cross-domain knowledge distillation on time series. Specifically, we utilize an adversarial domain discriminator to assist the compact student learn the comain-invvrianl knowledge from the cumbersome teacher. Meanwhile, we design reinforcement learning-based target sample selection module to effectively transfer teacher's knowledge which is suitable for compact student. The experimental results demonstrate the effectiveness of our proposed method in enhancing the generalization of compact student on target domain. There are also some limitations for our proposed framework. On the one hand, we still need to pre-train cumbersome teacher with advanced UDA methods, which involves more training time than others. On the other hand, only utilize the distance between teacher's and student's logits to assess sample's 's transfcrability, which might overlook some intrinsic information from feature space. In the future, will extend our work to (1) jointly train teacher and student for cross-domain knowledge distillation, and (2) consider feature representations into sample transfcrability assessment.
10

---

Acknowledgments
This work is supported by the Agency for Science, Technology and Research (A*STAR) Singapore under its NRF AME Young Individual Research Grant (Grant No. A2084c0167) and the National Research Foundation, Singapore under its Al Singapore Programme (AISG2-RP-2221-007))
References
[1] Hangwei Qian, Sinno Jialin Pan, Bingshui Da, and Chunyan Miao. A novel distribution- embedded neural network for sensor-based activity recognition. In IJCAI, volume 2019, pages 2019.
[2] Bin Zhou, Shenghua Liu, Bryan Hooi, Xueqi Cheng, and Jing Ye. Beatgan: Anomalous rhythm detection using adversarially generated time series. In ICCA,, volume 2019, pages 4433 4439, 2019.
[3] Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang. Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8980-8987, 2022.
[4] Qing Xu, Min Wu, Xiaoli Li, Kezhi Mao, and Zhenghua Chen. Distilling universal and joint knowledge for cross-domain model compression on time series data. In Edith Elkind, editor, Proceedings of the Thirty-Sccone International Joint Conference on Artificial Intelli- gence, IJCAI-23, pages 4460-4468. International Joint Conferences on Artificial Intelligence Organization, 2023. Main Track.
[5] Mohammad Mahfujur Rahman, Clinton Fookes, Mahsa Baktashmotlagh, and Sridha Sridharan. On minimum discrepancy estimation for deep domain adaptation. In Domain Adaptation for Visual Understanding, pages 81-94. Springer, 2020.
[6] Chao Chen, Zhihang Fu, Zhihong Chen, Sheng Jin, Zhaowei Cheng, Xinyu Jin, and Xian- Sheng Hua. Homm: Higher-order moment matching for unsupervised domain adaptation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 1422-9429, 2020.
[7] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. aonaiiiadversariall training of neural networks. The journal of machine learning research, (7(1):20962288) 2016.
[8] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. Conditional adversarial domain adaptation. Advances in neural information processing systems, 31, 2018.
(9] Junguang Jiang, Ximei Wang, Mingsheng Long, and Jianmin Wang. Resource efficient domain adaptation. In Proceedings of the 28th ACM International Conference Multimedia, pages 2220-2228, 2020.
[10] Wei Li, Lingqiao Li, and Huihua Yang. Progressive cross-domain knowledge distillation for efficient unsupervised domain adaptive object detection. Engineering Applications of Artificial Intelligence, 109774,774, 2023.
[11] Atif Belal, Madhu Kiran, Jose Dolz, Louis-Antoine Blain-Morin, Eric Granger, et al. Knowledge distillation methods for efficient unsupervised adaptation across multiple domains. Image and Vision Computing, 108: 104096, 2021.
[12] Divya Kothandaraman, Athira Nambiar, and Anurag Mittal. Domain adaptive knowledge distillation for driving scene semantic segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 134-143, 2021.
[13] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. NIPS Deep Learning and Representation Learning Workshop, 2015.
[14] Jianfei Yang, Han Zou, Shuxin Cao, Zhenghua Chen, and Lihua Xie. Mobileda: Toward edge-domain adaptation. IEEE Internet of Things Journal, 7(8):6909-611 2020.

---

[15] Minho Ryu, Geonseok Lee, and Kichun Lee. Knowledge distillation for bert unsupervised domain adaptation. Knowledge and Information Systems, 64(11):3113-3128, 2022.
[16] Sebastian Ruder, Parsa Ghaffari, and John G Breslin. Knowledge adaptation: Teaching to adapt. 2017.
arXiv preprint 1.Xiv:770..2200
[17] Aron Culotta and Andrew McCallum. Reducing labeling effort for structured prediction tasks. In AAAI, volume 5. pages 746-751, 2005.
[18] Jingbo Zhu, Huizhen Wang, Benjamin K Tsou, and Matthew Ma. Active learning with sampling by uncertainty and density for data annotations. IEEE Transuctions on audio, speech, and language processing, 18(6):1323-1331, 2009.
[19] Colin Campbell, Nello Cristianini, Alex Smola, et al. Query learning with large margin classifiers. In ICML, volume 20, page 0, 2000.
[20] Alaa Tharwat and Wolfram Schenck. A survey on active learning: State-of-the-art, practical challenges and research directions. Mathematics, 1 (4)):820, 2023.
[21] Michael C Burl and Esther Wang. Active learning for directed exploration of complex systems. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 89-96, 2009.
[22] Seokhwan Kim, Yu Song, Kyungduk Kim, Jeong-Wor Won Cha, and Gary Geunbae Lee. Mmr-based active machine learning for bio named entity recognition. In Proceedings of the human language technology conference of the NAACL, Companion Volume: Short Papers, pages 69-72, 2006.
[23] Zhihong Chen, Chao Chen, Zhaowei Cheng, Boyuan Jiang, Ke Fang. and Xinyu Jin. Selective transfer with reinforced transfer network for partial domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12706-127 14, 2020.
[24] Keyu Wu, Min Wu, Zhenghua Chen, Ruibing Jin, Wei Cui, Zhiguang Cao, and Xiaoli Li. Reinforced adaptation network for partial domain adaptation. IEEE Transactions on Circuits and Systems for Video Technology, 2022.
[25] Keyu Wu, Min Wu, Jianfei Yang. Zhenghua Chen, Zhengguo Li, and Xiaoli Li. Deep reinforce- ment learning boosted partial domain adaptation. In Zui-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 3192-3199. International Joint Conferences on Artificial Intelligence Organization, 8 2021.
[26] Andreas Damianou and Neil D Lawrence. Deep gaussian processes. In Artificial intelligence and statistics, pages 207-215. PMLR, 2013.
[27] Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, et al. Noisy networks for exploration. arXiv preprint arXiv:/706. 10295, 2017.
[28] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7167-7176, 2017.
[29] Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra Perez, and Jorge Luis Reyes Ortiz. A public domain dataset for human activity recognition using smartphones. In Proceedings of the 21th international European symposium artificial neural networks, computational intelligence and machine learning, pages 437-442, 2013.
[30] Garrett Wilson, Janardhan Rao Doppa, and Diane J Cook. Multi-source deep domain adaptation with weak supervision for time-series sensor data. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1768-1778, 2020.
[31] Allan Stisen, Henrik Blunck, Sourav Bhattacharya, Thor Siiger Prentow, Mikkel Baun Kjær- gaard, Anind Dey, Tobias Sonne, and Mads Møller Jensen. Smart devices are different: Assessing and mitigatingmobile sensing heterogeneities for activity recognition. In Proceedings of the 13th ACM conference on embedded networked sensor systems, pages 127-140, 2015.
12

---

[32] Qiao Liu and Hui Xue. Adversarial spectral kernel matching for unsupervised time series domain adaptation. IJCAL, pages 2744-2750, 2021.
[33] Christian Lessmeier, James Kuria Kimotho, Detmar Zimmer, and Walter Sextro. Condition monitoring of bearing damage in electromechanical drive systems by using motor current signals electric motors: benchmark data set for data-driven classification. PHM Society European Conference, volume 3, 2016.
[34] Mohamed Ragab, Emadeldeen Eldele, Wee Ling Tan, Chuan-Sheng Foo, Zhenghua Chen, Min Wu, Chee-Keong Kwoh, and Xiaoli Li. Adatime: A benchmarking suite for domain adaptation on time series data. ACM Transactions on Knowledge Discovery from Data, 17(8):1-18, 2023.
[35] Ary Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung- Kang Peng, and H Eugene Stanley. Physiobank, physiotoolkit, and physionet: components of new research resource for complex physiologic signals. circulation, 101:e215-e220 2000.
[36] Ruichu Cai, Jiawei Chen, Zijian Li, Wei Chen, Keli Zhang, Junjian Ye, Zhuozhang Li, Xiaoyan Yang, and Zhenjie Zhang. Time series domain adaptation via sparse associative structure alignment. In Proceedings the AAAI Conference on Artificial Intelligence, volume 35, pages 6859 6867, 2021.
[37] Markus Thill, Wolfgang Konen, and Thomas Bäck. Time series encodings with temporal convo- lutional networks. In International Conference Bioinspired Methods and Their Applications, pages 161-173. Springer, 2020.
[38] Kaiming He, Xiangyu Zhang, Shaoging Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
13

---

A Supplementary
A.1 Uncertainty Estimation with Monte Carlo Dropout Method
Given an input data set X {I11,.., Fn and the respective outputs Y = {y1..... yn the conven- tional machine learning methods intend find an optimal model (x;; 0), which is parameterized with 0 to map the input X to the Y. After training, the optimal model (a;; 0) will give a single point prediction for certain test sample with static 0. On the contrary, the Bayesian methods (e.g., Bayesian neural networks) can generate predictive distributions instead of a single point prediction for estimating model uncertainty. With defining (( 0) with prior P(0) over parameter space 0, the training objective is then turned to find an optimal posterior distribution over 0:
P(O = P(YXX, P))())
(11)
X)
The prediction value of with input x is the weighted average of model predictions over all possible sets parameters with various posterior probabilities as Eq. 12) shows.
Plylx,X,Y) = / P(ylæ, ()
(12)
P (1;;0))
However, the posterior distribution P(O X, is intractable as shown in previous works. Alter- natively, Gal and Ghahramani proved that a DNN with arbitrary non-linear depth and dropout is mathematically equivalent to Bayesian approximation of the probabilistic deep Gaussian process. They proposed method named Monte Carlo Dropout which utilizes dropout distribution P(0) to approximate P(0) X, Y). To be specific, for the I-th layer (1 1,... I.) in a model with total layers, the parameter distribution 0 defined as:
0: M, ((1,)
(13)
where M, E Di is matrix with variational parameters and diag(-) ) is an operator to map vector to a diagonal matrix. Zr,5 Bernoulli(g) is independently sampled from Bernoulli distribution, where 1, ..., Di-1. qi is the probability of dropout. Subsequently, the Eq. (12) is reformulated as:
N
E E (x;;
Σ n=1
(x; 0*).
(14)
N
Practically, Eq. (14) means to enable the dropout of model during test phase and forward N times for each sample z;. Furthermore, for classification tasks employ the entropy to measure teacher's uncertainty on target data as Eq. (5) shows:
H. --=- E( Σ ply = cla, Logt
1 Σ ply = c,,,, (*))), N
(15)
where ply = cl 0²) represents the probability of sample belong to class e and it is the softmax outputs of teacher model ((x; 0*) on the n-th forward pass. Intuitively, higher value of the predictive entropy Hi will be obtained when all classes are predicted to have equal probabilities, which means the teacher is less confident about the specific data.
A.2
Model Architecture
A.2.1 Teacher and Student model
As illustrated in Fig. a) and (b), the teacher and student model are constructed with three stacked CNN blocks as the backbone and one fully connected layer the classifier. Each CNN block consists
14

---

Conr 64)
ConvIDXlrpur. 16)
BN+ReLU
BN+ReLU
Mocppolld
Ma
Dropout
Drppout
Con4 1289
166/11D(16
BN+RaLU
BN+RaLU
F0/128, 64)
Merepool ld
Meeppoo 1d
RelUU
Dropout
Dropout
FO164,
ReLU
Con-1D(1228 123)
Conv CDn21D(33,, 32)
RO(64,
BN++Re
BN+ReLU
Manpoolld
Mappolld
Dropout
Dropout
AcaptiveAvarage Pooling
AdaptivAAeengg
Pooling
FC(128C)
FC(32)
(a)
(b)
(c)
Figure 4: Network Architectures for (a) ID-CNN Teacher, (b) ID-CNN Student and (c) Domain Discriminator.
of a ID convolutional layer, followed by BatchNorm layer, an activation layer (ReLU), ID MaxPooling layer and a Dropout layer. Here, 'ConvID' represents the ID convolutional layer and the first variable in the bracket represents the number of input channels and the second one represents the number of output channels. 'BN' is BatchNorm layer. 'FC' represents fully connected layer. 'C' represents the number of classes.
Meanwhile, to demonstrate the deployment on edge device we compared our ID-CNN teacher and student from four perspectives shown in Table Here, employ Raspberry Pi 3B+ the edge device for deployment. We can see that the student reduces 15.46 times parameters, 16.98 times FLOPs and 13.54 times memory usage compared its teacher. Besides, the inference of student is 21.67 times faster than teacher on the edge device. Meanwhile, in our manuscript we have already shown that the student trained with our method able achieve comparable performance as the teacher. This enables our compact student to potentially meet the real-time response and on-site deployment requirements for certain time series applications.
Table 8: Comparison of model complexity.
# Para. (M)
# FLOPs (M)
Memory Usage (Mb)
Inference Time (Sec)
T
0.201
0.917
83.73
4.16
S
0.013
0.054
6.33
0.192
Rate
15.46x
16.98x
13.54x
x1.67x
A.2.2 Domain discriminator
Fig. 4[c) is the network architecture of domain discriminator. It consists of three linear layer followed by ReLU activation layers. The output of domain discriminator is 2-classes probability distribution to indicate whether the feature maps come from the teacher with source domain data input or from the student with target domain data as input.
A.2.3 Dueling DDQN
Table 9
presents the details of our dueling DDQN. The left column is the state-value estimation stream and the right column the advantage estimation column. The 'NoisyFC" represents linear layer whose weights and biases are perturbed by a parametric function of the noise. The conventional
15

---

Table 9: Network Architecture for Dueling DDQN.
Layers
Dueling DDQN
#1
FC(Inpt,, 1024)+ReLU
#2
NoisyFC(1102.10024)RRRR NoisYFC(U024,004444RLL
#3
N014)FC(1104.11) +ReLU
NoisyFC(1024,, 2)+ReLU
#4
Aggregation
linear layer can be expressed as y wx where w E RR*P, b are trainable weights and biases, Iº and y E R2 the imputs and outputs, respectively. In the NoisyNets, the weights and biases are reformulated as Eq. (66) and 17 respectively. Here, μr E x o" Rixp, 4 R2, o" R are the trainable weights and biases. @ is the element wise multiplication. e" and e are the factorised Gaussian noise, where each entry ci2 f())f((), = f(c,) and f(x) sgn(x) /lal Adding such parametric noise to the deep reinforcement learning agent will enhance the efficiency of exploration.
1 + 06
(16)
== n¹ + + O
(17)
A.3 Additional Transfer Scenarios
We evaluate our proposed method another five additional transfer scenarios four datasets as shown in Table [0 and Table TT] From above two Tables, we can sse that our proposed method consistently achieves better performance than other SOTA methods, further indicating its effectiveness in transferring the knowledge under the cross-domain scenarios.
Table 10: Marco Fi-scores HAR and HHAR three independent runs.
HAR Transfer Scenarios
HHAR Transfer Scenarios
Methods
Avg
6-+1
7-44
8-33
Avg
Teacher
98.23
90.57
97.08
100)
92.21
95.62
66.56
33.25
94.47
94.99
96.68
77.19
Student-Only
98.37
48.78
77.38
61.17
76.41
72.42
61.94
27.43
69.10
77.72
80.51
63.34
KD-STDA
96.30
74.05
KA-MCD
85.22
78.03
86.14
91.19
74.28
82.97
43.90
33.35
92.32
94.27
97.02
72.17
MLD-DA
98.82
80.57
91.90
100
91.69
92.60
65.44
31.10
92.97
94.97
95.87
76.07
REDA
98.20
95.05
91.26
98.53
72.04
91.02
54,18
32.56
88.50
88 84
96.18
72.05
AAD
90.27
66.88
86.09
94.73
84.82
84,56
58.23
37.24
91.47
81.99
83.61
70.51
MobileDA
92.86
84.96
79.12
77.56
84,99
30.83
89.70
79.25
65,23
UNI-KD
100
94.42
100
92.26
87.10
94.76
62.33
39.01
92.89
96.90
96.52
77.53
Proposed
100
85.26
97.92
100
92.21
95.08
67.27
38.25
94.59
95.83
97.40
78.67
Table 11: Marco Fl-scores FD and SSC across three independent
FD Transter Scenarios
SSC Transfer Scenarios
Methods
1-30
3-10
3-11
Avg
3-
5
6-12
13-117
18-512
Avg
Teacher
66.40
100
62.30
100
81.65
82.07
71.85
73.69
72.21
50.74
52.96
64.29
Student-Only
45.13
91.14
44.84
93.03
70.55
68.94
43.65
41.04
48.21
38.78
47.28
43.79
KD-STDA
47.55
93.02
51.26
99.81
76.28
73.58
61.24
66.97
67.05
43.05
49.92
$7.65
KA-MCD
51.77
98.69
48.50
93.65
83.05
75.13
61.13
63.23
70.96
39.56
46.86
56.35
MLD-DA
51.98
99.67
52.14
96.01
75.62
75.08
66.23
70.30
69.33
44.22
44.13
57.65
REDA
53.65
96.21
52.48
96.60
80.85
75.96
56.09
61.96
53.59
40.50
36.26
49.68
AAD
46.42
94.65
$2.09
98.65
87.11
75.78
71.78
58.61
MobileDA
51.71
94.92
51.17
99.86
78.51
75.23
64.16
67.67
56.74
47.50
56.56
58.53
UNI-KD
60.91
99.97
61.00
100
87.08
81.79
66.84
70.76
65.70
50.19
49.77
60.65
Proposed
61.99
99.67
62.42
100
87.76
82.37
72.73
67.01
52.52
62.21
A.4
Sensitivity Analysis
A.4.1
Hyper parameter N
In our proposed method, one of the key hyper parameters is N, which is the number of teachers utilized for calculating the uncertainty specific sample. It relates to reward function, thus
16

---

-000
14:20
Figure 5: Sensitivity of N.
Figure 6: Sensitivity of K.
Figure 7: Sensitivity of A.
Table 12: Sensitivity Analysis 01 and
01=0.2
01=0.4
a0=6.6
01=0.8
01=1.0
01=1.2
01=1.4
01=1.6
Dataset
0g=11.8
02=1.6
0x=1.4
0g=1.2
02=1.0
02=0.8
0g=0.66
02=0.4
ag=0.2
HAR
94.68
94.23
94.49
93.58
93.25
92.65
92.68
92.72
92.05
HHAR
82.37
82.35
94.49
82.10
81.59
82.11
81.04
81.25
81.34
FD
92.63
92.34
92.87
92.10
91.89
91.74
92.01
91.34
92.05
SSC
68.26
68.47
68.15
67.65
67.80
66.65
66.95
66.35
66.01
affecting the learning process of target sample selection policy. Intuitively, the larger N will lead to more accurate estimation teacher's uncertainty and provide more accurate reward. As illustrated in Fig. 5 student's performance is gradually increased with N but will keep at some certain level when N 10. The larger value N also increases the total cost in terms of training time. Empirically,
N or 10 are appropriate, and we choose N 10 in our experiments for all the datasets.
A.5 Hyper parameter K
Another hyper parameter in our proposed approach is the episodes length for generating historical experience and we perform the analysis as illustrated in Fig. 6 From Fig. 6 we can see that our proposed method is not very sensitive to K. But a large value K will result in longer training time. K 5 is sufficient to generate informative historical experience for training the dueling DDQN.
A.5.1 Hyper parameter
Regarding hyper parameter which is to balance the contribution domain confusion loss LDC and the distillation loss LRKD, the student more focus on learning domain-ing iamaiin-nvariin knowledge from LDC- will result worse performance in datasets like HHAR and SSC. A higher value of À will obviously enhance student's generalization capability on target domain. À E [1, 5] is a suitable range based on our experiment results.
from Fig. 7 that small value of X (e.g. X = 0.1) will make
A.5.2 Hyper parameter aj and 0:2
To limit our reward within the range -I to I, ai and a2 should satisfy below constrains: 0i E (0, 2). 02 E (0, 2) and a1 + Q2 = 2. We perform grid search on four datasets as shown in Table 12 We can see that the student trained with low a₁ value and high Q2 value can achieve better performance than other configurations, indicating transferability contributes more to the final performance than uncertainty. In all of our experiments, we set 0.2 and ag = 1.8 for all datasets for simplicity.
A.5.3 Hyper parameter
For hyper parameter which the temperature to soften teacher's logits, we perform the analysis ranged from to 16 as shown in Table T3 We can see that higher value of temperature (e.g., 16) would over-smooth teacher's logits, resulting in poor distillation performance. Generally, 7 2 or 4 is good choice for our method.
17

---

Table 13: Sensitivity Analysis for temperature
Dataset
T=1
T= 2
T
16
HAR
92.14
94.68
94.23
91.35
89.45
HHAR
80.14
82.37
81.45
79.41
76.49
FD 90.79
92.63
92.74
88.51
85,41
SSC
65.10
67.49
66.98
63.21
59.01
Table 14: Comparison with different sample selection strategies in AL.
Methods
HAR
HHAR
FD
SSC
Baseline
89.32
78.99
89.13
60.65
LC
79.21
76.22
74.14
52.9
LC Consist.
82.01
75.43
74.45
56.11
LC Consist. + RL
84.9
76.24
81.45
60.01
M
80.55
75.9
82.05
58.03
M Consist.
83.55
78.91
81.9
59.45
M Consist. + RL
90.11
80.01
80.79
61.97
88.31
79.09
88.18
59.23
H Consist.
91.65
78.3
90.17
63.16
Consist, + RL
93.91
81.73
91.93
62.98
A.6
Comparison with sample selection strategies in Active Learning
We conduct the ablation study on three uncertainty based active learning (AL) strategies, including least confidence (LC), sample margin (M) and sample entropy (H). The results are presented Table We take student trained with our framework using whole target samples as the baseline (i.e., without RI.). 'I.C" refers to leveraging student's confidence to directly select samples. 'L.C Consist," refers to using the consistency of teacher's is and student's confidence for explicitly sample selection. "LC Consist. + RL' refers to leveraging 'LC Consist." as reward to learn optimal sample selection AL strategies exhibit performance
14
policy. We can see that: firstly, almost all uncertainty-baase degradation compared to the baseline. This could attributed to the unreliable uncertainty estimation from student's outputs, especially early training stage. Additionally, among these strategies, entropy performs the best, likely because considers the overall probability distribution which might partially address student's unreliable predictions issue. Secondly, utilizing uncertainty consistency instead of uncertainty alone could enhance performance most settings, as incorporating teacher's know knowledge through consistency provides more reliable measure. Lastly, RI. module could further enhance student's performance via employing any of uncertainty consistency as the reward, indicating its effectiveness.
A.7 Scalability to Larger Datasets
To verify the efficiency and scalability of our method on larger time series (TS) dataset, we conduct experiments on another Human Activity Recognition dataset named PAMAP2. Table compares the dataset complexity of PAMAP2 with the datasets employed in our manuscript. Note that it is meaningless compare total size of datasets our settings as our experiment evaluate transfer scenario between single subject. We summarize the averaged number of samples, channels, data length and classes across transfer scenarios for these datascts. We also report the time complexity of our method across these datasets (i.e., training time per epoch for single transfer scenario). From Table 15] we can see PAMAP2 is larger in terms of number samples and more complex in terms of number of channels and classes. Compared with FD, the epoch training time for PAMAP2 only increases times as the number of training samples increases about 4 times, indicating that our method scales well in terms of computational efficiency larger TS dataset.
Meanwhile, we also conduct the performance comparison between our method and benchmarks on PAMAP2 with randomly selected transfer scenarios. The experimental results are summarized as Table 16 We can see that our proposed method consistently outperform other benchmarks in terms of average Macro Fl Fl-score, even though it cannot achieve the best performance on some transfer
18

---

Table 15: Summary of Datasets.
Datasets
No. of Samples
No. of Channels
Data Length
No. of Classes
Training Time (Sec)
HAR
216
9
128
6
1.61
HHAR
1150
3
128
6
9.07
FD
1828
1
5120
3
16.43
SSC
1428
3000
5
7.09
PAMAP2
8180
36
256
11
31.64
Table 16: Performance Comparison on PAMAP2.
Methods
102->104
106-+103
107->105
105->106
107->102
Avg.
KD-STDA
66.19
53.12
46.34
67.87
59.75
58.65
KA-MCD
34.35
49.16
49.92
33.95
35.97
40.67
MLD-DA
68.14
50.85
61.23
75.03
63.32
63.71
REDA
71.49
53.31
59.11
74.75
64.86
64.70
AAD
60.28
51.61
48.01
73.64
45.55
55,82
MobileDA
67.14
54.09
64.21
74.67
63.35
64.69
UNI-KD
64.82
70,82
43.92
69.65
56.20
59.28
Proposed
68.33
68,86
59.56
75.44
62.50
66.96
scenarios. This observation indicates that the effectiveness of our proposed method also be generalized to larger time series dataset.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction include our main claims in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of our work in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification:
This paper does not include any theoretical assumptions and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
We have provided the details of model architecture, experimental configura- tions, the value of key parameters the paper. Meanwhile, we also provided the code the supplementary for result reproducibility.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
21

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The datasets in the paper are public available and the URL can be found in related references. A git repository URL for our proposed method will be provided upon acceptance.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
ssioooliccyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have specify the experimental settings in the paper
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Please refer the tables and figures in the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
22

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer the supplementary for computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have reviewed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no socictal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have clearly cite the original papers for the datasets used.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "inNQ3Kk8xD", "reviewer": "Reviewer_Ltvb", "summary": "This paper introduces a reinforcement learning-based active learning method designed to dynamically select target data for knowledge-transfer, whose goal is to bridge the network capacity gap between teacher and student networks within a domain adaptation framework incorporating knowledge distillation. Specifically, the authors propose a novel reward mechanism aimed at learning an optimal policy for selecting target data, considering the student network's capacity. \n\nAnother novel aspect of this paper is its successful demonstration of the framework's effectiveness on time-series data. This modality is less explored in both domain adaptation and knowledge distillation research fields.\n\n##################################### Post Rebuttal #####################################\n\nAll of my concerns have been properly addressed, and my questions have been answered during rebuttal. I am happy to raise my score to ***Strong Accept***.\n\n##################################### Post Rebuttal #####################################", "questions": "1. For Lines 90 to 91, the authors stated that the estimated entropy from the domain-shared feature extractor is deemed unreliable for the student. Could the authors provide further elaboration on this assertion? Specifically, what does this unreliability entail and what are its underlying causes?\n\n2. While DDQN isn't the method proposed in this paper, it should have significant impact on the overall performance. I am curious whether other active learning methods might outperform MCD. I recommend conducting an ablation study to examine how various active learning strategies influence overall performance.", "limitations": "N/A", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper is well-written and easy to follow.\n\n2. The proposed reward function for selecting target data in domain adaptation from a large network to a smaller one is novel. This method integrates principles from active learning, reinforcement learning, knowledge distillation, and domain adaptation.\n\n3. The authors offer a detailed description of their implementation in the paper, along with the accompanying code. This makes it straightforward for practitioners to apply their methods or build upon their work.\n\n4. Formulating network output entropy as a reward function for data selection represents a novel approach that effectively incorporates concepts from reinforcement learning into a classification model.\n\n5. The performance lift of the proposed method is very significant in time-series data, which promotes the interests of active-learning-based domain adaptation.", "weaknesses": "### Majors:\n\n1. Using model output entropy as a measure of uncertainty level is a common technique in active learning. While applying this to promote uncertainty consistency between teacher and student networks is novel, it is important to acknowledge the context of active learning within the paper. It would be not good to borrow ideas without attribution.\n\n2. Based on my experience, running such a reinforced loop for data selection is particularly time-consuming, especially concerning the Markov chain state update mentioned from Line 127 to 134. Therefore, it would be beneficial for the authors to conduct a time complexity analysis of their proposed method.\n\n### Minors:\n\n1. I believe active learning is highly related to the proposed work. Conducting a literature review on active learning would offer practitioners valuable context, enhancing their understanding of the proposed research.\n\n2. In Line 112, \"divergence\" would be a more appropriate term for describing the difference between distributions, rather than using \"distance\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to Reviewer Ltvb for all the comments. \n\n**Response to Major Weak-1 (Comparison with Active Learning):** We fully agree that our method is closely related to AL, particularly in selecting critical samples with uncertainty. We are also willing to include the acknowledge of AL in our updated version. Meanwhile, we'd like to clarify that there are also some distinguished differences between our approach and uncertainty-based AL sample selection methods. Firstly, as noted by the reviewer, our method introduces a novel concept by promoting uncertainty to uncertainty consistency. We use sample uncertainty consistency to assess the student's learning capability, rather than directly filtering samples based on their uncertainty. Additionally, this only forms part of our reward function, as we also incorporate sample transferability to evaluate student's network. Secondly, unlike AL, which explicitly leverages uncertainty for sample selection, our approach employs a RL-based module to heuristically learn the optimal target sample selection policy. And our experimental results on reinforced sample selection ablation (Table 6) have demonstrated its superior capability of performance enhancement compared to directly selecting samples as done in AL. Lastly, the selected target samples with our method will not be labeled as AL. Instead, they will continue to be evaluated without labelling after student's state is updated.\n\n**Response to Major Weak-2 (Computational Complexity):** We performed the time complexity analysis as suggested and the results are shown in below table. Specifically, we measure the training time for our proposed method and other benchmarks with a NVIDIA 2080Ti GPU. The reported results are measured with one epoch on single transfer scenario on **FD** dataset, which has the largest training samples (about 1,800 samples per transfer scenario) among evaluated datasets.\n\nMethods|Time(sec)|Macro F1-score\n:---:|:---:|:---:\nKD-STDA|1.68|55.81\nKA-MCD|4.55|57.74\nMLD-DA|1.91|60.90\nREDA|1.78|56.27\nAAD|0.91|59.12\nMobileDA|1.28|56.86\nUNI-KD|3.26|62.17\n**Ours**|16.42|67.49   \n\nWe can see that our method does require more training time compared to other benchmarks, reflecting its greater complexity. The primary computational costs arise from two factors. The first part is the generation of $K$ historical experiences at each step, which accounts for approximately 73\\% of total training time. This could be significantly reduced by using a smaller K. The second factor, which consumes about 21\\% of total training time, is the MCD module which conducts multiple inference processes for uncertainty estimation. This computational burden could be further decreased by adopting alternative uncertainty estimation methods. Nevertheless, although our training time is longer than other benchmarks, we argue that it is still within an acceptable range, especially considering the performance improvement it could bring.\n\n**Response to Minor Weak-1 (Literature Review):** As suggested, we performed a literature review on uncertainty-based active learning and will include it in our updated version (See Updates in global **Author Rebuttal**). \n\n**Response to Minor Weak-2 (Typo):**  We would like to revise it in the final revision.\n\n**Response to Q-1 (Unreliability):** As stated in UNI-KD, the authors employ a data-domain discriminator to estimate sample uncertainty, with inputs derived from feature extractor of compact student trained on both source and target domain (i.e., domain-shared). However, our experiments, which directly applied SOTA UDA methods to compact student (see Table 1 of our manuscript), demonstrated that the compact student cannot fully capture the fine-grained patterns in source and target domains as effectively as teacher. Due to its limited capacity, the student's adaptation performance is inferior, making the uncertainty estimated by UNI-KD unreliable. In contrast, our proposed method leverages more robust teacher to estimate uncertainty via MCD module. Our experimental results further demonstrate its effectiveness on enhancing student’s performance on target domain.\n\n**Response to Q-2 (Ablation with AL):** We conduct the ablation study on three uncertainty-based active learning (AL) strategies, including least confidence (LC), sample margin (M) and sample entropy (H). The results are presented in below table. We take student trained with our framework using whole target samples as the baseline (i.e., without RL).  'LC'  refers to leveraging student's confidence to directly select samples. 'LC Consist.' refers to using the consistency of teacher's and student's confidence for explicitly sample selection.  'LC Consist. +RL' refers to leveraging 'LC Consist.' as reward to learn optimal sample selection policy. \n\nMethods|HAR|HHAR|FD|SSC\n:---|:---:|:---:|:---:|:---:\nBaseline|89.32|78.99|89.13|60.65\nLC|79.21|76.22|74.14|52.9\nLC Consist.|82.01|75.43|74.45|56.11\nLC Consist.+RL|84.9|76.24|81.45|60.01\nM|80.55|75.9|82.05|58.03\nM Consist.|83.55|78.91|81.9|59.45\nM Consist.+RL|90.11|80.01|80.79|61.97\nH|88.31|79.09|88.18|59.23|\nH Consist.|91.65|78.3|90.17|63.16|\nH Consist.+RL|93.91|81.73|91.93|62.98|\n\nWe can see that: firstly, almost all uncertainty-based AL strategies exhibit performance degradation compared to the baseline. This could be attributed to the unreliable uncertainty estimation from student's outputs, especially at early training stage. Additionally, among these strategies, entropy performs the best, likely because it considers the overall probability distribution which might partially address student's unreliable predictions issue. Secondly, utilizing uncertainty consistency instead of uncertainty alone could enhance performance in most settings, as incorporating teacher's knowledge through consistency provides a more reliable measure. Lastly, our RL module could further enhance student's performance via employing any of uncertainty consistency as the reward, indicating its effectiveness.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aussRgqmpO", "reviewer": "Reviewer_E4vw", "summary": "This paper proposes a Reinforced Cross-Domain Knowledge Distillation (RCD-KD) framework for time series data, aiming to effectively transfer knowledge from a cumbersome teacher model to a compact student model across different domains. The RCD-KD framework leverages an adversarial domain discriminator to learn domain-invariant features and a reinforcement learning-based sample selection module to dynamically select informative target samples for knowledge distillation. The proposed method significantly improves the student model's generalization ability on the target domain compared to conventional knowledge distillation and domain adaptation techniques.", "questions": "See the weaknesses.", "limitations": "See the weaknesses.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. By incorporating an adversarial domain discriminator, the framework can effectively learn domain-invariant features, enabling the student model to generalize better to the target domain.\n2. The reinforcement learning module dynamically selects informative target samples based on the student model's capacity and uncertainty, mitigating negative transfer and improving the efficiency of knowledge distillation.\n3. Extensive experiments on four public datasets across three tasks demonstrate that the proposed RCD-KD framework consistently outperforms other state-of-the-art domain adaptation and knowledge distillation methods.", "weaknesses": "1. The framework utilizes the distance between teacher and student logits to assess sample transferability, potentially overlooking intrinsic information from the feature space. More comprehensive discussions on different feature distances could further enhance sample selection.\n2. While the experiments demonstrate the effectiveness of the proposed RCD-KD framework on several datasets, it is unclear how well the method would scale to larger and more complex time series datasets with higher dimensional feature spaces. The computational efficiency and scalability of the framework under such settings remain to be investigated.\n3. The paper acknowledges that grid search was used to tune hyperparameters such as α1, α2, λ, and τ. However, the sensitivity of the framework's performance to these hyperparameters is not thoroughly analyzed. The framework's robustness to different hyperparameter configurations could potentially limit its practical applicability without extensive tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to Reviewer E4vw for all the comments. \n\n**Response to Weak-1 (Feature Knowledge for Transferability Assessment):** Although we have discussed above weakness as one of our limitations in manuscript, we conducted some preliminary experiments as suggested. We investigate several feature-based KD methods to assess sample transferability on **SSC** as below table shows. Specifically, we utilized L2 [7], attention maps (AT) [8], CORAL [9], and probabilistic knowledge transfer (PKT) [10] to measure the distance between teacher's and student's feature maps. These feature distances are then used to calculate reward  $\\mathcal{R}_3$ in Eq.(3). From the table, we can see that transferability assessments using feature distance obviously underperform logits-based method in our proposed framework. The potential reason is that: in our current framework, we utilize MCD module to generate $N$ teachers and then average their logits for calculating uncertainty and transferability. However, unlike logits, each data point in the feature maps carries different meaning depending on its spatial location within its feature space. Simply averaging feature maps across multiple teachers appears unreasonable and impractical, potentially resulting in poor performance. Therefore, the feature-based knowledge cannot be directly integrated into our existing framework. A more comprehensive designs need to be carefully considered.\n\nMethods|0$\\to$1|12$\\to$5|16$\\to$1|7$\\to$18|9$\\to$14|Avg\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:\nL2|22.73|24.75|47.00|64.23|62.59|44.26\nAT|29.93|38.25|59.56|59.02|56.52|48.66\nCORAL|34.06|38.87|50.22|54.32|52.94|46.08\nPKT|28.20|57.70|47.74|55.70|74.84|52.84\nLogits (**Ours**)|48.39|69.56|70.52|72.20|76.79| 67.49 \n\n**Response to Weak-2 (Scalability to Larger Dataset):** To verify the efficiency and scalability of our method on larger time series (TS) dataset, we conduct experiments on another Human Activity Recognition dataset named PAMAP2 [11]. Below table compares the dataset complexity of PAMAP2 with the  datasets we employed in our manuscript. Note that it is meaningless to compare the total size of datasets in our settings as our experiment evaluate transfer scenario between single subject. We summarize the averaged number of samples, channels, data length and classes across all transfer scenarios for these datasets. We also report the time complexity of our method across these datasets (i.e., training time  per epoch for single transfer scenario). From this table, we can see PAMAP2 is larger in terms of number of samples and more complex in terms of number of channels and classes. Compared with **FD**, the epoch training time for PAMAP2 only increases 2 times as the number of training samples increases about 4 times, indicating that our method scales well in terms of computational efficiency on larger TS dataset. \n\nDatasets|No.of Samples|No.of Channels|Data Length|No.of Classes|Training Time (sec)\n:---:|:---:|:---:|:---:|:---:|:---:\nHAR|216|9|128|6|1.61\nHHAR|1150|3|128|6|9.07\nFD|1828|1|5120|3|16.43\nSSC|1428|1|3000|5|7.09\nPAMAP2|8180|36|256|11|31.64\n\nMeanwhile, we also conduct the performance comparison between our method and benchmarks on PAMAP2 with randomly selected 5 transfer scenarios. The experimental results are summarized as below table. We can see that our proposed method consistently outperform other benchmarks in terms of average Macro F1-score, even though it cannot achieve the best performance on some transfer scenarios. This observation indicates that the effectiveness of our proposed method can also be generalized to larger time series dataset.  \n\nMethods|102$\\to$104|106$\\to$103|107$\\to$105|105$\\to$106|107$\\to$102|Avg\n:---:|:---:|:---:|:---:|:---:|:---:|:---:|\nKD-STDA|66.19|53.12|46.34|67.87|59.75|58.65\nKA-MCD|34.35|49.16|49.92|33.95|35.97|40.67\nMLD-DA|68.14|50.85|61.23|75.03|63.32|63.71\nREDA|**71.49**|53.31|59.11|74.75|**64.86**|64.70\nAAD|60.28|51.61|48.01|73.64|45.55|55.82\nMobileDA|67.14|54.09|**64.21**|74.67|63.35|64.69\nUNI-KD|64.82|**70.82**|43.92|69.65|56.20|59.28\n**RCD-KD**|68.33|68.86|59.65|**75.44**|62.50|**66.96**\n\n**Response to Weak-3 (Hyperparameter Sensitivity):** Due to paper space constraints, our sensitivity analysis for hyperparameters $\\alpha_1$, $\\alpha_2$, $\\lambda$, $N$ and $K$ were included in the Supplementary (see Fig. 2,3,4 and Table 5). Please refer to our submitted Supplementary for the details. \n\nFor hyperparameter $\\tau$ which is the temperature to soften teacher's logits, we added additional analysis ranged from 1 to 16 as shown in below Table. We can see that higher value of temperature (e.g., $\\tau = 16$) would over smooth teacher's logits, resulting in poor distillation performance. Generally, $\\tau = 2$ or  $\\tau = 4$ is a good choice for our method.  \n\nDataset|$\\tau=1$|$\\tau=2$|$\\tau=4$|$\\tau=8$|$\\tau=16$\n:---:|:---:|:---:|:---:|:---:|:---:\nHAR|92.14|94.68|94.23|91.35|89.45\nHHAR|80.14|82.37|81.45|79.41|76.49\nFD|90.79|92.63|92.74|88.51|85.41\nSSC|65.10|67.49|66.98|63.21|59.01", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HWw5p6W8QZ", "reviewer": "Reviewer_bzJ9", "summary": "This paper proposes a knowledge distillation method for unsupervised domain adaptation models in time series classification. After pre-training the teacher model with existing domain adaptation methods, the proposed Reinforced Cross-Domain Knowledge Distillation (RCD-KD) method selects suitable target domain samples for knowledge distillation with reinforcement learning and distills knowledge from the pre-trained teacher model to a smaller student model. Empirical experimental results on four public time series datasets demonstrate the effectiveness of the proposed method over other state-of-the-art benchmarks.", "questions": "In addition to the Weaknesses, there are some minor points of suggestion:\n\n* Figure 1 is a little hard to understand, especially the relations between the reward module and other parts.\n\n* There are some typos, such as Line 118 ‘we consider (the) target sample selection task as a Markov Decision Process which can (be) addressed by reinforcement learning.’", "limitations": "Authors discussed the limitations in the conclusion part.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The writing and presentation of this paper are good. The setup of the proposed problem and the proposed method are described clearly. \n\n* This paper proposes a new distillation method. It makes some contributions in using reinforcement learning for target sample selection in knowledge distillation.\n\n* Experiments in four public datasets show the effectiveness of the proposed method, which outperforms some domain adaptation and knowledge distillation methods. There are also some ablation studies to validate the designs of the reward and the training losses.", "weaknesses": "* The proposed problem seems to be a simple two-stage combination of domain adaptation and knowledge distillation and may not be realistic enough, which does not show the significance of solving them together in one problem. Besides, authors claim that it can help ‘on edge devices with very limited computational resources’, but the experiments only distill from one bigger CNN to a smaller CNN, which does not make a difference in enabling deployment on edge devices. \n\n* Authors should explain more on why using reinforcement learning to select samples works better than directly using designed metrics such as uncertainty and transferability. Reinforcement learning will add a lot of computation costs to the training process and it is unclear what the learned selection policy looks like. It seems that the proposed reward cannot solve the claimed issue that ‘in the cross-domain scenario, teacher’s knowledge on each individual target sample may not be always reliable’. Besides, this paper proposes a distillation method for time series data, but the method does not show its special designs for time series. \n\n* The domain discrimination loss is confusing. How does it enable ‘transfer the domain-invariant knowledge’ to the student model? If the teacher model already learns domain-invariant knowledge, why don’t we achieve this by distilling teacher features to the student model?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to Reviewer bzJ9 for all the comments. \n\n**Response to Weak-1-1 (Simple Combination):** We clarify that our solution is not a simple two-stage combination of DA and KD. Particularly, we optimize DA loss $L_{DC}$ and KD loss $L_{RKD}$ together, addressing domain shift and model complexity simultaneously. Moreover, we consider the following two solutions as simple two-stage combination. 1. KD$\\to$DA:  first distill the knowledge from teacher to student on source data, and then directly perform DA on student. 2. DA$\\to$KD: first perform DA on the teacher and then conduct distillation. As shown in below table, these two solutions perform better than Source-Only model, but significantly worse than ours. The reason is that if performing KD on source data first, followed by DA, the student would become biased towards the source data during KD stage. And due to its compact network architecture, the student's adaptation performance would also be affected. If we perform DA first and then perform KD on target data, it would result in poor distillation efficiency in KD stage due to the lack of ground truth on target data. These issues  are consistent with findings from previous research [12][13].\n\nMethods|HAR|HHAR|FD|SSC\n:---:|:---:|:---:|:---:|:---:\nStudent|55.94|58.74|66.78|50.39\nKD$\\to$DA|73.66|72.67|67.87|54.26\nDA$\\to$KD|74.29|74.54|70.40|60.08\nOurs|94.68|82.37|92.63|67.49\n\n**Response to Weakness-1-2 (Model Complexity):** To demonstrate the deployment on edge device, we compared our 1D-CNN teacher and student from four perspectives as shown in below table. Here, we employ Raspberry Pi 3B+ as the edge device for deployment. We can see that the student reduces 15.46 times parameters, 16.98 times FLOPs and 13.54 times memory usage compared to its teacher. Besides, the inference of student is 21.67 times faster than teacher on the edge device. Meanwhile, in our manuscript we have already shown that the student trained with our method is able to achieve comparable performance as the teacher. This enables our compact student to potentially meet the real-time response and on-site deployment requirements for certain time series applications. \n\n||# Para.(M)|# FLOPs(M)|Memory Usage (Mb)|Inference Time (Sec)\n:---:|:---:|:---:|:---:|:---:| \nT|0.201|0.917|83.73|4.16\nS|0.013|0.054|6.33|0.192\nRate|15.46$\\times$|16.98$\\times$|13.54$\\times$|21.67$\\times$\n\n**Response to Weak-2-1 (Why RL works):** We argue that the primary reason why RL performs better is due to its inherent balance between exploitation and exploration. If we solely exploit the gained knowledge (i.e., directly using uncertainty or transferability for sample selection), the student is likely to become stuck at certain sub-optimal points (see comparison of $R_2$, $R_2^\\dagger$,  $R_3$ and $R_3^\\dagger$ in Table 6 of our manuscript). The exploration in RL allows the agent to explore new possibility from the environment. In our implementation, we utilize the ‘NoisyFC’ whose weights and biases are perturbed by a parametric function of the noise to enhance the efficiency of exploration. Meanwhile, we also agree that RL does involve more computational costs than others, but the total training time is still acceptable, especially considering the performance improvement it could bring (see **Response to Major Weak-2** for Reviewer **bzJ9**).\n\n**Response to Weak-2-2 (Claimed Issue and No Special Design for TS):** We'd like to clarify that our method does not intend to improve teacher's reliability but to adaptively transfer its target knowledge with our RL-based framework. The unreliability of teacher's target knowledge is the underlying reason why existing approaches that simply integrate KD with UDA frameworks often experience unsatisfactory adaptation performance. Thus, we are highly motivated to utilize RL to select samples aligning with student’s capability to mitigate such unreliable knowledge. Besides, we also agree that there is no special architecture design for TS in our proposed framework. We choose to evaluate on TS as model complexity and domain shift issues are very common in TS. As it is general, we will explore to extend our method to other research areas like CV and NLP in the future.\n\n**Response to Weak-3 (Domain-invariant Knowledge):** The domain discrimination loss originates from Reference [14], which is closely related to adversarial learning. The main idea is to pit two networks against each other. The student is expected to generate invariant representations for both domains and the discriminator is expected to fail to distinguish them. By minimizing this loss, the student would generate similar representations on target domain as the teacher on source domain. In other words, the domain-invariant knowledge would be transferred from teacher to student. \n\nBesides, there are two reasons why we cannot directly transfer the domain-invariant knowledge from teacher to student. First, the compact student model has very limited capacity and cannot capture the same fine-grained patterns in the data as teacher. Coarsely aligning their feature maps, as done in KD-STDA and MLD-DA, would lead to sub-optimal performance on target domain. Secondly, instead of focusing on learning domain-invariant representations, our objective is to improve student's generalization on target domain via teacher's knowledge, which motivates us to adaptively transfer teacher’s target knowledge based on student’s capability. Our ablation study results on framework (see Table 5 of manuscript) also suggest that the performance improvement from domain-invariant loss is very marginal. Conversely, our proposed RKD loss can significantly improve student's generalization capability on target domain.   \n\n**Response to Q-1 (Reward not clear):** We will try to improve the clarity of our framework via providing more description in updated version (See Updates in global **Author Rebuttal**).\n\n**Response to Q-2 (Typos):** We will improve the typos in updated version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H5cYtZHyz4", "reviewer": "Reviewer_nLZW", "summary": "The authors motivate the work by identifying limitations in existing approaches that integrate knowledge distillation into domain adaptation frameworks. Specifically, they note that coarsely aligning outputs over all source and target samples neglects the network capacity gap between teacher and student models, leading to poor distillation efficiency.\nThe proposed RCD-KD framework uses an adversarial discriminator module to align teacher and student representations between source and target domains. It also formulates target sample selection as a reinforcement learning problem, using a novel reward function based on uncertainty consistency and sample transferability. A dueling Double Deep Q-Network (DDQN) is employed to learn the optimal selection policy.", "questions": "How does the computational complexity of RCD-KD compare to existing methods, especially considering the reinforcement learning component?\nHave you explored the performance of the method on longer time series or datasets with a larger number of classes? How does it scale?\nThe paper mentions using DANN to pre-train the teacher model. How sensitive is the performance to the choice of the teacher's pre-training method?\nHow does the method perform when there is a significant domain shift between source and target domains? Are there cases where the performance degrades significantly?\nHave you considered extending the approach to handle multi-source domain adaptation scenarios?", "limitations": "The authors acknowledge some limitations of their work in the conclusion section. They note that pre-training a cumbersome teacher with advanced UDA methods involves more training time than other approaches. Additionally, they mention that using only the distance between teacher's and student's logits to assess sample transferability might overlook intrinsic information from the feature space.\nThese are valid limitations, and it's commendable that the authors have included them. However, the discussion could be expanded to include potential implications of these limitations and possible strategies to address them in future work.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Well-motivated: The authors clearly articulate the limitations of existing approaches and provide a compelling rationale for their proposed method.\nComprehensive evaluation: The experimental results are extensive, covering four different datasets across various time series tasks. The comparisons with state-of-the-art methods are thorough and demonstrate consistent improvements.", "weaknesses": "Theoretical foundation: While the paper provides a detailed description of the proposed method, it lacks a strong theoretical foundation or analysis. Adding theoretical insights or guarantees would strengthen the contribution.\nComputational complexity: The paper does not provide a detailed analysis of the computational complexity of the proposed method compared to existing approaches. Given the focus on resource-constrained devices, this information would be valuable.\nHyperparameter sensitivity: Although some hyperparameter settings are provided, a more comprehensive analysis of the method's sensitivity to different hyperparameters would be beneficial.\nLimited discussion on failure cases: While the paper shows impressive results, a more in-depth discussion of scenarios where the method might fail or underperform would provide a more balanced view.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to Reviewer nLZW for all the comments. \n\n**Response to Weak-1 (Theoretical Foundation):** We appreciate the suggestion to include a stronger theoretical foundation or analysis to further strengthen the contribution. While we acknowledge the value that theoretical insights and guarantees could bring to the paper, our primary focus in this work has been on the practical implementation and empirical validation of our method. Our goal was twofold: to tackle the prevalent issues of domain shift and model complexity in time series applications, and to demonstrate the superior effectiveness of our approach compared to existing benchmarks through extensive experiments on a range of real-world time series tasks. We believe that the empirical success of our method is also very crucial for research community, and thus, in this paper we emphasizes empirical results over theoretical analysis.\n\nNonetheless, we will definitely consider incorporating a theoretical foundation in future extensions of this research. Potential directions include: (1) Theoretical Justification for RL: include a theoretical analysis of using reinforcement learning to optimize sample selection, examining the exploration-exploitation trade-off and its impact on performance. (2) Uncertainty Consistency Analysis: explore the theoretical relationship between uncertainty consistency and model performance, supported by mathematical proofs and derivations.\n\n**Response to Weak-2 and Q-1 (Computational Complexity):** Please kindly refer to **Response to Major Weak-2** for Reviewer **Ltvb** for algorithm complexity analysis and **Response to Weak-1-2** for Reviewer **bzJ9** for model complexity analysis. \n\n**Response to Weak-3 (Hyperparameter Sensitivity):** Please kindly refer to  **Response to Weak-3** for Reviewer **E4vw** for sensitivity analysis of hyper parameters.\n\n**Response to Weak-4 (Failure Case Discussion):** Indeed, there are some cases where our approach does not perform as expected, such as $2\\to7$ in **HHAR** and $0\\to11$ in **SSC**. The poor adaptation performance in these scenarios may be attributed to the larger domain shift, as even the complex teacher model and other benchmarks struggle to achieve good adaptation. When there is a significant domain gap that the complex teacher model cannot effectively address, the target knowledge it provides may also be unreliable for the student. To explore this potential failure cause, we conducted further validation on another HAR dataset characterized by larger domain gaps (See **Response to Q-4**) and we are willing to incorporate a detailed discussion in the revised manuscript.\n\n**Response to Q-2 (Scalability to Larger Dataset):** Please kindly refer to **Response to Weak-2** for Reviewer **E4vw** for the analysis of our method on larger dataset. \n\n**Response to Q-3 (Teacher Sensitivity):**  As shown in Table 4 of our manuscript, we evaluate our framework with teachers pre-trained with different DA methods. Specifically, we utilize two discrepancy-based DA method (i.e., MDDA and SASA) and one adversarial-based DA method named CoDATS. The difference is that: for teachers pre-trained with discrepancy-based DA method, we have to train the domain discriminator. But for teachers pre-trained with adversarial-based DA methods, we can obtain the well-performed discriminator during teacher's training process. Thus, during student's training, the parameters of discriminator are frozen. From Table 4 of our manuscript, we can see that employing teachers from MDDA and SASA slightly underperforms CoDATS and DANN. The possible reason is that adding additional training steps for domain discriminator will inevitably increase training difficulty in terms of model convergence. But they still perform better than other benchmark methods. It indicates that our approach is not limited by the teacher pre-training strategies.\n\n**Response to Q-4 (Larger Domain shift):** To evaluate our method under larger domain shift, we conducted some preliminary experiments on another dataset called HPP HAR [15]. This dataset includes 12 subjects, each performing 6 activities, with the smartphone placed in three different positions: pants, a shirt, and a backpack. The results are shown in the table below. As observed, almost all methods experience performance degradation when faced with a significant domain gap. A potential reason for this is that these KD-based cross-domain methods (including ours) are highly dependent on the teacher's performance. If the teacher fails to capture domain-invariant representations, the student model may also be negatively impacted. Due to time constraints, we were unable to explore other SOTA domain adaptation methods to enhance the teacher's performance and verify the above hypothesis. We plan to conduct this verification in future extensions of our research.   \n\nScenario|Teacher|Student-Only|MobileDA|UNI_KD|Ours\n:---:|:---:|:---:|:---:|:---:|:---:|\nS01Pants $\\to$ S02Backpack|43.54|23.45|37.41|35.45|38.95\nS01Shirt $\\to$ S03Pants|56.10|33.45|49.41|41.12|47.77\n\n**Response to Q-5 (Extent to Multi-Source):** Some adjustments may be necessary if we extend our approach to multi-source domain adaptation (MSDA) scenarios. Firstly, instead of utilizing RL to select proper target samples, in MSDA we could leverage RL to dynamically select suitable source domains (or source samples), which are most relevant to target domain, to minimize the negative transfer as much as possible. Secondly, some of the key components in RL need to be re-defined. For instance, we need to re-formulate reward function to offer essential feedback to DDQN.  A potential solution could be some functions to measure the data distribution discrepancy between source and target domain. Lastly,  for MSDA, the teacher should be the expert who globally predicts well on the mixture of source domains. Methods like STEM [16] could be possible solutions for training a proper teacher in multi-source domain scenario.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## Summary

We sincerely thank all the reviewers for their insightful and valuable feedback. We are pleased that the reviewers recognized the novelty of our work and appreciated our motivation for designing a reinforcement learning-based sample selection approach for cross-domain knowledge transfer. Additionally, we are glad that the comprehensive experimental evaluation across four public datasets was noted, highlighting the superior performance and robustness of our approach. We have carefully considered all the suggestions and comments. The overall summary of our rebuttal is as follows.

* We performed the computational complexity analysis as suggested by Reviewer **Ltvb**,**bzJ9** and **nLZW**.
* We verified the scalability of our method to larger dataset as suggested by Reviewer **E4vw** and **nLZW**,  and scalability to dataset with significant domain shift as suggested by Reviewer **nLZW**. 
* We conducted the ablation study with additional active learning strategies as suggested by Reviewer **Ltvb**, 
* We added sensitivity analysis for temperature hyper parameter as suggested by **E4vw**, **nLZW**. 
* We included experiments on integrating feature distances for sample selection as suggested by Reviewer **E4vw**.
* We provided additional details and clarifications as requested by all reviewers. 

## Updates to manuscript:

* Include complexity analysis and additional hyperparameter sensitivity analysis as suggested by Reviewer **Ltvb**,**bzJ9** and **nLZW**.
* Include experimental results on larger dataset PAMAP2 as suggested by Reviewer **E4vw** and **nLZW**.
* Include literature review for active learning as suggested by Reviewer **Ltvb** as follows:
    >Meanwhile, our work also relates to active learning (AL) field specifically in terms of selecting the most critical instances from unlabeled data. Note that here we only discuss the uncertainty-based sampling strategies in active learning as other query strategies (e.g., instance correlation) are beyond the scope of our paper. Particularly, the uncertainty can be measured by three metrics: least confidence, sample margin, and sample entropy [1]. The least confidence methods like [2][3] select the instances which have the highest posterior probability and the margin sampling leverage the margin between posterior probabilities of the first two most probable classes [4]. Unlike above two, the entropy metric measures the uncertainty over the whole output prediction distribution [5][6]. In our method, instead of explicitly utilizing entropy-based uncertainty as AL methods, we propose to leverage the consistency between teacher' and student's entropy-based uncertainty to learn the optimal sample selection policy with dueling DDQN. The experimental results across various time series datasets demonstrate the effectiveness of our method.

* Update caption for Fig.1 and include more description to better illustrate our reward module in our framework as commented by Reviewer **bzJ9**. 
>As illustrated in Fig.1, the reward function consists of three parts. The first one is the action $a_k$ which is the output of dueling DDQN. The second part is the uncertainty consistency, estimated by entropy from student's logits $\boldsymbol{q}^S$ and the averaged logits $\overline{\boldsymbol{p}}^T$ of $N$ teachers generated from MCD module. The third part is the sample transferability based on the KL divergence between $\boldsymbol{q}^S$ and $\overline{\boldsymbol{p}}^T$. The output of reward module $r_k$ then will be utilized for the optimization of dueling DDQN for learning optimal sample selection policy.

## References in Rebuttal:

    [1] Tharwat, A., Schenck, W. (2023). A survey on active learning: State-of-the-art, practical challenges and research directions. Mathematics. 
    [2] Culotta A, McCallum A. Reducing labeling effort for structured prediction tasks. In: Proceedings of the 20th AAAI 2005
    [3] Zhu J, Wang H, Tsou B, Ma M (2010) Active learning with sampling by uncertainty and density for instances annotations. IEEE Trans Audio Speech Lang Process 18(6)
    [4] Campbell C, Cristianini N, Smola A. Query learning with large margin classifiers. In: Proceedings of the 17th ICML 2000
    [5] Burl MC, Wang E. Active learning for directed exploration of complex systems. In: Proceedings of the 26th (ICML 2009)
    [6] Kim J, Song Y, et al . MMr-based active machine learning for bionamed entity recognition. In: Human language technology and the North American association for computational linguistics
    [7]  A. Romero, N. Ballas, et al., “Fitnets: Hints for thin deep nets,” in ICLR, 2015.
    [8]. Zagoruyko and N. Komodakis, “Paying more attention to attention: Improving the performance of convolutional neural networks via attentiontransfer,” 5th ICLR, 2017.
    [9] B. Sun and K. Saenko, “Deep coral: Correlation alignment for deep do-main adaptation,” in European conference on computer vision. Springer,2016
    [10] N. Passalis and A. Tefas, “Learning deep representations with probabilistic knowledge transfer,” in Proceedings of the ECCV, 2018, 
    [11] Reiss, A., Stricker, D. Introducing a new benchmarked dataset for activity monitoring. In 2012 16th international symposium on wearable computers.
    [12] Atif Belal, Madhu Kiran, et al. Knowledge distillation methods for efficient unsupervised adaptation across multiple domains. Image and Vision Computing.
    [13] Granger, E., Kiran, M.. Joint progressive knowledge distillation and unsupervised domain adaptation. In 2020 IJCNN.
    [14] Tzeng, E., Hoffman, J., Saenko, K. Adversarial discriminative domain adaptation. In Proceedings of ICCV (2017).
    [15] Chen, Zhenghua, et al .:Smartphone sensor-based human activity recognition using feature fusion and maximum full a posteriori. IEEE TIM (2019)
    [16] Nguyen, V. A., Nguyen, T., et al. (2021). Stem: An approach to multi-source domain adaptation with guarantees. In Proceedings of ICCV.

Meta-review: Summary: 
The paper presents a novel RL-based active learning approach that dynamically selects target data used for transferring knowledge from the teacher network to the student network. The proposed framework is sensitive to the differences in the capacities of the teach and student networks, and its goal is to bridge this gap within a domain adaptation framework incorporating knowledge distillation. To accomplish this, the authors proposed a reward mechanism that learns an optimal policy for selecting data from the target domain, considering the student network’s capacity. The proposed model significantly improves the student model’s generalizability on the target domain compared to standard knowledge distillation and domain adaptation techniques. 

Strengths: 
* A novel method to select appropriate target data during domain adaptation from a large network to a smaller one. The potential impact of the proposed approach is significant. 
* A well-written, well-organized, and an easy to follow paper. 
* Experiments are fairly rigorous and are backed up by a detailed description of their implementation, accompanied by the code. 
* The performance gains in comparison with the baselines are also significant on the time-series datasets considered. 

Weaknesses: 
* Because the proposed method incorporates the RL loop during knowledge transfer, it will result in increased training time. While this is not a significant burden on the data sets used in the paper (as shown in their rebuttal), it could become a substantial burden in more complex settings. 
* The writing of the paper could be improved by focusing more on attribution to previous techniques/work that is closely related to the proposed techniques. 

Overall Recommendation: 
This paper presents a novel and elegant RL-based approach for knowledge transfer from the teacher network to the student network under the domain shift scenario. The method's experimental validation is thorough and convincing. The authors did a great job addressing the reviewers' concerns in their rebuttals. Since disseminating the ideas presented in this work can potentially benefit the broader machine-learning community, the recommendation is to accept this paper for NeurIPS.
Decision: Accept (poster)