Self-Adaptive Motion Tracking against On-body Displacement of Flexible Sensors
Authors: Chengxu Zuo
Jiawei Fang
Shihui Guo
Yipeng Qin
Venue: NeurIPS 2023 poster (2023)
Keywords: motion tracking
flexible sensor
on-body displacement
deep learning
domain adaptation

Abstract: Flexible sensors are promising for ubiquitous sensing of human status due to their flexibility and easy integration as wearable systems. However, on-body displacement of sensors is inevitable since the device cannot be firmly worn at a fixed position across different sessions. This displacement issue causes complicated patterns and significant challenges to subsequent machine learning algorithms. Our work proposes a novel self-adaptive motion tracking network to address this challenge. Our network consists of three novel components: i) a light-weight learnable Affine Transformation layer whose parameters can be tuned to efficiently adapt to unknown displacements; ii) a Fourier-encoded LSTM network for better pattern identification; iii) a novel sequence discrepancy loss equipped with auxiliary regressors for unsupervised tuning of Affine Transformation parameters.

Self-Adaptive Motion Tracking against On-body Displacement of Flexible Sensors
Chengxu Zuo
Jiawei Fang'
Shihui Guo!"
Yipeng Qin²
School of Informatics, Xiamen University, China School of Computer Science Informatics, Cardiff University, UK
Abstract
Flexible sensors promising for ubiquitous sensing of human status due to their flexibility and easy integration as wearable systems. However, on-body displace- ment of sensors is inevitable since the device cannot be firmly worn fixed position across different sessions. This displacement issue causes complicated patterns and significant challenges to subsequent machine learning algorithms. Our work proposes a novel self-adaptive motion tracking network to address this chal- lenge. Our network consists of three novel components: i) light-weight learnable Affine Transformation layer whose parameters can be tuned to efficiently adapt to unknown displacements; ii) Fourier-encoded L.TTM network for better pattern identification; iii) novel sequence discrepancy loss equipped with auxiliary regres- sors for unsupervised tuning of Alline Transformation parameters. Experimental results show that our method is robust across different on-body position configura- tions. Our dataset and code are available at: http::/github..oo//uuue 9996/Self- Alaptive-Motion Traooonn-rrangg-g Tracking-against against--nn hod-boy--DDe padyy-isplacemeettoou olaipaacemmttto--leeie-----ee Flexibe-SSensors.
Introduction
Following the philosophy of ubiquitous computing, wearable devices have emerged promising solution for motion tracking that has various applications in entertainment [1], human-computer interaction [2], healthcare [3, 4], etc. Among different types of such wearable devices, those using flexible sensors stand out for their advantages in long-term use scenarios. Specifically, flexible sensors can be easily integrated into wearable devices (e.g., sewn into ordinary clothing) while ensuring wearing comfort. These flexible sensors bend in response to human body movements (represented by joint angles), causing changes in their readings, which can be used for motion tracking [5].
In recent years, enabling wearable devices to be worn in non non-fixed positions has become a crucial objective in order to make these devices more suitable for daily use [6]. However, this flexibility poses new challenges for deep-learning powered motion capture (mocap) systems. That is, the inevitable on-body displacement across wearing sessions cause significant data distribution shifts, which degrades the performance of supervised learning models (Figure la).
A straightforward solution to the above problem is to collect a large dataset that covers wide spectrum of wearing positions for model training. However, this approach significantly amplifies data collection costs, including equipment expenses, manpower, and time investments. Furthermore, any alterations device design (sensor position, quantity, etc.) necessitate a fresh round of data collection, resulting in significant resource consumption and impeding swift product iteration. Another solution to fine-tune the model against "unscen" displacements with small amount of labeled data under new displacements, a.k.a., supervised domain adaptation. However, this is infeasible for our task as
*Corresponding author: Shihui Guo (aaahihui@xmu.cdu.cm))
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Fabric
840
(2) l= +
046 1+0 c40 1+0
o-5
1-0
880
Angle
780
800-
Cecular Dieplacement
Latena Dieplaeeent
720
720
740
780
780
750
800
850
F
1
(a) Device displacements
(b) Case of lateral displacement
(c) Case circular displacement
Figure 1: Demonstration of our problem setup. (a) Joint tracking with lateral and circular sensor displacements. (b) The data distribution shifts caused by lateral displacement, e,g., /=l: lateral displacement shifts upwards by I cm; and (c) circular displacements, e,g, c=:; circular displacement shifts clockwise by 5" T1, Tal sensor readings.
obtaining accurate joint angles (i.e,, labels) requires advanced optical motion capture systems that are difficult to set up in real-world scenarios (e.g., outdoor running, boating), hindering everyday use.
In this work, address the above challenges by proposing novel self-adaptive motion tracking network that can adapt itself to unknown displacements in an unsupervised way, using only a small amount of new sensor readings. Specifically, most existing unsupervised domain adaptation methods assume the source and target domains share identical feature distributions, which is not applicable to task as training contains data with multiple displacements, while the test data has only one unknown displacement. To this end, we propose a novel solution that allows the alignment of test data with subset of training data that also has one displacement, which encompasses three novel components: i) light-weight learnable Affine Transformation layer whose parameters effectively model the distributional shifts caused by displacements; ii) Fourier-encoded LSTM network that facilitates the learning of more frequency components for better pattern learning and identification; iii) a novel sequence discrepancy loss equipped with auxiliary regressors for unsupervised self adaptatttnnn
The contributions of our method are three-fold:
We propose a novel self-adaptive motion tracking network that can adapt unknown on- body displacements of flexible sensors in an unsupervised way, allowing for long-term and daily motion tracking.
We propose three novel components for network: i) light- weight learnable Affine Transformation layer, ii) a Fourier-encoded LSTM network, and iii) a novel sequence discrepancy loss, which together efficiently reduce the distributional shifts of sensor readings caused by on-body displacements.
Extensive experiments demonstrate the superior performance of our method against the state-of-the-an ahe-art methods of domain adaptation applied in scenario.
2
Related Work
2.1 Domain Adaptation
Domain adaptation aims mitigate the gap between the source and target domains that models trained in the source domain(s) can be applied to the target domain(s). Traditional methods perform adaptation effectively by either reweighting samples from the source domain [7, 8]. or seeking an explicit feature transformation that transforms the source and target samples into the same feature spaces [9, 10, 11, 12]. Subsequent studies have shown that deep neural networks can learn more transferable features for domain adaptation [13]. For example, Deep Domain Confusion (DDC) [14] first proposed the use of Maximum Mean Discrepancy (MMD) loss align the feature distribution of the target domain with that of the source domain for the domain adaption of deep neural networks; Deep Adaptation Networks (DAN) [15] extends the idea to the use of multiple-kernel MMD; Deep CORAL [16] proposed CORAL loss [17] for adaptation; Chen et al. propose Higher-order Moment Matching (HoMM) method for better distribution discrepancy minimizing. Other works within deep neural networks also illustrate immense success in learning transferable

---

features [19, 20, 21]. In addition, recent studies show that adversarial learning also contributes to learning more transferable and discriminative representations [22, 23, 24, 25, 26, 27, 28].
In summary, most existing domain adaptation methods assume that the source and target domains share the same feature distributions and thus aim minimize relevant distributional differences [29]. While in our task, the training set contains data with multiple displacements while the test data has only one unknown displacement, thus making the above-mentioned assumption invalid.
Addressing this issue, we propose a novel self-adaptive motion tracking network that tunes the parameters of an Affine Transformation layer with novel sequence discrepancy loss, which can align the test data with a subset of training data that also has one displacement, achieving robust and accurate domain adaptation for flexible sensors.
2.2 Human Motion Capture
Human motion capture records human body movements and has been widely applied in entertainment, healthcare, sports, etc. Existing mocap solutions can he roughly classified into two categories: vision- based and sensor-based.
Vision-based mocap solutions make use the latest deep learning techniques and have achieved great success in specific scenarios [30, 31J. However, they rely good visual conditions and are inherently weak against textureless clothes and environmental problems (e.g., challenging lighting, occlusion). Most existing censor-based mocap solutions rely Inertial Measurement Units (IMUs) to record motion inertia acceleration for the analysis of human posture [32, 33, 34]. Although robust against environmental conditions, the dense and tight placement of IMUs is intrusive and inconvenient, hampering performers from moving freely in their daily lives.
To this end, people turned to flexible sensor-based wearable devices indistinguishable from daily clothing. For example, Glauser et al. [35] designed a stretch-sensing soft glove and used it to interactively estimate hand poses with the aid of deep neural network; Ma et al. [36] proposed flexible all-textile dual tactile-tension sensors for precise monitoring of athletic and form, illustrating their potential application in robust physical training analysis. Zhou et al. [37] uses deep regressor to continuously predict the 3D upper body joints coordinates from 16-channel textile capacitive sensors.
Flexible Sensors
Given their advantages of bio-compatibility. high stretch-ability, lightweight, and ease of integration within clothing, flexible sensors have been used for long-term monitoring of human physical status, specifically motion capture [38], human-computer interfaces [39], soft robotics [40], etc. For human motion tracking, existing methods have explored the use of flexible sensors in tracking the motion of the upper body [41], fingers [35], lower limbs [42], elbow joints [43], and knee joints [44]. Along with such exploration, it has been noted that on-body displacement of sensors is inevitable as the device cannot be firmly worn a fixed position across different sessions [45]. Additionally, due to the deformation characteristics of flexible sensors, it is fairly complicated to achieve robust motion tracking in the presence of placement deviation [46, 47].
In this work, we start with an exemplary scenario: an elbow pad with two flexible and stretchable sensors for elbow joint tracking, and address this gap by proposing novel self-adaptive motion tracking network that can adapt to unknown on-body displacements in an unsupervised manner to achieve robust and accurate motion tracking.
Hardware
We design and develop a prototype by augmenting a standard elbow pad with two soft stretchable (i.e., flexible) sensors, which placed on the olecranon side of the elbow. The pad offers versatile size of 20 cm length and 25 em in circumference, accommodating a broad spectrum of users. The two sensors placed 2 apart.
Our method aims to estimate the bending angle, 0, of an elbow joint (Figure 1) from the sensor readings of the two flexible sensors. The bending angle is defined as the angle in the sagittal plane
3

---

(8)
Ausilary Regresors
ac
W; +- 11
Regaoss a-d
-- 8
Descrpanccy
Adapiation
Ro
Sequznce Discrepency
COS sin
X
Shape Diccrejancy
Unsoen
Sensor Readings R
Leamalle Affine Transfommation
Fourier-encaeed LSTM Network
Main Regressoc
Disliscement
fr
Figure 2: Overview of the proposed self-adaptive motion tracking network. Our network decomposes the mapping between sensor readings R. and joint angles 0 into parts: SR-O fR-RR, -Ro (a) The intuition behind the adaptation function jR-Ro, which aligns the test data R. to a subset of the training Ro that also has single displacement. (b) Illustration of the three novel components in the proposed network: i) a learnable Affine Transformation layer; ii) Fourier-encoded LSTM network; iii) sequence discrepancy loss L equipped with auxiliary regressors. Specifically, FR- is implemented with the Affine Transformation layer, whose parameters are adapted with the sequence discrepancy loss L; fRO- is implemented with the supervisedly pre-trained Fourier-encodee LSTM network, whose parameters are frozen during adaptation.
between the humerus and the central line between the radius and the ulna. Fabric sensors are purchased off-shelf products from ElasTech'. They capacitive, i.e., their capacitance increase with the stretch caused by the bending of the arm. The sensor readings are digitized to values in the range [0, 1023] and transmitted wirelessly via Bluetooth Low Energy frame rate of 50Hz.
Method
In our work, a motion tracking system using flexible sensors aims to construct a function f between sensor readings R and joint angles f (Figure la) that: @=f(R). Naively, f can be approximated through supervised learning by fitting a model to collected (R, 0) data. While in practice, it is infeasible collect enough (R, 0) data for training as R depends on the countless instances of on-body displacement of flexible sensors (c,2). where e (deg) and (cm) are circular and lateral displacement, respectively (Figure la). Addressing this issue, we propose novel self-adaptive motion tracking network for flexible sensors that can efficiently adapt to unseen on-body displacements in an unsupervised manner. As Figure 2 shows, our network decomposes f into two parts:
FR-9 FR- SR
where R, denotes the sensor readings with limited number of displacements in the training dataset, R. denotes the sensor readings with unknown displacements at the time of testing. Between them, fR-RR, is implemented with light-weight learnable Affine Transformation layer whose parameters can be tuned to align R to R, and thus adapt the network to "unseen" displacements (Sec. 4.1); JRA- is implemented with Fourier-encoded LSTM network reduce the learning bias towards
low-frequency functions and produce more accurate results (Sec. 4.2). The tuning of SR-RR is achieved by minimizing novel sequence discrepancy loss equipped with auxiliary regressors (Sec. 4.3).
'http://www. elas-tech. com/

---

aci)) 4.1 Learnable Affine Transformation
The proposed learnable Affine Transformation layer an efficient implementation of fR-RO- Specifically, it models the linear transformation components of flexible sensors in motion tracking system:
Initial Stretch, which is mainly affected by lateral displacements (Figure 1(b)) and can be modeled with the additive bias vector in affine transformation.
Stretching Range, which is mainly affected by circular displacement (Figure I(c) and can be modeled with scaling matrix in affine transformation.
Accordingly, let X [21, be the sensor readings, W, diag ($1, s2] be the scaling matrix, B [b1, b2]" the bias vector, we have:
b
fR-Ro(X) W.X BB=
+
(2)
82
122
b2
Both W, and B are learnable in the adaptation stage (Sec. 4.3) for the adaptation to unknown displacements.
4.2 Fourier-encoded LSTM Network
We implement fR, o by augmenting standard LSTM sequence prediction network with Fourier Feature Encoding (FFE) layer, which models the nonlinear mapping from flexible sensor signals to joint angles. Our key insight is that FFE helps to reduce the learning bias of deep neural networks towards low-frequency functions [48] that leads to underfitting of fRO-8.
The FFE layer used in our work is as follows:
2T
2xT
cos
sin
A
A
where E X is the input, A > |max(X) min(X)] I dence. Please note that our Fourier-encodee LSTM network is only trained in the pretraining stage and fixed during adaptation (Figure 2).
so that x' and x are in one-to-one correspon-
4.3 Unsupervised Adaptation to Displacements
Algorithm 1 Unsupervised Adaptation to Displacements
Pretrained model f with auxillary regressors; Scaling matrix W, and Bias vector B of the affine transformation layer in f: Test-time sensor readings It; Batch size
Input:
Parameter: Training epoch e: Learning rate y/;
Output: Adapted W. and B;
1: for = 1, .1......... do
2:
for j = 1,..., [[R/n' do
3:
X <- manc-batch of (sequence) samples from R
4:
Ô f(X)
acrè) 5: W. <- W. 1) OW,
B B 17 8B
6: end for
7: end for
8: return W., B
As mentioned above, our network decomposes the mapping into two parts fR-10 = FR-RR, TRO- Between them, the challenging nonlinear components are well-approximateddy by TRO-O, which greatly simplifies our adaptation tuning the parameters of the Affine Transformation layer of JR-ROO- Intuitively, we aim to align the distribution of R that of Ro so that SRo-0 can accurately map R to joint angle 0. Naively, this can be achieved by directly minimizing distributional difference
5

---

loss between Ry and R or maximizing the prediction accuracy when using R. However, neither approach is effective for our task as i) the former assumes and Rg share the same distribution after the affine transformation, which is incorrect because Ro (the training set) contains data with multiple displacements while R collected with a single unknown displacement; and ii) the latter requires ground truth joint angles A of RR which are unavailable during use. Addressing this challenge, we propose a novel sequence discrepancy loss equipped with auxiliary regressors as follows.
Sequence Discrepancy Loss. We observed that the domain gaps among different displacements are particularly evident when the elbow is bent and less when the elbow is flexed. Consequently, the domain gaps are not evenly distributed over time during elbow movement, leading to the discrepancy among different regressors that estimate the same joint angle 0g with different choices of time windows, where denotes the time step in a motion sequence. Our key idea that the adaptation to unknown displacements can be achieved by minimizing such discrepancy. Specifically, let n be the length of the time window for the main regressor, we have
8 =f f(Rinn+1, R Rd)
where 0.0)) is its joint angle estimate. Similarly, we include additional auxillary regressors that predict 0, with different time windows as:
0:00 f(B R(tt0)-+2 n22 Re+s)
(5)
where 0:00 = f( ... 8f 1(- for Ru used in supervised pretraining, i= 1. However, when R is used, discrepancies occur among regressors and we tune the affine transformation parameters (Sec. 4.1) minimize such discrepancies. Specifically, let (()) mcan(0) (0(*), ((()) ..... 0(()) be the
g( -min mean of joint angle estimates of regressor k, (R) max, (d(e)) min, (d(0))+ be the normalized joint
angles, E 1 for numerical purpose, define:
n-1
mmeon Σ
(k)
1
(6)
π
k=0
i=0
Eshape
(R)
2
(7)
8 km0
x=
Our overall sequence discrepancy loss is weighted sum of Emean and Lhaaee
L almean (1 ox) Sshoee
8
where a [0, is a weighting hyperparameter. Alg. shows the pseudo-code of our adaptation algorithm.
5 Experimental Results
5.1 Experimental Setup
The dataset used in this paper consists of sensor readings and joint angles collected by single user wearing the augmented elbow pad (Sec. 3) while performing elbow flexion. The joint angles are computed using the 3D positions captured by Nokov motion-capture system at rate of 60 FPS. As instances of different ondy displacements, we collect 11 groups of data
Dataset and Metrics
with circular displacements c E (-5.0, 5} and lateral displacements / E (-2, -1, 1, 0, 1, For each group, collected data for 8 consecutive elbow flexion. In total, we collected 5,3 10 frames of data across the 11 valid groups. Among the 11 groups, we randomly selected 5 of them as the training set Dirain and used the rest 6 groups the test set Dtest- We have obtained ethical approval for the publication of both datasets and results.
We use the Mean Absolute Error (MAE) of predicted joint angles (degrees) our main evaluation metric. For each experiment, we repeat times and report their mean and standard deviation (std). Unless specified, we use the same setups for all experiments.

---

Training Details Our model is trained in two stages: supervised pretraining and unsupervised
adaptation. For the supervised pretraining, we employ an MSE loss:
1 R-1
Lisse
(0 0,)²
where 0(*) is the output of regressor & time step t. 0, is the corresponding ground truth joint angle. use an Adam optimizer with a learning rate of le-3, B 0.9, B2 = 0.999, and training epoch e 30. For the adaptation,
3
Adam optimizer with learning rate of 5e and weight decay of 0.001, 31 0.9, A2 = 0.999, and training epoch e = 20. We use n = 10 for both pretraining and adaptation. All experiments were conducted on desktop PC with AMD Ryzen 3950X CPU and an NVIDIA RTX 3080 GPU, Please see the supplementary materials for the details of network architectures.
5.2 Comparisons with SOTA
As Table shows, we compare our method with state-of-the-ar (SOTA) domain adaptation methods. For fair comparison, we have adapted the official code provided by the authors to share the same input and output format as ours. Please see the supplementary materials for more details on the implementation of SOTA methods. It can be observed that our method outperforms all SOTA methods, which demonstrates its effectiveness.
GT
Ad/SKM
HuMM
Ous
160
(
120
80
40
Time Indes
Figure 3: Comparison with two most competitive SOTA methods (displacement condition: c=-5, [=1), As highlighted in the dashed boxes, our method (red) much closer to the ground truth (blue). GT: ground truth.
To gain more insights, visualize the results of method and its strongest competitors, AdvSKM [27] and HoMM [18), as time series in Figure 3. The results show that our method fits the ground truth with closer distance than other SOTA methods.
Comparison with Supervised Learning To demonstrate the effectiveness of un- supervised learning in our task, we tune the parameters of Affine Transformation layer by supervised learning (i.e., super- vised domain adaptation) to obtain its "up- per bound". Even trained in an unsuper- vised manner, our method achieves results comparable to its supervised version (Ta- ble 1). This difference of 1.5 degrees in- dicates that our unsupervised learning ap- proach is effective.
Table 1: Results of comparative experiments.
Method
MAE (deg)
MMD [49]
28.73 + 8.50
D-CORAL [16]
13.83 8.45
DAAN [23]
16.80 + 7.88
DSAN [50]
14.03 5.88
AdvSKM [27]
13.05 5.19
HoMM [18]
11.03 4.42
Ours
7.32 + 2.85
Supervised
1.63
Experiments on different participants To further demonstrate the generalization ability of our method, we conducted experiments with five additional participants of varying body types (see the supplementary materials for body profile details). Each of these five participants wore the devices in three supervised on-body positions and performed three distinet physical activities sequentially: ping-pong. basketball, and boxing. Each activity lasted for minimum duration of one minute. In total, this comprehensive evaluation approach yielded 15 (5 participants x on-body displacements) unique data segments across all participants, comprising 81,848 total frames. For each participant,
7

---

Table Experimental results on five additional participants.
Averaged MAE (deg)
User ID
D-CORAL
DSAN
HoMM
AdvSKM
Ours
11.84 + 1.49
11.86 + 1.57
11.96 + 1.73
11.84 1.80
12.07 + 1.04
2
16.59 + 1.81
17.04 + 1.80
16.53 + 1.70
16.63 + 0.95
16.69:::0 2.34
14.94
5.89
13.68 + 4.19
15.24 + 6.93
15.71 6.89
12.70
- 2.79
4
11.76 + 1.11
12.04
1.16
12.06 = 1.49
12.08 = 1.73 1.73
9.30
= 2.77
5
19.41
+ 4.37
1 4.34
20.58 + 5.88
19.44:4211
13.96 2.88
Avg
14.91 + 0.34
#.29
55.77:0.4
1
12.94 0.25
we conducted times 3-fold rross-validttion experiments and recorded the average result. As Table shows, our method achieves optimal or near-optimal results for all five additional users, indicating effectiveness of our method against user diversity.
5.3 Ablation Study
To demonstrate the effectiveness of the three novel techniques proposed, we conduct an ablation study follows:
FFE: Remove the Fourier Feature Encoding (FFE) layer from the LSTM network.
Affine: Remove the Affine Transformation layer and tune all the parameters of the network during adaptation after supervised pretraining. For the adaptation, we use a weighted sum of two loss terms: i) supervised loss Lmse applied on the training data; and ii) our sequence discrepancy loss C applied on the test data.
SD Loss: Replace our Sequence Discrepancy (SD) loss with more naive and stricter version:
C
(0
6,)
(10)
1. k=
where θ. mcan 0:0)
0(n-1)).
Table 3: Ablation study.
We have enumerated all combinations of the three components proposed. As Table shows, can be observed that: i) Ours achieves the best performance, indicating the effectiveness of the three novel techniques proposed; ii) FFE and affine transformation improve the performance in all cases; iii) the effectiveness of SD loss relies on affine transformation. In both cases with and without FFE, the co-existence of affine transformation and SD loss show better perfor- mance than using SD loss alone (Ours over Case
Case
FFE
Affine
SD Loss
MAE
13.40=5.58
+
16.81
+
12.58=8.62
4
54.36
+
+
10.27
12.36=6.99
+
+
9.38+8.78
Ours
+
+
7.32+12.85
6, Case over Case 3).
5.4 Justification of Motivation
Affine Transformation Figure 4 demonstrates the effective alignment of the target and source domains with our method. As shown in Figure 4a and 4b. although liggt, the proposed Affine Transformation layer effectively aligns the sensor reading distributions of test data (blue) to those in the training set (orange), thereby significantly improving the accuracy joint angle estimation.
Sequence Discrepancy Loss
The design rationale of this component is rooted in the insight that our sequence discrepancy loss L shares similar optimization landscape with the estimation error of joint angles during optimization (Figure 5). This consistency effectively offers the advantage of

---

160
160
120
120
88
88
7037
dim
1ug
dim
900
Before adaptation
Ours
HoMM
(d) AdvSKM
Figure 4: Original and adapted data distributions of our method, HoMM and AdvSKM. Note that due method characteristics, our method is aligned directly using the sensor reading, while the latter two are aligned in the latent space. "dim 1" and "dim 2" are obtained using t-SNE.
leading the network optimization toward the optimal point. As shown Figure i) before adaptation, the outputs of different regressors significantly differ from each other, leading to a high sequence discrepancy; ii) after adaptation, the discrepancy among regressors reduced and the accuracy joint angle estimates for all regressors was improved significantly at the same time.
Fourier Feature Encoding Existing literature shows the existence of spectral bias: the tendency of neural networks to prioritize learning www-frequency functions [48]. Figure 7a shows the frequency distribution difference between the prediction and ground truth. Without the component of FFE, the network output shows a large deviation in the region of low-frequency domain from the ground truth (on the left side of this plot). This over-emphasis the low-frequency components inevitably dominates the prediction capability and weakens the reconstruction of the high-frequency signals.
Our experiment confirms that Fourier Feature Encoding (FFE) reduces the spectral bias of neural networks. Therefore, as Figure 7 shows, our method can approximate the mapping between sensor readings and joint angles in more faithful way.
(ap
30,0
0.06
20.0
20.0
MAP
0.04
10.0
0.02
10.0
0.2
0.2
0.9
0.9
-0.1
b1
0.0
-0.4
0.3
0.0
0.1
-0.4
00.3
b2
81
L.
0.7
0.8
0.9
1.0
1.1
0.7
0.8
82
Error
Bias
(b) SD loss Bias
(c) Error
Scaling
(d) SD loss w.E.f. Scaling
Figure 5: Rationale of our Sequence Discrepancy (SD) loss (c=5, (=0). Our SD loss shares similar minima with the MAE errors. Error: errors of joint angle estimation. As defined in Eq. 2, Bias
B = [b,, bo]"
and Scaling W, = diag [$1, s2].
121 200
Matn- Al A5- Al
GT 200
Mat- Al AS-
200
100
160
100
Ampliikk
(deg))
(deg)
120
(dee))
123
50
120
50
100
100
651 101
100
150 140
Index
Thme Index
Time Indle
Fragancy (Hi)
(a) Before adaptation
(b) After adaptation
(a) Frequency Domain
(b) Time Domain
Figure 6: Justification of Sequence Discrepancy
Figure Justification of Fourier Feature Encod-
Loss (c=5, 1=0). The accuracy of joint angle timation improves along with the reduction of discrepancy among regressors. A1, A5, A9: aux- iliary regressors 1, 5, 9. Main: main regressor. GT: ground truth.
ing (FFE) (c=0, 1=-1). (a) Time series of network outputs with/without FFE and the ground truth in the time domain. (b) Differences of network out- puts between with/without FFE and the ground truth in the frequency domain.

---

Limitations and Future Work
The performance of the proposed method is dependent on the quality of the data in both the training and test sets. When the data quality is poor (e.g., noisy), the error in joint angle prediction may still be high even the sensor readings are well aligned (see Figure 8a and 8b).
Limitations
Specifically, according to the frequency char- acteristics of elbow movement, define the sensor signal greater than 5Hz as noise, and cal- culated the signal-to-noise ratio (SNR) for the sensor signals within the 11 data sets collected from single user. Furthermore, the mean absolute error (MAE) for each set after implementing our adaptive approach. It can be found that our proposed method exhibits signifi- cant effectiveness when the SNR exceeds 10 dB. Conversely, maintaining satisfactory outcomes becomes challenging when the SNR falls below 10 dB (see the supplementary materials for the visualization).
160
(Bx))
1.20
120
80
compiled
eno E
Pvp
680
880
340
(a) Before adaptation
(b) After adaptation
Figure 8: Failure cases (noisy data). x-axis and y-axis ((r1,2))) sensor readings. z-axis: joint angles. Blue and Orange: displacements in the training and test sets, respectively.
Future Work As abovementioned, we believe that extending our method to learning with noisy data would interesting future work. addition, our work opens up number of research directions for future efforts. First, in most domain adaptation methods, measuring the similarity between the distribution of the target domain and the source domain is a key step. While in our work, we found that the output of the auxiliary regressors shifts from the source domain, and verified the feasibility of domain adaptation using only target domain data in ficxible sensor applications. In future work, we hope to implement this idea in more diverse applications. Second, although we have only used translation and scaling, this work proves fefiee-based domain adaptation to be promising solution to mitigate the data distribution shifts of flexible sensors. The effectiveness of rotation and reflection remains to be studied in our future work.
be used
indicator of data distribution
7 Conclusion
Our work proposes novel self-adaptive motion tracking network to address the challenging data distributional shifts caused by on-body displacements of flexible sensors. To mitigate the effects of such displacements, we propose three novel techniques. First, we propose an Affine Transformation layer that can remap shifted data distributions to those in the training set efficiently. Second, we propose a Fourier-encoded LSTM network that can learn richer frequency components in the input signals and thus improves the accuracy of joint angle estimation. Finally, we propose Sequence Discrepancy loss equipped with auxiliary regressors that can adapt the parameters of Affine Transformation effectively in unsupervised manner. Experimental results show that method can effectively adapt to unknown displacements of flexible sensors worn at different positions.
Acknowledgements
This work is supported by National Natural Science Foundation of China (62072383), the Fundamen- tal Research Funds for the Central Universities (0720210044)) and partially supported by Royal Society (IEC ANSFC 10222..
10

---

References
[1] Arata Horie, Ryo Murata, Zendai Kashino, and Masahiko Inami. Seeing is feeling: A novel haptic display for wearer-observer mutual haptic understanding. In SIGGRAPH Asia 2022 Emerging Technologies, pages 1-2. 2022.
[2] Young Jin Yoo, Yeon Heo, Yeong Jae Kim, Joo Hwan Ko, Zafrin Ferdous Mira, and Young Min Song. Functional photonic structures for external interaction with flexible/wearable devices. Nano Research, 14(9):2904 2918, 2021.
[3] Sheikh Iqbal, Imadeldin Mahgoub, Du, Mary Ann Leavitt, and Waseem Asghar. Advances in healthcare wearable devices. NPJ Flexible Electronics, 5(1):1-14, 2021.
[4] Tianyiyi He, Feng Wen, Yanqin Yang, Xianhao Le. Weixin Liu, and Chengkuo Lec. Emerging wearable chemical sensors enabling advanced integrated systems toward personalized and preventive medicine. Analytical Chemistry, 95(1):490-51) 514, 2023.
[5] Xiaowei Chen, Xiao Jiang, Jiawei Fang, Shihui Guo, Juncong Lin, Minghong Liao, Guoliang Luo, and Hongbo Fu. Dispad: Flexible on-body displacement of fabric sensors for robust joint- motion tracking. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 7(1):1-27, 2023.
[6] Takao Someya and Masayuki Amagai. Toward a new generation of smart skins. Nature biotechnology, 37(4))382 3888 2019.
[7] Yiqiang Chen, Jindong Wang, Meiyu Huang, and Han Yu. Cross-position activity recognition with stratified transfer learning. Pervasive and Mobile Computing, 57:1-13, 2019.
[8] Jindong Wang. Yiqiang Chen, Han Yu, Meiyu Huang, and Qiang Yang. Easy transfer learning by exploiting intra-domain structures. In 2019 IEEE international conference on multimedia and expo (ICME), pages 1210-1215. IEEE, 2019.
[9] Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE conference on computer vision and pattern recognition, pages IEEE, 2012.
[10] Jindong Wang, domain adaptation with manifold embedded distribution alignment. In Proceedings of the 26th ACM international conference on Multimedia, pages 402-410, 2018.
Wenjie Feng, Yiqiang Chen, Han Yu, Meiyu Huang, and Philip Yu. Visual
[11] Jindong Wang, Yiqiang Chen, Shuji Hao, Wenjie Feng, and Zhiqi Shen. Balanced distribution adaptation for transfer learning. In 2017 IEEE international conference data mining (ICDM), pages 1129-112 1134. IEEE, 2017.
[12] Jindong Wang. Yiqiang Chen, Lisha Hu. Xiaohui Peng, and S Yu Philip. Stratified transfer learning for cross-domain activity recognition. In 2018 IEEE international conference on pervasive computing and communications (PerCom), pages 1-10. IEEE, 2018.
[13] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in neural information processing systems, 27, 2014.
[14] Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474, 2014.
[15] Arthur Gretton, Dino Sejdinovic, Heiko Strathmann, Sivaraman Balakrishnan, Massimiliano Pontil, Kenji Fukumizu, and Bharath K Sriperumbudur. Optimal kernel choice for large-scale two-sample tests. Advances in neural information processing systems, 25, 2012.
[16] Baochen Sun and Kate Sacnko. Deep coral: Correlation alignment for deep domain adaptation. In European conference computer vision, pages 443-450. Springer, 2016.
[17] Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
11

---

[18] Chao Chen, Zhihang Fu, Zhihong Chen, Sheng Jin, Zhaowei Cheng, Xinyu Jin, and Xian- Sheng Hua. Homm: Higher-orden moment matching for unsupervised domain adaptation. In Proceedings of the AAAI conference artificial intelligence, volume 34, pages 3422-3429, 2020.
[19] Yuang Liu, Wei Zhang, and Jun Wang. Source-free domain adaptation for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition,
pages 1215-1224, 2021.
[20] Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Model adaptation: Historical contrastive learning for unsupervised domain adaptation without source data. Advances in Neural Information Processing Systems, 34:3635- 3649,, 2021.
[21] Jonghyun Lee, Dahuin Jung, Junho Yim, and Sungroh Yoon. Confidence score for source-free unsupervised domain adaptation. In International Conference on Machine Learning, pages 12365-12377. PMLR, 2022.
[22] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pages 1180-1189. PMLR, 2015.
[23] Chaohui Yu, Jindong Wang. Yiqiang Chen, and Meiyu Huang. Transfer learning with dynamic adversarial adaptation network. In 2019 IEFE International Conference on Data Mining (ICDM), pages 778-786. IEEE, 2019.
[24] Chao Chen, Zhihong Chen, Boyuan Jiang, and Xinyu Jin. Joint domain alignment and discrimi- native feature learning for unsupervisee deep domain adaptation. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3296-3303, 2019.
[25] Jingfeng Zhang, Jianing Zhu, Gang Niu, Bo Han, Masashi Sugiyama, and Mohan Kankanhalli. Geometry-awe aware rnstihee adversarial training. arXiv preprint arXiv:2010.01736. 2020.
[26] Zhun Deng, Linjun Zhang, Kailas Vodrahalli, Kenji Kawaguchi, and James Zou, Adversarial training helps transfer learning via better representations. Advances in Neural Information Processing Systems, 34:251 79-25191, 2021.
[27] Qiao Liu and Hui Xue. Adversarial spectral kernel matching for unsupervised time series domain adaptation. In IJCAI, pages 2744-2750, 2021.
[28] Yang Li, Hong-Ning Dai, and Zibin Zheng. Selective transfer learning with adversarial training for stock movement prediction. Connection Science, 34(1):492-510, 2022.
[29] Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1):43-76, 2020.
[30] Long Chen, Haizhou Ai, Rui Chen, Zijie Zhuang, and Shuang Liu. Cross-view tracking for multi-human 3d pose estimation at over 100 fps. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3279-332233 3288, 2020.
[31] Dushyant Mehta, Oleksandr Sotnychenko, Franziska Mueller, Weipeng Xu, Mohamed Elgharib, Pascal Fua, Hans-Peter Seidel, Helge Rhodin, Gerard Pons-Moll, and Christian Theobalt. Xnect: Real-time multi-person 3d motion capture with single rgb camera. Acm Transactions On Graphics (TOG), 39-4) 2020.
[32] Martin Schepers, Matteo Giuberti, Giovanni Bellusci, et al. Xsens mvn: Consistent tracking of human motion using inertial sensing. Xsens Technol, 1(8), 2018.
[33] Yinghao Huang, Manuel Kaufmann, Emre Aksan, Michael Black, Otmar Hilliges, and Gerard Pons-Moll. Deep inertial poser: Learning to reconstruct human pose from sparse inertial measurements in real time. ACM Transactions on Graphics (TOG), 37(5) 2018.
[34] Xinyu Yi, Yuxiao Zhou, and Feng Xu. Transpose: real-time 3d human translation and pose estimation with six incrtial sensors. ACM Transactions Graphics (TOG), 40(4):1-13, 2021.
12

---

[35] Oliver Glauser, Shihao Wu, Daniele Panozzo, Otmar Hilliges, and Olga Sorkine- Hornung. Interactive hand pose estimation using a stretch-sensing soft glove. ACM Transactions on Graphics (ToG), 38(4):1-15, 2019.
[36] Yulong Ma, Jingyu Ouyang, Tahir Raza, Pan Li, Aijia Jian, Zengging Li, Hong Liu, Min Chen, Xueji Zhang, Lijun Qu, et al. Flexible all-textile dual tactile-tension sensors for monitoring athletic motion during tackwondo. Nano Energy, 85:10594,, 2021.
[37] Bo Zhou, Daniel Geissler, Mare Faulhaber, Clara Elisabeth Gleiss, Esther Friederike Zahn, Lala Shakti Swarup Ray, David Gamarra, Vitor Fortes Rey, Sungho Suh, Sizhen Bian, et al. Mocapose: Motion capturing with textile-integrated capacitive sensors in loose-fitting smart gar- ments. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 7(1):1-40, 2023.
[38] Quan Zhang, Tao Jin, Jianguo Cai, Liang Xu, Tanyiyi He, Tianhong Wang, Yingzhong Tian, Long Li, Yan Peng, and Chengkuo Lee. Wearable triboelectric sensors enabled gait analysis and waist motion capture for iot-based smart healthcare applications. Advanced Science, 9(3)220,3,,, 2022.
[39] Zekun Liu, Tianxue Zhu, Junru Wang, Zijian Zheng, Yi Li, Jiashen Li, and Yuekun Lai. Functionalized fiber-based strain sensors: Pathway to next-generation wearable electronics. Nano-Micro Letters, 14(1):1-39, 2022.
[40] Binbin Ying, Ryan Zeyuan Chen, Runze Zuo, Jianyu Li, and Xinyu Liu. An anti-freezing, ambient-stable and highly stretchable ionic skin with strong surface adhesion for wearable sensing and soft robotics. Advanced Functional Materials, (142)22104466, 2021.
[41] Ying Jin, Guoning Chen, Kete Lao, Songhui Li, Yong Lu, Yufeng Gan, Zhundong Li, Jiajia Hu, Jingcheng Huang. Jinxiu Wen, et Identifying human body states by using a flexible integrated sensor. npj Flexible Electronics, 4(1):1-8, 2020.
[42] Jean Won Kwak, Mengdi Han, Zhaoqian Xie, Ha Uk Chung, Jong Yoon Lee, Raudel Avila, Jessica Yohay, Xuexian Chen, Cunman Liang, Manish Patel, et al. Wireless sensors for continuous, multimodal measurements at the skin interface with lower limb prostheses. Science translational medicine, x2a$44):a 2020.
[43] Zamir Ahmed Abro, Zhang Yi-Fan, Chen Nan-Liang, Hong Cheng-Yu, Rafique Ahmed Lakho, and Habiba Halepoto. A novel flex sensor-based flexible smart garment for monitoring body postures. Journal of Industrial Textiles, 49(2):262-274, 2019.
[44] Miao Yu, Jiaqin Jin, Xia Wang. Xu Yu,, Dingjia Zhan, and Jia Gao. Development and design of flexible sensors used in Journal, 21(22):2540-225 25402,550820 2021.
sports pants for human knee joints. IEEE Sensors
[45] Yi Tan and Limao Zhang. Computational methodologies for optimal sensor placement in structural health monitoring: A review. Structural Health Monitoring, 19(4):1287-133) 2020.
[46] Tao Wu, Panlong Yang, Haipeng Dai, Wanru Xu, and Mingxue Xu. Charging oriented sensor placement and flexible scheduling in rechargeable wsns. In IEEE INFOCOM 2019-IEEE Conference on Computer Communications, pages 73-81. IEFF, 2019.
[47] Tao Wu, Panlong Yang, Haipeng Dai, Chaocan Xiang, and Wanru Xu. Optimal charging oriented sensor placement and flexible scheduling in rechargeable wsns. ACM Transactions on Sensor Networks (TOSN), 18(3):1-27, 2022.
[48] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning, pages 5301-5310, PMLR, 2019.
[49] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pages 97-105. PMLR, 2015.
[50] Yongchun Zhu, Fuzhen Zhuang, Jindong Wang, Guolin Ke, Jingwu Chen, Jiang Bian, Hui Xiong, and Qing He. Deep subdomain adaptation network for image classification. IEEE transactions neural networks and learning systems, 1714)1171-1 2020.
13


Reviews:
[{"review_id": "ZdkE4a7VFJ", "reviewer": "Reviewer_ncj6", "summary": "This paper presents an approach to self-adaptive motion tracking using on-body, flexible sensors that in real world applications may be subject to displacements. The authors present a network that contains a component that automatically learns an affine transformation between training data (with annotations in form of sensor data with their corresponding target joint angles), a Fourier encoded LSTM part for the actual pattern recognition, and a bespoke loss function that uses auxiliary regressors for automated, unsupervised tuning of the parameters of the aforementioned affine transformation. The authors evaluate their method in practical experiments where participants wore elbow sleeves that contain flexible sensors and performed various bend movements. The objective of the evaluation is to measure the accuracy of the predicted joint angles for motion tracking applications. Comparisons to related SOTA models are made and it is shown that the proposed method outperforms all other analyzed models in terms of much better accuracy (distance from ground truth). An ablation study sheds light on the contributions of the individual components of the network and the supplementary material provides details on the implementations, data characteristics etc. \n", "questions": "See my comment above re limited impact beyond the particular application domain. I wonder if the authors could elaborate on this a bit. \n\nAlso, see my concern about the evaluation. Can the authors provide more information about their participants?\n", "limitations": "Discussion on limitations is there and ok.\n", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "This paper is very well written, technically sound, and the experimental results are convincing. I applaud the author for what seems like rigorous work and great thoroughness in execution. Well done. The papers is very application driven, which is fine of course. The tracked problem is relevant for the targeted application domain of motion tracking using body-worn sensors. As such, I can see that the paper may have impact in this field. The authors carefully designed a technical solution in form of a network that specifically, and successfully, addresses domain specific aspects. The experimental evaluation is relevant at least in parts and comparisons to related work / SOTA are done in a very careful and thorough manner. The reported results look promising. \n", "weaknesses": "The experimental evaluation looks a bit limited. While I appreciate the range of displacement that were tackled, the number of participants is a bit small, and so are their individual recordings. I would imagine that the displacements are dependent on a number of factors including the physique of the participants and the tasks performed. Not much is said about either of these and I fear that the promising results may not (or maybe they will — hard to tell!) generalize too much.\n\nOn a higher level note: “application papers” typically have a more challenging time at NeurIPS. I do not want to discount the paper merely by the fact that it is mainly an application paper. However, the presented method is very much tailored to motion tracking and I cannot see how the method itself would generalize beyond this. This is not a criticism per se but rather an assessment of the potential impact that paper might have to the broader NeurIPS audience, which would judge as rather limited. I wonder if the authors could branch out their method beyond the specific motion tracking application here? They argue through domain shifts and such, which is a hot topic in the field, but I fail to see how this domain adaptation etc. community would gain much insight from this paper.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "sLwHuDKtxb", "reviewer": "Reviewer_f2Et", "summary": "Flexible sensors are useful for tracking human status as wearable systems, but they can become displaced when worn, causing challenges for machine learning algorithms. The proposed solution of this paper is a self-adaptive motion tracking network that includes a learnable Affine Transformation layer, a Fourier-encoded LSTM network for pattern identification, and a sequence discrepancy loss with auxiliary regressors for unsupervised tuning of Affine Transformation parameters.", "questions": "Please answer 2 of the weakness", "limitations": "Please answer 2 of the weakness", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method of this paper is useful for real-world applications.\n2. The proposed method is intuitive and easy to follow.\n3. The experiment results show the effectiveness of their method.", "weaknesses": "1. This paper mainly works for the real-world sensor. I would like to know how it works in real-world applications. \n2. In the experiments, the author should compare more advanced methods, for example, the least method the author compared is published in 2020. I would like to know how this framework works together with state-of-the-art modules, for example, transformer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "hIxAtrRbkz", "reviewer": "Reviewer_fsrB", "summary": "This paper shows an approach for adaptive learning to track motion trajectories from elbow pad sensor, especially concentrating on modelling of sensor displacements in unsupervised manner during the operation. Tracking method is based on multi-layer neural network architecture with learnable affine layer, Fourier feature encoded LSTM, and multiple outputs regression layers. Furthermore, novel sequence discrepancy loss is introduced to reduce data shift caused by sensor displacements. Proposed techniques are evaluated on collected real datasets with ablation study and comparison to other domain adaptation methods from the literature.", "questions": "- How well the prior SOTA methods evaluated in the paper compares to particular time-series type of domain shift?\n- What are the boundaries of sensor displacement (i.e., how much noise or data shifts method can handle)?", "limitations": "Some of the limitations of proposed approach are shortly analysed in relation to specific issues such as data quality.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This is application oriented paper with novel combination of existing techniques (i.e., affine transformation layer for data shift, Fourier LSTM for pattern matching, and sequence discrepancy loss with aux. regressor for unsupervised/self-adaptive learning) for specific problem, which is evaluated with SOTA comparison and ablation study to work in practice (see also weaknesses). Paper is well-structured and its main processing pipeline is illustrated with related formulation.\n\nSummary of strengths\n- Practical solution for (very specific) problem in domain adaptation\n- Good combination of different techniques for new application \n- Empirical evaluation with comparison and ablation study\n- Clearly illustrated processing pipeline and formulation", "weaknesses": "Paper only considers very specific problem with one type of sensor and small dataset. It is difficult to assess if this could generalise to different settings. Although there are comparison to SOTA models, paper lacks some basic baseline from signal processing field, e.g., low dimensional data could be estimated using state-space models such as extended kalman or particle filters or at least give more justification of need for learnable methods with adaptive capabilities. Also, most of the SOTA methods presented in the paper are designed for the image data domain adaptation, not for low-dimensional time-series data, so there could be some more related approaches. In addition to utilisation of more comprehensive dataset, paper lack detailed analysis of boundaries in relation to how much noise and sensor displacement it could handle.\n\nSummary of weaknesses \n- Very narrow application of domain adaptation (could it be extend to other settings or problems)\n- Small datasets (one person, four in supplement)\n- Lack of basic baseline (e.g., state space models (EKF, PF etc.))\n- Missing the study of data noise and/or boundaries for amount of sensor displacement handled \n- Most of the compared SOTA methods designed for different data (e.g., images)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "vhn8DqzUVc", "reviewer": "Reviewer_Bo9S", "summary": "In the context of ubiquitous sensing, the paper addresses the issue that on-body devices cannot be firmly worn at a fixed position across different sessions by adapting to unknown displacements. The authors propose three main contributions: (1) A transformation layer adapts to unknown displacements, (2) an LSTM networks identifies patterns, and (3) a discrepancy loss for unsupervised learning.", "questions": "none.", "limitations": "The limitation of this work is that it is not clear whether the proposed approach can be applied to other time series domain adaptation applications or the method is specifically designed for this on-body displacement of flexible sensors and overfitted on these dataset.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The problem setup is well motivated. The addressed problem exists in many application areas and research fields.\n- The paper is well written and easily understandable.\n- The soundness of the technical claims are good and the experiments support the proposed method.\n- The authors will make the datasets and source code publicly available.\n", "weaknesses": "\n- Section 2.3 contains introductory applications that is repeated from Chapter 1, and hence, can be integrated in the Introduction.\n- Experiments are only performed on this specific dataset. It is not clear, whether the proposed method can be generalized to other applications, e.g., on datasets such as HHAR, UCI HAR, WISDM, and uWave (time series domain adaptation datasets).\n- Experiments in Sec. 5.1 should also cover data from more than one participant\n- References are missing and state-of-the-art methods on which experiments are performed are limited. The following methods can also be considered for comparison: MMCD [1], MMDA [2], DDC [3], DAN [4], HoMM [5], CDAN [6], DIRT-T [7], CoDATS [8], AdvSKM [9], and Sinkhorn methods [10,11]. \n\nIn my opinion, the paper needs an extensive re-write if published at an ML-venue such as NeurIPS. From a methodological perspective the main contribution is the novel sequence discrepancy loss (as (1) learnable affine transformations and (2) Fourier-encoded LSTMs or similar ideas already exist [12]). The paper should be re-written around this main contribution and the on-body placement use-case should be one dataset of many time series data set where the method should be applied on.\n\n**Some minor points:**\n- L34ff: But if the features are transformed accurately they still share the same feature space, right?\n- L101: it is uncommon to name a specific journal of a citation within the main text\n- L115: increase[s]\n\n**Missing references:**\n- [1] Naimeh Alipour and Jafar Tahmoresnezhad: Heterogeneous Domain Adaptation with Statistical Distribution Alignment and Progressive Pseudo Label Selection. In Applied Intelligence, volume 52, pp. 8038–8055, October 2021. doi: 10.1007/s10489-021-02756-x.\n- [2] Mohammad Mahfujur Rahman et al.: On Minimum Discrepancy Estimation for Deep Domain Adaptation. In Domain Adaptation for Visual Understanding, Springer, Cham., January 2020. doi:10.1007/978-3-030-30671-7_6.\n- [3] Eric Tzeng et al.: Deep Domain Confusion: Maximizing for Domain Invariance. In arXiv preprint arXiv:1412.3474v1 [cs.CV], December 2014.\n- [4] Mingsheng Long et al.: Learning Transferable Features with Deep Adaptation Networks. In Proc. of the Intl. Conf. on Machine Learning (ICML), volume 37, pp. 97–105, July 2015. doi: 10.5555/3045118.3045130.\n- [5] Chao Chen et al.: HoMM: Higher-Order Moment Matching for Unsupervised Domain Adaptation. In Proc. of the AAAI Conf. on Artificial Intelligence (AAAI), volume 34(4), pp. 3422–3429, April 2020. doi: 10.1609/aaai.v34i04.5745.\n- [6] Mingsheng Long et al.: Conditional Adversarial Domain Adaptation. In Advances of Neural Information Processing Systems (NIPS), volume 31, pp. 1647–1657, December 2018. doi: 10.5555/3326943.3327094.\n- [7] Rui Shu et al.: A DIRT-T Approach to Unsupervised Domain Adaptation. In Intl. Conf. on Learning Representations (ICLR), 2018.\n- [8] Garrett Wilson et al.: A Survey of Unsupervised Deep Domain Adaptation. In Proc. of the ACM Trans. on Intelligent Systems and Technology (TIST), volume 11(5), pp. 1–46, October 2020. doi: 10.1145/3400066.\n- [9] Qiao Liu and Hui Xe. Adversarial Spectral Kernel Matching for Unsupervised Time Series Domain Adaptation. In Proc. of the Intl. Joint Conf. on Artificial Intelligence (IJCAI), pp. 2744–2750, August 2021. doi: 10.24963/ijcai.2021/378.\n- [10] Felix Ott et al.: Domain Adaptation for Time-Series Classification to Mitigate Covariate Shift. In Proceedings of the ACM International Conference on Multimedia (ACMMM), pages 5934–5943, Lisboa, Portugal, October 2022. doi:10.1145/3503161.3548167.\n- [11] Huan He et al.: Domain Adaptation for Time Series Under Feature and Label Shift. arXiv preprint arXiv:2302.03133, February 2023.\n- [12] Zhang et al.: Learning Long Term Dependencies via Fourier Recurrent Units. ICML 2018.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:
# Reviewer #ncj6
## Q1. Limited participants / recording length / physique of participants / task.
To further demonstrate the generalization ability of our method, we conducted additional experiments with five new participants of varying body types (see Table 1 in the uploaded pdf file).
Each participant wore the devices in three on-body positions and performed three distinct physical activities sequentially: __ping-pong__, __basketball__, and __boxing__. In total, we collected 15 (5 participants × 3 on-body displacements) unique data segments across all participants, comprising 81,848 total frames.

Table 2 in the uploaded PDF shows the experimental results of our method on the aforementioned dataset, including two additional SOTA methods, HoMM [1] and AdvSKM [2], as recommended by Reviewers f2Et, fsrB, and Bo9S.
It can be observed that the proposed method demonstrates the lowest average error across data from five users, supporting its applicability to varied user profiles and movement patterns.

## Q2. Branch out proposed method beyond the specific motion tracking application?
To demonstrate the generalizability of our method beyond the task in our paper, we conducted further experiments on the DIP-IMU dataset [3], which includes IMU sensor data from 10 subjects executing diverse motions, along with corresponding 3D human body poses. The objective is to estimate 3D rotations of 15 SMPL model joints using only six IMU readings from head, pelvis, hands, and feet. Similar to DIP [3], we employed an LSTM network for this, yielding 15 joint rotations in axis-angle notation as output. 
We trained the model on data from 8 of the 10 subjects, reserving the remaining 2 subjects' data as test set.

As shown in Table 3 in the uploaded PDF, the proposed method achieves slightly improved performance compared to the SOTA HoMM and AdvSKM methods. This demonstrates the strong generalizability of our approach across different input modalities and sensing technologies.

# Reviewer #f2Et
## Q1. How it works in real-world applications.
Our method can be applied in real-world scenarios through the following steps:
1) To begin, users wear the device and perform a flexed-arm calibration pose.
2) Using the collected sensor data, our method fine-tunes the model to adapt to the present on-body device displacement.

## Q2. Comparison to more advanced methods.
Thanks for the suggestion and please see Table 2 in the uploaded PDF for the comparison to two more advanced methods, HoMM [1] and AdvSKM [2], which further demonstrates the superiority of the proposed method.

# Reviewer #fsrB
## Q1. Narrow application of domain adaptation.
Please kindly review our reply to Reviewer #ncj6, Q2, and Table 3 in the provided PDF, showcasing our method's enhanced performance on IMU data. This underscores our approach's robust adaptability across diverse input modes and sensing technologies.
## Q2. Small datasets.
We performed extra experiments on five new participants. Please refer to our response to Reviewer #ncj6, Q1, and Tables 1 and 2.

## Q3. Missing the study of data noise and/or boundaries for amount of sensor displacement handled.
Thank you for your suggestion. We computed SNR for the sensor signals and aggregated MAE for each set of data. As shown in Fig 1 (in the uploaded PDF), our method demonstrates significant efficacy for SNR > 10 dB, while achieving satisfactory outcomes becomes challenging for SNR < 10 dB.
## Q4. Most of the compared SOTA methods designed for different data.
Thank you for your suggestion! To make a fairer comparison, we included comparison results against AdvSKM [2], witch specifically designed for time series data. Please see Table 2 for more details.
# Reviewer \#Bo9S
## Q1. Experiments are only performed on this specific dataset.
Thank you for your comment. Please see our response to Reviewer #ncj6, Q2 and Table 3 in the uploaded PDF, which demonstrates that our method also achieves improved performance on inertial measurement unit (IMU) data. 
we carefully assessed the suggested datasets, discovering their focus on classification tasks that our method does not currently accommodate. Hence, we are unable to gauge our approach's performance on these specific datasets.
## Q2. Experiments in Sec. 5.1 should also cover data from more than one participant.
Thank you for your constructive suggestion. We have conducted preliminary experiments with additional participants, and the results indicate that our conclusions remain valid. We will include the updated results from these expanded experiments in the revision.
## Q3. Missing SOTAs.
Thanks for the suggestion and please see Table 2 for the comparison with two suggested SOTAs, HoMM and AdvSKM, which further demonstrates the superiority of the proposed method. 
Please note that HoMM and AdvSKM are the two most suitable benchmark methods for our task, which involves unlabeled target domain data and regression modeling.
## Q4. Paper organization.
To clarify, as Reviewer #ncj6 noted, ours is an ''application paper focused on solving an important real-world problem, rather than a ''methodology paper'' aiming to develop a new algorithm applicable to multiple potential applications.
This is within the remit of NeurIPS 2023 Call for Papers,'' Applications (e.g., vision, language, speech and audio)''.

## Q5. Missing references and minor issues.
Thanks for the suggestions. We will include all references in our revision and correct the minor issues.

# References
[1] Chen, Chao, et al. "HoMM: Higher-order moment matching for unsupervised domain adaptation." Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 04. 2020.

[2] Liu, Qiao, and Hui Xue. "Adversarial Spectral Kernel Matching for Unsupervised Time Series Domain Adaptation." IJCAI. 2021.

[3] Huang, Yinghao, et al. "Deep inertial poser: Learning to reconstruct human pose from sparse inertial measurements in real time." ACM Transactions on Graphics (TOG) 37.6 (2018): 1-15.


Meta-review: Overall this is a borderline paper trending toward accept.  In summary. all reviewers believe that this is a technically sound paper but with limited or specialized interest to the NeurIPS community.  In discussion with the reviewers, no reviewer is entirely against it and because it would bring technical diversity to the conference I would like to see it accepted.
Decision: Accept (poster)