UniMTS: Unified Pre-training for Motion Time Series
Authors: Xiyuan Zhang
Diyan Teng
Ranak Roy Chowdhury
Shuheng Li
Dezhi Hong
Rajesh K. Gupta
Jingbo Shang
Venue: NeurIPS 2024 poster (2024)
Keywords: motion time series classification
pre-training
contrastive learning
physics-based simulation
human activity recognition
Primary area: machine_learning_for_healthcare
TL;DR: We present the first unified pre-training procedure for motion time series that can generalize to diverse device locations, orientations, and activity types.

Abstract: Motion time series collected from low-power, always-on mobile and wearable devices such as smartphones and smartwatches offer significant insights into human behavioral patterns, with wide applications in healthcare, automation, IoT, and AR/XR. However, given security and privacy concerns, building large-scale motion time series datasets remains difficult, hindering the development of pre-trained models for human activity analysis. Typically, existing models are trained and tested on the same dataset, leading to poor generalizability across variations in device location, device mounting orientation, and human activity type. In this paper, we introduce UniMTS, the first unified pre-training procedure for motion time series that generalizes across diverse device latent factors and activities. Specifically, we employ a contrastive learning framework that aligns motion time series with text descriptions enriched by large language models. This helps the model learn the semantics of time series to generalize across activities. Given the absence of large-scale motion time series data, we derive and synthesize time series from existing motion skeleton data with all-joint coverage. We use spatio-temporal graph networks to capture the relationships across joints for generalization across different device locations. We further design rotation-invariant augmentation to make the model agnostic to changes in device mounting orientations. Our model shows exceptional generalizability across 18 motion time series classification benchmark datasets, outperforming the best baselines by 340% in the zero-shot setting, 16.3% in the few-shot setting, and 9.2% in the full-shot setting.

UniMTS: Unified Pre-training for Motion Time Series
Xiyuan Zhang UC San Diego
Diyan Teng
Qualcomm diyateng@qti
xiyuanzh&ucad.ed edu
com
Ranak Roy Chowdhury
Shuheng Li UC San Diego
Dezhi Hong*
UC San Diego rrchoudh@ucsd.co edu
Amazon
sh1060@ucsd.cm edu
hondezhi Camazon. com
Rajesh K. Gupta
Jingho Shang
UC San Diego
UC San Diego
rgupta@ucsd. edu
shanggucsd. edu
Abstract
Motion time series collected from low-power, always-on mobile and wearable devices such as smartphones and smartwatches offer significant insights into human behavioral patterns, with wide applications in healthcare, automation, IoT, and AR/XR. However, given security and privacy concern,, building large-scale motion time series datasets remains difficult, hindering the development of pre-trained models for human activity analysis. Typically, existing models are trained and tested on the same dataset, leading to poor generalizability across variations in device location, device mounting orientation, and human activity type. this paper, we introduce UniMTS[] the first unified pre-training procedure for motion time series that generalizes across diverse device latent factors and activities. Specifically, we employ contrastive learning framework that aligns motion time series with text descriptions enriched by large language models. This helps the model learn the semantics of time series to generalize across activities. Given the absence of large-scale motion time series data, we derive and synthesize time series from existing motion skeleton data with all-joint coverage. We use spatio-temporal graph networks to capture the relationships across joints for generalization across different device locations. We further design iiaationinvaaiant augmentation make the model agnostic to changes in device mounting orientations. Our model shows exceptional generalizability across 18 motion time series classification benchmark datasets, outperforming the best baselines by 340 % the zero-shot setting, 16.3% in the few-shot setting, and 9.2 % in the full-shot setting.
1 Introduction
Recognition of human motion using time series from mobile and wearable devices, such as accelera- tions and angular velocities, is widely adopted as key context information for various applications from health condition monitoring 4 sports activity analysis I user habit studies (50). Compared with vision- based approaches, methods based on motion sensor time series offer more energy-efficient and cost-effective solutions with enhanced privacy protection [54] making them preferable.
'Work unrelated to Amazon. 'Code is available Github: httpp:///iithub com/stiyuanzzh/UhiitS Model is available Hugging Face: htttps://uugiinfface co/ wttp:///w...oocccaaa//aa//aaaa
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Challenges Given Insufficient
Training Activities
Real-world
Training
Wrist Smartwwachh
Deployment Activities
Deployment Orientation
OOOO
Generalization across
Locations
Generalization across Activities
Upper 1og Suaapponne
Generalization across
Mounting Orientations
Figure 1:
Our framework addresses three key generalization challenges (variation in device location, orientation, and activity) where existing methods fall short.
While valuable, collecting motion time series data at large scale remains challenging due to security privacy concerns. Labeling motion time series proves even more difficult as such data cannot be easily interpreted by humans for post annotation. This results in data insufficiency that impedes development of supervised learning methods. In other fields such as natural language processing (39 56 and
pre-trained foundation models have shown remarkable performance in such settings with insufficient data. However, in the motion time series domain, lack of comprehensive datasets and effective pre-training task makes it difficult to similarly develop pre-trained models that can operate with limited data. Typically, existing models perform training and testing on the same dataset, and struggle to generalize across different datasets given the following three unique challenges within the motion time series problem domain.
computer vision 44 30
We summarize these three unique generalization challenges in Figure First of all, variation in device placement during deployment poses a significant issue; for instance, data from a smartwatch on the wrist vary considerably from data gathered from a smartphone near the upper leg. Therefore, models trained on data from one body location can barely generalize to others during the testing experience arbitrary orientations during data collection, making
phase. Secondly, devices difficult for models trained on specific device orientations to adapt to new ones during deployment. Thirdly, different motion time series datasets can be focused on different types of human activities. For example, some datasets aim to identify stationary activities such as lying or sitting, while others concentrate on dynamic movements such walking or cycling. Models trained on specific types of activities typically struggle to generalize to new activities introduced by other datasets.
We introduce UniMTS, the first Unified pre-trained model for Motion Time Series to address all the above three generalization issues, achieving state-of-the-arr zero-shot and fine-tuning performance. UniMTS follows contrastive learning framework that aligns motion time series with I.LMMenriched textual descriptions to learn the time series semantics for activity generalization. To prepare large- scale motion time series for pre-training, we synthesize these time series based on existing extensive motion skeleton data [19] with comprehensive coverage of different body locations. We model these synthesized time series using graph networks to capture the spatio-temporal relationships across devices for location generalization. We further implement rotation-invarial augmentation to ensure the model's robustness to any device orientation during testing.
We summarize our primary contributions as follows:
We introduce the first unified pre-training procedure for motion time series, UniMTS, which successfully generalizes to various device locations, device orientations and activities.
We design contrastive learning framework to align motion time series with corresponding semantic meanings for activity generalization. For device location generalization, we propose to synthesize motion time series covering various body locations and model their spatio-temporal correlations using graph convolutional neural networks. We also design rotation-invarit augmentation to make the model agnostic to different device orientations.
Our pre- pre-trained model demonstrates state-of of of-the-art performance across 18 real-world motion time series benchmark datasets, notably with performance improvement of 340 % in the zero-shot setting, 16.3% in the few-shot setting, and 9.2% in the full-shot setting, compared with the respective perf-pegg baselines.
Related Work
Conventional motion time series classification approaches train dedicated classifier for each dataset, and can be categorized into statistical feature extraction methods [15] and deep learning

---

dp d
Pre-training
person moves slowly ahead and subsequently turns to the rightt
71
12
35
wwww
Text Encoder
mrraa a
Text Augmentation
person walks slowly and than makes right turn
Graph Encoder
Motion
Rotation Invariance
Skeleton
Physies Engine
Contrastive Learning
Motion Descriptions
I.I.M
The physics engine computes motion time series for each joint based on motion skeleton data and enhances time series through ootation-invariant augmentation. During pre-training, we adopt contrastive learning align motion time series encoded by graph convolutional neural networks with corresponding text descriptions augmented by an LLM.
Figure UniMTS pre-training framework:
methods, including convolutional neural networks (MA-CNN 45 SenseHAR 23 Rocket [2] )
recurrent neural network (DeepConvL.STY) (40). and the attention mechanism based models (At-
tnSense [36]. THAT (291). Recently, IMUGPT 228 27 generates motion sequences given activity textual descriptions and trains conventional classification models such DeepConvLSTM [40].
TimesNet [60] GPT4TS 722 and TEST [52
propose task-general time-series models for multiple tasks including classification. SHARE 7 presents a sequence-to-sequeee framework that leverages shared structures of label names. However, these models perform training and testing on the same dataset, and cannot generalize across datasets.
Self-supervised motion time series representation learning methods first learn time series represen- tations based on mask reconstruction (TST 688, TARNet [1]. LIMU-BERT (62) contrastive learn-
ing (TNC (55). TS-TCC [1,, TS2Vec (67 TF-C 70, FOCAL
32 CL.-HAR 42 DDLeamn 43)
or other self-supervised learning objectives (BioBankSSL [66
13 [0]. Step2Heart (5I)). Subse-
quently, they fine-tune classifier heads for specific downstream tasks. However, the representation learning and fine-tuning phases of these methods generally occur the same highly similar datasets, which continues to face challenges in generalization across diverse datasets.
Pre-trained models for motion time series are inspired by the recent success of large language or multimodal models. ImageBind [16] and IMU2CLIP 38 leverage recent large vision-lann languge models [44] to learn joint embedding across multiple modalities including motion time series and text. However, both ImageBind and IMU2CLIP are trained on motion time series collected from head-mounted devices [17). limiting their generalizability across different device locations and orientations. Furthermore, several studies have explored directly applying LLMs for motion time series classification. For example, HARGPT [44] processes motion time series through LLMs and incorporates role-play and chain-of-thought strategies for prompting. ContexIGPT designs prompt engineering approaches leveraging context information. However, since LLMs are not directly trained on raw motion time series, such methods require extensive context information that is not usually available, and struggle with accurately recognizing complex activities.
Other related works on motion classification include multimodal action recognition and domain adaptation methods. Multimodal action recognition such as the Ego4D [17] and Ego-Exo4D [T8 benchmarks incorporates video and audio modalities, whereas we focus on a more enery--efficiem and challenging scenario of action recognition based purely on motion time series. Domain adaptation methods mostly assume that source and target datasets share the same label names and have the same number of classes, such cross-user domain adaptation and cross-dataset domain adaptation only those common classes [22] 35 21. We aim for a more generic yet challenging generalization scenario where pre-training and downstream datasets share different label names.
Method
UniMTS takes contrastive learning-basee approach that aligns paired motion time series with text descriptions to enable activity generalization, as shown in Figure 2 We simulate motion time series from motion skeleton data (Section 3.1 and augment them for orientation generalization (Section 3.2

---

We use graph encoder to model the simulated motion time series, capturing correlations among joints generalize across different device locations (Section 3.3.I) To enhance semantics learning, we use large language models to augment text descriptions (Section 3.3.2).
3.1 Physies Engine for Motion Time Series Simulation
Motion skeleton data [9] describe the movements of human skeleton joints over time, containing positions and orientations for each joint. On the other hand, motion time series captured by physical sensors typically measure higher-order data such as accelcrations and angular velocities. Conse- quently, we apply motion equations [65] to synthesize these time series of accelerations and angular velocities from motion skeleton data. More specifically, for each skeleton joint J,, we input both positions PJ..g (mapped from time domain T to R3 defined in global frame 9), and orientation quaternions qJ..cc (mapped from time domain T to the Special Orthogonal Group SO(3). defined in Hamilton convention with subscript GL representing a frame rotation from local frame L global frame G). We drop the subscript G and GL from here on for simplicity of notation. Based on motion equations [65. calculate velocities VJ, and accelerations a,, by taking the first and second order derivatives of positions PJ.- These derivatives are then transformed from global frames to local frames using the corresponding orientation sequences qu,- Similarly, angular velocities w., are computed by taking the first order derivatives of orientation quaternions qJ,- Mathematically,
v., (t) = 9., (t) @ P',, (t) @ qt)(t),
a., (1) q0000...00 (1) @ P (1) @ (t),
(2)
w 2q3, (1) (,,0),
where & and represent the quaternion multiplication operator and the quaternion conjugate.
Recognizing the inherent presence of noise carried by sensors practice, the physics engine incorpo- rates Gaussian noise with a zero mean into the simulated data. Representing the above motion time series as xt;(),, (t). which can denote either a., (t) (accelerations) or wJ, (t) (angular velocities), the noisy time series N,, (t) are formulated as
(t) (t) nJ, (t), ~((,o))
(4)
3.2 Rotation-Invarian Augmentation
A common limitation we have identified from prior studies that leads to their poor generalization is that they fail consider the impact of latent device orientation factors on the motion time series. For example, end users can potentially wear devices in various orientations, such as with phone facing towards or against the body in a pocket. Additionally, the software driver API for axis definition can be arbitrarily configured by the developers. For example, the iOS system defines acceleration in an opposite direction compared to the Android system With the listed risk factors considered, we apply data augmentation technique to simulate random orientations during pre-training, that our learned model achieves pre-training, for each iteration we sample random rotation matrix for each joint J/.
during deployment 77] 57 61]. Specifically, during
RJ, Uniform(SO(3),
(5)
and compute the augmented time series 84, at timesteps ( t= 1,2, 2,...,77
85. R
During one iteration, the same R is consistently applied to J, for every time series and every timestep / = 1,2.... . The rotation-invariant augmentation ensures that the simulated time series adaptable any downstream orientation, thereby enhancing the generalization capabilities.
3.3 Contrastive Learning
The physics engine generates sufficient motion time series data, which are subsequently encoded by graph networks and aligned with their corresponding text embeddings through contrastive learning.
md

---

Inference
Sitting
Fine- Tuning
Walkinng
Graph Encoder
Real signals at wrist
Walline
Real signals
Cycling
wrist
Encoder
Real signals
Real signals at ankle
ankle
The Proficted Activity ohe
Gruph Encoder
Text Encoder
Figure 3: Inference (left) and fine-tuning (right) phases of UniMTS. We assign real signals to the nearest location in the skeleton graph. During inference, we compute the similarity score between the graph embedding and each label candidate, and predict the with the highest score. During fine-tuning, we freeze the text encoder and update weights of the graph encoder and linear layer.
3.3.1
Graph Encoder
To capture the spatio-temporal correlations among different joints over time, we adopt spatio-temporal graph convolutional network [63] our motion time series encoder. We denote the initial input graph representation as follows,
g (V {*, 1, (0f==((s,,,
{(Ks, & (,,,, M0, 1(8), . -
Nodes contain skeleton joints with features X CXTVV where C,T, V represent the number of signal channels, temporal steps and joint nodes. Spatial edges E, connect adjacent nodes defined by the skeleton structure 7/ and temporal edges & connect temporally adjacent frames.
In practice, devices may not cover the complete joints but are rather positioned at arbitrary subsets of the complete joints. To simulate this, during each pre-training iteration, we randomly select a subset of joints and mask data from the remaining joints with zeros. We denote the mask at one iteration as M E RCTxVV where M, E RCXT if joint J, is selected, and M; if joint Jj is masked:
X X O M,
(8)
The graph convolution network Gs first computes the spatial output features as
Xout MKK P (X(A, A
where K2 denotes the spatial kernel size, A represents whether node xJ, belongs to the spatial convolution sampling subset St, of node xJ,, and A (A + a represents the normalized diagonal matrix, with set to 0.001 prevent empty rows 63 48 weights of the x convolution operation with C" denoting output channel dimension. Following spatial convolution,
4 " represents
further perform K, x temporal convolution the spatial output features Xout similar to classical convolution operations, where K; represents the temporal kernel size. The final graph representation Io (X) is derived by averaging features across both spatial and temporal dimensions with a graph average pooling layer at the end.
3.3.2 Text Encoder
To increase the diversity of paired text descriptions in the pre-training motion corpus 19). apply large language models (GPT-3.5) to augment original motion text descriptions with the following prompt template: The following one multiple descriptions are describing the same human activities: <motion descriptions>. Generate k paraphrases to describe the same activities.
We denote the original text descriptions combined with the LLM-augmented ones Y. We encode them using the same text encoder fo as CLIP [44), utilizing its pre-trained weights for initialization.
3.3.3 Training and Inference
During pre-training, maximize the similarities of paired simulated motion time series and text descriptions through contrastive learning:
B log
exp(sim (X.) (X)))
Lerr
(10)
Σ K
exp(sim (X.),
5

---

where B, y represent batch size and temperature parameter that controls distribution concentrations, and sim represents similarity score computed as inner product:
(g (X,), (Y)) = (o
(11)
We pre-train the graph and text encoders using simulated motion time series and augmented text descriptions. During inference, we evaluate the model on real-world motion time series, illustrated in the left part of Figure 3 For the text encoder, we input all label candidates. For the graph encoder, we assign real motion time series to the nearest joint the skeleton graph and assign zeros the remaining joints. The random mask M during pre-training emulates the zero mark-ig process. We compute the similarity score between the graph embedding with text embedding from each label candidate, and choose the label with the highest similarity score as the predicted activity.
We can further fine-tune the pre-trained model on downstream real-world data, as depicted in the right part of Figure 3 Specifically, we freeze the text encoder fo and update weights of the graph encoder Is followed by linear classifier hy- Following the same process as inference, we assign the real motion time series to the nearest joint in the skeleton graph and assign zeros to the remaining joints construct the graph input representation X. We fine-tune the model using X and one-hot encoded labels z with D classes based on cross-entropy loss, where o-) represents the softmax operation:
B
D
Le =
Zsy (xo))))))(
(12)
B
We report both zero-shot and fine-tuning performance in the subsequent experiment section.
4 Experiments
4.1 Datasets and Experimental Setting
We simulate motion time series from existing motion skeleton dataset HumanML3D 19) which contain both motion skeleton data and corresponding text descriptions as detailed in Section A.1 in Appendix. We further augment the text descriptions as described in Section 3.3.2
We evaluate the most extensive motion time series classification benchmark to date, comprising 18 real-world datasets that cover diverse activities. These datasets are collected from various body locations such as head, chest, back, arm, wrist, waist, hip, leg, knee and ankle. We categorize these datasets into three difficulty levels: (1) easy level (with fewer than 10 activities): Opportunity [47), UCI-HAR 12]. MotionSense [37], w-HAR (5). Shoaib [49], HAR70+ (58), RealWorld [53), TNDA-
HAR 664 (2) medium level (with 10 to 20 activities): PAMAP2 46) USC-HAD [69], Mhealth [4)
Harth 3 UT-Complex (50) Wharf (6, WISDM (59, DSADS M: (3) hard level (with more than
20 activities): UTD-MHAD 8 MMAct 26). We provide the specific number of activities for each
dataset in Table and Table 2 and detail their collection settings in Section A.2 in Appendix.
We re-sam the real-world test data to the same sampling frequency as the simulation data (20 Hz), and apply normalization to ensure consistency in unit measurements, e.g., standardizing accelerations to m/s² We pre-train UniMTS using Adam optimizer (55 with a learning rate of 0.0001 on a single NVIDIA A100 GPU. The pre-training process consumes approximately 13 GB of memory given a batch size of 64. For text augmentation, we prompt GPT-3.5 ("spt-3.5-turbo" to generate = 3 paraphrases. During each iteration, we randomly generate the mask M by selecting mask the remaining joints as zeros. We adopt learnable temperature parameter 7 initialized from CLIP. We evaluate the models using accuracy, macro-F1 and the top-2 retrieval performance R@2.
5 joints and
4.2 Zero-Shot Results
We pre-train UniMTS exclusively on simulated data and evaluate on 18 real-world motion time series classification benchmark datasets. We compare UniMTS against classification models with zero-shot capabilities: ImageBind [16], IMU2CLIP (38, IMUGPT (28] and HARGPT (24]. We also input
the 2D visualizations of motion time series to pre-trained lasion-language model LLaVA [30 for comparison. We detail the configurations of baselines in Section A.3 in Appendix. As shown in Table [] UniMTS significantly outperforms all baselines in the zero-shot setting. We also apply the Wilcoxon-signed rank test with Holm's 's a (5%) following previous works (20 [ The Wilcoxon-
signed rank test indicates that the improvement of UniMTS compared with all the basclines is

---

Table I: Zero-Shot performance. We bold the best and underline the second best. UniMTS performs the best compared with both baselines and our model ablations. The last column shows the average performance across 18 datasets with standard deviation.
Merrics
Opportuiity
UCIHAR
Senien
HAR
Stonah
HAR70+
Roal
INDA-RAR
PAMAP2
USC-HAD
Mhealth
Horth
UT-Comple
Whest
WISDM
DSADS
UTD-MHAD
MMAct
Average
Datasct
Nomber of Classes
6
7
8
8
12
12
13
14
18
19
27
35
Level
Easy
Medium
Hand
Avg
Acc
50.2
14.1
18.1
13.1
19.4
0.0
18.9
19.3
14.6
11.
7.9
9.8
9.7
3.2
7.1
2.1
3.3
2.9
12.5
ImageBind 16
F1
30.0
5.0
16.4
8.6
15.5
0.0
10.1
13.7
8.1
7.3
1.7
6.1
6.3
1.9
4.5
1.2
1.6
1.5
7.8
R@2
83.6
21.4
42.8
54.1
35.5
0.2
23.9
32.8
15.6
25.0
21.3
22.5
17.8
6.8
13.7
6.9
5.1
3.7
24.0
Acc
25.9
17.8
15.5
6.6
16.7
5.6
6.1
8.5
1.9
12.8
7.9
2.5
73
14
4.6
3.7
6.4
8.6sar
IMU2CLIP 38
F1
10.3
11.4
8.6
3.3
9.2
1.8
4.5
4.2
1.1
9.3
1.3
1.1
3.4
0.7
2.7
1.1
2.1
1.6
4.3
R@2
35.1
34.6
19.7
22.4
19.8
13.4
5.5
8.3
6.5
20.
Acc
10.1
1.1
118
67.2
12.4
0.0
16.9
14.3
8.9
6.0
9.8
4.8
11.6
2.7
8.3
7.5
3.7
2.0
11.4440
IMUGPT 28
F1
10.4
0.3
3.5
38.8
6.2
4.0
6.1
2.5
8.3
1.8
2.0
0.3
5.7
R@2
33.7
18.2
40.4
67.2
32.1
0.0
33.7
28.5
19.3
31.8
19.5
27.8
17.4
17.7
13.9
14.6
8.8
5.1
23.9(149)
Acc
28.8
15.0
11.6
4.9
21.0
34.3
12.7
13.7
11.1
9.5
10.4
28.8
7.5
5.9
5.5
5.8
3.3
2.3
12.9
HARGPT 24
F1
17.3
12.7
5.6
3.1
12,4
10.6
5.3
5.4
2.1
3.6
7.4
7.5
4.4
1.4
3.5
3.4
1.5
1.1
6.0kmm
R@2
47.0
31.4
35.9
11.5
38.6
51.9
31.7
25.2
23.0
17.6
27.4
48.6
16.5
11.4
11.8
12.1
9.3
5.0
25.3q142)
Acc
40.1
16.3
22.8
0.0
16.7
10.3
16.8
12.9
10.3
11.1
18.9
16.3
2.1
3.6
5.3
3.7
12.0wa)
LLaVA 30
F1
14.3
6.5
6.2
0.0
4.8
3.7
3.7
2.9
1.6
2.6
7.4
5.0
0.6
0.7
0.6
0.5
0.3
0.2
3.4(55)
R@2
67.6
34.6
33.3
25.2
19.6
33.5
10.5
7.4
22.4
Acc
35.2
59.0
63.6
43.6
30.5
70.7
34
18.2
27.8
31.5
43.5
UniMTS
F1
42.2
22.0
33.7
42.9
57.2
34.8
36.7
53.7
43.6
27.8
61.8
41.1
29.2
13.7
25.5
23.7
18.5
10.0
34.3
R@2
80.0
53.1
57.2
60.7
82,1
86.6
64.0
77.5
63.2
45.4
78.7
85,0
44.2
38.6
47.1
46.0
32.6
18.7
58.9
Acc
37.7
18.6
25.1
36
19.4
53.4
55.5
35.6
32.5
20.7
27
59
9.9
3.6
13.5
21.5
13.5
4.3
27. k163)
weo not aug
F1
30.4
8.2
11.2
26.4
13.5
27.7
41.1
29.6
28.3
10.5
24.0
20.5
4.9
4.8
10.6
13.4
7.2
4.7
17.6
R@2
14.3
40.1
64.3
52.5
40.1
76.5
76.3
54.4
29.7
49.4
61.2
28.5
6.8
23.3
34.0
32.1
44.4(2000
Acc
52.7
36.3
43.4
57.4
55.6
61.0
40.7
40.9
38.4
29.6
59.2
62.8
30.3
28.2
31.3
29.3
6.5
12.6
39.8,133)
w/o text aug
39.4
21.3
41.4
49.5
32.8
29.7
72.7
19.0
26.7
21.2
19.6
22.2
4.7
28.8
R@2
70.9
39.6
63.7
57.4
78.7
77.4
60.4
63.5
48.8
49.1
63.4
77.9
4144
43.2
45.1
41.7
10.7
23,2
53.1
Acc
37.6
23.0
50.9
42.7
33.8
22.4
28.7
23.4
0.5
19.8
1.9
11.2
25.1gs
wlo graph
F1
20.5
28.5
21.6
17.9
38.6
9.1
23.7
28.9
23.6
23.0
19.3
15.3
11.5
1.1
8.5
16.0
2.7
7.7
17.6ux)
R@2
63.7
66.7
54,2
34.4
66.1
35.3
51.1
62.1
41.8
41.7
43.9
51.6
38.6
8.2
23.7
29.1
4.7
18.0
40.7a1
statistically significant, with p-values significantly lower than 0.05 (c.g., p-value = 8 x 10 for ImageBind, which has the highest F1 score among the baselines).
Compared with UniMTS, ImageBind and IMU2CLIP are trained on data from single location (head- mounted devices), limiting their generalization to data collected from other locations. IMUGPT struggles to generalize across datasets featuring different activities and requires individual training for each downstream dataset. Both HARGPT and LLa VA focus on simple and easily distinguishable activities as these language or vision models are not originally trained on motion time series, and they also require careful prompt designs. Another limitation for all the above models is that they do not generalize across device orientations. In contrast, UniMTS shows remarkable generalizability to vari- ous downstream device locations, orientations and activities, achieving state-of-the-as art performance. We also compare with few ablations of UniMTS as illustrated in the Ablation Study section.
4.3 Few-Shot Fine-tuning Results
Apart from the zero-shot setting, we provide few real samples for each activity and fine-tune UniMTS and the baselines. More specifically, we provide 1, 2. 3, 5, 10 samples for each activity and compare UniMTS against ImageBind (16). IMU2CLIP (38), IMUGPT (28), GPT4TS (72). BioBankSSL [66] and a randomly initialized model with the same model architecture as UniMTS (referred to as Random). We report both the mean and the standard deviation in Figure 4 UniMTS also demonstrates tate-oofttheart performance in the few-shot fine-tuning setting. showing the effectiveness of pre-training. Following the same Wilcoxon-signed rank test as in the zero-shot setting, we observe p-values far below 0.05 (e.g., p-value = 2 x 10-25 for the best-performing baseline ImageBind), indicating the statistical significance of our improvement.
4.4 Full-Shot Results
We also compare the full-shot performance where UniMTS and the baselines are fine-tuned or trained using all the available training samples of the downstream datasets. We compare UniMTS with pre- trained models (ImageBind [16), IMU2CLIP (38), self-supervised models (TST (68). TARNet (Л].
7

---

imageSind
IMU2CIIP
IMUGPT
GPT4TS
BioBankSSL
Random
UniMTS
Opportunity
UCI-AR
MotionSense
W-HAR
Shoa
HAR70+
100
75
75
Mocro
50
25
10
10
10
RealWorld
TNDD--AR
PAMAP2
USC-HAD
Mhoalth
Harth
Macro.FF F1
75
75-
75
50
25
50
25-
25
10
10
10
UTTCComplex
Wharf
WISDM
DSADS
UTD-MHAD
MMAct
F1
13
75
50
15
20
25
25
10
Samples
Number Samples
Number Samples
Number of Samples
Number of Samples
Number Samples
Figure 4: Few-shot fine-tuning results. UniMTS consistently outperforms both baselines and our model ablation. We repeat 3 runs and report both mean and standard deviation.
#ong
aeding nalading
Asll
Org
20
Haaddigg
vis ber Bruadd
lmming
fortard
(a) PAMAP2
(b) Mhealth
(c) TNDA HAR
Figure 5: T-SNE visualizations show that signal clusters align with their semantic meanings.
Ve2 [67]. BioBankSSL (66]), and conventional models (DeepConvLSTM 40), MA-CNN 45
XGBoost [9. THAT (29, IMUGPT 288, TimesNet [60], GPT4TS 722) SHARE (7T). Baselines detailed in Section A.3 in Appendix. We also compare pre-trained UniMTS with randomly initialized UniMTS (referred to as Random). As shown in Table 2 UniMTS also demonstrates
state-of-the-art performance in the full-shot setting, outperforming pre-trained, self-supervised and conventional models. Due to space limit, we report baselines before 2021 in Table Appendix. Following the same Wilcoxon-signed rank test, observe p-values far below 0.05 (e.g., p-value 0.018 for the best-performing baseline), indicating the statistical significance of our improvement. UniMTS also demonstrates space and time efficiency, as detailed in Section A.5 Appendix.
4.5 Ablation Study
In the zero-shot setting. we compare UniMTS with few ablations by removing -otation-invariant augmentation (w/o rot aug), removing text augmentation (w/o text aug) and by replacing the graph encoder with a CNN-based encoder that directly concatenates joints without modeling their spatial relationships (w/o graph). We can observe in Table [ that the performance declines after removing each of the above components, verifying their respective importance in improving generalization across locations (graph encoder), orientations augmentation). We also compare the pre-trained UniMTS with randomly initialized UniMTS both few-shot and full-shot settings. As shown in Figure 4 and Table pre-trained UniMTS consistently outperforms randomly initialized UniMTS, highlighting the benefits pre-training.
augmentation) and activities (text
4.6 Case Study
UniMTS's time series embeddings align with corresponding semantic meanings, As shown in Figure the t-SNE visualizations of UniMTS's time series embeddings form distinguishable clusters that align with their semantic meanings. Notably, UniMTS is only pre-trained on the simulated data but its embeddings for real-world data closely align with the semantic space, which again demonstrates our model's zero-shot generalization due to contrastive learning. For example, in Figure 5a stationary activities such lying and sitting group together; light-movement activities such as standing, ironing, and vacuum cleaning are close to each other; while high high-intensity activities such as running and cycling cluster closer in the embedding space.
8

---

Table 2: Full-Shot performance. We bold the best and underline the second best. UniMTS performs the best compared with both pre-trained, self-supervised and conventional models. The last column shows the average performance across 18 datasets with standard deviation.
INDA-HHR
Complox
UTD MHAD
Merris
Opporusiit
CCI-AR
Motion
HAR
Shoudb
HAR30+
ResWorld
PAMAP2
USC-RAD
Mhealth
Horth
Wharf
WISDM
DSADS
MMA
Avange
Datnset
Number of Classes
12
12
12
13
14
18
19
|
35
Level
Easy
Medium
Hard
Avg
ImageBind 16
82.8
90.9
94.0
98.5
95.0
94.3
52.1
82.9
97.0
71.4
77.2
90.0
14.0
65.8
85.0
F1
84.6
90.7
93.1
61.9 98.5
67.7
86.1
94.9
94.6
52.2
83.2
62.7
96.9
43.3
76.8
89.3
71.6
58.
78. 1ows
IMU2CLIP
79.3
98.2
90.7
55.7
90.6
52.3
62.7
88.7
62.8
80.5
F1
70.0
87.0
77.2
75.1
98.1
64.4
80.5
97.1
91.2
52.0
95.0
65.7
90.2
35.6
62.8
88.2
60.3 59.6
75.00ze
TST x8
Acc
F1
89.1
91.1
91.6
91.4
91.4
89.6
75-4
79.3
72.3
73.5
95.8
62.9
67.2
68.9
94.1
94.1
88.9
89.8
66.2
60.9
83.8
86.0
98.8
72.7
92.9
92.8
59.1
34.5
69.1
70.2
76.1
80.0
61.4
59.6
48.4
54.5
19.9
74.2.6
TARNet
F1
82.6
83:4
91.2
72.0
70.0
41.0
66.1
67.9
76.9
54.4
92.5
92.4
92.4
90.6
48.7
57.6
77.5
47.8
94.4
94.4
68.2
38.1
58.5
58.5
23.0
30.7
75.5
56.7
55.9
75. krts:
65.9
65
V82 67
Acc
80.7
92.1
94.2
93.9
100.0
87.4
98.4
74.6
95.9 9
95.9
91.8
92.9
56.1
95.3
98.2
98.5
98.5
47.9
73.2
69.5
84.3
80.0
53.4
53.4
19.8
84.5
BieBankSSL 66
Acc
85599
86.1
92.7
92.9
99.2
99.4
93.4
99.1
99.1
93.2
85.9
68.0
75.5
89
89.4
87.3
89.3
72.6
65.8
94.0
95.1
68.1
98.0
96.4
96.4
62.7
81.4
82,9
83.3
65.8
83.1
18.4
80.5
62.5
58.4
82.2
85. 85.8811.:
THAT
Acc
83.1
86.8
86.7
87.9
86.5
77.1
65.0
92.2
92.3
95.8
73.9
67.7
62.9
97.7
97.7
9167
96.5
55.0
53.9
89.9
89.0
97.6
71.9
86.9
87.0
60.0
29.5
60.9
60.3
79.0
82.1
71.2
70.0
55.0
52.3
744.517% 9
IMUGPT 28
Acc
84.8
84.9
87.0
87.0
85.2
86.2
85.33
48.8
61.7
61.8
98.3
78.0
56.0
62.2
87.7
87.5
83.8
85.6
42.4
41.1
63.6
71.3
65.2
98.3
85.8
86.5
54.6
24.6
67.8
67.4
71.0
71.9
17.7
53.2
49.0
$1.9
74.4.6
16.4
TimesNet 60
Acc
F1
82.4
80.0
88.5
88.6
88.3
90.5
88.5
79.5
82.6
82.4
77.2
97.5
62.2
65.4
93.2
93.1
88.0
88.0
48.7
58.3
81.7
79.1
97.7
61
92.8
92.7
41.4
30.2
67.1
67.4
78.5
77.3
62.3
61.0
45.7
50.9
72.6677
GPT4TS
F1
86.1
88.4
88.5
92.3
92.1
54.8
74.1
76.6
56.6
95.7
95.7
92.6
93.4
61.7
58.9
85.2
58.2
95.3
95.2
51.8
34.4
64.5
79.9
78.0
66.1
63.7
48.8
52.6
78.77557
12.5
SHARE
Acc
89.0
92.2
99.6
96.7
77.5
97.7
66.0
95.9 9
94.1
72.9
97.0
99.1
98.7
63.2
77.0
92.0
73.0
62.1
85.8 18c
90.2
92.1
99.6
85.5
78.0
77.5
57.1
95 8
94.8
66.7
97.1
74.0
98.8
40.6
76.5
91.9
69.3
56.5
80. 1eps
UniMTS
92.1
99.8
98.4
99.7
96.9
99.6
9880
58.3
97.0
98.0
99.1
82.7
81.3
85.6
77.1
90.6 10s::
F1
89.1
92.2
99.8
98.8
99.7
87.9
81.2
99.6
98.1
63.1
97.3
86.7
99.3
51.7
80.9
94.2
83.2
71.7
87.5
Random
Acc
87.7
88.9
89.3
89.4
95.5
95.5
60.1
95.1
66.1
65.0
85.9
98.1
74.2
74.0
98. 7
98.7
96.2
96.8
48.5
51.9
78.2
82.9
98.6
69.1
98.5
98.5
50.8
76.8
79.3
79.4
90.1
90.3
69.7
74.4
73.9
71.6
84.7
78.5
Simulation
PAMAP2
Simulation
PAMAP2
Simulation
PAMAP2
0
10
0
0
25
25
-10
-10
100
M4MWWMMMM
-2
-25
200
200
200
(a) Sitting
(b) Walking
(c) Rope Jumping
Figure 6: Simulated motion time series closely resemble patterns of the real PAMAP2 time series.
UniMTS generalizes well to new activities unseen in pre-
Acthvity
200
Imagetins ours
training. To show UniMTS's capability to generalize to new
75
activities not seen during pre-training, we visualize the zero-
shot performance for some example new activities in Fig-
ure 7 Compared with the best- persforeffo baseline Image-
Eniiing chstees
Wishing oi fnctrodd trauisal
saking cheps
eleuttor down
frotts edvantion of arms
ineai dending ceucting
soise
draw carlle mhnnia caackeii serng
tho bandi Fost ciss
carry figte
tmntt torf Eving List
Bind, UniMTS shows significant performance improvement on these previously unseen activities, verifying the effective- ness of semantic generalization via contrastive learning.
Simulated data from our physics engine closely resemble real signal patterns. In Figurelo] we compare some example simulated data alongside their real-world counterparts. We show three example activities of different intensity levels (i.e., sitting, walking, rope jumping), where both simulated data and real-world data near the wrist. We can observe that the patterns in simulated data closely resemble those of real data, in terms of both magnitude and frequency. Although the tri-axial distributions might differ, these variations mostly due to orientation differences and effectively managed by our and real data for other device locations, shown in Section A.4 in Appendix.
Figure 7: UniMTS shows significant performance improvement compared with the best baseline when evaluated on new activities not seen.
augmentation. We observe similar patterns between simulated data

---

Conclusion and Discussion
Conclusion. In this paper, we present the first unified pre-training procedure, UniMTS, for motion time series classification. Our model is pre-trainne pre-trained only on physics-simulate data, and yet demon- strates remarkable generalization across diverse real-world motion time series datasets featuring different device locations, orientations and activities. The simulated data with all-joint coverage are augmented for rotation invariance and modeled by graph encoder, improving generalization across various device factors. During pre-training, contrastive learning aligns time series with their semantic meanings to improve generalization across activities. Extensive evaluation in zero-shot, few-shot and full-shot settings consistently demonstrates the state-of-the-an performance of UniMTS.
Limitation and Future Work. We acknowledge a few limitations which we leave as future work. (1) Simulated motion time series only be approximations of real signals, which usually collected near - rather than directly on - the body joints. For example, sensors on smartwatches collect data near the wrist, not on the wrist joint itself. We plan to incorporate random offset vectors to better simulate real-world signal variations near joints. (2) While our framework effectively addresses the classification task, we intend to extend its applicability
other motion time series tasks such as inertial navigation. (3) Our current pre- pre-training utilizes existing motion datasets, and we plan to enrich our pre-training corpus with additional motion data extracted from large-scale video-based pose estimation. (4) We also plan to integrate our model with efficient inference optimization techniques such as quantization, pruning and distillation for deployment on edge devices.
Broad Impact. UniMTS is the first pre-trained motion time series classification model that gener- alizes to diverse downstream datasets, irrespective of device locations, orientations and activities. The primary societal concern centers around privacy as motion time series might reveal personal information, so we ensure strict privacy controls the earliest stages of model development by pre-training exclusively on synthetic data. With UniMTS's s state-of-the-art performance in zero-shot, few-shot and full-shot settings, we believe would bring broad, positive impact to the community.
6 Acknowledgement
Our work is supported in part by ACE, one of the seven centers in JUMP 2.0, a Semiconductor Research Corporation (SRC) program sponsored by DARPA.
Our work is also sponsored part by NSF CAREER Award 2239440, NSF Proto-OKN Award 2333790, as well as generous gifts from Google, Adobe, and Teradata. Any opinions, findings, and conclusions or recommendations expressed herein are those of the authors and should not be interpreted as necessarily representing the views, either expressed or implied, of the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for government purposes not withstanding any copyright annotation hereon.
10

---

References
[I] Kerem Altun, Billur Barshan, and Orkun Tunçel. Comparative study on classifying human activities with miniature inertial and magnetic sensors. Pattern Recognition, 43(10):3605-3620 2010.
[2] Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, Jorge Luis Reyes-Ortiz, et al. A public domain dataset for human activity recognition using smariphones. In Esann, volume 3, page 3, 2013.
[3] Luca Arrotta, Claudio Bettini, Gabriele Civitarese, and Michele Fiori. Contextgpt: Infusing Ilms knowledge into neuro-symbolic activity recognition models. arXiv preprint MrXXi:2403.065880 2024.
[4] Oresti Banos, Rafael Garcia, Juan A Holgado-Tenriza, Miguel Damas, Hector Pomares, Ignacio Rojas, Alejandro Saez, and Claudia Villalonga. mhealthdroid: a novel framework for agile development of mobile health applications. In Ambient Assisted Living and Daily Activities: 6th International ConfeConfee,,ee IWAAL 2014, Belfast, UK, December 2-5, 2014. Proceedings 6, pages 91-98. Springer, 2014.
[5] Ganapati Bhat, Nicholas Tran, Holly Shill, and Umit Y Ogras. w-har: An activity recognition dataset and framework using low-power wearable devices. Sensors, 20(18):5356, 2020.
[6] Barbara Bruno, Fulvio Mastrogiovanni, Antonio Sgorbissa, Tullio Verazza, and Renato Zac- caria. Analysis of human behavior recognition algorithms based on acceleration data. In 2013 IEEE International Conference on Robotics and Automation, pages 1602-1607. IEEE, 2013.
[7] Sara Caramaschi, Gabriele B Papini, and Enrico G Caiani. Device orientation independent human activity recognition model for patient monitoring based on triaxial acceleration. Applied Sciences, 13(7):4175, 2023.
[8] Chen Chen, Roozbeh Jafari, and Nasser Kehtarnavaz Utd-mhad: multimodal dataset for human action recognition utilizing depth camera and a wearable inertial sensor. In 2015 IEEE International conference on image processing (ICIP), pages 168-172. IEEE, 2015.
[9] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference knowledge discovery and data mining, pages 785-794, 2016.
[10] Yuanyuan Chen, Shing Chan, Derrick Bennett, Xiaofang Chen, Xianping Wu, Yalei Ke, Jun Lv, Dianjianyi Sun, Lang Pan, Pei Pei, et al. measured movement behaviours in over 20,000 china kadoorie biobank participants. International Journal of Behavioral Nutrition and Physical Activity, 20( 1):188 2023.
[11] Ranak Roy Chowdhury, Xiyuan Zhang, Jingbo Shang, Rajesh Gupta, and Dezhi Hong. Tarnet: Task-aware reconstruction for time-series transformer. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, pages 14-18, 2022.
[12] Angus Dempster, François Petitjean, and Geoffrey I Webb. Rocket: exceptionally fast and accurate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery, 34(5): 1495, 2020.
[13] Aiden Doherty, Dan Jackson, Nils Hammerla. Thomas Plötz, Patrick Olivier, Malcolm Granat, Tom White, Vincent T Van Hees, Michael I Trenell, Christoper G Owen, al. Large scale population assessment of physical activity using wrist worn accelerometers: the uk biobank study. PloS one, 12(2):e0169649, 2017.
[14] Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting. arXiv preprint MXXiv:2106.14122 2021.

---

[15] Davide Figo, Pedro C Diniz, Diogo R Ferreira, and Joao MP Cardoso. Preprocessing tech- niques for context recognition from accelerometer data. Personal and Ubiquitous Computing. 14(7):645-66) 662, 2010.
[16] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15180-15199 2023.
[17] Kristen Grauman, Andrew Westbury, Eugene Byrne Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995-19012, 2022.
[18] Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyl- los Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19383-19400, 2024.
[19] Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5152-51611 June 2022.
[20] Sture Holm. simple sequentially rejective multiple test procedure. Scandinavian journal of statistics, pages 65-70, 1979.
[21] Zhiqing Hong, Zelong Li, Shuxin Zhong, Wenjun Lyu, Haotian Wang, Yi Ding, Tian He, and Desheng Zhang. Crosshar: Generalizing cross-dataset human activity recognition via hierarchical self-supervised pretraining. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 8(2):1-26, 2024.
[22] Rong Hu, Ling Chen, Shenghuan Miao, and Xing Tang. Swl-adapt: An unsupervised domain adaptation model with sample weight learning for cross-user wearable human activity recog- nition. In Proceedings of the AAAI Conference on artificial intelligence, volume 37, pages 2020, 2023.
[23] Jeya Vikranth Jeyakumar, Liangzhen Lai, Naveen Suda, and Mani Srivastava. Sensehar: a robust virtual activity sensor for smartphones and wearables. In Proceedingy of the 17th Conference on Embedded Networked Sensor Systems, 2019.
[24] Sijie Ji, Xinzhe Zheng, and Chenshu Wu. Hargpt: Are Ilms zero-shot human activity recogniz- ers? arXiv preprini arXiv:2403.02722 2024.
[25] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
[26] Quan Kong, Ziming Wu, Ziwei Deng, Martin Klinkigt, Bin Tong, and Tomokazu Murakami. Mmact: A large-scale dataset for cross modal human action understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 0658-8667, 2019.
[27] Zikang Leng, Amitrajit Bhattacharjee, Hrudhai Rajasekhar, Lizhe Zhang, Elizabeth Bruda, Hyeokhyen Kwon, and Thomas Plötz. Imugpt 2.0: Language-based cross modality transfer for bassor human activity recognition. arXiv preprint arXiv:2402.01049 2024.
[28] Zikang Leng, Hyeokhyen Kwon, and Thomas Plötz. Generating virtual on- on-body accelerometer data from virtual textual descriptions for human activity recognition. In Proceedings of the 2023 ACM International Symposium on Wearable Computers, pages 39-43, 2023.
[29] Bing Li, Wei Cui, Wei Wang, Le Zhang, Zhenghua Chen, and Min Wu. Two-stream convolution augmented transformer for human activity recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 286-293, 2021.
12

---

[30] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[31] Jun Liu, Amir Shahroudy. Mauricio Perez, Gang Wang, Ling- Yu Duan, and Alex C Kot. Ntu rgb+ d 120: large-scale benchmark for 3d human activity understanding. IEEE transactions on puttern analysis and machine intelligence, 42(10))2684-270) 2019.
[32] Shengzhong Liu, Tomoyoshi Kimura, Dongxin Liu, Ruijie Wang. Jinyang Li, Suhas Diggavi, Mani Srivastava, and Tarek Abdelzaher. Focal: Contrastive learning for multimodal time- series sensing signals factorized orthogonal latent space. Advances Neural Information Processing Systems, 36, 2024.
[33] Aleksej Logacjov, Kerstin Bach, Atle Kongsvold, Hilde Bremseth Bårdstu, and Paul Jarle Mork. Harth: human activity recognition dataset for machine learning. Sensors, 21(23):7853 2021.
[34] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 851-866. 2023.
[35] Wang Lu, Jindong Wang. Yiqiang Chen, Sinno Jialin Pan, Chunyu Hu, and Xin Qin. Semantic- discriminative mixup for generalizable sensor-based cross-domain activity recognition. Pro- ceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 6(2):1-19, 2022.
[36] Haojie Ma, Wenzhong Li, Xiao Zhang, Songcheng Gao, and Sanglu Lu. Attnsense: Multi-level attention mechanism for multimodal human activity recognition. I/CAI, pages 3109-3115, 2019.
[37] Mohammad Malekzadeh, Richard G Clegg, Andrea Cavallaro, and Hamed Haddadi. Mobile sensor data anonymization. In Proceedings of the international conference on internet of things design and implementation, pages 49-58, 2019.
[38] Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Alireza Dirafzoon, Aparajita Saraf, Amy Bearman, and Babak Damavandi. Imu2clip: Multimodal contrastive learning for imu motion sensors from egocentric videos and text. arXiv preprint arXiv:2210. 14395, 2022.
[39] OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
[40] Francisco Javier Ordóñez and Daniel Roggen. Deep convolutional and Istm recurrent neural networks for multimodal wearable activity recognition. Sensors, 16(1):115, 2016.
[41] Matthias Plappert, Christian Mandery, and Tamim Asfour. The kit motion languae dataset. Big data, 4(4):236-25 252, 2016.
[42] Hangwei Qian, Tian Tian, and Chunyan Miao. What makes good contrastive learning on small-scale wearable-bar based tasks? In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, pages 3761-3771, 2022.
[43] Xin Qin, Jindong Wang, Shuo Ma, Wang Lu, Yongchun Zhu, Xing Xie, and Yiqiang Chen. Generalizable low-resource activity recognition with diverse and discriminative representation learning. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1943-1953, 2023.
[44] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 774488766. PMLR, 2021.
[45] Valentin Radu, Catherine Tong, Sourav Bhattacharya, Nicholas D Lane, Cecilia Mascolo, Mahesh K Marina, and Fahim Kawsar. Multimodal deep learning for activity and context recog- nition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 1(4):1-27, 2018
13

---

[46] Attila Reiss and Didier Stricker. Introducing a new benchmarked dataset for activity monitoring. In 2012 16th international symposium on wearable computers, pages 108-109. IEEE, 2012.
[47] Daniel Roggen, Alberto Calatroni, Mirco Rossi, Thomas Holleczek, Kilian Förster, Gerhard Tröster, Paul Lukowicz, David Bannach, Gerald Pirkl, Alois Ferscha, et al. Collecting complex activity datasets in highly rich networked sensor environments. In 2010 Seventh international conference on networked sensing systems (INSS), pages 233-240. IEEE, 2010.
[48] Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu. Two-stream adaptive graph convolutional networks for skeleton- based action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12026-12035, 2019.
[49] Muhammad Shoaib, Stephan Bosch, Ozlem Durmaz Incel, Hans Scholten, and Paul JM Havinga. Fusion of smartphone motion sensors for physical activity recognition. Sensors, 14(6):10146- 10176, 2014.
[50] Muhammad Shoaib, Stephan Bosch, Ozlem Durmaz Incel, Hans Scholten, and Paul JM Havinga. Complex human activity recognition using smartphone and wrist-worn motion sensors. Sensors, 16(4):426, 2016.
[51] Dimitris Spathis, Ignacio Perez-Pozuee Soren Brage, Nicholas J Wareham, and Cecilia Mascolo. Self-supervised transfer learning of physiological representations from free-living wearable data. In Proceedings of the Conference Health, Inference, and Learning, pages 69-78, 2021.
[52] Chenxi Sun, Yaliang Li, Hongyan Li, and Shenda Hong. Test: Text prototype aligned embedding to activate IIm's ability for time series. arXiv preprint arXiv:2308.08241 2023.
[53] Timo Sztyler and Heiner Stuckenschmidt. On-body localization of wearable devices: An investigation of position-aware activity recognition. In 2016 IEEE International Conference on Pervasive Computing and Communications (PerCom), pages 1 1-9. IEEE, 2016.
[54] Shuhan Tan, Tushar Nagarajan, and Kristen Grauman. Egodistill: Egocentric head motion distillation for efficient video understanding. Advances in Neural Information Processing Systems, 36:33455-334988 2023.
[55] Sana Tonekaboni, Danny Bytan, and Anna Goldenberg. Unsupervised represnnttionn learning for time series with temporal neighborhood coding. arXiv preprint arXiv:2 2021.
[56] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.0922 09288, 2023.
[57] Terry T Um. Franz MJ Pfister, Daniel Pichler, Satoshi Endo, Muriel Lang, Sandra Hirche, Urban Fietzek, and Dana Kulić. Data augmentation of wearable sensor data for parkinson's 's disease monitoring using convolutional neural networks. In Proceedings of the 19th ACM international 2017.
conference on multimodal interaction, pages
[58] Astrid Ustad, Aleksej Logacjov, Stine Øverengen Trollebo, Pernilee Thingstad, Beatrix Verei- jken, Kerstin Bach, and Nina Skjæret Maroni. Validation of an activity type recognition model classifying daily physical behavior in older adults: the har70+ model. Sensors, 23(5):2368, 2023.
[59] Gary M Weiss. Wisdm smartphone and smartwatch activity and biometrics dataset. UCI Machine Learning Repository: WISDM Smartphone and Smartwatch Activity and Biometrics Dataset Data Set, 7:133190-13333
2019.
[60] Haixu WU, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. The eleventh international conference on learning representations, 2022.
[61] Huatao Xu, Pengfei Zhou, Rui Tan, and Mo Li. Practically adopting human activity recogni- tion. In Proceedings of the 29th Annual International Conference on Mobile Computing and Networking, pages 1-15, 2023.
14

---

[62] Huatao Xu. Pengfei Zhou, Rui Tan, Mo Li, and Guobin Shen. Limu-bert: Unleashing the potential of unlabeled data for imu sensing applications. In Proceedings of the 19th ACM Conference Embedded Networked Sensor Systems, pages 220-233, 2021.
[63] Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton-based action recognition. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[64] Yan Yan, Dali Chen, Yushi Liu, Jinjin Zhao, Bo Wang, Xuankun Wu, Xiaohao Jiao, Yuqian Chen, Huihui Li, and Xuchao Ren. Tnda-har, 2021.
[65] Alexander D Young, Martin J Ling, and Damal K Arvind. Imusim: A simulation environment for inertial sensing algorithm design and evaluation. In Proceedings of the 10th ACM/IEEE International Conference on Information Processing in Sensor Networks, pages 199-210 IEEE, 2011.
[66] Hang Yuan, Shing Chan, Andrew Creagh, Catherine Tong, Aidan Acquah, David A Clifton, and Aiden Doherty. Self-supervised learning for human activity recognition using 700,000 person-days of wearable data. NPJ digital medicine, 7(1):91, 2024.
[67] Zhihan Yue, Yujing Wang. Juanyong Duan, Tianmeng Yang. Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 3980-8987 2022.
[68] George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A cansformer-basel framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 2114-2124, 2021.
[69] Mi Zhang and Alexander Sawchuk, Usc-had: A daily activity dataset for ubiquitous activity recognition using wearable sensors. In Proceedings of the 2012 ACM conference on ubiquitous computing, pages 1036-1043, 2012.
[70] Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised contrastive pre-trainin for time series via time-frequency consistency. Advances in Neural Information Processing Systems, 35:3988-4003 2022.
[71] Xiyuan Zhang, Ranak Roy Chowdhury, Jiayun Zhang, Dezhi Hong, Rajesh K. Gupta, and Jingbo Shang. Unleashing the power of shared label structures for human activity recognition. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM '23, page 3340-3350, 2023.
[72] Tian Zhou, Peisong Niu, Liang Sun, Rong Jin, et al. One fits all: Power general time series analysis by pretrained Im. Advances in neural information processing systems, 36, 2024.
15

---

A Appendix supplemental material
A.1 Pre-training Datasets
HumanML3D [9 is large-scale motion skeleton data consisting of 14,616 3D human motion skeletons spanning 28.59 hours. The average motion skele- ton sequence length is 7.1 seconds, Paired with each motion skeleton sequence there is an average of 3 textual descriptions, resulting in a total of 44,970 textual descriptions with vocabulary size of 5.371. The average and median lengths of these descriptions are 12 and 10 words. We further augment the textual descriptions using large language models as described in Section [33.2 All motion skeletons follow the skeleton structure of SMPL [34] with 22 joint nodes. Figure provides an example skeleton of "a person waves his hands".
We also tried to incorporate additional motion skeleton datasets into pre- training, such KIT-ML [4] and NTU RGB+D 120 [T. However, these data are relatively less diverse in terms of both motion skeletons and textual descriptions. We did not observe performance improvement from adding them, and therefore use H3manML33D as our primary pre-training corpus.
Figure 8: Skeleton of "waving hands".
A.2 Downstream Evaluation Datasets
We detail the information for each downstream real-world evaluation dataset as follows.
Opportunity 47 contains data collected from back, upper arms and lower arms, and features multiple sets of activities. We aim to predict the modes of locomotion such as standing and walking.
UCI-HAR 2 collects motion data from smartphone located on the subject's waist. The subject performs daily activities such walking upstairs and walking downstairs.
collects data from a smartphone in the participant's front pocket, featuring daily activities such sitting and jogging.
MotionSense 37
w-HAR [5] contains motion time series data collected from the ankle. It captures daily physical activities such as jumping and lying down.
contains daily activities such as biking. Each participant is equipped with five smart- phones on five positions: right jean's 's and left jean's 's pockets, belt, right upper arm and right wrist.
Shoaib 49
tracks activities such shuffling for older adult subjects. The motion time series collected from the right front thigh and the lower back.
HAR70+ 58
records daily activities such as climbing stairs from multiple body positions including chest, forearm, head, shin, thigh, upper arm, and waist.
RealWorld 53
TNDA-HAR 64 collects static as well as periodic daily activities such as cycling, from devices located at multiple body positions such as wrist, ankle and back.
PAMAP2 46
monitors physical activities such as ironing, vacuum cleaning and rope jumping using devices located the wrist, chest and ankle.
USC-HAD 69
records daily activities such as sleeping and taking the elevator with devices attached to the subject's s front right hip.
Mhealth 4 comprises body motion for common activities such waist bending forward, frontal elevation of arms and knees bending. Devices are placed on the user's 's chest, right wrist and left ankle.
Harth 33 records data in free-living setting with devices located at the right thigh and lower back.
UT-Complex [50] contains different smartphone sensor data such as typing, drinking coffee and giving a talk, with devices positioned at wrist and pocket positions.
Wharf 6 records activities from wrist-worn devices, such combing hair and getting up bed.
WISDM (59 collects diverse daily activities such as brushing teeth, eating soup, playing balls, and folding clothes, using data from the smartphone in the pocket and smartwatch on hand.
16

---

Table 3: Full-Shot performance additional baselines before 2021. We bold the best results. The
last column shows the average performance across 18 datasets with standard deviation.
Metrics
Oppartiity
UCT-AR
MotionSense
w-HAR
Sbouth
HAR70+
RealWord
INDA-HAR
PAMAP2
USC-HAD
Mhcalth
UT-Complex
Whatf
WISDM
DSADS
UTD MHAD
MMAc
Average
Dataset
Number of Classes
4
7
12
12
12
12
13
14
18
19
27
35
Level
Easy
Medium
Avg
DeepCL 2D
Acc
84.4
86.9
86,2
93.4
62,4
97.7
69.9
89.6
82.7
45.2
73.8
95.5
88.0
46.8
69.0
69.6
54.9
50.2
11.8
86.4
92.2
88.2
69.0
46.9
68.6778)
MA-CNN 43
Acc
82.9
89.8
92.0
67.2
93.2
86.1
73.9
96.0
94.7
37.3
78.7
97.9
93.8
20.9
51.5
84.5
48.8
28.9
73.2
84.8
91.7
51.7
93.1
59.8
71.2
96.0
95.2
35.3
70.7
94.0
17.2
51.8
83.9
48.5
18.4
67.0gss
XGBoost
Acc
83.1
90.2
91.6
92.9
68.9
94.8
77.5
97.7
78.2
93.2
93.2
93.9
93.9
47.8
74.1
79.9
97.2
96.7
96.6
52.3
66.6
80.5
61.9
53.0
73.9
79.4.155)
UnIMTS
Acc
89.1
88.2
92.1
99.8
99.8
98.4
99.7
96.9
87.9
84.0
99.6
99.6
98.0
98.1
58.3
97.0
98.0
99.2
99.1
82.7
81.3
80.9
94.3
83.2
85.6
71.7
77.1
87.5
90.6
DSADS comprises daily and sports activities such as exercising and rowing. Multiple devices are positioned at the torso, right arm, left arm, right leg, and left leg.
UTD-MHAD contains diverse activities such as swiping arms, hand clapping, throwing, arm crossing, drawing and squatting. The devices are worn on the subject's right wrist or the right thigh depending on whether the action is mostly an arm or leg type of action.
MMAct 26 presents a large-scale activity dataset covering wide range of daily life activities such as carrying, talking on phone and falling. Devices recording motion time series include smartwatch as well as a smartphone inside the pocket of the subject's ' pants.
A.3 Baselines
We detail the baseline settings as follows.
We employ the pre- traitre weights from "imagebind huge" for zero-shot evaluation. During fine-tuning, we add linear layer to map Image embeddings the number of activity classes. We fine-tune both ImageBind and the linear layer during fine-tuning, which performs better than simply tuning the linear layer.
ImageBind [[6]
IMU2CLIP 38 The pre- pre-trained weights of IMUCLIP are not released. Therefore, we first follow their pre-training implementationn to pre-train on Ego4D datasets (17). During fine-tuning, we add linear layer after IMU2CLIP embeddings and fine-tune both IMU2CLIP and the linear layer.
We choose DeepConVLSTM the backbone model, which shows the best perfor- mance as reported in their original paper. We remove the supervised distribution calibration phase, which relies labeled downstream data and conflicts with the shot setting objectives.
IMUGPT 28
The method directly prompts large language models to classify motion time series. We down-sample motion time series to 10 Hz. as used in their paper and follow their prompt template.
HARGPT 24
VA 30 We visualize motion time series 2D plots and use these visualizations as input for the pre-trained model of 1llav-vv.55-75
TST [68] This is Transformer-bast based representation learning framework with several downstream tasks including multivariate time-series classification. We follow the framework to first pre-train the Transformer model in an unsupervised fashion and then fine-tune the pre-traine pre-trained model on the downstream classification task.
TARNet [1
The model proposes task-aware representation learning that reconstructs important timestamps guided by self-attention score distribution from end-task training. We jointly train the reconstruction task and the classification task.
17

---

Simulation
Simulation
PAMAP2
Simulation
PAMAP2
25
25
25
-10
0
-25
25
200
200
200
200
200
200
(a) Sitting
(b) Walking
(c) Rope Jumping
Figure 9: Simulated motion time series show similar patterns as real PAMAP2 time series.
The method performs contrastive learning learn contextual representations of time series. We follow their implementation to first apply contrastive learning and then train a linear regression model for each dataset.
TS2Vec [67
BioBankSSL [66] The paper proposes a pre-trained model for human activity recognition using a large-scale UK Biobank wrist accelcrometer dataset with multi-tast self-supervised learning.
(shown as "DeepCI." in Table 3 due to space limit): The model applies convolutional layers to automatically learn feature representations and further applies LSTM to capture the temporal dependencies between their activations.
DeepConVL.STM 40
The model first extracts preliminary features for each motion time series modal- ity through its own dedicated convolutional layers, then the extracted intra-modality features combined through fully-connected layers for motion time series classification.
MA-CNN 45
XGBoost 99:
This is scalable end-to-end machine learning system for tree boosting, which has been widely recognized in machine learning and time series analysis.
THAT [29
The model proposes a two-stream convolution augmented human activity transformer which captures both over-one--hannel and channel-over-tiee time features in a two-stream structure.
TimesNet [60]: This is a task-general backbone for time series analysis including classification by modeling the multi-periodicity and extracting temporal variations.
This is also a task-general framework that includes time series classification. The model is based on a frozen pre- pre-trained language model, baseline. However, the method is not suitable for zero-shot evaluation, as it requires training a separate classifier head for each downstream dataset and does not generalize across activities.
GPT4TS 72]
also adopt it as a few-shot fine-tuning
SHARE 71
This is sequence-to-sequenee to-sequnnce model that contains an encoder extract motion time series features, as well as decoder to generate label name sequences to capture label semantics.
A.4 Simulated Data
In addition the simulated data for wrist as shown in Figure 6 we present more examples for other device locations such as ankle in Figure 9 We observe consistently similar patterns between simulated motion time series and real PAMAP2 data across activities of various intensity levels, ranging from sitting to walking and rope jumping.
A.5 Efficiency Analysis
For space complexity, the graph encoder of UniMTS contains only 4.94M parameters, which is significantly smaller compared with the 18.69M used in the IMU encoder of the best existing baseline ImageBind. For time complexity, fine-tuning of UniMTS is also efficient. On example dataset of UCI-HAR, full-shot fine-tuning of UniMTS takes approximately 1.3 minutes to converge while takes approximately 9.8 minutes for ImageBind converge. Moreover, we have run a power estimate assuming 0. Hzz cadence (i.c., 10-second window size), and it takes approximately 22.64 mW to run the whole graph model on an eNPU (embedded Neural Processing Unit), which is much smaller than ImageBind IMU encoder's 's power consumption of approximately 702 mW. Therefore, UniMTS is efficient for real-world applications and suitable be deployed on edge devices.
18

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the
paper's contributions and scope?
Answer: [Yes]
The main claims in the abstract and introduction are that we build the first unified pre-trained model for motion time series that is able generalize various device locations, orientations and activities, which are verified by our experimental results.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss four limitations and corresponding future works as detailed in "Limitation and Future Work" of Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
19

---

Answer: [NA]
Justification: Our paper does not involve new theoretical results. We have properly cited the literature for prior results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide detailed experimental settings in Section 4.1 as well as Section A.1 Section A.2 and Section A.3 in Appendix. We also provide the code in the supplementary
material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
20

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide them as zip file in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
public/guides/ https///oooeeSuueiiiiiiieeneeeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //nips. fttaps.cccpublicee wwtppbblccgguueeeeeoooooocpppped
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify all the training and test details in Section 4.1
well as Section A.1 Section (A.2 and Section A.3 in Appendix. We also provide the code the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report error bars and apply statistical significance tests for all experiments, including both zero-shot, few-shot and full-shot settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We specify computer resource details in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: htteee://eeuuuiee www.rpsu.p.../. httppp.ccpppppp cccppbllic/tthicccuueeeel.......
Answer: [Yes]
Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics in every aspect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts and limitations in Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
22

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper does not pose high risk for misusing data or models. We pre-train the model exclusively synthetic data ensure strict privacy control at the earliest stages model development.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cite all the code and data used in this paper, and respect their license and terms of use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
23

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, datasets.paperssiii paperswi thcode. www.com//aaaa datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide the new assets as a zip file in the supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
24

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "uGo9O3UeZK", "reviewer": "Reviewer_4CUw", "summary": "The paper introduces a new pre-training approach designed specifically for motion time series data from mobile and wearable devices. The proposed method, UniMTS, employs a contrastive learning framework to align motion time series with text descriptions, enhanced by large language models. This approach addresses the challenges of device placement variation, orientation differences, and activity variability. The model is evaluated on 18 benchmark datasets and demonstrates significant improvements in zero-shot, few-shot, and full-shot settings.", "questions": "Can you provide a comparison with the Ego4D and Exo benchmarks?", "limitations": "None", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality:\nThe paper presents a highly original approach by combining contrastive learning with synthetic data generation and graph networks to pre-train on motion time series data. This is a novel application of existing techniques to a new domain, addressing the unique challenges posed by motion data.\n\nQuality:\nThe methodology is well-developed and the experiments are thorough. The authors provide a detailed description of their approach, including the use of synthetic data and graph networks. The evaluation on a wide range of datasets strengthens the validity of their findings.\n\nClarity:\nThe paper is clearly written and well-organized. Each section logically follows from the previous one, making the complex methodology easy to understand. The appropriate use of figures and tables helps illustrate key points and results effectively.\n\nSignificance:\nUniMTS addresses a critical gap in the field of motion time series analysis. By improving generalization across different device placements, orientations, and activities, this method has the potential to significantly impact applications in healthcare, sports, and IoT. The impressive performance improvements demonstrated in the experiments underscore the method's potential.", "weaknesses": "Evaluation Comparisons:\nThe paper could be strengthened by comparing the proposed method with more recent benchmarks such as Ego4D and Exo. This would provide a clearer picture of how UniMTS stands relative to the current state-of-the-art methods in the field.\n\nGeneralizability of Results:\nWhile the method shows strong results across multiple datasets, there is limited discussion on the potential limitations or specific conditions where the model might underperform. A more detailed analysis of the generalizability and potential edge cases would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the insightful feedback, and for recognizing the novelty, experimental rigor, and clarity of our presentation. We have addressed the comments as below.\n\n[1] Ego4D and Ego-Exo4D benchmark\n\na. We acknowledge that these are important action recognition benchmarks; however, we would like to highlight that these two benchmarks are mainly aimed for multimodal action recognition by incorporating video and audio modalities [1,2]. The motion time series in these datasets are collected from head-mounted devices, and recognizing activities based purely on head movement is a very challenging task. Therefore, only very few papers target this task, such as IMU2CLIP. For example, IMU2CLIP reports an F1 score of only 31.89 for a 4-class action recognition task. \n\nb. We choose Ego4D as an example benchmark for comparison with IMU2CLIP. It’s important to note that models on the Ego4D benchmark are already pre-trained on the Ego4D dataset, but our UniMTS has never seen the Ego4D dataset during pre-training. Therefore, for a fair comparison, we split the Ego4D dataset into a 80% training set and a 20% test set, continue pre-training of UniMTS on the training set, and compare with IMU2CLIP pre-trained on the same training set. We report both accuracy and F1 in the following table. As the original Ego4D dataset only contains text descriptions without label IDs, for the test set, we derive class labels by extracting stem verbs from these original text descriptions, and choose top 10, top 20, top 50 activities for evaluation. During evaluation, for both UniMTS and IMU2CLIP, we compute the similarity of signal embeddings and embeddings of each label candidate. The activity with the highest similarity score is predicted as the label.\n\nSetting|Top-10 Acc|Top-10 F1|Top-20 Acc|Top-20 F1|Top-50 Acc|Top-50 F1\n---------|---------------|--------------|---------------|--------------|--------------|--------------\nIMU2CLIP|25.2|24.6|16.3|15.5|10.8|8.5\nUniMTS|**29.1**|**26.8**|**18.9**|**15.7**|**12.0**|**9.4**\n\nWe observe that UniMTS also consistently outperforms the current state-of-the-art method on the Ego4D benchmark, demonstrating the effectiveness and generalizability of our pre-training framework. \n\nc. We will explore combining visual features with inertial signals for action recognition as future work. We believe that inertial signals will help the visual features to learn implicit physical constraints, potentially leading to better generalization. We will also include these benchmarks as references in the final version.\n\n[2] Generalizability and potential edge cases\n\nAs we discussed in the “Limitation and Future Work” section in the paper, the simulated signals are for body joints, but real signals might be collected near – rather than directly on – the body joints. For example, sensors on smartwatches collect data near the wrist, not on the wrist joint itself. We plan to incorporate random offset vectors to better simulate real-world signal variations near joints. Other potential edge cases include missing data and multi-device synchronization for real-time motion time series classification, which we plan to address by simulating missing data across time and devices during pre-training.\n\nReference\n\n[1] Tan, S., Nagarajan, T. and Grauman, K., Egodistill: Egocentric head motion distillation for efficient video understanding, NeurIPS 2023.\n\n[2] Liu, M., Ma, L., Somasundaram, K., Li, Y., Grauman, K., Rehg, J.M. and Li, C., Egocentric activity recognition and localization on a 3d map, ECCV 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tJIn1sZTdf", "reviewer": "Reviewer_jvaP", "summary": "This paper presents a novel pretraining method for IMU time series (i.e., acceleration + angular velocity). The authors make use of human skeleton sequences and simulate IMU measurements by calculating the acceleration and angular speed at each joint. These simulated data are then used to pre-train a spatial-temporal graph encoder. Since each skeleton is paired with texts, the graph encoder is trained with contrastive learning against CLIP text features. Augmentations on IMU data and texts are performed to boost the performance.", "questions": "Please refer to weakness", "limitations": "The authors have adequately addressed the limitations", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of generating IMU measurements from skeleton sequences is very interesting. On one hand, it brings a huge amount of plausible motion time series for pre-training. On the other hand, it could draw connections between IMU and other modalities. \n\n2. The authors conduct extensive experiments and the proposed method consistently outperforms other baselines in both zero-shot, few-shot, and full-shot settings. The results look promising to me.", "weaknesses": "1. I have some concerns about the quality of the generated IMU data as it is constrained to 20Hz due to the original skeleton sequences. This sampling frequency might be enough for action recognition but it is unknown whether it can be used for other tasks such as inertial navigation. Besides, some sensor-level distortion such as IMU drift or some spike noises cannot be simulated using the proposed method.\n\n2. What is the model size of the graph encoder? Since the model is trained on IMUs from many joints (>20?), the model might be huge when fine-tuned on a dataset that has fewer IMU sensors (e.g., 2 sensors), which could make the model inefficient for real-world applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the valuable feedback, and for recognizing the novelty of our approach and the promising results from our experiments. We have addressed each of the comments as below.\n\n[1] Quality of generated IMU data\n\nUniMTS is aimed for action recognition, so the sampling frequency of 20Hz is sufficient. We leave other tasks such as inertial navigation as future work as we discussed in the “Limitation and Future Work” section in the paper. Our current simulation process has incorporated IMU drift as offset vectors. We will make this clear in the final version. Spike noises are relatively uncommon to occur over a long period of observation window (e.g., a few seconds), especially considering that modern wearable devices have built–in low pass filters, and thus have minimal impact on downstream classification. Therefore, we have chosen to incorporate the more common Gaussian noise during simulation, as specified in Equation 4 in the paper. We will leave exploring other types of noises as future work.\n\n[2] Efficiency of graph encoder\n\nOur graph encoder contains only 4.94M parameters, which is significantly smaller compared with the 18.69M used in the IMU encoder of the best existing baseline ImageBind. On one example dataset of UCI-HAR, full-shot fine-tuning of UniMTS takes ~1.3 minutes to converge while it takes ~9.8 minutes for ImageBind to converge. Moreover, we have run a power estimate assuming 0.1Hz cadence (i.e., 10-second window size), and it takes ~22.64 mW to run the whole graph model on an eNPU (embedded Neural Processing Unit), which is much smaller than ImageBind IMU encoder’s power consumption of ~702 mW. Therefore, our model is efficient for real-world applications and suitable to be deployed on edge devices.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jPrSICAQHK", "reviewer": "Reviewer_7idQ", "summary": "This paper introduces a unified pre-training procedure for motion time series that generalizes across diverse device latent factors and activities. A contrastive learning framework that aligns motion time series with text descriptions enriched by large language models and the spatio-temporal graph networks are utilized to capture the relationships across joints for generalization across different device locations. Experimental results show the superior performance of the proposed method that selected baselines.", "questions": "1. What is the motivation for the usage of LLM?\n2. Can the proposed method improve the efficiency of existing methods (e.g., baselines)?\n3. How do you obtain the text descriptions?", "limitations": "Please see the weaknesses and questions.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. The paper propose a unified pre-training framework for motion time series based on contrastive learning. Specifically, the proposed framework introduce LLM to incorporate the knowledge of text descriptions into time series modeling.\n3. Extensive experiments demonstrate the effectiveness of the proposed method.", "weaknesses": "1. It would be better to provide more descriptions regarding the usage of LLM.\n2. It would be better to discuss the space and time complexities of the proposed framework.\n3. As efficiency is important in real-world applications. It would suggest conducting experiments to compare the proposed method's training time (including pre-training time) with baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the valuable comments and for recognizing the effectiveness and clarity of our work. We have addressed each comment as below.\n\n[1] How to obtain text descriptions\n\nWe use the original text descriptions in the motion skeleton dataset HumanML3D. As detailed in Appendix A.1 of our paper, there is an average of 3 textual descriptions paired with each motion skeleton sequence in the HumanML3D dataset, resulting in a total of 44,970 textual descriptions with a vocabulary size of 5,371. Some example text descriptions are “the standing person kicks with their left foot before going back to their original stance”, “a man lifts something on his left and places it down on his right”, “a person walks slowly forward then toward the left hand side and stands facing that direction”. \n\n[2] Usage of LLM\n\nWe apply LLM to generate paraphrases of original text descriptions in the HumanML3D dataset to further increase the diversity. This allows UniMTS to better learn the text semantics and generalize to varied label descriptions in the downstream datasets. For instance, consider the initial description of a skeleton motion: “a person falls to the ground in a sitting motion and then pops back up in a standing position”. From this, the LLM generates three distinct paraphrases: “a person drops into a seated position before quickly rising to their feet”, “a man crouches down and quickly jumps up”, and “someone squats low and then springs up”. These variations substantially broaden the range of text descriptions, helping the model in generalizing across various label names in the downstream datasets.\n\n[3] Space and time complexities compared with baselines\n\na. For space complexity, our graph encoder contains only 4.94M parameters, which is significantly smaller compared with the 18.69M used in the IMU encoder of the best existing baseline ImageBind.\n\nb. For time complexity, our pre-training takes a one-time 32 hours. Our fine-tuning is also efficient. On one example dataset of UCI-HAR, full-shot fine-tuning of UniMTS takes ~1.3 minutes to converge while it takes ~9.8 minutes for ImageBind to converge. Moreover, we have run a power estimate assuming 0.1Hz cadence (i.e., 10-second window size), and it takes ~22.64 mW to run the whole graph model on an eNPU (embedded Neural Processing Unit), which is much smaller than ImageBind IMU encoder’s power consumption of ~702 mW. Therefore, our model is efficient for real-world applications and suitable to be deployed on edge devices.\n\n[4] Our proposed method can improve the efficiency of existing methods \n\na. As we discussed in Response [3], UniMTS is more efficient than baselines in terms of both time and space complexities, and converges faster during fine-tuning.\n\nb. Our synthetic data pre-training also provides an effective initialization which further improves the fine-tuning efficiency. For example, fine-tuning on downstream datasets takes on average 10 epochs to converge. Moreover, such benefits are model-agnostic and we can also improve the fine-tuning efficiency of baseline models when they are pre-trained on our synthetic data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aOPeNCDRA8", "reviewer": "Reviewer_G59f", "summary": "In the context of time series tasks, the paper addresses motion time series data collected from mobile and wearable devices such as smartphones and smartwatches. However, training and testing on the same dataset leads to poor generalizability of the models. The authors propose UniMTS, a unified pre-training procedure for motion time series that generalizes across diverse device latent factors and activities.", "questions": "- Why cannot domain adaptation be utilized to address the shift in data or continual learning to adapt to new activities, etc.?", "limitations": "- The limitation is that the main contribution of the paper is utilizing LLMs to improve the generalizability of time series models.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is well written and easily understandable.\n- The soundness of the technical claims are good and the experiments support the proposed method.\n- The problem setup is well motivated. The addressed problem exists in many application areas and research fields.\n- The paper proposed a large experiments on many different datasets showing a higher performance than state-of-the-art methods.", "weaknesses": "- After reading the Abstract, the actual contribution of the paper is unclear. The Abstract describes several minor contributions, but what is the focus of the paper?\n- It is unclear which large language model is used (should be mentioned earlier).\n- Does the proposed method actually fit to NIPS? The contributions are limited/unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude for the reviewer’s constructive feedback, and for recognizing the soundness of our technical claims, the scale and performance of our experiments, as well as the clarity of our paper’s presentation. We have addressed each of the concerns as below.\n\n[1] Contribution of the paper\n\na. We propose the **first unified pre-training framework** for motion time series. While large-scale pre-training is common in vision and audio domains, it is challenging to develop similar pre-trained models given the diversity of motion time series data due to variations in device locations, device mounting orientations, and human activity types across different datasets. Existing works mostly train and test their models on a single type of datasets with limited generalizability. Our proposed UniMTS framework **for the first time** addresses all these challenges and generalizes to diverse downstream datasets.\n\nb. **UniMTS generalizes across datasets with various device locations**. Real-world motion time series datasets typically focus on different device locations, and no single real-world dataset provides comprehensive location coverage. To address this gap, we propose to synthesize motion time series from motion skeletons with all-joint coverage. By utilizing a graph encoder, we effectively model the spatio-temporal correlations across these joints, enhancing the generalizability of UniMTS across diverse device locations.\n\nc. **UniMTS also generalizes across device orientations**. The device mounting orientations could significantly affect the downstream performance. To ensure the model remains robust regardless of how the devices are oriented, we propose to use rotation-invariant augmentation techniques during pre-training.\n\nd. **Additionally, UniMTS generalizes across different activity types**. We align motion time series with text semantics through contrastive learning, enabling classification beyond predefined class labels and supporting arbitrary activity types. To improve the diversity and generalization of activity type representations, we further use text augmentations from LLMs.\n\n**We would like to highlight that utilizing LLMs is *not* our main contribution**. Our main contribution is the novel pre-training framework for motion time series, which represents a significant advancement unseen in existing literature. LLMs are just one of the techniques we use to augment text descriptions for enhanced pre-training activity diversity. In addition to this technique, we have proposed various methods, as outlined above, to improve generalization across multiple aspects.\n\n[2] Which LLM is used\n\nWe use GPT-3.5 (“gpt-3.5-turbo”) as discussed in Section 4.1 in the paper. We will make sure to mention this in earlier sections in the final version.\n\n[3] Paper scope\n\nOur work is of significant relevance to various key topics that have a wide audience in NeurIPS, including time series analysis, machine learning for healthcare, pre-training, physics-based simulation, and motion modeling. One of the key challenges in these communities is to improve model’s generalizability across diverse datasets. Our contribution falls under this area by designing the first unified pre-training procedure for motion time series, which successfully generalizes to various device locations, device orientations and activities. We believe our insights are valuable to the community and align well with the scope of NeurIPS.\n\n[4] Domain adaptation and continual learning methods\n\nExisting domain adaptation methods mostly assume that source and target datasets share the same label names and have the same number of activities, such as cross-user domain adaptation and cross-dataset domain adaptation **only for those common classes** [1-3]. We aim for a more generic yet challenging generalization scenario where pre-training and downstream datasets share different label names. Therefore, existing domain adaptation methods generally cannot be applied to address the activity generalization challenge that our work seeks to overcome. Moreover, continual learning [4] typically involves training on the new activities with the risk of catastrophic forgetting. In contrast, our model is able to generalize to new activities in a zero-shot fashion. We will include domain adaptation and continual learning methods as references and make this clear in the final version. \n\nReference\n\n[1] Hu, R., Chen, L., Miao, S., & Tang, X, SWL-Adapt: An unsupervised domain adaptation model with sample weight learning for cross-user wearable human activity recognition, AAAI 2023.\n\n[2] Lu, W., Wang, J., Chen, Y., Pan, S. J., Hu, C., & Qin, X, Semantic-discriminative mixup for generalizable sensor-based cross-domain activity recognition, IMWUT 2022.\n\n[3] Hong, Z., Li, Z., Zhong, S., Lyu, W., Wang, H., Ding, Y., He, T. and Zhang, D., CrossHAR: Generalizing Cross-dataset Human Activity Recognition via Hierarchical Self-Supervised Pretraining, IMWUT 2024.\n\n[4] Jha, S., Schiemer, M., Zambonelli, F. and Ye, J., Continual learning in sensor-based human activity recognition: An empirical benchmark analysis, Information Sciences, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7buIVnqsva", "reviewer": "Reviewer_6pyv", "summary": "This paper proposes a unified pre-training framework using simulated data from body skeleton model for motion time series. The authors rightly pointed out the challenges of collecting motion time series at scale due to privacy concerns despite the fact that the motion time series data have great promises in a wide range of applications. Thus, this is an area with few pre-trained models unlike other data modalities. \n\nThe simulation framework proposed was used to generate the synthetic data needed for pre-training. The author then utilised the generated motion time series in contrastive learning using a graph neural network that learns a joint embedding for sensor data at different body locations. The extensive evaluations of the pre-trained model demonstrated superior performance for human activity recognition in several settings, including zero-hot, few-shot and full-shot performance.", "questions": "1. What your input data format during fine-tuning and sampling frequency? \n2. Are Model weights and pre-trained model made available?", "limitations": "The authors have addressed this adequately.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "* The manuscript was well-written. I thoroughly enjoyed reading it. I particularly like that the paper clearly points out the methodological gap for motion time series data to the ML audience at NeurIPS. \n* The simulation framework is an important contribution to the ubiquitous computing community, for which large-scale datasets are difficult to obtain.\n* The use of GNNs to learn a joint embedding for wearable sensors is novel. It is great that the GNN implementation can also account for downstream tasks by using a masking trick when not all sensor modes are available! \n* The performance evaluations were extensive in terms of both benchmark datasets and baseline models included.", "weaknesses": "1. Despite my general excitement about this paper, this paper has not recognised the body of work already done in large-scale pre-trained models for motion time series. The authors might not be familiar with this area of work because existing pre-trained models generally use Biobank and are published in journals. It would be great if the authors could compare the following work in the context of their results: \n    1. Yuan, Hang, et al. \"Self-supervised learning for human activity recognition using 700,000 person-days of wearable data.\" NPJ digital medicine 7.1 (2024): 91.\n    2. Spathis, Dimitris, et al. \"Self-supervised transfer learning of physiological representations from free-living wearable data.\" Proceedings of the Conference on Health, Inference, and Learning. 2021.\n2. Pre-training from synthetic data is a cost-effective approach. Its performance is likely to plateau at some point as compared to pre-trained using real data. It would be interesting to see a comparison with Yuan et al. 2024 as this model was pre-trained on 700,000 person-days of real data. I also appreciate that adding an experiment might not be realistic, but I do believe that it should be fairly easy to do the fine-tuning. \n3. The HumanML3D human selection data only spans over a mere 29 hours. I am very suspicious that any pre-training on just 29 hours of synthetic data will be enough for pre-training as scaling laws go. Even though the current results show good performance, if I am not wrong, the majority of the benchmarks were collected in the lab, which doesn't represent realistic human behaviour in a free-living environment. CAPTURE-24 could be another interesting benchmark as it is one of the largest open-access free-living activity recognition datasets. More importantly, it would be good to get a sense of how the downstream performance scales to your pre-training data so that we can determine whether the benefits of pre-training have plateaued. \n4. Also, you probably don't have to add the references below, but I just want you to know that there are several biobanks that contain large-scale wearable sensor data that can be used for pre-training. Your work is still valuable as your can support multi-sensors, whilst most of the existing large-scale datasets only contain wrist-worn devices.\n\t1. Doherty, Aiden, et al. \"Large scale population assessment of physical activity using wrist worn accelerometers: the UK biobank study.\" PloS one 12.2 (2017): e0169649.\n\t2. Chen, Yuanyuan, et al. \"Device-measured movement behaviours in over 20,000 China Kadoorie Biobank participants.\" International Journal of Behavioral Nutrition and Physical Activity 20.1 (2023): 138.\n\t3. Master, Hiral, et al. \"Association of step counts over time with the risk of chronic disease in the All of Us Research Program.\" Nature medicine 28.11 (2022): 2301-2308.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer for all the valuable feedback and for recognizing our technical contributions, novelty of our method, thoroughness of our experiments and the clarity of our paper presentation. We have addressed the comments as below.\n\n[1] Baselines trained on BioBank\n\nWe thank the reviewer for mentioning these related works. We will make sure to include them in the final version. We have conducted experiments to compare UniMTS with the most recent work of Yuan et al. 2024 which is pre-trained on the BioBank dataset. We would like to emphasize that\n\na. The BioBank dataset is limited to signals collected from wrist-worn devices, offering less diversity compared to our synthetic dataset that covers joint locations across the entire body.\n\nb. Yuan et al. 2024 is pre-trained through self-supervised learning and does not involve text semantics aligning, so it cannot be applied for zero-shot recognition as UniMTS does. This highlights the advantages and generalizability of UniMTS. \n\nWe compare the few-shot fine-tuning and full-shot performances of UniMTS vs Yuan et al. 2024 respectively in **Figure 1** and **Table 3** of the attached PDF in the general response. UniMTS performs consistently better in both few-shot and full-shot scenarios, demonstrating the effectiveness of our pre-training pipeline. \n\n[2] Add Capture-24 in pre-training\n\nWe have conducted experiments by incorporating Capture-24 dataset into pre-training. We randomly sample it to 10% of our synthetic dataset size, in order to mitigate any bias due to its larger size. We report the zero-shot performance of UniMTS after pre-training with Capture-24 in **Table 2** of the attached PDF in the general response. \n\nNotably, performance improvements are primarily observed in downstream datasets that utilize devices attached near the wrist (e.g., WISDM, Wharf) and datasets with simpler activities (e.g., UCI-HAR), which correlates with Capture-24’s collection procedure with wrist-worn devices and limited activity descriptions. However, there is a slight decline in overall performance, attributed mainly to the dataset's focus on a single device location and the poorer quality of label descriptions. \n\nThese findings highlight the significance of our synthetic data pre-training, which has high-quality all-joint coverage and detailed, diverse text descriptions mostly absent in current real-world datasets, facilitating generalization across varied activities and devices. We also note a similar trend when increasing the proportion of Capture-24 in our pre-training corpus.\n\n[3] New references \n\nWe thank the reviewer for pointing to these related works. We will ensure to add these references in the final version.\n\n[4] Data format during fine-tuning and sampling frequency\n\nThe input data are of shape [batch size, sequence length, number of joints, number of feature channels] during fine-tuning. We assign data to specific joint locations according to where the downstream devices are attached, and assign zeros to the remaining joint locations. The sampling frequency is 20 Hz, consistent with the HumanML3D dataset, which is enough for activity recognition tasks. \n\n[5] Release of pre-trained model weights \n\nWe will release both pre-trained model weights and all the code upon acceptance of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bC7uNTckti", "reviewer": "Reviewer_s9ZD", "summary": "Overall, this is a paper about human activity recognition from motion time series data. Authors design a unified framework for data generation, pre-training, and evaluation. The data used for pre-training is generated by motion skeleton data. LLM is also involved in text augmentation. Experimental results on 18 datasets demonstrate the superior performance of this paper.", "questions": "I have some questions regarding the experimental design, baselines, related works, novelty, etc. I put them in the limitation section.", "limitations": "[1] The data generation is interesting. I am thinking about what if we replace the generated data with a few real data for model training, how will the performance look like. Or, please see my question in [8], how to justify the importance of synthesized data. \n\n[2] Missing related work. There are important studies of pretraining motion time series missing. Authors can check more carefully, e.g., from sensys, kdd. \n\n[3] For the random rotation method, there are multiple existing approaches. Authors need to justify the difference or add citations. For example, \"Device Orientation Independent Human Activity Recognition Model for Patient Monitoring Based on Triaxial Acceleration”, “Data augmentation of wearable sensor data for Parkinson’s disease monitoring using convolutional neural networks”,  and a few other papers from Mobicom and Ubicomp also applied similar methods.  \n\n[4] Also, for the rotation method, does Acc and Gyro share the same rotation? \n\n[5] I am wondering is there any data normalization? Given the distribution discrepency among synthesized data and real data, it might hurt the performance. Currently, I only see a normalized matrix in Line 156. \n\n[6] For the data generation module, I like the idea of utilizing physical laws. However, it seems this generation method is not proposed by this manuscript. Therefore, authors might want to justify the technical contribution. \n\n[7] Fig 7 looks weird. Does it mean ImageBind cannot get any samples correct for most activities? Also, how the proposed method and ImageBind infer the activities not seen before should be described in more detail. \n\n[8] The performance gain of the zero-shot setting is impressive. For zero-shot setting, how are baseline models are trained? Are they fine-tuned also with the generated data? \n\n[9] In line 131, it would be great if there is some documents can support the claim. \n\n[10] I guess the authors aim to address the generalization issue of HAR, e.g., device locations, orientations. There are quite a few studies in these area, for domain adaptation  domain generalization. Some of them explore augmentation, some explore pretraining, etc., to address the challenges mentioned in the introduction. \n\n[11] For zero-shot setting, the performance seems too low for real-world usage, e.g., for datasets with only 4 or 6 classes, the acc is smaller than 50%. \n\n[12] Authors mention cross-dataset a few times in the paper, is there some experiments focusing on this setting except the zero-shot experiment? \n\n[13] Another concern I have is the full-shot setting, there are a bunch of SoA models from AAAI, KDD, IMWUT. I am not sure whether the provided baselines are representative. However, it is good to see authors replace some baselines from zero-shot settings.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[1] This paper is well-written, figures are well-designed. \n\n[2] The idea of synthesizing data for pretraining motion time series is interesting. \n\n[3] 18 real-world datasets are involved for evaluation, multiple activities are included.", "weaknesses": "Overall, I think this is an interesting paper and is promising. Therefore, I provide lots of comments. However, it has some obvious potential flaws in the current version. Please see limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the valuable feedback and for recognizing our idea, promising results and clear presentation. We have addressed the comments as below.\n\n[1] Importance of synthesized data\n\nWe add one experiment by incorporating Capture-24 (one of the largest real data from free-living environments) into pre-training. We randomly sample it to 10% of our synthetic dataset size to mitigate any bias due to its larger size. **Table 2** in the general response PDF shows performance gains mainly in datasets with wrist-worn devices (e.g., WISDM, Wharf) and simpler activities (e.g., UCI-HAR), as Capture-24 was collected with wrist-worn devices and limited activity descriptions. However, overall performance slightly declines due to its single device focus and poor label description quality. This highlights the significance of our synthesized data, which has all joint locations and detailed text descriptions mostly absent in current real-world datasets, facilitating generalization across diverse activities and devices.\n\n[2-3] More related works\n\nWe appreciate the additional references (pre-training motion time series [1-3], random rotations [4-6]) and will make sure to include them in the final version. We are also open to including any further related works as suggested by the reviewer.\n\n[4] Rotation\n\nYes, they share the same rotation. Conversations with leading sensor vendors like STM, Invensense and Bosch confirm that acc and gyro are typically calibrated to have negligible misalignment factors, so we can use the same rotation. \n\n[5] Data normalization\n\nWe normalize the data to ensure consistency in unit measurements, e.g., standardizing accelerations to $m/s^2$. We will make this clear in the final version.\n\n[6] Data generation \n\nWe acknowledge that there exist simulators that help generate motion time series. However, unlike previous works, we jointly synthesize data across the entire body, and model their correlations via graph encoder for better location generalization. We also align time series with text semantics for activity generalization, presenting the first unified pre-training framework.\n\n[7] Zero-shot generalization\n\nImageBind pre-trained on Ego4D data with only head-mounted devices cannot generalize to most activities involving different device placements. This highlights the importance of synthesized data covering all joints. UniMTS and ImageBind predict unseen activities by matching time series embeddings with text label candidates, selecting the one with the highest similarity score as shown in Figure 3 of the paper.\n\n[8] Zero-shot setting\n\nBaselines are trained on their original pre-training corpus (e.g., ImageBind, IMU2CLIP on Ego4D) without fine-tuning on synthetic data. We add one experiment by continually training IMU2CLIP on our synthetic dataset as shown in **Table 1** of the general response PDF. IMU2CLIP improves after training on the synthetic data, which again highlights the importance of our synthesized data pre-training. Yet, it still underperforms UniMTS, as our graph encoder better models cross-device spatio-temporal relationship. \n\n[9] Documents\n\niOS uses inertial force and Android uses accelerating force. We noted such sign discrepancy when collecting data using devices from both OS platforms. This is also confirmed in the Sensor Logger App manual. Rebuttal policy excludes external links, and we will include them in the final version.\n\n[10] Domain adaptation (DA)\n\nExisting DA methods mostly assume identical label names and activity counts between source and target, such as cross-user and cross-dataset DA only for common classes [7-9]. We aim for a more challenging scenario with varying label names across datasets, which existing DA generally cannot address. We will include DA methods as references and make this clear in the final version.\n\n[11] Zero-shot performance\n\nReal-world motion time series classification in a zero-shot manner is extremely challenging, yet we’ve shown a 342.3% improvement over the best baseline. The score also averages \\~60% when we retrieve the top-2 activities. Moreover, fine-tuning on 1-shot/10-shot per class further enhances the model to \\~60%/\\~75% for most datasets. \n\n[12] Cross-dataset experiment\n\nThe zero-shot experiment shows how our pipeline generalizes across datasets. It works the same by substituting the synthetic data with others like Capture-24 (see Response [1]). We use synthetic data for its high quality with all-joint coverage and detailed descriptions.\n\n[13] Full-shot baseline\n\nOur compared baselines like GPT4TS (NeurIPS'23), SHARE (CIKM'23), ImageBind (ICCV'23), IMU2CLIP (EMNLP'23) are all SOTA general motion time series classification models. There are more recent papers but they mostly focus on sub-domains such as low-resource [1] or federated learning [10] scenarios instead of general classification.\n\nReference\n\n[1] Generalizable low-resource activity recognition with diverse and discriminative representation learning, KDD 2023\n\n[2] What makes good contrastive learning on small-scale wearable-based tasks? KDD 2022\n\n[3] Limu-bert: Unleashing the potential of unlabeled data for imu sensing applications, SenSys 2021\n\n[4] Device orientation independent human activity recognition model for patient monitoring based on triaxial acceleration\n\n[5] Data augmentation of wearable sensor data for Parkinson’s disease monitoring using convolutional neural networks\n\n[6] Practically Adopting Human Activity Recognition, MobiCom 2023\n\n[7] SWL-Adapt: An unsupervised domain adaptation model with sample weight learning for cross-user wearable human activity recognition, AAAI 2023\n\n[8] Semantic-discriminative mixup for generalizable sensor-based cross-domain activity recognition, IMWUT 2022\n\n[9] CrossHAR: Generalizing Cross-dataset Human Activity Recognition via Hierarchical Self-Supervised Pretraining, IMWUT 2024\n\n[10] Flames2graph: An interpretable federated multivariate time series classification framework, KDD 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all the reviewers for their valuable comments and feedback. We appreciate that nearly all the reviewers recognized the strengths of our work as follows:

a. **Novelty**: high original approach (4CUw), novel method (jvaP, 6pyv).

b. **Importance**: address a critical gap, significant impact (4CUw), well motivated (G59f), important contribution to the community (6pyv).

c. **Promising results supported by extensive experiments**: thorough experiments, impressive performance improvements (4CUw), extensive experiments, promising results (jvaP), superior performance (7idQ, 6pyv, s9ZD), sound technical claims supported by large experiments showing higher performance than SOTA (G59f).

d. **Clarity**: clearly written and well-organized (4CUw), thoroughly enjoyed reading (6pyv), well-written (7idQ, G59f, 6pyv, s9ZD).

We have addressed all the comments from the reviewers, and the major results are as below.

a. We observe better performance on the Ego4D benchmark (4CUw).

b. We show that UniMTS is efficient in terms of both space and time complexities (jvaP, 7idQ).

c. We clarify that our main contribution is designing the first unified pre-training framework for motion time series that generalizes to various downstream device latent factors and activities (G59f).

d. We show better performance compared with pre-trained models on BioBank (6pyv).

e. Our synthetic data generation is critical and model-agnostic. It can also improve the training efficiency for baselines (s9ZD).

f. We show the importance and diversity of our synthesized data by comparing it with pre-training that incorporates real-world datasets with limited joint coverage and text semantics (Capture-24) (6pyv, s9ZD).

Meta-review: The paper proposes a pre-training strategy, specifically designed for motion time series from mobile sensors. The contributions lie in a contrastive learning framework and the alignment of time series with textual descriptions from LLMs. IMU measurements are simulated for each joint and used to pre-train a graph encoder. Doing so, the approach addresses common technical challenges such as variations in the device placements, difference in sensor orientations, and variable activity. On a large benchmark study of 18 datasets the authors show significant improvements over the state of the art in key metrics and in settings such as zero-sho, few-shot and full-shot. As a results, the approach also allows to use a pre-trained model that generalizes well to slight variations in the setup (e.g. across orientations, placements, ...). The reviewers mainly raised questions about the clarity of technical details which have all been resolved during the rebuttal period. 

Decision:
After an intensive discussion all reviewers voted for clear acceptance. I therefore recommend to accept the paper and encourage the authors to use the feedback provided to improve the paper for its final version.
Decision: Accept (poster)