UniMTS: Unified Pre-training for Motion Time Series

Authors: xiyuan  zhang
diyan  teng
ranak roy chowdhury
shuheng  li
dezhi  hong
rajesh k gupta
jingbo  shang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-08 09:04:17

Abstract: Motion time series collected from low-power, always-on mobile and wearable devices such as smartphones and smartwatches offer significant insights into human behavioral patterns, with wide applications in healthcare, automation, IoT, and AR/XR. However, given security and privacy concerns, building large-scale motion time series datasets remains difficult, hindering the development of pre-trained models for human activity analysis. Typically, existing models are trained and tested on the same dataset, leading to poor generalizability across variations in device location, device mounting orientation, and human activity type. In this paper, we introduce UniMTS, the first unified pre-training procedure for motion time series that generalizes across diverse device latent factors and activities. Specifically, we employ a contrastive learning framework that aligns motion time series with text descriptions enriched by large language models. This helps the model learn the semantics of time series to generalize across activities. Given the absence of large-scale motion time series data, we derive and synthesize time series from existing motion skeleton data with all-joint coverage. We use spatio-temporal graph networks to capture the relationships across joints for generalization across different device locations. We further design rotation-invariant augmentation to make the model agnostic to changes in device mounting orientations. Our model shows exceptional generalizability across 18 motion time series classification benchmark datasets, outperforming the best baselines by 340% in the zero-shot setting, 16.3% in the few-shot setting, and 9.2% in the full-shot setting.

Main paper: 
 introduction: 
 Recognition of human motion using time series from mobile and wearable devices, such as accelerations and angular velocities, is widely adopted as key context information for various applications from health condition monitoring [4] , sports activity analysis [1] to user habit studies [50] . Compared with vision-based approaches, methods based on motion sensor time series offer more energy-efficient and cost-effective solutions with enhanced privacy protection [54] , making them preferable. While valuable, collecting motion time series data at large scale remains challenging due to security or privacy concerns. Labeling motion time series proves even more difficult as such data cannot be easily interpreted by humans for post annotation. This results in data insufficiency that impedes development of supervised learning methods. In other fields such as natural language processing [39, 56] and computer vision [44, 30] , pre-trained foundation models have shown remarkable performance in such settings with insufficient data. However, in the motion time series domain, lack of comprehensive datasets and an effective pre-training task makes it difficult to similarly develop pre-trained models that can operate with limited data. Typically, existing models perform training and testing on the same dataset, and struggle to generalize across different datasets given the following three unique challenges within the motion time series problem domain.We summarize these three unique generalization challenges in Figure 1 . First of all, variation in device placement during deployment poses a significant issue; for instance, data from a smartwatch on the wrist vary considerably from data gathered from a smartphone near the upper leg. Therefore, models trained on data from one body location can barely generalize to others during the testing phase. Secondly, devices can experience arbitrary orientations during data collection, making it difficult for models trained on specific device orientations to adapt to new ones during deployment. Thirdly, different motion time series datasets can be focused on different types of human activities. For example, some datasets aim to identify stationary activities such as lying or sitting, while others concentrate on dynamic movements such as walking or cycling. Models trained on specific types of activities typically struggle to generalize to new activities introduced by other datasets.We introduce UniMTS, the first Unified pre-trained model for Motion Time Series to address all the above three generalization issues, achieving state-of-the-art zero-shot and fine-tuning performance. UniMTS follows a contrastive learning framework that aligns motion time series with LLM-enriched textual descriptions to learn the time series semantics for activity generalization. To prepare largescale motion time series for pre-training, we synthesize these time series based on existing extensive motion skeleton data [19] with comprehensive coverage of different body locations. We model these synthesized time series using graph networks to capture the spatio-temporal relationships across devices for location generalization. We further implement rotation-invariant augmentation to ensure the model's robustness to any device orientation during testing.We summarize our primary contributions as follows:• We introduce the first unified pre-training procedure for motion time series, UniMTS, which successfully generalizes to various device locations, device orientations and activities. • We design a contrastive learning framework to align motion time series with corresponding semantic meanings for activity generalization. For device location generalization, we propose to synthesize motion time series covering various body locations and model their spatio-temporal correlations using graph convolutional neural networks. We also design rotation-invariant augmentation to make the model agnostic to different device orientations. • Our pre-trained model demonstrates state-of-the-art performance across 18 real-world motion time series benchmark datasets, notably with performance improvement of 340% in the zero-shot setting, 16.3% in the few-shot setting, and 9.2% in the full-shot setting, compared with the respective best-performing baselines. 
related work: 
 Conventional motion time series classification approaches train a dedicated classifier for each dataset, and can be categorized into statistical feature extraction methods [15] and deep learning K c C Z y W e q n G h L I x H W L X U k k j 1 H 4 2 P 3 R K z q w y I G G s b E l D 5 u r v i Y x G W k + i w H Z G 1 I z 0 s j c T / / O 6 q Q m v / Y z L J D U o 2 W J R m A p i Y j L 7 m g y 4 Q m b E x B L K F L e 3 E j a i i j J j s y n Z E L z l l 1 d J + 6 L q X V Z r z V q l f p P H U Y Q T O I V z 8 O A K 6 n A H D W g B A 4 RM S q H V C N g k t s G G 4 E t h O F N A o E t o L R 7 c x v P a L S P J b 3 Z p y g H 9 G B 5 C F n 1 F i p / t Q r l t y y O w d Z J V 5 G S p C h 1 i t + d f s x S y O U h g m q d c d z E + N P q D K c C Z w W u q n G h L I R H W D H U k k j 1 P 5 k f u i U n F m l T 8 J Y 2 Z K G z N X f E x M a a T 2 O A t s Z U T P U y 9 5 M / M / r p C a 8 9 i d c J q l B y R a L w l Q Q E 5 P Z 1 6 T P F T I j x p Z Q p r i 9 l b A h V Z Q Z m 0 3 B h u A t v 7 x K m h d l 7 7 J c q V d K 1 ZM S q H V C N g k t s G G 4 E t h O F N A o E t o L R 7 c x v P a L S P J b 3 Z p y g H 9 G B 5 C F n 1 F i p / t Q r l t y y O w d Z J V 5 G S p C h 1 i t + d f s x S y O U h g m q d c d z E + N P q D K c C Z w W u q n G h L I R H W D H U k k j 1 P 5 k f u i U n F m l T 8 J Y 2 Z K G z N X f E x M a a T 2 O A t s Z U T P U y 9 5 M / M / r p C a 8 9 i d c J q l B y R a L w l Q Q E 5 P Z 1 6 T P F T I j x p Z Q p r i 9 l b A h V Z Q Z m 0 3 B h u A t v 7 x K m h d l 7 7 J c q V d K 1 ZK c C Z y W e q n G h L I x H W L X U k k j 1 H 4 2 P 3 R K z q w y I G G s b E l D 5 u r v i Y x G W k + i w H Z G 1 I z 0 s j c T / / O 6 q Q m v / Y z L J D U o 2 W J R m A p i Y j L 7 m g y 4 Q m b E x B L K F L e 3 E j a i i j J j s y n Z E L z l l 1 d J + 6 L q X V Z r z V q l f p P H U Y Q T O I V z 8 O A K 6 n A H D W g B A 4 RM S q H V C N g k t s G G 4 E t h O F N A o E t o L R 7 c x v P a L S P J b 3 Z p y g H 9 G B 5 C F n 1 F i p / t Q r l t y y O w d Z J V 5 G S p C h 1 i t + d f s x S y O U h g m q d c d z E + N P q D K c C Z w W u q n G h L I R H W D H U k k j 1 P 5 k f u i U n F m l T 8 J Y 2 Z K G z N X f E x M a a T 2 O A t s Z U T P U y 9 5 M / M / r p C a 8 9 i d c J q l B y R a L w l Q Q E 5 P Z 1 6 T P F T I j x p Z Q p r i 9 l b A h V Z Q Z m 0 3 B h u A t v 7 x K m h d l 7 7 J c q V d K 1 ZM S q H V C N g k t s G G 4 E t h O F N A o E t o L R 7 c x v P a L S P J b 3 Z p y g H 9 G B 5 C F n 1 F i p / t Q r l t y y O w d Z J V 5 G S p C h 1 i t + d f s x S y O U h g m q d c d z E + N P q D K c C Z w W u q n G h L I R H W D H U k k j 1 P 5 k f u i U n F m l T 8 J Y 2 Z K G z N X f E x MK c C Z y W e q n G h L I x H W L X U k k j 1 H 4 2 P 3 R K z q w y I G G s b E l D 5 u r v i Y x G W k + i w H Z G 1 I z 0 s j c T / / O 6 q Q m v / Y z L J D U o 2 W J R m A p i Y j L 7 m g y 4 Q m b E x B L K F L e 3 E j a i i j J j s y n Z E L z l l 1 d J + 6 L q X V Z r z V q l f p P H U Y Q T O I V z 8 O A K 6 n A H D W g B A 4 RM S q H V C N g k t s G G 4 E t h O F N A o E t o L R 7 c x v P a L S P J b 3 Z p y g H 9 G B 5 C F n 1 F i p / t Q r l t y y O w d Z J V 5 G S p C h 1 i t + d f s x S y O U h g m q d c d z E + N P q D K c C Z w W u q n G h L I R H W D H U k k j 1 P 5 k f u i U n F m l T 8 J Y 2 Z K G z N X f E x MM S q H V C N g k t s G G 4 E t h O F N A o E t o L R z d R v P a H S P J a P Z p y g H 9 G B 5 C F n 1 F j p 4 b b n 9 c o V t + r O Q J a J l 5 M K 5 K j 3 y l / d f s z S C K V h g m r d 8 d z E + B l V h j O B k 1 I 3 1 Z h Q N q I D 7 F g q a Y T a z 2 a n T s i J V f o k j J U t a c h M / T 2 R 0 U j r c R T Y z o i a o V 7 0 p u J / X i c 1 4 Z W f c Z m k B i W b L w p T Q U x M p n + T P l f I j B h b Q p n i 9 l b C h l R R Z m w 6 J R u C t /Q c 4 W X U N K O 4 + O b Q 3 g o v Y 5 g y C M = " > A A A B 9 H i c b V D L S g N B E J y N r x h f U c G L l 8 E g e A q 7 I u o x x I M e E 8 g L k m W Z n Z 0 k Q 2 Z n 1 5 n e Q F j y H V 4 8 K O J R v 8 I v 8 O b F b 3 H y O G h i Q U N R 1 U 1 3 l x 8 L r s G 2 v 6 z M y u r a + k Z 2 M 7 e 1 v b O 7 l 9 8 / a O g o U Z T V a S Q i 1 f K J Z o J L V g c O g r V i x U j o C 9 b 0 B z c T v z l k S v N I 1 m A U M z c k P c m 7 n B I w k n v r O b h D g w h w z X O 8 f M E u 2 l P g Z e L M S a F 0 V P 3 m b + W P i p f / 7 A Q R T U I m g Q q i d d u x Y 3 B T o o B T w c a 5 T q J Z T O i A 9 F j b U E l C p t 1 0 e v Q Y n x o l w N 1 I m Z K A p + r v i Z S E W o 9 C 3 3 S G B P p 6 0 Z u I / 3 n t B L r X b s p l n A C T d L a o m w g M E Z 4 k g A O u G A U x M o R Q x c 2 t m P a J I h R M T j k T g r P 4 8 j J p n B e d y + J F 1 a R R R j N k 0 T E 6 Q W f I Q V e o h O 5 Q B d U R R f f o A T 2 h Z 2 t o P V o v 1 u u s N W P N Z w 7 RV C 7 S g N B F L 3 r M 8 Z X V L C x G Q y C V d g V U c s Q C 6 0 k Q f O A Z A m z k 9 l k y M z s M j M r h C W f Y G O h i K 2 t f + E X 2 N n 4 L U 4 e h S Y e u H A 4 5 1 7 u v S e I O d P G d b + c h c W l 5 Z X V z F p 2 f W N z a z u 3 s 1 v T U a I I r Z K I R 6 o R Y E 0 5 k 7 R q m O G 0 E S u K R c B p P e h f j v z 6 P V W a R f L O D G L q C 9 y V L G Q E G y v d X r V v 2S J h N p Q E n 6 c V x X U s d R V 9 T c d 3 c = " > A A A B 7 H i c b V D L S s N A F L 2 p r x p f V Z d u B o v g K i Q i 1 Y 1 Y d O O y g m k L b S i T 6 a Q d O p m E m Y l Q Q r / B j Q t F X An + i n s 3 4 t 8 4 f S y 0 9 V w u H M 6 5 l / s I U 8 6 U d t 1 v q 7 C 0 v L K 6 V l y 3 N z a 3 t n d K u 3 t 1 l W S S U J 8 k P J H N E C v K m a C + Z p r T Z i o p j k N O G + H g e u w 3 7 q l U L B F 3 e p j S I M Y 9 w S J G s D a S 7 9 g m O q W y 6 7 g T o E X i z U j 5 8 s O + S N + + 7 F q n 9 N n u J i S L q d C E Y 6 V a n p v q I M d S M 8 L p y G 5 n i q a Y D H C P t g wV O K Y q y C f L j t C R U b o o S q R J o d F E / d 2 R 4 1 i p Y R y a y h j r v p r 3 x u J / X i v T 0 X m Q M 5 F m m g o y H R R l H O k E j S 9 H X S Y p 0 X x o C C a S m V 0 R 6 W O J i T b /s c 0 T v P m T F 0 n 9 x P E q z u m t W 6 5 e w R R F O I B D O A Y P z q A K N 1 A D H w g w e I A n e L a E 9 W i 9 W K / T 0 o I 1 6 9 m H P 7 D e f w A X P J C Q < / l a t e x i t > ... < l a t e x i t s h a 1 _ b a s e 6 4 = " 2 q W S J hN p Q E n 6 c V x X U s d R V 9 T c d 3 c = " > A A A B 7 H i c b V D L S s N A F L 2 p r x p f V Z d u B o v g K i Q i 1 Y 1 Y d O O y g m k L b S i T 6 a Q d O p m E m Y l Q Q r / B j Q t F X An + i n s 3 4 t 8 4 f S y 0 9 V w u H M 6 5 l / s I U 8 6 U d t 1 v q 7 C 0 v L K 6 V l y 3 N z a 3 t n d K u 3 t 1 l W S S U J 8 k P J H N E C v K m a C + Z p r T Z i o p j k N O G + H g e u w 3 7 q l U L B F 3 e p j S I M Y 9 w S J G s D a S 7 9 g m O q W y 6 7 g T o E X i z U j 5 8 s O + S N + + 7 F q n 9 N n u J i S L q d C E Y 6 V a n p v q I M d S M 8 L p y G 5 n i q a Y D H C P t g wV O K Y q y C f L j t C R U b o o S q R J o d F E / d 2 R 4 1 i p Y R y a y h j r v p r 3 x u J / X i v T 0 X m Q M 5 F m m g o y H R R l H O k E j S 9 H X S Y p 0 X x o C C a S m V 0 R 6 W O J i T b /s c 0 T v P m T F 0 n 9 x P E q z u m t W 6 5 e w R R F O I B D O A Y P z q A K N 1 A D H w g w e I A n e L a E 9 W i 9 W K / T 0 o I 1 6 9 m H P 7 D e f w A X P J C Q < / l a t e x i t > ... < l a t e x i t s h a 1 _ b a s e 6 4 = " 2 q W S J hN p Q E n 6 c V x X U s d R V 9 T c d 3 c = " > A A A B 7 H i c b V D L S s N A F L 2 p r x p f V Z d u B o v g K i Q i 1 Y 1 Y d O O y g m k L b S i T 6 a Q d O p m E m Y l Q Q r / B j Q t F X An + i n s 3 4 t 8 4 f S y 0 9 V w u H M 6 5 l / s I U 8 6 U d t 1 v q 7 C 0 v L K 6 V l y 3 N z a 3 t n d K u 3 t 1 l W S S U J 8 k P J H N E C v K m a C + Z p r T Z i o p j k N O G + H g e u w 3 7 q l U L B F 3 e p j S I M Y 9 w S J G s D a S 7 9 g m O q W y 6 7 g T o E X i z U j 5 8 s O + S N + + 7 F q n 9 N n u J i S L q d C E Y 6 V a n p v q I M d S M 8 L p y G 5 n i q a Y D H C P t g wV O K Y q y C f L j t C R U b o o S q R J o d F E / d 2 R 4 1 i p Y R y a y h j r v p r 3 x u J / X i v T 0 X m Q M 5 F m m g o y H R R l H O k E j S 9 H X S Y p 0 X x o C C a S m V 0 R 6 W O J i T b /s c 0 T v P m T F 0 n 9 x P E q z u m t W 6 5 e w R R F O I B D O A Y P z q A K N 1 A D H w g w e I A n e L a E 9 W i 9 W K / T 0 o I 1 6 9 m H P 7 D e f w A X P J C Q < / l a t e x i t > ... < l a t e x i t s h a 1 _ b a s e 6 4 = " 2 q W S J hN p Q E n 6 c V x X U s d R V 9 T c d 3 c = " > A A A B 7 H i c b V D L S s N A F L 2 p r x p f V Z d u B o v g K i Q i 1 Y 1 Y d O O y g m k L b S i T 6 a Q d O p m E m Y l Q Q r / B j Q t F X An + i n s 3 4 t 8 4 f S y 0 9 V w u H M 6 5 l / s I U 8 6 U d t 1 v q 7 C 0 v L K 6 V l y 3 N z a 3 t n d K u 3 t 1 l W S S U J 8 k P J H N E C v K m a C + Z p r T Z i o p j k N O G + H g e u w 3 7 q l U L B F 3 e p j S I M Y 9 w S J G s D a S 7 9 g m O q W y 6 7 g T o E X i z U j 5 8 s O + S N + + 7 F q n 9 N n u J i S L q d C E Y 6 V a n p v q I M d S M 8 L p y G 5 n i q a Y D H C P t g wV O K Y q y C f L j t C R U b o o S q R J o d F E / d 2 R 4 1 i p Y R y a y h j r v p r 3 x u J / X i v T 0 X m Q M 5 F m m g o y H R R l H O k E j S 9 H X S Y p 0 X x o C C a S m V 0 R 6 W O J i T b /s c 0 T v P m T F 0 n 9 x P E q z u m t W 6 5 e w R R F O I B D O A Y P z q A K N 1 A D H w g w e I A n e L a E 9 W i 9 W K / T 0 o I 1 6 9 m H P 7 D e f w A X P J C Q < / l a t e x i t > ... < l a t e x i t s h a 1 _ b a s e 6 4 = " 2 q W S J hN p Q E n 6 c V x X U s d R V 9 T c d 3 c = " > A A A B 7 H i c b V D L S s N A F L 2 p r x p f V Z d u B o v g K i Q i 1 Y 1 Y d O O y g m k L b S i T 6 a Q d O p m E m Y l Q Q r / B j Q t F X An + i n s 3 4 t 8 4 f S y 0 9 V w u H M 6 5 l / s I U 8 6 U d t 1 v q 7 C 0 v L K 6 V l y 3 N z a 3 t n d K u 3 t 1 l W S S U J 8 k P J H N E C v K m a C + Z p r T Z i o p j k N O G + H g e u w 3 7 q l U L B F 3 e p j S I M Y 9 w S J G s D a S 7 9 g m O q W y 6 7 g T o E X i z U j 5 8 s O + S N + + 7 F q n 9 N n u J i S L q d C E Y 6 V a n p v q I M d S M 8 L p y G 5 n i q a Y D H C P t g wV O K Y q y C f L j t C R U b o o S q R J o d F E / d 2 R 4 1 i p Y R y a y h j r v p r 3 x u J / X i v T 0 X m Q M 5 F m m g o y H R R l H O k E j S 9 H X S Y p 0 X x o C C a S m V 0 R 6 W O J i T b /s c 0 T v P m T F 0 n 9 x P E q z u m t W 6 5 e w R R F O I B D O A Y P z q A K N 1 A D H w g w e I A n e L a E 9 W i 9 W K / T 0 o I 1 6 9 m H P 7 D e f w A X P J C Q < / l a t e x i t > ...Q T i M s / P M 5 V k J M Z i l g L 5 Q O T a N Q S 8 = " > A A A B 6 n i c b V C 7 S g N B F L 0 b X z G + o o K N z W A Q r M K u i F q G 2 F g m m B c k S 5 i d z C Z D Z m e W m V k h L P k E G w t F b G 3 9 C 7 / A z s Z v c f I o N P H A h c M 5 9 3 L v P U H M m T a u + + V k V l b X 1 j e y m 7 m t 7 Z 3 d v f z + Q U P L R B F a J 5 J L 1 Q q w p p w J W j f M c N q K F c V R w G k z G N 5 M / O Y 9 V Z p J U T O j m P o R 7 g s W M o K N l e 5 q X a + b L 7 h F d w q 0 T L w 5 K Z S O q t / s v f x R 6 e Y / O z 1 J k o g K Q z j W u u 2 5 s f F T r A w j n I 5 z n U T T G J M h 7 t O 2 p Q J H V P v p 9 N Q x O r V K D 4 V S 2 R I G T d X f E y m O t B 5 F g e 2 M s B n o R W 8 i / u e 1 E x N e + y k T c W K o I L N F Y c K R k W j y N + o x R Y n h I 0 s w U c z e i s g A K 0 y M T S d n Q / A W X 1 4 m j f O i d 1 m 8 q N o 0 y j B D F o 7 h B M 7 A g y s o w S 1 U o A 4 E + v A A T / D s c O f R e X F e Z 6 0 Z Z z 5 z C H / g v P 0 A t l + R I g = = < / l a t e x i t > T1 < l a t e x i t s h a 1 _ b a s e 6 4 = " G Q 2 o 0 K c J q v L i N u k U P P F Q n O m 8 / r M = " > A A A B 6 n i c b V C 7 S g N B F L 3 j M 8 Z X V L C x G Q y C V d g N o p Y h N p Y J 5 g X J E m Y n s 8 m Q 2 d l l Z l Y I S z 7 B x k I R W 1 v / w i + w s / F b n D w K T T x w 4 X D O v d x 7 j x 8 L r o 3 j f K G V 1 b X 1 j c 3 M V n Z 7 Z 3 d v P 3 d w 2 N B R o i i r 0 0 h E q u U T z Q S X r G 6 4 E a w V K 0 Z C X 7 C m P 7 y Z + M 1 7 p j S P Z M 2 M Y u a F p C 9 5 w C k x V r q r d Y v d X N 4 p O F P g Z e L O S b 5 0 X P 3 m 7 + W P S j f 3 2 e l F N A m Z N F Q Q r d u u E x s v J c p w K t g 4 2 0 k 0 i w k d k j 5 r W y p J y L S X T k 8 d 4 z O r 9 H A Q K V v S 4 K n 6 e y I l o d a j 0 L e d I T E D v e h N x P + 8 d m K C a y / l M k 4 M k 3 S 2 K E g E N h G e / I 1 7 X D F q x M g S Q h W 3 t 2 I 6 I I p Q Y 9 P J 2 h D c x Z e X S a N Y c C 8 L F 1 W b R h l m y M A J n M I 5 u H A F J b i F C t S B Q h 8 e 4 A m e k U C P 6 A W 9 z l p X 0 H z m C P 4 A v f 0 A t + O R I w = = < / l a t e x i t > T2 < l a t e x i t s h a 1 _ b a s e 6 4 = " 3 l f M 3 + h x j M 1 g M T v V W 4 o U q x y D b 8 4 = " > A A A B 6 n i c b V C 7 S g N B F L 3 r M 8 Z X V L C x G Q y C V d g V U c s Q G y t J M C 9 I l j A 7 m U 2 G z M w u M 7 N C W P I J N h a K 2 N r 6 F 3 6 B n Y 3 f 4 u R R a O K B C 4 d z 7 u X e e 4 K Y M 2 1 c 9 8 t Z W l 5 Z X V v P b G Q 3 t 7 Z 3 d n N 7 + 3 U d J Y r Q G o l 4 p J o B 1 p Q z S W u G G U 6 b s a J Y B J w 2 g s H 1 2 G / c U 6 V Z J K t m G F N f 4 J 5 k I S P Y W O m u 2 r n t 5 P J u w Z 0 A L R J v R v L F w 8 o 3 e y 9 9 l D u 5 z 3 Y 3 I o m g 0 h C O t W 5 5 b m z 8 F C v D C K e j b D v R N M Z k g H u 0 Z a n E g m o / n Z w 6 Q i d W 6 a I w U r a k Q R P 1 9 0 S K h d Z D E d h O g U 1 f z 3 t j 8 T + v l Z j w y k + Z j B N D J Z k u C h O O T I T G f 6 M u U 5 Q Y P r Q E E 8 X s r Y j 0 s c L E 2 H S y N g R v / u V F U j 8A P n L c f 4 l O R P w = = < / l a t e x i t > TN < l a t e x i t s h a 1 _ b a s e 6 4 = " 2 q W S J h N p Q E n 6 c V x X U s d R V 9 T c d 3 c = " > A A A B 7 H i c b V D L S s N A F L 2 p r x p f V Z d u B o v g K i Q i 1 Y 1 Y d O O y g m k L b S i T 6 a Q d O p m E m Y l Q Q r / B j Q t F X A n + i n s 3 4 t 8 4 f S y 0 9 V w u H M 6 5 l / s I U 8 6 U d t 1 v q 7 C 0 v L K 6 V l y 3 N z a 3 t n d K u 3 t 1 l W S S U J 8 k P J H N E C v K m a C + Z p r T Z i o p j k N O G + H g e u w 3 7 q l U L B F 3 e p j S I M Y 9 w S J G s D a S 7 9 g m O q W y 6 7 g T o E X i z U j 5 8 s O + S N + + 7 F q n 9 N n u J i S L q d C E Y 6 V a n p v q I M d S M 8 L p y G 5 n i q a Y D H C P t g w V O K Y q y C f L j t C R U b o o S q R J o d F E / d 2 R 4 1 i p Y R y a y h j r v p r 3 x u J / X i v T 0 X m Q M 5 F m m g o y H R R l H O k E j S 9 H X S Y p 0 X x o C C a S m V 0 R 6 W O J i T b / s c 0 T v P m T F 0 n 9 x P E q z u m t W 6 5 e w RA n e L a E 9 W i 9 W K / T 0 o I 1 6 9 m H P 7 D e f w A X P J C Q < / l a t e x i t > ...S J h N p Q E n 6 c V x X U s d R V 9 T c d 3 c = " > A A A B 7 H i c b V D L S s N A F L 2 p r x p f V Z d u B o v g K i Q i 1 Y 1 Y d O O y g m k L b S i T 6 a Q d O p m E m Y l Q Q r / B j Q t F X A n + i n s 3 4 t 8 4 f S y 0 9 V w u H M 6 5 l / s I U 8 6 U d t 1 v q 7 C 0 v L K 6 V l y 3 N z a 3 t n d K u 3 t 1 l W S S U J 8 k P J H N E C v K m a C + Z p r T Z i o p j k N O G + H g e u w 3 7 q l U L B F 3 e p j S I M Y 9 w S J G s D a S 7 9 g m O q W y 6 7 g T o E X i z U j 5 8 s O + S N + + 7 F q n 9 N n u J i S L q d C E Y 6 V a n p v q I M d S M 8 L p y G 5 n i q a Y D H C P t g w V O K Y q y C f L j t C R U b o o S q R J o d F E / d 2 R 4 1 i p Y R y a y h j r v p r 3 x u J / X i v T 0 X m Q M 5 F m m g o y H R R l H O k E j S 9 H X S Y p 0 X x o C C a S m V 0 R 6 W O J i T b / s c 0 T v P m T F 0 n 9 x P E q z u m t W 6 5 e w RA n e L a E 9 W i 9 W K / T 0 o I 1 6 9 m H P 7 D e f w A X P J C Q < / l a t e x i t > ... < l a t e x i t s h a 1 _ b a s e 6 4 = " 2 q W S J hN p Q E n 6 c V x X U s d R V 9 T c d 3 c = " > A A A B 7 H i c b V D L S s N A F L 2 p r x p f V Z d u B o v g K i Q i 1 Y 1 Y d O O y g m k L b S i T 6 a Q d O p m E m Y l Q Q r / B j Q t F X A n + i n s 3 4 t 8 4 f S y 0 9 V w u H M 6 5 l / s I U 8 6 U d t 1 v q 7 C 0 v L K 6 V l y 3 N z a 3 t n d K u 3 t 1 l W S S U J 8 k P J H N E C v K m a C + Z p r T Z i o p j k N O G + H g e u w 3 7 q l U L B F 3 e p j S I M Y 9 w S J G s D a S 7 9 g m O q W y 6 7 g T o E X i z U j 5 8 s O + S N + + 7 F q n 9 N n u J i S L q d C E Y 6 V a n p v q I M d S M 8 L p y G 5 n i q a Y D H C P t g w V O K Y q y C f L j t C R U b o o S q R J o d F E / d 2 R 4 1 i p Y R y a y h j r v p r 3 x u J / X i v T 0 X m Q M 5 F m m g o y H R R l H O k E j S 9 H X S Y p 0 X x o C C a S m V 0 R 6 W O J i T b / s c 0 T v P m T F 0 n 9 x P E q z u m t W 6 5 e w R R F O I B D O A Y P z q A K N 1 A D H w g w e IA n e L a E 9 W i 9 W K / T 0 o I 1 6 9 m H P 7 D e f w A X P J C Q < / l a t e x i t > ... < l a t e x i t s h a 1 _ b a s e 6 4 = " 2 q W S J hN p Q E n 6 c V x X U s d R V 9 T c d 3 c = " > A A A B 7 H i c b V D L S s N A F L 2 p r x p f V Z d u B o v g K i Q i 1 Y 1 Y d O O y g m k L b S i T 6 a Q d O p m E m Y l Q Q r / B j Q t F X A n + i n s 3 4 t 8 4 f S y 0 9 V w u H M 6 5 l / s I U 8 6 U d t 1 v q 7 C 0 v L K 6 V l y 3 N z a 3 t n d K u 3 t 1 l W S S U J 8 k P J H N E C v K m a C + Z p r T Z i o p j k N O G + H g e u w 3 7 q l U L B F 3 e p j S I M Y 9 w S J G s D a S 7 9 g m O q W y 6 7 g T o E X i z U j 5 8 s O + S N + + 7 F q n 9 N n u J i S L q d C E Y 6 V a n p v q I M d S M 8 L p y G 5 n i q a Y D H C P t g w V O K Y q y C f L j t C R U b o o S q R J o d F E / d 2 R 4 1 i p Y R y a y h j r v p r 3 x u J / X i v T 0 X m Q M 5 F m m g o y H R R l H O k E j S 9 H X S Y p 0 X x o C C a S m V 0 R 6 W O J i T b / s c 0 T v P m T F 0 n 9 x P E q z u m t W 6 5 e w R R F O I B D O A Y P z q A K N 1 A D H w g w e IA n e L a E 9 W i 9 W K / T 0 o I 1 6 9 m H P 7 D e f w A X P J C Q < / l a t e x i t > ...A A A B 9 H i c b V D L S g N B E J z 1 G e M r K n j x M h g E T 2 E 3 i H o M 8 a D H B P K C Z F l m Z y f J k N n Z d a Y 3 E J Z 8 h x c P i n j U r / A L v H n x W 5 w 8 D p p Y 0 F B U d d P d 5 c e C a 7 D t L 2 t l d W 1 9 Y z O z l d 3 e 2 d 3 b z x 0 c N n S U K M r q N B K R a v l E M 8 E l q w M H w V q x Y i T 0 B W v 6 g 5 u J 3 x w y p X k k a z C K m R u S n u R d T g k Y y b 3 1 H N y h Q Q S 4 5 h W 9 X N 4 u 2 F P g Z e L M S b 5 0 X P 3 m b + W P i p f 7 7 A Q R T U I m g Q q i d d u x Y 3 B T o o B T w c b Z T q J Z T O i A 9 F j b U E l C p t 1 0 e v Q Y n x k l w N 1 I m Z K A p + r v i Z S E W o 9 C 3 3 S G B P p 6 0 Z u I / 3 n t B L r X b s p l n A C T d L a o m w g M E Z 4 k g A O u G A U x M o R Q x c 2 t m P a J I h R M T l k T g r P 4 8 j J p F A v O Z e G i a t I o o x k y 6 A S d o n P k o C t U Q n e o g u q I o n v 0 g J 7 Q s z W 0 H q 0 X 6 3 X W u m L N Z 4 7 Q H 1 j v P / h W l K Q = < / l a t e x i t > G1 • T2 < l a t e x i t s h a 1 _ b a s e 6 4 = " k N g u S 5 q o h l u X g c x 8 Y 2 i 3 y k 0 Q o 1 0 = " > A A A B 9 H i c b V D L S g N B E O z 1 G e M r K n j x M h g E T 2 F X R D 2 G e N C T J J A X J M s y O 5 l N h s z u r D O z g b D k O 7 x 4 U M S j f o V f 4 M 2 L 3 + L k c d D E g o a i q p v u L j / m T G n b / r K W l l d W 1 9 Y z G 9 n N r e 2 d 3 d z e f l 2 J R B J a I 4 I L 2 f S x o p x F t K a Z 5 r Q Z S 4 p D n 9 O G 3 7 8 e + 4 0 B l Y q J q K q H M X V D 3 I 1 Y w A j W R n J v P A e 1 S U d o V P X u v F z e L t g T o E X i z E i + e F j 5 Z m + l j 7 K X + 2 x 3 B E l C G m n C s V I t x 4 6 1 m 2 K p G e F 0 l G 0 n i s a Y 9 H G X t g y N c E i V m 0 6 O H q E T o 3 R Q I K S p S K O J + n s i x a F S w 9 A 3 n S H W P T X v j c X / v F a i g y s 3 Z V G c a B q R 6 a I g 4 U g L N E 4 A d Z i k R P O h I Z h I Z m 5 F p I c l J t r k l D U h O P M v L 5 L 6 W c G 5 K J x X T B o l m C I D R 3 A M p + D A J R T h F s p Q A w L 3 8 A B P 8 G w N r E f r x X q d t i 5 Z s 5 k D + A P r / Q c i 1 Z T A < / l a t e x i t > G1 • TN < l a t e x i t s h a 1 _ b a s e 6 4 = " A C y Q G X h / Q e 3 1 s r W 4 j 3 H n y x U Z X N 8 = " > A A A B 9 H i c b V D L S g N B E J z 1 G e M r K n j x M h g E T 2 E 3 i H o M 8 a D H B P K C Z F l m Z y f J k N n Z d a Y 3 E J Z 8 h x c P i n j U r / A L v H n x W 5 w 8 D p p Y 0 F B U d d P d 5 c e C a 7 D t L 2 t l d W 1 9 Y z O z l d 3 e 2 d 3 b z x 0 c N n S U K M r q N B K R a v l E M 8 E l q w M H w V q x Y i T 0 B W v 6 g 5 u J 3 x w y p X k k a z C K m R u S n u R d T g k Y y b 3 1 i r h D g w h w z X O 8 X N 4 u 2 F P g Z e L M S b 5 0 X P 3 m b + W P i p f 7 7 A Q R T U I m g Q q i d d u x Y 3 B T o o B T w c b Z T q J Z T O i A 9 F j b U E l C p t 1 0 e v Q Y n x k l w N 1 I m Z K A p + r v i Z S E W o 9 C 3 3 S G B P p 6 0 Z u I / 3 n t B L r X b s p l n A C T d L a o m w g M E Z 4 k g A O u G A U x M o R Q x c 2 t m P a J I h R M T l k T g r P 4 8 j J p F A v O Z e G i a t I o o x k y 6 A S d o n P k o C t U Q n e o g u q I o n v 0 g J 7 Q s z W 0 H q 0 X 6 3 X W u m L N Z 4 7 Q H 1 j v P / h g l K Q = < /l a t e x i t > G2 • T1 < l a t e x i t s h a 1 _ b a s e 6 4 = " D K 8 U z 4 G R T 1 p y q a O 6 S 1 Z F Y 9 i e l 8 Q = " > A A A B 9 H i c b V D L S g N B E J z 1 G e M r K n j x M h g E T 2 E 3 i H o M 8 a D H B P K C Z F l m Z 2 e T I b O z 6 0 x v I I R 8 h x c P i n j U r / A L v H n x W 5 w 8 D p p Y 0 F B U d d P d 5 S e C a 7 D t L 2 t l d W 1 9 Y z O z l d 3 e 2 d 3 b z x 0 c N n S c K s r q N B a x a v l E M 8 E l q w M H w V q J Y i T y B W v 6 / Z u J 3 x w w p X k s a z B M m B u R r u Q h p w S M 5 N 5 6 R d y h Q Q y 4 5 h W 9 X N 4 u 2 F P g Z e L M S b 5 0 X P 3 m b + W P i p f 77 A Q x T S M m g Q q i d d u x E 3 B H R A G n g o 2 z n V S z h N A + 6 b K 2 o Z J E T L u j 6 d F j f G a U A I e x M i U B T 9 X f E y M S a T 2 M f N M Z E e j p R W 8 i / u e 1 U w i v 3 R G X S Q p M 0 t m i M B U Y Y j x J A A d c M Q p i a A i h i p t b M e 0 R R S i Y n L I m B G f x 5 W X S K B a c y 8 J F 1 a R R R j N k 0 A k 6 R e f I Q V e o h O 5 Q B d U R R f f o A T 2h Z 2 t g P V o v 1 u u s d c W a z x y h P 7 D e f w D 5 5 J S l < / l a t e x i t > G2 • T2< l a t e x i t s h a 1 _ b a s e 6 4 = " T E L V c T G x p v g R I 5 E 8 l + X 1 J B c 2 Z E Y = " > A A A B 9 H i c b V D L S g M x F M 3 U V 6 2 v q u D G T b A I r s p M E X V Z 6 k J X 0 k J f 0 A 5 D J p N p Q z P J m G Q K Z e h 3 u H G h i E v 9 C r / A n R u / x f S x 0 N Y D F w 7 n 3 M u 9 9 / g x o 0 r b 9 p e V W V l d W 9 / I b u a 2 t n d 2 9 / L 7 B 0 0 l E o l J A w s m Z N t H i j D K S U N T z U g 7 l g R F P i M t f 3 A 9 8 V t D I h U V v K 5 H M X E j 1 O M 0 p B h p I 7 k 3 X g l 2 c S A 0 r H t 3 X r 5 g F + 0 p 4 D J x 5 q R Q P q p 9 0 7 f K R 9 X L f 3 Y D g Z O I c I 0 Z U q r j 2 L F 2 U y Q 1 x Y y M c 9 1 E k R j h A e q R j q E c R U S 5 6 f T o M T w 1 S g B D I U 1 x D a f q 7 4 k U R U q N I t 9 0 R k j 3 1 a I 3 E f / z O o k O r 9 y U 8 j j R h O P Z o j B h U A s 4 S Q A G V B K s 2 c g Q h C U 1 t 0 L c R x J h b X L K m R C c x Z e X S b N U d C 6 K 5 z W T R g X M k A X H 4 A S c A Q d c g j K 4 B V X Q A B j c g w f w B J 6 t o f V o v V i vl E o l J A w s m Z N t H i j D K S U N T z U g 7 l g R F P i M t f 3 A z 8 V t D I h U V v K 5 H M X E j 1 O M 0 p B h p I 7 m 3 X g V 2 c S A 0 r H s V L 1 + w i / Y U c J k 4 c 1 I o H d W + 6 V v 5 o + r l P 7 u B w E l E u M Y M K d V x 7 F i 7 K Z K a Y k b G u W 6 i S I z w A P V I x 1 C O I q L c d H r 0 G J 4 a J Y C h k K a 4 h l P 1 9 0 S K I q V G k W 8 6 I 6 T 7 a t G b i P 9 5 n U S H 1 2 5 K e Z x o w v F s U Z g w q A W c J A A D K g n W b G Q I w p K a W y H u I 4 m w N j n l T A j O 4 s v L p H l e d C 6 L F z W T R h n M k A X H 4 A S c A Q d c g R K 4 A 1 X Q A B j c g w f w B J 6 t o f V o v V i vl Y q J q K q H M X V D 3 I 1 Y w A j W R n J v v D v U J h 2 h U d V z v F z e L t g T o E X i z E i + e F j 5 Z m + l j 7 K X + 2 x 3 B E l C G m n C s V I t x 4 6 1 m 2 K p G e F 0 l G 0 n i s a Y 9 H G X t g y N c E i V m 0 6 O H q E T o 3 R Q I K S p S K O J + n s i x a F S w 9 A 3 n S H W P T X v j c X / v F a i g y s 3 Z V G c a B q R 6 a I g 4 U g L N E 4 A d Z i k R P O h I Z h I Z m 5 F p I c l J t r k l D U h O P M v L 5 L 6 W c G 5 K J x X T B o l m C I D R 3 A M p + D A J R T h F s p Q A w L 3 8A B P 8 G w N r E f r x X q d t i 5 Z s 5 k D + A P r / Q c j 9 5 T A < / l a t e x i t > GN • T1 < l a t e x i t s h a 1 _ b a s e 6 4 = " s w R B t K f Z W T a p s t t u P L a L C e I f p b I = " > A A A B 9 H i c b V D L S g M x F M 3 U V 6 2 v q u D G T b A I r s p M E X V Z 6 k J X 0 k J f 0 A 5 D J p N p Q z P J m G Q K Z e h 3 u H G h i E v 9 C r / A n R u / x f S x 0 N Y D F w 7 n 3 M u 9 9 / g x o 0 r b 9 p e V W V l d W 9 / I b u a 2 t n d 2 9 / L 7 B 0 0 l E o l J A w s m Z N t H i j D K S U N T z U g 7 l g R F P i M t f 3 A 9 8 V t D I h U V v K 5 H M X E j 1 O M 0 p B h p I 7 k 3 3 h 3 s 4 k B o W P d K X r 5 g F + 0 p 4 D J x 5 q R Q P q p 9 0 7 f K R 9 X L f 3 Y D g Z O I c I 0 Z U q r j 2 L F 2 U y Q 1 x Y y M c 9 1 E k R j h A e q R j q E c R U S 5 6 f T o M T w 1 S g B D I U 1 x D a f q 7 4 k U R U q N I t 9 0 R k j 31 a I 3 E f / z O o k O r 9 y U 8 j j R h O P Z o j B h U A s 4 S Q A G V B K s 2 c g Q h C U 1 t 0 L c R x J h b X L K m R C c x Z e X S b N U d C 6 K 5 z W T R g X M k A X H 4 A S c A Q d c g j K 4 B V X Q A B j c g w f w B J 6 t o f V o v V i v s 9 a M N Z 8 5 B H 9 g v f 8 A J X u U w Q = = < / l a t e x i t > GN • T2Motion Descriptions methods, including convolutional neural networks (MA-CNN [45] , SenseHAR [23] , Rocket [12] ), recurrent neural network (DeepConvLSTM [40] ), and the attention mechanism based models (At-tnSense [36] , THAT [29] ). Recently, IMUGPT [28, 27] generates motion sequences given activity textual descriptions and trains conventional classification models such as DeepConvLSTM [40] . TimesNet [60] , GPT4TS [72] and TEST [52] propose task-general time-series models for multiple tasks including classification. SHARE [71] presents a sequence-to-sequence framework that leverages shared structures of label names. However, these models perform training and testing on the same dataset, and cannot generalize across datasets.Contrastive LearningSelf-supervised motion time series representation learning methods first learn time series representations based on mask reconstruction (TST [68] , TARNet [11] , LIMU-BERT [62] ), contrastive learning (TNC [55] , TS-TCC [14] , TS2Vec [67] , TF-C [70] , FOCAL [32] , CL-HAR [42] , DDLearn [43] ) or other self-supervised learning objectives (BioBankSSL [66, 13, 10] , Step2Heart [51] ). Subsequently, they fine-tune classifier heads for specific downstream tasks. However, the representation learning and fine-tuning phases of these methods generally occur on the same or highly similar datasets, which continues to face challenges in generalization across diverse datasets.Pre-trained models for motion time series are inspired by the recent success of large language or multimodal models. ImageBind [16] and IMU2CLIP [38] leverage recent large vision-language models [44] to learn a joint embedding across multiple modalities including motion time series and text. However, both ImageBind and IMU2CLIP are trained on motion time series collected from head-mounted devices [17] , limiting their generalizability across different device locations and orientations. Furthermore, several studies have explored directly applying LLMs for motion time series classification. For example, HARGPT [24] processes raw motion time series through LLMs and incorporates role-play and chain-of-thought strategies for prompting. ContextGPT [3] designs prompt engineering approaches leveraging context information. However, since LLMs are not directly trained on raw motion time series, such methods require extensive context information that is not usually available, and struggle with accurately recognizing complex activities.Other related works on motion classification include multimodal action recognition and domain adaptation methods. Multimodal action recognition such as the Ego4D [17] and Ego-Exo4D [18] benchmarks incorporates video and audio modalities, whereas we focus on a more energy-efficient and challenging scenario of action recognition based purely on motion time series. Domain adaptation methods mostly assume that source and target datasets share the same label names and have the same number of classes, such as cross-user domain adaptation and cross-dataset domain adaptation only for those common classes [22, 35, 21] . We aim for a more generic yet challenging generalization scenario where pre-training and downstream datasets share different label names. 
method: 
 UniMTS takes a contrastive learning-based approach that aligns paired motion time series with text descriptions to enable activity generalization, as shown in Figure 2 . We simulate motion time series from motion skeleton data (Section 3.1) and augment them for orientation generalization (Section 3.2).We use graph encoder to model the simulated motion time series, capturing correlations among joints to generalize across different device locations (Section 3.3.1). To enhance semantics learning, we use large language models to augment text descriptions (Section 3.3.2).Motion skeleton data [19] describe the movements of human skeleton joints over time, containing positions and orientations for each joint. On the other hand, motion time series captured by physical sensors typically measure higher-order data such as accelerations and angular velocities. Consequently, we apply motion equations [65] to synthesize these time series of accelerations and angular velocities from motion skeleton data. More specifically, for each skeleton joint J i , we input both positions p Ji,G (mapped from time domain T to R 3 , defined in global frame G), and orientation quaternions q Ji,GL (mapped from time domain T to the Special Orthogonal Group SO(3), defined in Hamilton convention with subscript GL representing a frame rotation from local frame L to global frame G). We drop the subscript G and GL from here on for simplicity of notation. Based on motion equations [65] , we calculate velocities v Ji and accelerations a Ji by taking the first and second order derivatives of positions p Ji . These derivatives are then transformed from global frames to local frames using the corresponding orientation sequences q Ji . Similarly, angular velocities ω Ji are computed by taking the first order derivatives of orientation quaternions q Ji . Mathematically,EQUATIONEQUATIONEQUATIONwhere ⊗ and * represent the quaternion multiplication operator and the quaternion conjugate.Recognizing the inherent presence of noise carried by sensors in practice, the physics engine incorporates Gaussian noise with a zero mean into the simulated data. Representing the above motion time series as x Ji (t), which can denote either a Ji (t) (accelerations) or ω Ji (t) (angular velocities), the noisy time series xJi (t) are formulated asEQUATIONA common limitation we have identified from prior studies that leads to their poor generalization is that they fail to consider the impact of latent device orientation factors on the motion time series. For example, end users can potentially wear devices in various orientations, such as with a phone facing towards or against the body in a pocket. Additionally, the software driver API for axis definition can be arbitrarily configured by the developers. For example, the iOS system defines acceleration in an opposite direction compared to the Android system 2 . With the listed risk factors considered, we apply a data augmentation technique to simulate random orientations during pre-training, so that our learned model achieves rotation-invariance during deployment [7, 57, 61] . Specifically, during pre-training, for each iteration we sample a random rotation matrix for each joint J i ,EQUATIONand compute the augmented time series xt Ji at timesteps t = 1, 2, • • • , T asEQUATIONDuring one iteration, the same R δ Ji is consistently applied to J i for every time series and every timestep t = 1, 2, • • • , T . The rotation-invariant augmentation ensures that the simulated time series are adaptable to any downstream orientation, thereby enhancing the generalization capabilities.The physics engine generates sufficient motion time series data, which are subsequently encoded by graph networks and aligned with their corresponding text embeddings through contrastive learning. e N U M W y w W M S q H V C N g k t s G G 4 E t h O F N A o E t o L R z d R v P a H S P J a P Z p y g H 9 G B 5 C F n 1 F j p 4 b b n 9 c o V t + r O Q J a J l 5 M K 5 K j 3 y l / d f s z S C K V h g m r d 8 d z E + B l V h j O B k 1 I 3 1 Z h Q N q I D 7 F g q a Y T a z 2 a n T s i J V f o k j J U t a c h M / T 2 R 0 U j r c R T Y z o i a o V 7 0 p u J / X i c 1 4 Z W f c Z m k B i W b L w p T Q U x M p n + T P l f I j B h b Q p n i 9 l b C h l R R Z m w 6 J R u C t / j y M m m e V b 2 L 6 v n 9 e a V 2 n c d R h C M 4 h l P w 4 B J q c A d 1 a A C D A T z D K 7 w 5 w n l x 3 p 2 P e W v B y W c O 4 Q + c z x / D 6 Y 1 4 < / l a t e x i t > G1 < l a t e x i t s h a 1 _ b a s e 6 4 = " 9 6 h Q c 4 W X U N K O 4 + O b Q 3 g o v Y 5 g y C M = " > A A A B 9 H i c b V D L S g N B E J y N r x h f U c G L l 8 E g e A q 7 I u o x x I M e E 8 g L k m W Z n Z 0 k Q 2 Z n 1 5 n e Q F j y H V 4 8 K O J R v 8 I v 8 O b F b 3 H y O G h i Q U N R 1 U 1 3 l x 8 L r s G 2 v 6 z M y u r a + k Z 2 M 7 e 1 v b O 7 l 9 8 / a O g o U Z T V a S Q i 1 f K J Z o J L V g c O g r V i x U j o C 9 b 0 B z c T v z l k S v N I 1 m A U M z c k P c m 7 n B I w k n v r O b h D g w h w z X O 8 f M E u 2 l P g Z e L M S a F 0 V P 3 m b + W P i p f / 7 A Q R T U I m g Q q i d d u x Y 3 B T o o B T w c a 5 T q J Z T O i A 9 F j b U E l C p t 1 0 e v Q Y n x o l w N 1 I m Z K A p + r v i Z S E W o 9 C 3 3 S G B P p 6 0 Z u I / 3 n t B L r X b s p l n A C T d L a o m w g M E Z 4 k g A O u G A U x M o R Q x c 2 t m P a J I h R M T j k T g r P 4 8 j J p n B e d y + J F 1 a R R R j N k 0 T E 6 Q W f I Q V e o h O 5 Q B d U R R f f o A T 2 h Z 2 t o P V o v 1 u u s N W P N Z w 7 R H 1 j v P / b S l K M = < / l a t e x i t > G1 • T1 < l a t e x i t s h a 1 _ b a s e 6 4 = " 2 q W S J h N p Q E n 6 c V x X U s d R V 9 T c d 3 c = " > A A A B 7 H i c b V D L S s N A F L 2 p r x p f V Z d u B o v g K i Q i 1 Y 1 Y d O O y g m k L b S i T 6 a Q d O p m E m Y l Q Q r / B j Q t F X An + i n s 3 4 t 8 4 f S y 0 9 V w u H M 6 5 l / s I U 8 6 U d t 1 v q 7 C 0 v L K 6 V l y 3 N z a 3 t n d K u 3 t 1 l W S S U J 8 k P J H N E C v K m a C + Z p r T Z i o p j k N O G + H g e u w 3 7 q l U L B F 3 e p j S I M Y 9 w S J G s D a S 7 9 g m O q W y 6 7 g T o E X i z U j 5 8 s O + S N + + 7 F q n 9 N n u J i S L q d C E Y 6 V a n p v q I M d S M 8 L p y G 5 n i q a Y D H C P t g wV O K Y q y C f L j t C R U b o o S q R J o d F E / d 2 R 4 1 i p Y R y a y h j r v p r 3 x u J / X i v T 0 X m Q M 5 F m m g o y H R R l H O k E j S 9 H X S Y p 0 X x o C C a S m V 0 R 6 W O J i T b / s c 0 T v P m T F 0 n 9 x P E q z u m t W 6 5A n e L a E 9 W i 9 W K / T 0 o I 1 6 9 m H P 7 D e f w A X P J C Q < / l a t e x i t > ... Q T i M s / P M 5 V k J M Z i l g L 5 Q O T a N Q S 8 = " > A A A B 6 n i c b V C 7 S g N B F L 0 b X z G + o o K N z W A Q r M K u i F q G 2 F g m m B c k S 5 i d z C Z D Z m e W m V k h L P k E G w t F b G 3 9 C 7 / A z s Z v c f I o N P H A h c M 5 9 3 L v P U H M m T a u + + V k V l b X 1 j e y m 7 m t 7 Z 3 d v f z + Q U P L R B F a J 5 J L 1 Q q w p p w J W j f M c N q K F c V R w G k z G N 5 M / O Y 9 V Z p J U T O j m P o R 7 g s W M o K N l e 5 q X a + b L 7 h F d w q 0 T L w 5 K Z S O q t / s v f x R 6 e Y / O z 1 J k o g K Q z j W u u 2 5 s f F T r A w j n I 5 z n U T T G J M h 7 t O 2 p Q J H V P v p 9 N Q x O r V K D 4 V S 2 R I G T d X f E y m O t B 5 F g e 2 M s B n o R W 8 i / u e 1 E x N e + y k T c W K o I L N F Y c K R k W j y N + o x R Y n h I 0 s w U c z e i s g A K 0 y M T S d n Q / A W X 1 4 m j f O i d 1 m 8 q N o 0 y j B D F o 7 h B M 7 A g y s o w S 1 U o A 4 E + v A A T / D s c O f R e X F e Z 6 0 Z Z z 5 z C H / g v P 0 A t l + R I g = = < / l a t e x i t > T1 < l a t e x i t s h a 1 _ b a s e 6 4 = " G Q 2 o 0 K c J q v L i N u k U P P F Q n O m 8 / r M = " > A A A B 6 n i c b V C 7 S g N B F L 3 j M 8 Z X V L C x G Q y C V d g N o p Y h N p Y J 5 g X J E m Y n s 8 m Q 2 d l l Z l Y I S z 7 B x k I R W 1 v / w i + w s / F b n D w K T T x w 4 X D O v d x 7 j x 8 L r o 3 j f K G V 1 b X 1 j c 3 M V n Z 7 Z 3 d v P 3 d w 2 N B R o i i r 0 0 h E q u U T z Q S X r G 6 4 E a w V K 0 Z C Xf M 3 + h x j M 1 g M T v V W 4 o U q x y D b 8 4 = " > A A A B 6 n i c b V C 7 S g N B F L 3 r M 8 Z X V L C x G Q y C V d g V U c s Q G y t J M C 9 I l j A 7 m U 2 G z M w u M 7 N C W P I J N h a K 2 N r 6 F 3 6 B n Y 3 f 4 u R R a O K B C 4 d z 7 u X e e 4 K Y M 2 1 c 9 8 t Z W l 5 Z X V v P b G Q 3 t 7 Z 3 d n N 7 + 3 U d J Y r Q G o l 4 p J o B 1 p Q z S W u G G U 6 b s a J Y B J w 2 g s H 1 2 G / c U 6 V Z J K t m G F N f 4 J 5 k I S P Y W O m u 2 r n t 5 P J u w Z 0 A L R J v R v L F w 8 o 3 e y 9 9 l D u 5 z 3 Y 3 I o m g 0 h C O t W 5 5 b m z 8 F C v D C K e j b D v R N M Z k g H u 0 Z a n E g m o / n Z w 6 Q i d W 6 a I w U r a k Q R P 1 9 0 S K h d Z D E d h O g U 1 f z 3 t j 8 T + v l Z j w y k + Z j B N D J Z k u C h O O T I T G f 6 M u U 5 Q Y P r Q E E 8 X s r Y j 0 s c L E 2 H S y N g R v / u V F U j 8TN < l a t e x i t s h a 1 _ b a s e 6 4 = " 2 q W S J hN p Q E n 6 c V x X U s d R V 9 T c d 3 c = " > A A A B 7 H i c b V D L S s N A F L 2 p r x p f V Z d u B o v g K i Q i 1 Y 1 Y d O O y g m k L b S i T 6 a Q d O p m E m Y l Q Q r / B j Q t F X An + i n s 3 4 t 8 4 f S y 0 9 V w u H M 6 5 l / s I U 8 6 U d t 1 v q 7 C 0 v L K 6 V l y 3 N z a 3 t n d K u 3 t 1 l W S S U J 8 k P J H N E C v K m a C + Z p r T Z i o p j k N O G + H g e u w 3 7 q l U L B F 3 e p j S I M Y 9 w S J G s D a S 7 9 g m O q W y 6 7 g T o E X i z U j 5 8 s O + S N + + 7 F q n 9 N n u J i S L q d C E Y 6 V a n p v q I M d S M 8 L p y G 5 n i q a Y D H C P t g wV O K Y q y C f L j t C R U b o o S q R J o d F E / d 2 R 4 1 i p Y R y a y h j r v p r 3 x u J / X i v T 0 X m Q M 5 F m m g o y H R R l H O k E j S 9 H X S Y p 0 X x o C C a S m V 0 R 6 W O J i T b / s c 0 T v P m T F 0 n 9 x P E q z u m t W 6 5A n e L a E 9 W i 9 W K / T 0 o I 1 6 9 m H P 7 D e f w A X P J C Q < / l a t e x i t > ...S J h N p Q E n 6 c V x X U s d R V 9 T c d 3 c = " > A A A B 7 H i c b V D L S s N A F L 2 p r x p f V Z d u B o v g K i Q i 1 Y 1 Y d O O y g m k L b S i T 6 a Q d O p m E m Y l Q Q r / B j Q t F X An + i n s 3 4 t 8 4 f S y 0 9 V w u H M 6 5 l / s I U 8 6 U d t 1 v q 7 C 0 v L K 6 V l y 3 N z a 3 t n d K u 3 t 1 l W S S U J 8 k P J H N E C v K m a C + Z p r T Z i o p j k N O G + H g e u w 3 7 q l U L B F 3 e p j S I M Y 9 w S J G s D a S 7 9 g m O q W y 6 7 g T o E X i z U j 5 8 s O + S N + + 7 F q n 9 N n u J i S L q d C E Y 6 V a n p v q I M d S M 8 L p y G 5 n i q a Y D H C P t g wV O K Y q y C f L j t C R U b o o S q R J o d F E / d 2 R 4 1 i p Y R y a y h j r v p r 3 x u J / X i v T 0 X m Q M 5 F m m g o y H R R l H O k E j S 9 H X S Y p 0 X x o C C a S m V 0 R 6 W O J i T b / s c 0 T v P m T F 0 n 9 x P E q z u m t W 6 5A n e L a E 9 W i 9 W K / T 0 o I 1 6 9 m H P 7 D e f w A X P J C Q < / l a t e x i t > ... A A A B 9 H i c b V D L S g N B E J z 1 G e M r K n j x M h g E T 2 E 3 i H o M 8 a D H B P K C Z F l m Z y f J k N n Z d a Y 3 E J Z 8 h x c P i n j U r / A L v H n x W 5 w 8 D p p Y 0 F B U d d P d 5 c e C a 7 D t L 2 t l d W 1 9 Y z O z l d 3 e 2 d 3 b z x 0 c N n S U K M r q N B K R a v l E M 8 E l q w M H w V q x Y i T 0 B W v 6 g 5 u J 3 x w y p X k k a z C K m R u S n u R d T g k Y y b 3 1 H N y h Q Q S 4 5 h W 9 X N 4 u 2 F P g Z e L M S b 5 0 X P 3 m b + W P i p f 7 7 A Q R T U I m g Q q i d d u x Y 3 B T o o B T w c b Z T q J Z T O i A 9 F j b U E l C p t 1 0 e v Q Y n x k l w N 1 I m Z K A p + r v i Z S E W o 9 C 3 3 S G B P p 6 0 Z u I / 3 n t B L r X b s p l n A C T d L a o m w g M E Z 4 k g A O u G A U x M o R Q x c 2 t m P a J I h R M T l k T g r P 4 8 j J p F A v O Z e G i a t I o o x k y 6 A S d o n P k o C t U Q n e o g u q I o n v 0 g J 7 Q s z W 0 H q 0 X 6 3 X W u m L N Z 4 7 Q H 1 j v P / h W l K Q = < / l a t e x i t > G1 • T2 < l a t e x i t s h a 1 _ b a s e 6 4 = " k N g u S 5 q o h l u X g c x 8 Y 2 i 3 y k 0 Q o 1 0 = " > A A A B 9 H i c b V D L S g N B E O z 1 G e M r K n j x M h g E T 2 F X R D 2 G e N C T J J A X J M s y O 5 l N h s z u r D O z g b D k O 7 x 4 U M S j f o V f 4 M 2 L 3 + L k c d D E g o a i q p v u L j / m T G n b / r K W l l d W 1 9 Y z G 9 n N r e 2 d 3 d z e f l 2 J R B J a I 4 I L 2 f S x o p x F t K a Z 5 r Q Z S 4 p D n 9 O G 3 7 8 e + 4 0 B l Y q J q K q H M X V D 3 I 1 Y w A j W R n J v P A e 1 S U d o V P X u v F z e L t g T o E X i z E i + e F j 5 Z m + l j 7 K X + 2 x 3 B E l C G m n C s V I t x 4 6 1 m 2 K p G e F 0 l G 0 n i s a Y 9 H G X t g y N c E i V m 0 6 O H q E T o 3 R Q I K S p S K O J + n s i x a F S w 9 A 3 n S H W P T X v j c X / v F a i g y s 3 Z V G c a B q R 6 a I g 4 U g L N E 4 A d Z i k R P O h I Z h I Z m 5 F p I c l J t r k l D U h O P M v L 5 L 6 W c G 5 K J x X T B o l m C I D R 3 A M p + D A J R T h F s p Q A w L 3 8 A B P 8 G w N r E f r x X q d t i 5 Z s 5 k D + A P r / Q c i 1 Z T A < / l a tTo capture the spatio-temporal correlations among different joints over time, we adopt spatio-temporal graph convolutional network [63] as our motion time series encoder. We denote the initial input graph representation as follows,EQUATION)Nodes V contain skeleton joints with features X ∈ R C×T ×V , where C, T, V represent the number of signal channels, temporal steps and joint nodes. Spatial edges E s connect adjacent nodes defined by the skeleton structure H and temporal edges E t connect temporally adjacent frames.In practice, devices may not cover the complete joints but are rather positioned at arbitrary subsets of the complete joints. To simulate this, during each pre-training iteration, we randomly select a subset of joints and mask data from the remaining joints with zeros. We denote the mask at one iteration asEQUATIONThe graph convolution network g ϕ first computes the spatial output features asEQUATIONwhere K s denotes the spatial kernel size, A il k represents whether node x J l belongs to the spatial convolution sampling subset S k Ji of node x Ji , and Λ ii k = Σ l (A il k ) + α represents the normalized diagonal matrix, with α set to 0.001 to prevent empty rows [63, 48] . Φ k ∈ R C ′ ×C×1×1 represents weights of the 1 × 1 convolution operation with C ′ denoting output channel dimension. Following spatial convolution, we further perform K t × 1 temporal convolution on the spatial output features X out , similar to classical convolution operations, where K t represents the temporal kernel size. The final graph representation g ϕ (X) is derived by averaging features across both spatial and temporal dimensions with a graph average pooling layer at the end.To increase the diversity of paired text descriptions in the pre-training motion corpus [19] , we apply large language models (GPT-3.5) to augment original motion text descriptions with the following prompt template: The following one or multiple descriptions are describing the same human activities: <motion descriptions>. Generate k paraphrases to describe the same activities.We denote the original text descriptions combined with the LLM-augmented ones as Y. We encode them using the same text encoder f θ as CLIP [44] , utilizing its pre-trained weights for initialization.During pre-training, we maximize the similarities of paired simulated motion time series and text descriptions through contrastive learning:EQUATION)where B, γ represent batch size and temperature parameter that controls distribution concentrations, and sim represents similarity score computed as inner product:EQUATION)We pre-train the graph and text encoders using simulated motion time series and augmented text descriptions. During inference, we evaluate the model on real-world motion time series, as illustrated in the left part of Figure 3 . For the text encoder, we input all label candidates. For the graph encoder, we assign real motion time series to the nearest joint in the skeleton graph and assign zeros to the remaining joints. The random mask M during pre-training emulates the zero-masking process. We compute the similarity score between the graph embedding with text embedding from each label candidate, and choose the label with the highest similarity score as the predicted activity.We can further fine-tune the pre-trained model on downstream real-world data, as depicted in the right part of Figure 3 . Specifically, we freeze the text encoder f θ and update weights of the graph encoder g ϕ followed by a linear classifier h ψ . Following the same process as inference, we assign the real motion time series to the nearest joint in the skeleton graph and assign zeros to the remaining joints to construct the graph input representation X. We fine-tune the model using X and one-hot encoded labels z with D classes based on cross-entropy loss, where σ(•) represents the softmax operation:EQUATION)We report both zero-shot and fine-tuning performance in the subsequent experiment section. 
datasets and experimental setting: 
 We simulate motion time series from existing motion skeleton dataset HumanML3D [19] , which contain both motion skeleton data and corresponding text descriptions as detailed in Section A.1 in Appendix. We further augment the text descriptions as described in Section 3.3.2.We evaluate on the most extensive motion time series classification benchmark to date, comprising 18 real-world datasets that cover diverse activities. These datasets are collected from various body locations such as head, chest, back, arm, wrist, waist, hip, leg, knee and ankle. We categorize these datasets into three difficulty levels: (1) easy level (with fewer than 10 activities): Opportunity [47] , UCI-HAR [2] , MotionSense [37] , w-HAR [5] , Shoaib [49] , HAR70+ [58] , RealWorld [53] , TNDA-HAR [64] ;(2) medium level (with 10 to 20 activities): PAMAP2 [46] , USC-HAD [69] , Mhealth [4] , Harth [33] , UT-Complex [50] , Wharf [6] , WISDM [59] , DSADS [1] ;(3) hard level (with more than 20 activities): UTD-MHAD [8] , MMAct [26] . We provide the specific number of activities for each dataset in Table 1 and Table 2 , and detail their collection settings in Section A.2 in Appendix.We re-sample the real-world test data to the same sampling frequency as the simulation data (20 Hz), and apply normalization to ensure consistency in unit measurements, e.g., standardizing accelerations to m/s 2 . We pre-train UniMTS using Adam optimizer [25] with a learning rate of 0.0001 on a single NVIDIA A100 GPU. The pre-training process consumes approximately 13 GB of memory given a batch size of 64. For text augmentation, we prompt GPT-3.5 ("gpt-3.5-turbo") to generate k = 3 paraphrases. During each iteration, we randomly generate the mask M by selecting 1 to 5 joints and mask the remaining joints as zeros. We adopt learnable temperature parameter γ initialized from CLIP. We evaluate the models using accuracy, macro-F1 and the top-2 retrieval performance R@2.We pre-train UniMTS exclusively on simulated data and evaluate on 18 real-world motion time series classification benchmark datasets. We compare UniMTS against classification models with zero-shot capabilities: ImageBind [16] , IMU2CLIP [38] , IMUGPT [28] and HARGPT [24] . We also input the 2D visualizations of motion time series to pre-trained vision-language model LLaVA [30] for comparison. We detail the configurations of baselines in Section A.3 in Appendix. As shown in Table 1 , UniMTS significantly outperforms all baselines in the zero-shot setting. We also apply the Wilcoxon-signed rank test with Holm's α (5%) following previous works [20, 71] . The Wilcoxonsigned rank test indicates that the improvement of UniMTS compared with all the baselines is statistically significant, with p-values significantly lower than 0.05 (e.g., p-value = 8 × 10 -6 for ImageBind, which has the highest F1 score among the baselines).Compared with UniMTS, ImageBind and IMU2CLIP are trained on data from single location (headmounted devices), limiting their generalization to data collected from other locations. IMUGPT struggles to generalize across datasets featuring different activities and requires individual training for each downstream dataset. Both HARGPT and LLaVA focus on simple and easily distinguishable activities as these language or vision models are not originally trained on motion time series, and they also require careful prompt designs. Another limitation for all the above models is that they do not generalize across device orientations. In contrast, UniMTS shows remarkable generalizability to various downstream device locations, orientations and activities, achieving state-of-the-art performance.We also compare with a few ablations of UniMTS as illustrated in the Ablation Study section.Apart from the zero-shot setting, we provide a few real samples for each activity and fine-tune UniMTS and the baselines. More specifically, we provide 1, 2, 3, 5, 10 samples for each activity and compare UniMTS against ImageBind [16] , IMU2CLIP [38] , IMUGPT [28] , GPT4TS [72] , BioBankSSL [66] and a randomly initialized model with the same model architecture as UniMTS (referred to as Random). We report both the mean and the standard deviation in Figure 4 . UniMTS also demonstrates state-of-the-art performance in the few-shot fine-tuning setting, showing the effectiveness of pre-training. Following the same Wilcoxon-signed rank test as in the zero-shot setting, we observe p-values far below 0.05 (e.g., p-value = 2 × 10 -25 for the best-performing baseline ImageBind), indicating the statistical significance of our improvement.We also compare the full-shot performance where UniMTS and the baselines are fine-tuned or trained using all the available training samples of the downstream datasets. We compare UniMTS with pretrained models (ImageBind [16] , IMU2CLIP [38] ), self-supervised models (TST [68] , TARNet [11] , TS2Vec [67] , BioBankSSL [66] ), and conventional models (DeepConvLSTM [40] , MA-CNN [45] , XGBoost [9] , THAT [29] , IMUGPT [28] , TimesNet [60] , GPT4TS [72] , SHARE [71] ). Baselines are detailed in Section A.3 in Appendix. We also compare pre-trained UniMTS with a randomly initialized UniMTS (referred to as Random). As shown in Table 2 , UniMTS also demonstrates state-of-the-art performance in the full-shot setting, outperforming pre-trained, self-supervised and conventional models. Due to space limit, we report baselines before 2021 in Table 3 in Appendix.Following the same Wilcoxon-signed rank test, we observe p-values far below 0.05 (e.g., p-value = 0.018 for the best-performing baseline), indicating the statistical significance of our improvement. UniMTS also demonstrates space and time efficiency, as detailed in Section A.5 in Appendix.In the zero-shot setting, we compare UniMTS with a few ablations by removing rotation-invariant augmentation (w/o rot aug), removing text augmentation (w/o text aug) and by replacing the graph encoder with a CNN-based encoder that directly concatenates joints without modeling their spatial relationships (w/o graph). We can observe in Table 1 that the performance declines after removing each of the above components, verifying their respective importance in improving generalization across locations (graph encoder), orientations (rotation-invariant augmentation) and activities (text augmentation). We also compare the pre-trained UniMTS with randomly initialized UniMTS in both few-shot and full-shot settings. As shown in Figure 4 and Table 2 , pre-trained UniMTS consistently outperforms randomly initialized UniMTS, highlighting the benefits of pre-training.UniMTS's time series embeddings align with corresponding semantic meanings. As shown in Figure 5 , the t-SNE visualizations of UniMTS's time series embeddings form distinguishable clusters that align with their semantic meanings. Notably, UniMTS is only pre-trained on the simulated data but its embeddings for real-world data closely align with the semantic space, which again demonstrates our model's zero-shot generalization due to contrastive learning. For example, in Figure 5a , stationary activities such as lying and sitting group together; light-movement activities such as standing, ironing, and vacuum cleaning are close to each other; while high-intensity activities such as running and cycling cluster closer in the embedding space. Simulated data from our physics engine closely resemble real signal patterns. In Figure 6 , we compare some example simulated data alongside their real-world counterparts. We show three example activities of different intensity levels (i.e., sitting, walking, rope jumping), where both simulated data and real-world data are near the wrist. We can observe that the patterns in simulated data closely resemble those of real data, in terms of both magnitude and frequency. Although the tri-axial distributions might differ, these variations are mostly due to orientation differences and are effectively managed by our rotation-invariant augmentation. We observe similar patterns between simulated data and real data for other device locations, as shown in Section A.4 in Appendix. DSADS [1] comprises daily and sports activities such as exercising and rowing. Multiple devices are positioned at the torso, right arm, left arm, right leg, and left leg.UTD-MHAD [8] contains diverse activities such as swiping arms, hand clapping, throwing, arm crossing, drawing and squatting. The devices are worn on the subject's right wrist or the right thigh depending on whether the action is mostly an arm or a leg type of action.MMAct [26] presents a large-scale activity dataset covering a wide range of daily life activities such as carrying, talking on phone and falling. Devices recording motion time series include a smartwatch as well as a smartphone inside the pocket of the subject's pants.We detail the baseline settings as follows.ImageBind [16] : We employ the pre-trained weights from "imagebind_huge" for zero-shot evaluation.During fine-tuning, we add a linear layer to map ImageBind embeddings to the number of activity classes. We fine-tune both ImageBind and the linear layer during fine-tuning, which performs better than simply tuning the linear layer.IMU2CLIP [38] : The pre-trained weights of IMUCLIP are not released. Therefore, we first follow their pre-training implementation3 to pre-train on Ego4D datasets [17] . During fine-tuning, we add a linear layer after IMU2CLIP embeddings and fine-tune both IMU2CLIP and the linear layer.IMUGPT [28] : We choose DeepConvLSTM as the backbone model, which shows the best performance as reported in their original paper. We remove the supervised distribution calibration phase, which relies on labeled downstream data and conflicts with the zero-shot setting objectives.HARGPT [24] : The method directly prompts large language models to classify motion time series. We down-sample motion time series to 10 Hz as used in their paper and follow their prompt template.LLaVA [30] : We visualize motion time series as 2D plots and use these visualizations as input for the pre-trained model of "llava-v1.5-7b".TST [68] : This is a Transformer-based representation learning framework with several downstream tasks including multivariate time-series classification. We follow the framework to first pre-train the Transformer model in an unsupervised fashion and then fine-tune the pre-trained model on the downstream classification task.TARNet [11] : The model proposes task-aware representation learning that reconstructs important timestamps guided by self-attention score distribution from end-task training. We jointly train the reconstruction task and the classification task. TS2Vec [67] : The method performs contrastive learning to learn contextual representations of time series. We follow their implementation to first apply contrastive learning and then train a linear regression model for each dataset.BioBankSSL [66] : The paper proposes a pre-trained model for human activity recognition using a large-scale UK Biobank wrist accelerometer dataset with multi-task self-supervised learning.DeepConvLSTM [40] (shown as "DeepCL" in Table 3 due to space limit): The model applies convolutional layers to automatically learn feature representations and further applies LSTM to capture the temporal dependencies between their activations.MA-CNN [45] : The model first extracts preliminary features for each motion time series modality through its own dedicated convolutional layers, then the extracted intra-modality features are combined through fully-connected layers for motion time series classification.XGBoost [9] : This is a scalable end-to-end machine learning system for tree boosting, which has been widely recognized in machine learning and time series analysis.THAT [29] : The model proposes a two-stream convolution augmented human activity transformer which captures both time-over-channel and channel-over-time features in a two-stream structure.TimesNet [60] : This is a task-general backbone for time series analysis including classification by modeling the multi-periodicity and extracting temporal variations.GPT4TS [72] : This is also a task-general framework that includes time series classification. The model is based on a frozen pre-trained language model, so we also adopt it as a few-shot fine-tuning baseline. However, the method is not suitable for zero-shot evaluation, as it requires training a separate classifier head for each downstream dataset and does not generalize across activities.SHARE [71] : This is a sequence-to-sequence model that contains an encoder to extract motion time series features, as well as a decoder to generate label name sequences to capture label semantics.In addition to the simulated data for wrist as shown in Figure 6 , we present more examples for other device locations such as ankle in Figure 9 . We observe consistently similar patterns between simulated motion time series and real PAMAP2 data across activities of various intensity levels, ranging from sitting to walking and rope jumping.For space complexity, the graph encoder of UniMTS contains only 4.94M parameters, which is significantly smaller compared with the 18.69M used in the IMU encoder of the best existing baseline ImageBind. For time complexity, fine-tuning of UniMTS is also efficient. Justification: We specify computer resource details in Section 4.1.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics in every aspect.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss the broader impacts and limitations in Section 5.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: The paper does not pose high risk for misusing data or models. We pre-train the model exclusively on synthetic data to ensure strict privacy control at the earliest stages of model development.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We properly cite all the code and data used in this paper, and respect their license and terms of use.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: We provide the new assets as a zip file in the supplementary material.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.https://github.com/tszheichoi/awesome-sensor-logger/blob/main/CROSSPLATFORM.mdhttps://github.com/facebookresearch/imu2clip 
 
 Appendix:  


Decision: The paper proposes a pre-training strategy, specifically designed for motion time series from mobile sensors. The contributions lie in a contrastive learning framework and the alignment of time series with textual descriptions from LLMs. IMU measurements are simulated for each joint and used to pre-train a graph encoder. Doing so, the approach addresses common technical challenges such as variations in the device placements, difference in sensor orientations, and variable activity. On a large benchmark study of 18 datasets the authors show significant improvements over the state of the art in key metrics and in settings such as zero-sho, few-shot and full-shot. As a results, the approach also allows to use a pre-trained model that generalizes well to slight variations in the setup (e.g. across orientations, placements, ...). The reviewers mainly raised questions about the clarity of technical details which have all been resolved during the rebuttal period. 

Decision:
After an intensive discussion all reviewers voted for clear acceptance. I therefore recommend to accept the paper and encourage the authors to use the feedback provided to improve the paper for its final version.

Reviews: review: paper_summary: paper_summary: The paper introduces a new pre-training approach designed specifically for motion time series data from mobile and wearable devices. The proposed method, UniMTS, employs a contrastive learning framework to align motion time series with text descriptions, enhanced by large language models. This approach addresses the challenges of device placement variation, orientation differences, and activity variability. The model is evaluated on 18 benchmark datasets and demonstrates significant improvements in zero-shot, few-shot, and full-shot settings.
strength_weakness: strength_weakness: Evaluation Comparisons:
The paper could be strengthened by comparing the proposed method with more recent benchmarks such as Ego4D and Exo. This would provide a clearer picture of how UniMTS stands relative to the current state-of-the-art methods in the field.

Generalizability of Results:
While the method shows strong results across multiple datasets, there is limited discussion on the potential limitations or specific conditions where the model might underperform. A more detailed analysis of the generalizability and potential edge cases would be beneficial.
questions: questions: Can you provide a comparison with the Ego4D and Exo benchmarks?
limitations: limitations: None
paper_summary: paper_summary: This paper presents a novel pretraining method for IMU time series (i.e., acceleration + angular velocity). The authors make use of human skeleton sequences and simulate IMU measurements by calculating the acceleration and angular speed at each joint. These simulated data are then used to pre-train a spatial-temporal graph encoder. Since each skeleton is paired with texts, the graph encoder is trained with contrastive learning against CLIP text features. Augmentations on IMU data and texts are performed to boost the performance.
strength_weakness: strength_weakness: 1. I have some concerns about the quality of the generated IMU data as it is constrained to 20Hz due to the original skeleton sequences. This sampling frequency might be enough for action recognition but it is unknown whether it can be used for other tasks such as inertial navigation. Besides, some sensor-level distortion such as IMU drift or some spike noises cannot be simulated using the proposed method.

2. What is the model size of the graph encoder? Since the model is trained on IMUs from many joints (>20?), the model might be huge when fine-tuned on a dataset that has fewer IMU sensors (e.g., 2 sensors), which could make the model inefficient for real-world applications.
questions: questions: Please refer to weakness
limitations: limitations: The authors have adequately addressed the limitations
paper_summary: paper_summary: This paper introduces a unified pre-training procedure for motion time series that generalizes across diverse device latent factors and activities. A contrastive learning framework that aligns motion time series with text descriptions enriched by large language models and the spatio-temporal graph networks are utilized to capture the relationships across joints for generalization across different device locations. Experimental results show the superior performance of the proposed method that selected baselines.
strength_weakness: strength_weakness: 1. It would be better to provide more descriptions regarding the usage of LLM.
2. It would be better to discuss the space and time complexities of the proposed framework.
3. As efficiency is important in real-world applications. It would suggest conducting experiments to compare the proposed method's training time (including pre-training time) with baselines.
questions: questions: 1. What is the motivation for the usage of LLM?
2. Can the proposed method improve the efficiency of existing methods (e.g., baselines)?
3. How do you obtain the text descriptions?
limitations: limitations: Please see the weaknesses and questions.
paper_summary: paper_summary: In the context of time series tasks, the paper addresses motion time series data collected from mobile and wearable devices such as smartphones and smartwatches. However, training and testing on the same dataset leads to poor generalizability of the models. The authors propose UniMTS, a unified pre-training procedure for motion time series that generalizes across diverse device latent factors and activities.
strength_weakness: strength_weakness: - After reading the Abstract, the actual contribution of the paper is unclear. The Abstract describes several minor contributions, but what is the focus of the paper?
- It is unclear which large language model is used (should be mentioned earlier).
- Does the proposed method actually fit to NIPS? The contributions are limited/unclear.
questions: questions: - Why cannot domain adaptation be utilized to address the shift in data or continual learning to adapt to new activities, etc.?
limitations: limitations: - The limitation is that the main contribution of the paper is utilizing LLMs to improve the generalizability of time series models.
paper_summary: paper_summary: This paper proposes a unified pre-training framework using simulated data from body skeleton model for motion time series. The authors rightly pointed out the challenges of collecting motion time series at scale due to privacy concerns despite the fact that the motion time series data have great promises in a wide range of applications. Thus, this is an area with few pre-trained models unlike other data modalities. 

The simulation framework proposed was used to generate the synthetic data needed for pre-training. The author then utilised the generated motion time series in contrastive learning using a graph neural network that learns a joint embedding for sensor data at different body locations. The extensive evaluations of the pre-trained model demonstrated superior performance for human activity recognition in several settings, including zero-hot, few-shot and full-shot performance.
strength_weakness: strength_weakness: 1. Despite my general excitement about this paper, this paper has not recognised the body of work already done in large-scale pre-trained models for motion time series. The authors might not be familiar with this area of work because existing pre-trained models generally use Biobank and are published in journals. It would be great if the authors could compare the following work in the context of their results: 
    1. Yuan, Hang, et al. "Self-supervised learning for human activity recognition using 700,000 person-days of wearable data." NPJ digital medicine 7.1 (2024): 91.
    2. Spathis, Dimitris, et al. "Self-supervised transfer learning of physiological representations from free-living wearable data." Proceedings of the Conference on Health, Inference, and Learning. 2021.
2. Pre-training from synthetic data is a cost-effective approach. Its performance is likely to plateau at some point as compared to pre-trained using real data. It would be interesting to see a comparison with Yuan et al. 2024 as this model was pre-trained on 700,000 person-days of real data. I also appreciate that adding an experiment might not be realistic, but I do believe that it should be fairly easy to do the fine-tuning. 
3. The HumanML3D human selection data only spans over a mere 29 hours. I am very suspicious that any pre-training on just 29 hours of synthetic data will be enough for pre-training as scaling laws go. Even though the current results show good performance, if I am not wrong, the majority of the benchmarks were collected in the lab, which doesn't represent realistic human behaviour in a free-living environment. CAPTURE-24 could be another interesting benchmark as it is one of the largest open-access free-living activity recognition datasets. More importantly, it would be good to get a sense of how the downstream performance scales to your pre-training data so that we can determine whether the benefits of pre-training have plateaued. 
4. Also, you probably don't have to add the references below, but I just want you to know that there are several biobanks that contain large-scale wearable sensor data that can be used for pre-training. Your work is still valuable as your can support multi-sensors, whilst most of the existing large-scale datasets only contain wrist-worn devices.
	1. Doherty, Aiden, et al. "Large scale population assessment of physical activity using wrist worn accelerometers: the UK biobank study." PloS one 12.2 (2017): e0169649.
	2. Chen, Yuanyuan, et al. "Device-measured movement behaviours in over 20,000 China Kadoorie Biobank participants." International Journal of Behavioral Nutrition and Physical Activity 20.1 (2023): 138.
	3. Master, Hiral, et al. "Association of step counts over time with the risk of chronic disease in the All of Us Research Program." Nature medicine 28.11 (2022): 2301-2308.
questions: questions: 1. What your input data format during fine-tuning and sampling frequency? 
2. Are Model weights and pre-trained model made available?
limitations: limitations: The authors have addressed this adequately.
paper_summary: paper_summary: Overall, this is a paper about human activity recognition from motion time series data. Authors design a unified framework for data generation, pre-training, and evaluation. The data used for pre-training is generated by motion skeleton data. LLM is also involved in text augmentation. Experimental results on 18 datasets demonstrate the superior performance of this paper.
strength_weakness: strength_weakness: Overall, I think this is an interesting paper and is promising. Therefore, I provide lots of comments. However, it has some obvious potential flaws in the current version. Please see limitations.
questions: questions: I have some questions regarding the experimental design, baselines, related works, novelty, etc. I put them in the limitation section.
limitations: limitations: [1] The data generation is interesting. I am thinking about what if we replace the generated data with a few real data for model training, how will the performance look like. Or, please see my question in [8], how to justify the importance of synthesized data. 

[2] Missing related work. There are important studies of pretraining motion time series missing. Authors can check more carefully, e.g., from sensys, kdd. 

[3] For the random rotation method, there are multiple existing approaches. Authors need to justify the difference or add citations. For example, "Device Orientation Independent Human Activity Recognition Model for Patient Monitoring Based on Triaxial Acceleration”, “Data augmentation of wearable sensor data for Parkinson’s disease monitoring using convolutional neural networks”,  and a few other papers from Mobicom and Ubicomp also applied similar methods.  

[4] Also, for the rotation method, does Acc and Gyro share the same rotation? 

[5] I am wondering is there any data normalization? Given the distribution discrepency among synthesized data and real data, it might hurt the performance. Currently, I only see a normalized matrix in Line 156. 

[6] For the data generation module, I like the idea of utilizing physical laws. However, it seems this generation method is not proposed by this manuscript. Therefore, authors might want to justify the technical contribution. 

[7] Fig 7 looks weird. Does it mean ImageBind cannot get any samples correct for most activities? Also, how the proposed method and ImageBind infer the activities not seen before should be described in more detail. 

[8] The performance gain of the zero-shot setting is impressive. For zero-shot setting, how are baseline models are trained? Are they fine-tuned also with the generated data? 

[9] In line 131, it would be great if there is some documents can support the claim. 

[10] I guess the authors aim to address the generalization issue of HAR, e.g., device locations, orientations. There are quite a few studies in these area, for domain adaptation  domain generalization. Some of them explore augmentation, some explore pretraining, etc., to address the challenges mentioned in the introduction. 

[11] For zero-shot setting, the performance seems too low for real-world usage, e.g., for datasets with only 4 or 6 classes, the acc is smaller than 50%. 

[12] Authors mention cross-dataset a few times in the paper, is there some experiments focusing on this setting except the zero-shot experiment? 

[13] Another concern I have is the full-shot setting, there are a bunch of SoA models from AAAI, KDD, IMWUT. I am not sure whether the provided baselines are representative. However, it is good to see authors replace some baselines from zero-shot settings.

Comments: title: Response to the authors
Some concerns are addressed
comment: We sincerely thank the reviewer for all the insightful feedback and for supporting our work!
We sincerely thank the reviewer for the insightful comments. Our text encoder leverages pre-trained weights from CLIP, which inherently understands semantic similarities between words. During pre-training, we didn’t pre-define any categories; we fed the model with extensive text sentence descriptions to align motion time series with CLIP’s semantic space. Some example text descriptions in our pre-training dataset are “the standing person kicks with their left foot before going back to their original stance”, “a man lifts something on his left and places it down on his right”, “a person walks slowly forward then toward the left hand side and stands facing that direction”. 

During evaluation, UniMTS leverages such semantic awareness to effectively generalize to new activities. For example, during pre-training the model has encountered “walking” and “running”, during evaluation it can leverage the knowledge that “jogging” represents a movement with intensity level in between “walking” and “running” to successfully generalize and recognize “jogging”. Other examples include generalizing from “ascending stairs” to “walking on inclined treadmill”, and from “waving hands” to “drawing circle counter clockwise”. 

We sincerely thank the reviewer once again for the invaluable feedback and support.
I want to thank the authors for their thorough response and new experiments.
Thank you for the detailed responses. As you mentioned, the model is only exposed to old datasets and old categories for the training process, which makes sense to me. I am also interested in the new category prediction. Is the model capable of predicting any new categories? Or, you need to pre-define a few new categories (e.g., running and jogging), so the model can link the representation with the pre-defined new categories.
We sincerely thank the reviewer for the insightful feedback and for supporting our work. We have addressed the concerns as below.

[1] We follow the same zero-shot setting as CLIP [1] for both UniMTS and ImageBind. During pre-training, the model learns to connect motion time series with text descriptions on the pre-training dataset. During evaluation, the model predicts new labels on new downstream datasets not seen during training in a zero-shot manner. The term zero-shot refers to the model’s exposure to both new downstream datasets as well as new categories without prior direct training on them. 

[2] Yes, the zero-shot setting is extremely challenging. Apart from the 342.3% improvement over the best existing baseline, we believe that with the availability of high-quality, large-scale real-world datasets in the future, our framework can be directly applied to model these datasets during pre-training and achieve even more significant improvements.

Thank you once again for your support of our work!

[1] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J. and Krueger, G., Learning transferable visual models from natural language supervision, ICML 2021
Dear authors,

Thank you for your timely response! 

While I agree with you on some points, some concerns remain. 

1.The pairwise alignment makes sense to me. However, there are still labels needed to enable the model to know, what kind of time series looks like a specific label, say, "walking". Then, in future cases, when the model is given a time series of new labels, say, "running", the model can make the prediction. Therefore, in terms of zero-shot, I guess you mean the model has seen "zero" training samples of data from new categories but still needs some labeled data from at least some labels. Hopefully I have made myself clear. 

2.Your responses make sense to me and I admit the improvement compared to SoAs is significant enough. I still think there is some steps to go to make it practical, at least in the zero-shot setting. But this is totally fine. 

Given the comprehensive rebuttal and most of my concerns are addressed, I have raised my score. Hope to see the camera-ready version, great work!
We sincerely thank the reviewer for all the valuable feedback and for recognizing the potential broad impact of our work. We greatly appreciate your support!
Thank you for your comprehensive response and the further clarifications provided. The paper's contribution is now well understood. The primary contribution lies in the innovative pre-training framework, which enables generalization across datasets with varying device locations, orientations, and different activity types. The experiments presented substantiate this contribution. I also appreciate the additional references provided.

All my concerns have been addressed, and I have increased my rating to acceptance, recognizing the potential for broad impact.
We sincerely thank the reviewer for all the insightful feedback and we greatly appreciate your support!
Thank you for your timely response. Combining your previous actual space and time complexity, your analyses are comprehensive. I have raised my score. Nice work!
We sincerely thank the reviewer for all the insightful feedback, and for the recognition of the quality and potential broad impact of our work. We are committed to releasing the datasets and pre-trained models upon acceptance of the paper. Thank you once again for your valuable feedback and support.
I would like to thank the authors for their responses. Most of my concerns have been resolved. It is a good work with potential broad impact. Look forward to the release of dataset and pre-trained models.
We sincerely thank the reviewer for the insightful feedback. We compare the theoretical space and time complexities with the best-performing baseline ImageBind as follows. Since UniMTS and ImageBind share similar text encoders, our analysis mostly focuses on their respective IMU encoders. Let $B$, $L$, and $F$ represent the batch size, sequence length, and feature dimension, respectively. Additionally, let $V$ and $V'$ denote the total number of joints (with $V=22$ for the HumanML3D dataset) and the number of joints with attached devices in downstream datasets ($V' \leq V$). We note that the skeleton graph is sparse and the number of edges is $E=V-1$. 

For UniMTS, the time complexity is $\mathcal{O}(BVLF^2 + BVLF)$ and the space complexity is $\mathcal{O}(BVLF + F^2 + V)$. For ImageBind, the time complexity is $\mathcal{O}(BV'LF^2 + BV' L^2F)$ and the space complexity is $\mathcal{O}(BV'LF + F^2 + BV'L^2)$. We observe that UniMTS does not contain the $L^2$ terms present in ImageBind (i.e., complexity grows linearly rather than quadratically with respect to sequence length), making it more efficient in terms of both time and space complexities.
Thanks to the authors for answering my questions. I am satisfied with the responses except for the Space and time complexities compared with the baselines. I mean the theoretical space and time complexities analysis. Thanks.
Thank you so much for your insightful comments and for recognizing that our work is of high quality and high impact!

We have tried our best to evaluate the model on a diverse set of real-world datasets, and to reduce bias by splitting train/test data based on subjects. We believe that future availability of larger-scale benchmarks will further validate the robustness of our model. We greatly appreciate your interest in our work and are committed to releasing the model upon acceptance.

Thank you once again for all your invaluable feedback!
Thank you very much for the helpful feedback. 

Your evals make sense. I fear that the issue with existing benchmark is that it is very easy to overfit on tiny test set (data with a few subjects) with a large model. So I wouldn't necessarily trust the test results even though they conform with existing literature for consistency sake. 

This work is of high quality and has high impact to the field of ubiquitous computing and moderate impact on several other fields including pre-training using synthetic data and multi-modal learning.  

I thoroughly enjoyed reading this paper. Looking forward to seeing the final version this manuscript and the open release of the model!
We sincerely thank the reviewer for the insightful comments, and for recognizing that our paper has good presentation, clear motivation, and high potential for a broad impact. We have addressed the concerns as follows.

[1] References and explanations

We will make sure to include the additional related works, as well as detailed explanations such as the same rotation for gyro and acc. 

[2] Zero-shot setting

ImageBind is pre-trained to use image modality as a common bridge to bind multiple modalities together, including motion time series and text. More specifically, the model is pre-trained to align pairwise images and motion time series from the Ego4D daily-life activity dataset, as well as to align large-scale web images and text. By having image modality as the common bridge, the model therefore has the knowledge to connect motion time series with text descriptions. However, as Ego4D only contains motion time series from head-mounted devices, the model struggles to generalize to downstream datasets with different device placements. 

[3] Practical impact

We would like to note that recognizing new classes in a zero-shot fashion is extremely challenging. Despite the difficulties, UniMTS has outperformed the best existing baseline by an impressive 342.3%. Furthermore, the top-2 retrieval score has increased to approximately 60. In practice, it is often feasible to obtain a few labels for downstream datasets. With just one labeled example per class, the F1 score can improve to about 60, and it further enhances to about 75 with ten labeled examples per class, which significantly outperforms the best baseline by 16.3%. Additionally, even in the full-shot setting, UniMTS continues to outperform the best baseline by 9.2%, making it practically impactful for a broad range of real-world use cases. 

Thank you once again for all your insightful feedback!
I would like to thank the authors for providing detailed feedback, which addressed some of my concerns. 

First, the additional experiments are valuable. Results in both Table 1 and Table 2 provide interesting insights about the synthetic data. 

Second, the additional related works are convincing, I believe these studies will benefit this paper and put this paper in a better position. 

Third, I hope some explanations could be incorporated in the final version, e.g., the same rotation for gyro and acc. 

However, I still have one concern about the zero-shot setting in 7 and 8. If methods like ImageBind are only trained with unsupervised pre-training, it will not have the knowledge to connect motion data with corresponding labels. Or, the zero-shot means ImageBind is also fine-tuned with labels (such as walking and running), but zero-shot for recognizing new activities (such as still and biking)? 

Another concern is the practical impact. Given the low performance of recognizing new activities, I doubt how such performance could generate a practical impact for real-world usage. 

Again, overall this is a good paper with a good presentation, clear motivation, and high potential for a broad impact. Great work.
Dear Reviewer,

Thank you so much for your insightful comments and for raising the score! We have answered your questions as follows.

[1] Few-shot fine-tuning performance

For datasets with a public train/test split (e.g., UCI-HAR), we adopt the available divisions. For the remaining datasets, we split them based on subjects, except for only very few datasets where we didn’t find subject specification (e.g., UT-Complex). Our test window size is 10 seconds, which carries sufficient information for the model to infer the activities and to minimize bias from random noise.  

UniMTS shows great generalizability in few-shot scenarios due to its pre-training on our synthesized data with diverse device locations, orientations and activity types. In contrast, without such pre-training on diverse device latent factors and text semantics aligning, Yuan et al. 2024 initially performs near-randomly (e.g., achieving an F1 score of 25 for 4-class Opportunity dataset). However, self-supervised pre-training on large-scale BioBank data also provides good initialization for Yuan et al. 2024 to converge faster, so the model is able to quickly improve its performance as more samples per class are given.

[2] Discussion on synthetic and real-world data

Yes, we believe that the community would greatly benefit from large-scale real-world data with comprehensive joint coverage and detailed text descriptions, if such a dataset becomes available in the future. Should such datasets become available, our framework of graph encoding and contrastive semantic aligning is still directly applicable to model these new datasets, and incorporating these data into our pre-training framework will potentially lead to more significant improvements. However, in the absence of such comprehensive real-world data, currently synthetic data still present higher quality than real-world datasets with limited joint coverage and limited text descriptions. Therefore, we generate synthetic data as the first step to address the challenges, establishing the first pipeline for pre-training motion time series. We believe our approach will bring insights to the community to address this long-standing challenge of data generalization, and we will make sure to discuss these comparisons between synthetic and real-world data in the final version. 

For the full-shot experiments of BioBank SSL, we observe that datasets with wrist-worn devices, such as WISDM and Wharf, also show slightly better performance compared to UniMTS, which is consistent with the observation that incorporating Capture-24 slightly improves performance on these wrist-worn datasets. For few-shot experiments, given that BioBank SSL is not pre-trained to align with text semantics, it would be hard for it to generalize to new activities given few shots, even for datasets with wrist-worn devices. Therefore, UniMTS consistently outperforms BioBank SSL in the few-shot scenarios, demonstrating the significance of our pre-training pipeline. 

Thank you once again for your invaluable feedback!
Thank you for all the additional experiments that provided pre-training based on synthetic data compared to biobank pre-training. 
I just have two more comments and questions: 

1. In your Figure 1 comparing BioBankSSL and UniMTS, the few-shot few-tuning performance might be too good to be true, at least in the first instance. How are you doing the train/test split? Currently, the axis is the number of samples from each activity class. If your train/test split is not subject-wise, then the test result is biased.  

2. When you incorporated real data for pre-training (wrist-worn data), your evals on wrist datasets improved despite the evals on other datasets decreased. This is telling me that realistic data is still better than synthetic data if available. I wouldn't expect a model pre-trained on the wrist to work well with other placements, for example. It may be worth discussing the value of realistic and synthetic data in different use cases. Given this observation, I was surprised to find that your BioBankSSL experiments showed that synthetic pre-training on wrist data did better. 

Overall, human activity recognition has poor benchmark datasets, often extremely limited in scale (n<50 participants). I wouldn't hold the authors accountable for addressing the benchmark issue for the community. It will take time, but once this paper is accepted and externally validated by other researchers, we can better test the generability of the proposed framework.

You have done a great job addressing most of my concerns, and I have adjusted my score from 6->7.

Obviously, it would be great if you could reply to my new queries above :D