MoPFormer: Motion-Primitive Transformer for Wearable-Sensor Activity Recognition
Authors: Hao Zhang
Zhan Zhuang
Xuehao Wang
Xiaodong Yang
Yu Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Human Activity Recognition
Wearable Sensors
Deep Learning
Primary area: applications
TL;DR: MoPFormer: Motion-Primitive Transformer for Wearable-Sensor Activity Recognition

Abstract: Human Activity Recognition (HAR) with wearable sensors is challenged by limited interpretability, which significantly impacts cross-dataset generalization. To address this challenge, we propose Motion-Primitive Transformer (MoPFormer), a novel self-supervised framework that enhances interpretability by tokenizing inertial measurement unit signals into semantically meaningful motion primitives and leverages a Transformer architecture to learn rich temporal representations. MoPFormer comprises two stages. The first stage is to partition multi-channel sensor streams into short segments and quantize them into discrete ``motion primitive'' codewords, while the second stage enriches those tokenized sequences through a context-aware embedding module and then processes them with a Transformer encoder. The proposed MoPFormer can be pre-trained using a masked motion-modeling objective that reconstructs missing primitives, enabling it to develop robust representations across diverse sensor configurations. Experiments on six HAR benchmarks demonstrate that MoPFormer not only outperforms state-of-the-art methods but also successfully generalizes across multiple datasets. More importantly, the learned motion primitives significantly enhance both interpretability and cross-dataset performance by capturing fundamental movement patterns that remain consistent across similar activities, regardless of dataset origin.

MoPFormer: Motion-Primitive Transformer for Wearable-Sensor Activity Recognition
Hao Zhang'; Zhan Zhuang!2; Xuehao Wang', Xiaodong Yang*, Yu Zhang't
'Souther University of Science and Technology, "City University of Hong Kong 'Zhejiang University, "Institute of Computing Technology, Chinese Academy of Sciences zhanghao@mail sustech. xuehaowangf @gmail com, yangxiiadong@@icom ac. yu. zhang. ust@gmail. com
sustech edu. en, zhazhuang3-cany htt.icctttctttt...m cityu. ettyy.edu.... edu.hk
Abstract
Human Activity Recognition (HAR) with wearable sensors is challenged by lim- ited interpretability. which significantly impacts cross-dataset generalization. To address this challenge, we propose Motion-Primitive Transformer (MoPFormer), novel self-super framework that enhances interpretability by tokenizing inertial measurement unit signals into semantically meaningful motion primitives and leverages Transformer architecture learn rich temporal representations. MoPFormer comprises two stages. The first stage is to partition multi-channel sen- sor streams into short segments and quantize them into discrete "motion primitive" codewords, while the second stage enriches those tokenized sequences through a context-aware embedding module and then processes them with a Transformer encoder. The proposed MoPFormer can be pre- pre-trained using masked motion- modeling objective that reconstructs missing primitives, enabling it to develop robust representations across diverse sensor configurations. Experiments HAR benchmarks demonstrate that MoPFormer not only outperforms state-of- the-art methods but also successfully generalizes across multiple datasets. More importantly, the learned motion primitives significantly enhance both interpretabil- ity and cross-dataset performance by capturing fundamental movement patterns that remain consistent across similar activities, regardless of dataset origin.
six
Introduction
Human Activity Recognition (HAR) has wide range of applications and can be achieved through various methods [42 4]. While vision-based and environmental sensor-based methods have been extensively explored 23 18), systems leveraging Inertial Measurement Units (IMUs) offer distinct advantages, including compact size, low power consumption, and minimal computational overhead for data acquisition and preprocessing. IMUs also inherently preserve privacy compared to visual alternatives 10 3 and are robust to environmental factors like lighting variations and occlusions (25) Such characteristics make IMUs a highly suitable sensing modality for capturing human motion across diverse real-world scenarios.
Despite these strengths, IMU-based HAR confronts a significant hurdle: interpretability. IMU data, comprising multi-channel time series of linear acceleration and angular velocity, is inherently less intuitive for human to interpret than video data, making hard understand what the model has learned 19]. Moreover, many human activities exhibit hierarchical structures, composed of more
Equal contribution. Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Motion
VQ Index
Context- Awrext-Awwre Embedding Module
Primitive
11
43
Module
Sensor
Sensor 2
Data Windows
Stat
Feature
Transformer
Task Heads
M
M
E
S
Encoder
M
...
M
E
S
Backbone
Figure 1: Architecture of the proposed MoPFormer model, showcasing the flow from raw data windows through tokenization, embedding, Transformer encoding. task-specific heads.
fundamental activities movements. However, there is common absence of fine-grained labels for the sub-actioo -ebbviiivisees that constitute these complex, hierarchical activities. For example, an activity like "washing hands" can be deconstructed into sub-activitics such "turning the tap". "applying soap". "scrubbing hands", and "rinsing hands". An HAR model might learn to identify "washing hands" by detecting the "turning on the tap" sub-activity, especially if it is prominent signal. However, the model encounters a different activity. such as "filling a water bottle", which also begins with "turning the tap", it may struggle for HAR models to differentiate between the two activities. This is because the model might rely on the common initial sub-activity rather than grasping the complete, semantically meaningful sequence (e.g.. the subsequent application of soap and scrubbing unique to handwashing) that defines the core of the activity. Each of these sub-activities might, in turn, be composed of even finer finer-graine grained movements. The black-box nature of contemporary HAR models often obscures whether these models are learning the underlying activity structure merely exploiting superficial data correlations, thereby exacerbating the interpretability challenge [27] (
Another challenge in sensor-based HAR data heterogeneity. Variations in sensor types, sampling rates, on-body placement (e.g., sensors worn the wrist the waist), subject characteristics (e.g.. age, height, fitness level of the individual), and environmental conditions can introduce significant distributional shifts [32). This heterogeneity impedes model generalization across different contexts. Consequently, developing strategies to address data heterogeneity is crucial for building robust HAR systems.
To address these challenges, we introduce Motion-Primitive TransFormer (MoPFormer). a novel pre-training architecture for IMU-based HAR that significantly enhances model interpretability. Inspired by advancements in language modeling [20]. MoPFormer conceptualizes IMU motion sequences series of "Motion Primitives" that serve semantically meaningful, discrete building blocks of human activity. As illustrated in Fig. [ MoPFormer first divides raw IMU sequences into short segments, analogous to "words" in natural language, creating interpretable vocabulary of fundamental movements. Each segment integrates data from various sensor channels at concurrent time points, forming feature vectors that then arranged sequentially into two-dimensional feature matrix. This construction not only mirrors the arrangement of words in a sentence but also enables transparent analysis that which motion primitives contribute to activity recognition decisions. To further enrich semantic understanding, we embed metadata of each sensor using context-aware embedding module. The resulting representation allows for direct examination motion primitive similarities, frequencies, and their distribution across different activities, providing unprecedented subsequently processed by
insights into model behavior. This comprehensive representation a Transformer backbone [48), enabling MoPFormer to effectively handle heterogeneous channel configurations and learn robust, interpretable features for diverse downstream tasks.
In summary, the primary contributions of this work are:
We propose MoPFormer, an effective pre-training framework that achieves state-of-the-ar perfor- mance.
We introduce representation method that enables unified training across heterogeneous datasets.
We extract motion primitives and provide detailed interpretability analysis of them.

---

2 Related Work
Traditional human activity recognition approaches train model parameters on different datasets separately, including statistical feature extraction methods and deep learning approaches. Deep learning methods include CNN-based architectures such as CALANet (33 COA-HAR (50). MA- CNN [37], and SenseHAR (24). RNN-based approaches like DeepConvLSTM (30), and attention-
based models such THAT (26) and PA-HAR [5I]. Recent research has proposed purppoe time series models applicable to various tasks, including classification, such as TimesNet [49], TSLANet [17], and FITS [52]. However, these methods typically trained and tested splits from the original datasets and are limited in their ability to achieve cross- cross-dataset generalization.
Self-supervised human activity recognition methods perform representation learning through various approaches. Reconstruction-bent
methods such as TST (55) and TARNet (13) focus on rebuilding
input signals. Contrastive learning methods such as TS2Vec (53), CL-HAR (35). DDLearm (36),
TS-TCC (16), FOCAL [28]. and ModCL [22] execute discriminative representation learning tasks. Other self self-super objectives like BioBankSSL (55) [4] and Step2Heart (45) also contribute to
learning time series representations. These methods typically learn feature representations on specific datasets and fine-tune classifier heads the target dataset, but they can only perform datasets with similar structures and still require labeled data from the target dataset.
Research on sensor-based HAR for composite activity recognition is relatively limited (11). For instance, Chen et al. decomposed composite activities into multiple simple activities [Ti], where each simple activity is represented by sequence of sensor signal segments. These segments are first fed into a CNN to extract representations for identifying simple activities. Concurrently, the CNN- extracted features from all segments are passed to an LSTM network to achieve high-level semantic activity classification. Similarly, prior work [12) inferred composite activities using estimated activity sequences, where temporal correlations of simple activities were extracted for composite activity classification. Conversely, predicted composite activities were used to aid the derivation of simple activity sequences for the next time step, with predictions for both simple and composite activity sequences mutually updating during inference. While these studies present viable approaches for composite activity research, datasets with comprehensive, multi-level hierarchical labels for composite activities are scarce, and existing hierarchical annotations are often incomplete.
The concept of motion primitives has gained traction in robotics research (4 These primitives are fundamental movement patterns that can be sequenced and combined to generate or understand a diverse repertoire of complex human activities. For instance, Calvo et al. utilized eight such primitive movements, including "static" and partial "squat", to train Hidden Markov Models for activity classification. Similarly, Sanzari et al. developed framework for automatically discovering human motion primitives from 3D pose data by optimizing "motion flux" and then clustering the results (4T). However, the application of motion primitives to enhance the interpretability of IMU-based HAR has seen limited progress. This limitation stems largely from the inherently non-intuitive nature of IMU sensor data, which makes it challenging to define and interpret primitives directly from these signals.
Much of the research on interpretability in HAR has focused on attention mechanisms. The principle behind deep attention models is to weight input components, with the assumption that components assigned higher weights are more relevant to the recognition task and exert greater influence on the model's decisions (43). For example, Shen et al. (44) designed a segment-level attention method to determine which time periods contain more information; combined with a gated CNN, this segment- level attention can extract temporal dependencies. Zeng et al. (54) developed attention mechanisms from two perspectives: they first proposed sensor attention on the input to extract salient sensory patterns, and then applied temporal attention to LSTMs to filter out inactive data segments. Ma et al. (29) employed spatial and temporal attention mechanisms, extracting spatial dependencies by fusing patterns with self-attention. However, their interpretability often remains superficial level, indicating what the model focuses on, rather than why or how these features contribute to the recognition of complex, hierarchically structured activities.
3 Methodology
The architecture of MoPFormer, as illustrated in Fig. 2 comprises four modules: the motion primitive module, the context-aware embedding module, and two task heads. The motion primitive module first
3

---

(a)
(b)
(c)
MAE Head
VQ Index
Stat Feature
M
M
E
S
...
42
Linear+VQ
Pasitional Encoding
Metadata Embedding Adapter
MLP+ MLfP+Softmat tmax
Instance Normalization
Motion Primitive Embedding Layer
Stat Feature Embedding Layer
CLS Head
Motion Primitive Partitioning
Special Token
Stat Feature
MLP+Softmax
Insertion
VQ Index
Data Windows
Figure 2: Detailed illustration of key modules in motion-centric framework. (a) Motion Primitive Module: raw data windows from multiple sensor channels are partitioned into motion primitives and processed through instance normalization generate Vector Quantization (VQ) indices and statistical features. (b) Context-Aware Embedding Module: special tokens are inserted alongside masked motion embedding tokens (M) form the complete imput representation, which combines motion primitive embeddings, positional encodings, and statistical feature embeddings. (c) Task Heads: The diagram shows transformed features X representing the corresponding position vectors after Transformer Encoder processing. The MAE head utilizes M positions for masked token prediction during pretraining, while the trainable CLS head operates exclusively on the transformed [CLS] token representation for downstream task fine-tuning.
transforms multi multi-dimennsonal time-series data windows into index sequences from a fixed vocabulary. To further extract patterns, the context-awand aware embedding module extracts semantic information from contextual relationships and embeds nigh-ddmensiioal vector sequences for the Transformer backbone. The Masked anto-Fncodiig (MAE) head, as a pre-training task, strengthens the context- aware embedding module's 's ability to understand motion primitives, while the Classification (CI.S head enables the Transformer backbone to learn effective features for classification. Fig. 2 Fillustrates the detailed architecture of each key module in our framework.
3.1 Motion Primitive Module
The Motion Primitive Module transforms data windows into discrete motion primitives, serving as the foundational, interpretable units for our motion-centric representation. For each data window of shape T C, where T is the number of time steps and C is the number of channels (e.g., accelerometer, gyroscope), we first partition the sequence into non-overlapping segments across each channel independently. Each channel divided into segments of length L. resulting in S [ segments per channel and a total of S x C' segments per window.
Each segment undergoes feature extraction process where we compute both low-level temporal patterns and statistical characteristics. For each segment & E RR' (a single-channel segment), we apply instance normalization in Eq. (1) to standardize the input:
8, m(s:)
(s:) +
where ((s;) and ar (8:) represent the mean and standard deviation of segment 8,, and is a small con- stant for numerical stability. This normalization removes sensor-specific scale differences, ensuring all segments are comparable scale before quantization.
The normalized segments are then encoded into discrete motion primitives using a Vector Quantization (VQ) (47) approach. We maintain a learnable codebook Z = {21, .2, zK ). which contains K prototype vectors (or "motion primitives"),, each of dimension L. As illustrated in Fig. a), for each normalized segment &,, we compute its VQ index through Eq. (2) by finding the nearest prototype in the codebook:
=
arg arg min ke (1.2.....)
|| 24 11.
(2)

---

This quantization process maps continuous sensor data to discrete motion primitives, establishing a "vocabulary" of motion primitives. Alongside the VQ indices, we extract statistical features from each segment (mean, variance) to capture complementary information that might be lost during quantization. These statistical features Sa are concatenated with the VQ indices to form the complete motion primitive representation.
The codebook is learned end-to-end through a commitment loss defined in Eq. (3) that encourages consistency between the input segments and their quantized representations:
Lv | 2q. 1 + B|| sg -9[2y.]1 11,
where sg[:] denotes the stop-grade gradiet operator and B a hyperparameter controlling the commitment cost.
3.2 Context-Aware Embedding Module
The Context-Aware Embedding Module transforms the discrete motion primitive indices and their associated statistical features into rich, contextualized representations suitable for the Transformer backbone. This module aims capture not only the type of motion primitive but also its intensity, variability, sensor origin, and temporal position. For each data window, the module processes the sequence of VQ indices {41, 42....:9.. } and corresponding statistical features {ʃ1 f2, .... fs} from all channels.
For the special tokens [MASK], [START], and [END], we assign reserved indices (K + 1, K + 2. and K +3, respectively) our vocabulary, ensuring they are consistently represented across different sensor channels. Using a learnable embedding matrix EvQ E R(K-3)x where D is the embedding dimension, we embed the VQ indices through Eq. (4)
VQ Evqlai]
The [CLS] token is separate learnable parameter outside the VQ embedding matrix.
The statistical features obtained during instance normalization (mean, variance) projected into the same embedding space via linear projection in Eq. (
stat
W WstntJi batat-
(5)
To incorporate sensor metadata (e.g., sensor type. mounting position) and enhance the model's ability to generalize across different sensor configurations and distinguish between data from different sources, we introduce the Metadata Embedding Adapter for sensor-specific embeddings. For each segment its corresponding sensor channel has associated metadata. This metadata is first converted into fixed-length vector emeta IR using a pre-trained text embedding model. The Metadata Embedding Adapter uses a linear layer (W Wadopter E IR Dmodet X. badapter E IR Dmodet), then maps this N-dimensional vector
model's 's embedding dimension Dmodel.
For each segment i, its complete token embedding e, is formed by summing these constituent embeddings, shown in Eq. (6. This fusion represents the motion primitive in way that includes its quantized shape (e) Q magnitude/variaaiiiit information (eştat), and sensor context (e (endapter_enm) _emb):
e; e VQ + etaat + Wadepter (e + badapter;
where e meta is the pretrained text embedding of dimension for the channel c that segment belongs to.
The model incorporates various special tokens to provide structural cues in the input sequence.
The final input sequence to the transformer backbone is structured as in Eq. 7
X [[CLS], [ST ART"), ..., e LEN D),..,, ..., [ST ART"), of .... nnc DI],
(7)
where ef represents the j-th embedding from sensor channel c, and ne is the number of segments for channel
To provide temporal context, positional encodings are added through Eq. N
X
X +
(8)
where P E SDD contains learnable position embeddings. Importantly, our positional encoding scheme assigns identical positional embeddings to data from different channels at the same time step, meaning that ef and ef receive the same positional encoding if they represent data from the same time point but different channels.
5

---

3.3 Task Heads
As shown in Fig, Ce), MoPFormer employs dual-task learning approach with two specialized heads built on top of the transformer encoder:
Masked Auto-Encoding (MAE) Head. During pre-training, we randomly mask a portion of the motion embeddings in the input sequence and replace them with [MASK] tokens. The MAE head aims reconstruct the original VQ indices of these masked positions based on the contextual information processed by the transformer encoder. Eq. (9) shows how for each masked position i, the reconstruction is performed:
q softmax(W mmmach; + bmae);
where hę is the transformer output at position i, and W Wmae and bmaee are learnable parameters. The MAE objective in Eq. 10 is formulated as a cross-entropy loss between the predicted and true VQ indices:
Liaa
log (:1)),
(10)
JM
where M is the set of masked positions and de 0.14:] is the probability assigned to the true index 4s-
Classification (CLS) Head. For downstream activity recognition tasks, we utilize a classification head that operates exclusively on the transformed [CLS] token representation. The [CLS] token aggregates information from the entire sequence through self-attention mechanisms in the transformer encoder. Eq. 11 defines how the classification is performed:
softmax (Wess + bels),
(11)
where higs is the transformed [CLS] token representation and Wals and bas, are learnable parameters. The classification objective is formulated as cross cross-entroee consopy loss in Eq. 12
Lote
Σ My Ú, j=1
(12)
where y is the true activity label. During pre- vre-training, we primarily focus on the MAE task, while during different training phases, we balance various objectives through Eq. (I3):
Ammam Emae + Aatshals + Augfrogy
(13)
where Amans Aata. and Ang are hyperparameters balancing the different loss components from Eqs. (T). (12), and (3), respectively. By adjusting these hyperparameters, can control which tasks are optimized during different training stages.
For fine-tuning downstream tasks, freeze the Motion Primitive Module and Context-Aware Em- bedding Module while only updating the classification head parameters, enabling efficient adaptation datasets with minimal computational overhead.
4 Experiments
4.1 Experimental Setup
Datasets. We conducted extensive evaluations using six publicly available benchmark datasets:
PAMAP2 38 39), DSADS []. MHealth 6). Realworld (46), UCI-HAR [40] 2]. and USC-
HAD 56 To ensure fair comparison with existing self-supervised methods, we pre-trained our model on five of these datasets while using the remaining dataset for evaluation, maintaining consistent experimental settings with other self- celf-super approaches. All datasets are resampled to 100 Hz and segmented with a 500-sample window. To eliminate information leakage from overlapping windows, we adopt the setting by using strides equal to the window size for all labeled training segments. This strict non-overlapping approach may reduce absolute performance compared with more permissive settings, but ensures a more rigorous evaluation. We use accuracy and macro-F1 as evaluation metrics to assess model performance.

---

Table 1: Comparison with supervised pretrain- pretrain-and-transfer baselines (upper block) and supervised cenn-from--cratcee baselines (middle block) across six HAR datasets. Accuracy and macro-FI are reported in percent; the rightmost column lists the mean over all datasets.
Method
PAMAP2
DSADS
MHeallh
Realworld
UCI-HAR
USC-HAD
Average
Acc
80.88
91.30
88,94
85.18
82.66
74.04
83.83
BYOL
21
pperm_ji
Fl
79.01
89.22
88,59
86.81
80.32
73,92
82.98
Acc
76.76
90.80
89,74
86.08
81.18
72.82
82.90
BYOL 21
+lfe
F1
74.64
89.39
88.84
85.99
80.87
69.23
81.49
Acc
82.38
93.87
90.06
88.80
85.31
75.59
86.00
BYOL 21
+ixxpp
F1
80.93
93.29
89.32
87.95
86.78
70.04
84.72
Acc
82.49
92.46
90.19
89.69
89.79
76.64
86.88
ModCL 22
Fl
80.63
91.09
89.98
90.43
90.15
73.61
85.98
Acc 86.76
98.12
89.84
90.03
91.02
79.93
89.28
TSLANet
F1 84.08
97.43
87.85
91.81
89.84
77.74
88.13
Acc 85.10
96.01
86.05
90.07
88,38
77.37
87.16
CALANet
34
F1
83.94
95.58
83.80
91.59
86.91
72.36
85.70
Acc 86,08
97.60
93.22
92.05
90.58
81.46
90.17
MoPFormer
F1 84.83
97.38
92.28
93.25
89.98
77.67
89.23
Acc
85.76
94.68
90.53
91.84
77.58
78.30
86.45
w/o Pretrain
F1
83.94
94.64
89.95
91.33
76.79
74.86
85.25
Training Protocol. All experiments were conducted on Quadro RTX 8000 GPU. Our training protocol followed a two-stage approach. In the first pre-training stage, we trained the model across multiple datasets to extract a diverse set of motion primitives. We set the segment size to 50 samples for Motion Primitive length, used an internal embedding dimension of 256, and masked 25% of motion primitives prediction targets for the MAE task. In the second stage, the classification head and transformer layers are fine-tuned on each downstream dataset.
Metadata Embedding Adapter. Sensor descriptors standardized to consistent string format (c.g., "body __trttrrttt Chest, sensor: acc, axis: x"). We then obtained embeddings with Google's 's text- embedding- 004 API, which enables the model to Iearn from contextual sensor metadata effectively.
4.2 Results
Baseline Comparisons. We compared MoPFormer with six different baseline approaches. Follow- ing the comprehensive evaluation in (35), we selected the BYOL framework (2T). which demonstrated the best average performance among various contrastive learning frameworks. For time-domain and frequency-domali augmentations, we used perm_ jrr and Ifc, respectively, which showed superior performance in their domains. As noted [15), Mixup provides distinctive benefits compared to other temporal augmentation methods, so we included it as a baseline. ModCL. [22] represents state- of-the-art contrastive learning method that leverages both intra-modal and inter-modal consistency in wearable sensor data. We included it as a contemporary contrastive learning approach. For all contrastive learning methods, we maintained consistent 0.2:0.8 train-test split ratio.
Additionally, we compared against recent supervised approaches, including TSLANet [1 (ICML 2024), a widely recognized foundational model in the time-series domain, and CALANet (34) (NeurIPS 2024), which was specifically designed for HAR tasks [8] Detailed parameter configura- tions for all baseline models are provided in Appendix A.
Performance Analysis. Tab. presents comprehensive results across all six datasets. MoPFormer achieves state-of-the-art performance, obtaining the highest average accuracy and Fl scores. Specif- ically, our model achieved the best performance on MHealth (93.22% Accuracy, 92.28% F1) and USC-HAD (81.46% Accuracy), while maintaining competitive results on other datasets.
The ablation study (w/o pretrain) demonstrates that our pretraining strategy significantly contributes model performance, particularly on datasets like UCI-HAR with limited data but complex activity patterns, where performance drops by 13 percentage points without pretraining. This confirms the value of our two-stage training approach with motion primitive extraction and masked autoencoding.
7

---

1.25
00000
0.33
4.20
3.20
0.56
0.56-6.32
4.07 0.04
0.15
0.33
0.5
20
0:42
0.14
0:33
6.44
0.40
0.20
01225
00
30
0:91
0.31
0.45
50
1.50
-0.5
3
4
0:60
0:04
0.12
0.33
0.25
0.54
0.43
6:83
0.00
1:00
100
420
1.00
0.334
0.30
03
XX0
0.00
2:04
5.00
3.00
+0.30
0.61
0.30
20
3.0C
0.50
000
6-84
0.00
5.00
0.30
0.39
0.00
0.28
-20
0.32
0.34
0.33
0.10
0.30
0.27
-0.75
8.61
4.33
-00
420
(c)
(d)
Figure 3: Motion primitive segmentation and similarity analysis. (a) 5-second raw accelerometer trace from USC-HAD dataset, segmented into ten 0.5-second motion primitives. (b) simila-similarity ma- trix of motion primitive embeddings from accelerometer data. (c) Corresponding 5-second gyroscope trace with identical segmentation. (d) Cosine-similarity matrix embeddings for pyroscope-baser motion primitives. The matrices reveal pattern correlations between different motion primitives after Motion Primitive Embedding processing.
Table 2: Ablation study for each component on the PAMAP2 and DSADS datasets.
Ablation Study. To validate each compo-
nent's contribution, conduct ablation study on two representative datasets (Tab. (2). First, removing the pretraining stage leads modest accuracy drop, highlighting the impor- tance of nottion-rrimitive initialization in cross- modal generalization. Next, further omitting the statistical features excluding metadata embedding causes a dramatic performance loss.
PAMAP2
DSADS
Method
Acc
Acc
MoPFormer
86.08
84.83
97.60
97.38
w/o Pretrain
85,76
83.94
94.68
94.64
wlo Statistical Feature
69.95
53.60
71.15
62.34
w/o Metadata Fimhedding
58.18
53.10
80.94
75.40
Together, these findings show that pretraining, statistical features, and metadata embeddings each deliver unique information, and that the combination is critical to MoPFormer's superior performance. More results are put Appendix B.
MoPFormer's strong performance across diverse datasets validates our motion-centric approach and demonstrates its generalization capabilities. In the following section, will further analyze the extracted motion primitives better to understand their contribution to the model's effectiveness.
5
Analysis
5.1
Motion Primitive Similarity
To examine what the VQ codebook captures, we analyzed two example segments of five seconds each, shown in Fig. Each trace is divided into ten 0.5s motion primitives, and compute pairwise cosine similarity between the embeddings those motion primitives. A consistent pattern emerges: primitives describing similar kinematic shapes, such as indices 8 and 9 in (a) or indices I and 3 in

---

Activity Label
lying sitting
standing walking
running cycling
Nordic welking ascending
descending stairs cleaning
Trening
jumping
x10
1.0
0.3
Proportion
Total Count
0.6
0.4
0.0
170
50
776
390
684
151
1008
174
985
202
VQ Index
Figure 4: Frequency and activity composition of the 32 most common motion primitives from The stacked bars (left axis) show the proportion of each activity label for every VQ index, while the black line (right axis) plots the absolute occurrence count of motion primitives.
(c), cluster into blocks with high cosine similarity. In contrast, primitives with opposite dissimilar trends (c.g., indices and 3 in (a), and indices O and in (c)) exhibit low similarity.
5.2 Motion Primitive Frequency
To further explain the rationale of the learned codebook, we analyzed both the overall frequency of each primitive and its distribution across activity classes (using the PAMAP2 dataset for illustration). Fig. presents the 32 most frequent VQ indices: each stacked bar depicts the relative share of 12 activity classes, and the black curve shows the absolute number of occurrences. Because any macroscopic activity
be decomposed into sequence of fine- fnn-grained motion primitives, the codehook naturally captures micro-movements that are reused across different activities. The resulting distribution is highly skewed: the most common primitives cover the vast majority of windows, whereas the remaining indices form a long-tailed set that represents rare or transitional motions. Notably, locomotion classes (e.g., walking, running, and cycling) dominate the high-frequency primitives (e.g., primitive #170 and primitive #567), while low-variance postures such as lying, sitting, and standing are scattered among the less frequent primitives (e.g., primitive #72 and primitive #751). This frequency analysis could allow us to identify which primitive motions are more common and how they relate high-level activities, which is a step toward explaining model decisions.
Conclusion and Future Work
In this work, we introduced MoPFormer, a motion-prinet siriiio-primiiive--bee Transformer framework for wearable-sensor activity recognition. Our approach tackles two key HAR challenges: interpretability and cross-domain generalization. Through comprehensive experiments, we showed that MoPFormer achieves state-of-the-ar classification performance six diverse datasets, outperforming both super- vised and self-supervised baselines. We also demonstrated that MoPFormer's learned primitives are semantically meaningful. This interpretability lets peek inside the "black box" of the HAR model. Overall, MoPFormer combines the strengths of sequence modeling and symbolic representation learning, yielding HAR system that is both accurate and explainable.
Our future work will focus on two key directions. First, developing more robust foundational models by pre-training larger, more diverse sensor datasets to extract richer motion primitives. Second, aligning our learned motion primitive embeddings with large language models as novel input modality enables more flexible activity recognition and potentially expresses transfer ability.

---

Acknowledgments
This work was supported by National Key Research and Development Program of China (No. 2022YFA1004102) and NSFC grant (No. 62136005 and 62202455).
References
[1] K. Altu, B. Barshan, and O. Tunçel. "Comparative study on classifying human activities with miniature inertial and magnetic sensors". In: Pattern Recognit. 43 (2010), pp. 3605-3620. URL:
https://app
Org/CorpusID. 18488847
[2] D. Anguita et al. "A Public Domain Dataset for Human Activity Recognition using Smart- phones". In: The European Symposium on Artificial Neural Networks. 2013. URL.: https
/aapi.semanti cscholar org/ CorpusID: 6975432
[3] S. Arabzadeh, F. Almasganj, and M. M. Ahmadi. "CNN Autoencoders for Hierarchical Feature Extraction and Fusion in Multi-sensor Human Activity Recognition". In: arXiv preprint TrXiv:2502.04455 (2025).
[4] M. H. Arshad, M. Bilal, and Gani. "Human activity recognition: Review, taxonomy and open challenges". In: Sensors 22.17 (2022), p. 6463.
[5] O. Banos, R. Garcia, and A. Saez. MHEALTH. UCI Machine Learning Repository. DOI: 2014.
[6] O. Banos et al. "Design, implementation and validation of a novel open framework for agile development of mobile health applications". In: Biomedical engineering online 14 (2015),
Pp. 1-20.
[7]
B. Barshan and K. Altun. Daily and Sports Activities. UCI Machine Learning Repository. DOI: 2010.
http://oi.org11..4443322 [8] C. Bergmeir. Fundamental limitations of foundational forecasting models: The need for multi- modality and rigorous evaluation. Talk at the Conference on Neural Information Processing
Systems (NeurIPS 2024). Accessed: 2025-05-07. Dec. 2024. URL.: https: c gmeir
[9] O. Celik, A. Taranovic, and G. Neumann. "Acquiring diverse skills using curriculum reinforee- ment learning with mixture of experts". In: Proceedings of the 41st International Conference on Machine Learning. ICML'22. Vienna, Austria: JMI.R.org. 2024.
[10] B. Chen et al. "Comodo: Cross-modal video-to-imu distillation for efficient egocentric human activity recognition". In: arXiv preprint arXiv:2503.0722 07959 (2025).
[11] Chen et "Deep learning for sensor-based human activity recognition: Overview, chal- lenges, and opportunities". In: ACM Computing Surveys (CSUR) 54,4 (2021). pp. 1-40.
[12] W. Cheng et al. "Predicting Complex Activities from Ongoing Multivariate Time Series". In: Proceedings of the Twenty-Seventl International Joint Conference on Artificial Intelligence, IJCAI-18. International Joint Conferences Artificial Intelligence Organization, July 2015 8, pp. 3022-33222222 DO1: 10. 24222/ 2018/ /
URL: atttss://doi org/10. 229933. 24963/
ijcai. 2018/4611
[13] R. R. Chowdhury et "Tarnet: Task-aware reconstruction for time-series transformer". In: Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data Mining.
2022. pp. 212-220.
[14] A. Doherty et al. "Large scale population assessment physical activity using wrist worn accelerometers: the UK biobank study". In: PloS one 12.2 (2017), e0169649.
[15] E. Eldele et al. "Contrastive Domain Adaptation for Time-Series Via Temporal Mixup". In: IEEE Transactions on Artificial Intelligence 5.3 (2024), pp. 1185-1194. DOI: 10. 1109/TAI
2023. 3293473
[16] E. Eldele al. "Time-series representation learning via temporal and contextual contrasting".
In: arXiv preprint arXiv:2 106. (2021).
[17] E. Eldele al. "TSLANet: rethinking transformers for time series representation learning". In:
Proceedings of the 4/ International Conference on Machine Learning. ICML'24. Vienna,
Austria: JMLR.org, 2024.
[18] M. Al-Faris et al. "A review on computer vision-basse methods for human action recognition".
In: Journal of imaging 6.6 (2020), p. 46.
10

---

[19] D. Geissler, B. Zhou, and P. Lukowicz. "Strategies and Challenges of Efficient White-Boo Box
Training for Human Activity Recognition". In: arXiv preprint arXiv:2412.08507 (2024).
[20] A. Grattafiori et al. "The Ilama herd of models". In: arXiv preprint arXiv:2407.21783 (2024).
[21] J.-B. Grill et al. "Bootstrap your own latent-a new approach to self-supervised learning". In:
Advances in neural information processing systems 33 (2020), pp. 21271-21284,
[22] C. Guo et al. "Modality Consistency- Activity Recogmition". In: IEEE Internet of Things Journal 11. 1.12 (2024), pp. 21750-21762
Contrastive Learning for Wearable-Ban Based Human
D01:110. 100.01109/JO0 2024. 3379019
[23] Z. Hussain, M. Sheng, and W. E. Zhang. "Different approaches for human activity recognition: (2019).
A survey". In: arXiv preprint 1906.
[24] J. V. Jeyakumar et al. "SenseHAR: a robust virtual activity sensor for smartphones and wearables". In: Proceedings of the 17th Conference on Embedded Networked Sensor Systems. SenSys '19, New York, New York: Association for Computing Machinery, 2019, pp. 15-28. ISBN; 9781450369503. 1145/3356250. 3360032
DO1: 10. 1145/3356250 3360032
URL: https://doi: org/10
[25] A. Kamboj and M. Do. "A Survey of IMU Based
Transfer Learning in Human Activity Recognition". In: arXiv preprini arXiv:2403. / 15444 (2024).
[26] B. Li et al. "Two-stream convolution augmented transformer for human activity recognition".
In: Proceedings of the AAAI conference on artificial intelligence. Vol. 35. 1. 2021, pp. 286-293.
[27] Z. C. Lipton. "The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery." In: Queue 16.3 (2018), pp. 31-57.
[28] S. Liu et al. "Focal: Contrastive learning for multimodal time-series sensing signals in fac- torized orthogonal latent space". In: Advances in Neural Information Processing Systems 36 (2023), pp. 47309-47338.
[29] H. Ma et al. "AttnSense: Multi-level attention mechanism for multimodal human activity recognition." " In: I/CAI. 2019, pp. 310--3115.
[30] F. J. Ordóñez and D. Roggen. "Deep convolutional and Istm recurrent neural networks for multimodal wearable activity recognition". In: Sensory 16.1 (2016), p. 115.
[31] A. Padmanabha et al. "FgoCHARM: Resource-Efficient Hierarchical Activity Recognition using an Egocentric IMU Sensor". In: arXiv preprint arXiv:2504. /7735 (2025).
[32] G. Panahandeh et al. "Continuous hidden Markov model for pedestrian activity classification and gait analysis". In: IEEE Transactions on Instrumentation and Measurement 62.5 (2013), pp. 1073-1083.
[33] J. Park, D.-W. Kim, and J. Lee. "CALANet: Cheap All-Layer Aggregation for Human Activity Recognition".. In: The Thirty-eighth Annual Conference on Neural Information Processing Systems. 2024. URL: https: /ttt:ttt/////perrree //tp:://///eerver//
[34] J. Park, D.-W. Kim, and J. Lee. "CALANet: Cheap All-Layer Aggregation for Hu- man Activity Recognition". In: Advances in Neural Information Processing Systems. Ed. by A. Globerson al. Vol. 37. Curran Associates, Inc., 2024, pp. 69419-69444 URL: https / proceedings neurips cc / paper files paper / 2024 / file / http:/84000000000000002-----0-0- Paper- Conf erence. pdf [35] H. Qian, T. Tian, and C. Miao. "What Makes Good Contrastive Learning on Small-Scale Wearable-based Tasks?" In: Proceedings of the 28th ACM SIGKDI) Conference on Knowledge
Discovery and Data Mining. KDD '22. Washington DC, USA: Association for Computing
Machinery, 2022, pp. 3761-3771. ISBN: 9781450393850, DO1: (0. 1ww11114/33467..000 35391334
URL: https: httpa:://doiooo// .rgg / 11w.1115//336788..8 3555334448883331144444
[36] Radford et al. "Learning transferable visual models from natural language supervision". In:
International conference on machine learning. PmLR. 2021, pp. 8748-87866
[37] V. Radu al. "Multimodal Deep Learning for Activity and Context Recognition". In: Proc.
ACM Interact. Mob. Wearable Ubiquitous Technol. 1.4 (Jan. 2018). DO1: [0. 114./31661117
URL: https: //doi .rr...rg/111 10. 1145/3161174
[38] A. Reiss. PAMAP2 Physical Activity Monitoring. UCI Machine Learning Repository. DOI:
https://doi.org/1.. htg//00.2442//55NW2 2012.
[39] A. Reiss and D. Stricker. "Introducing New Benchmarked Dataset for Activity Monitoring".
In: 2012 16th International Symposium on Wearable Computers (2012), pp. 108-109. URI.:
https: api. www.nticschooar..c.m.mo http//CopusI::10333300
11

---

[40] J. L. Reyes- Reyes-Ortize et al. Human Activity Recognition Using Smartphones. UCI Machine Learn- ing Repository. DOI: http://....rrr00022233333/4544 2013.
[41] M. Sanzari, V. Ntouskos, and F. Pirri. "Discovery and recognition of motion primitives in human activities". In: PloS one 14.4 (2019), e0214499.
[42] N. sedaghati, S. ardebili, and A. Ghaffari. "Application of human activity/action recognition:
review". In: Multimedia Tools and Applications (2025), pp. 1-30.
[43] S. Serrano and N. A. Smith. "Is attention interpretable?" In: arXiv preprint arXiv:1906.03331
(2019).
[44] Y.-H. Shen, K.-X. He, and W.-Q. Zhang. "SAM-GCNN: a gated convolutional neural network with segment-level attention mechanism for home activity monitoring". In: 2018 JEEE inter- national symposium on signal processing and information technology (ISSPIT). IEEE. 2018, pp. 679-684.
[45] D. Spathis al. "Selt-supervised transfer learning of physiological representations from free- living wearable data". In: Proceedings of the Conference on Health, Inference, and Learning. 2021, pp. 69-78.
[46] T. Sztyler and H. Stuckenschmidt. "On-body localization of wearable devices: An investigation of position-aware aware activity recognition". " In: 2016 IEEE International Conference Pervasive Computing and Communications
Ceeeiittttiinneeeeaae (PerCom). 2016, pp. 1-9. DOT: 10. N0.1109/PERCOK 2016
7456521
[47]
Van Den Oord, O. Vinyals, et "Neural discrete representation learning". In: Advances in neural information processing systems 30 (2017).
[48] A. Vaswani et al. "Attention is all you need". In: Advances in neural information processing systems 30 (2017).
[49] H. Wu et al. "Timesnet: Temporal 2d-variation modeling for general time series analysis". In: arXiv preprint 10.iv:2210.02/11 186 (2022).
[50] Xu et al. "A shallow convolution network based contextual attention for human activity recognition". In: International conference on mobile and ubiquitous systems: Computing, networking, and services. Springer. 2022, pp. 155-171.
[51] C. Xu et al. "An Enhanced Human Activity Recognition Algorithm with Positional Attention". In: Proceedings of The 14th Asian Conference Machine Learning. Ed. by E. Khan and M. Gonen. Vol. 189. Proceedings of Machine Learning Research. PMLR, Dec. 2023, pp. 1181
1 119.. URL.: pattps://proceeeeee
.mlr...rr... pressppess//eee /ttes/ss//v//////e// /tt///22..... html
[52] Z. Xu, A. Zeng, and Q. Xu. "FITS: Modeling time series with 10k parameters". In: arXiv preprint arXiv:2.307.08. 03756 (2023).
[53] Z. Yue al. "Ts2vec: Towards universal representation of time series". In: Proceedings of the AAAI conference on artificial intelligence. Vol. 36. 8. 2022, pp. 8980-8987.
[54] M. Zeng et al. "Understanding and improving recurrent networks for human activity recognition
by continuous attention". In: Proceedings of the 2018 ACM international symposium on wearable computers. 2018, pp. 56-63.
[55] G. Zerveas et al. "A transformer- based framework for multivariate time series representation learning". Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining. 2021, pp. 2114-2124.
[56] M. Zhang and A. A. Sawchuk. "USC-HAD: A Daily Activity Dataset for Ubiquitous Activity Recognition Using Wearable Sensors". In: ACM International Conference on Ubiquitous Computing (Ubicomp) Workshop on Situation, Activity and Goal Awareness (SAGAware). Pittsburgh, Pennsylvania, USA, Sept. 2012.
12

---

A Experimental Settings
In this section, we provide detailed parameter settings for our proposed MoPFormer model and baseline models used in our experiments. All experiments can run normally on the RTX 8000 with 40GB VRAM (requiring the use of gradient accumulation techniques).
A.1 Parameter Settings for MoPFormer
We implement the MoPFormer model with the following configuration. For motion representation, we use 50 as the motion primitive length, which corresponds to 0.5s of motion when resampled to 100Hz. The codebook size is set to 1024 with an embedding dimension of 256. The architecture consists of standard Transformer Encoder layers with 8 attention heads per layer and an MLP ratio of 1. We use GELU as the activation function throughout the network. Notably, we do not employ any dropout in our model. During training, we apply a masking ratio of 0.25 for the masked modeling objective. For optimization, we employ the AdamW optimizer with a learning rate of le-4 and a weight decay coefficient of le-5. We use batch size of 512, implemented with gradient accumulation using Py PyTorch Lightning to optimize memory usage.
A.2 Parameter Settings for Baseline Models
BYOL For BYOL 21] (Bootstrap Your Own Latent), we implement the model following the CL-HAR (35 library's configuration. The architecture consists of two networks: online network architecture as the backbone encoder. The online network includes a projection head that maps the backbone's 's output to 128-dimensional embedding space with a hidden layer of the same dimension, structured as a sequence of Linear, BatchNorm1d, ReLU, Linear, and BatchNormld layers. The prediction head follows with the same dimensionality (128) and is structured as Linear, BatchNorm ReLU, and Linear layers. The target network's parameters are updated via an exponential moving average of the online network's parameters with decay rate of 0.996. We employ three different data augmentation techniques, all following the settings provided in (35). The model is trained using negative cosine similarity as the loss function with the Adam optimizer. We set the learning rate to le-4 for the online encoder and le-3 for the online predictor, with weight decay of 1.5e-6, using a Cosine AnnealingLLR scheduler. For downstream activity recogmition, we freeze the backbone encoder and train linear classifier with the Adam optimizer a learning rate of le-3 for the same number of epochs.
and target network, both utilizing the DCL (DeepConvLSTI)
ModCL For ModCL, we follow the default parameter settings described in the original paper. Since the official code was not publicly available, we implemented the model based on the paper's specifications. During our reproduction, we enabled several data augmentation techniques, specifically applying five methods: Jittering, Scaling, Permutation, Masking, and Time Warping, all using the same proportions and parameters as specified in the original work.
TSLANet For TSLANet, we implement the model using the default parameter settings specified in the original paper. For parameters not explicitly mentioned in the paper, we adopt the default configurations from the authors' publicly released code repository. Following the original architecture, enable both the ICB (Input Channel Block) and ASB (Attention Sub-Bloch modules. The model is trained with the same optimization strategy and hyperparameters as described in the original TSLANet paper.
CALANet For CALANet, we follow the default parameter settings described in the original paper. For configuration details not explicitly stated in the paper, authors' official code implementation. We set the number of layers in the layer aggregation pool to 9 based the authors' ablation study results, which identified this the optimal configuration. All other hyperparameters and training procedures follow the specifications in the original paper.
the default settings provided in the
B Supplementary Ablation Studies
Since our work primarily focuses on investigating interpretable motion primitives in human activity recognition (HAR), the main paper demonstrates the necessity of cach architectural component
13

---

through module-wiss ablation experiments. This appendix provides additional ablation studies examining the impact of key hyperparameters on model performance provide comprehensive insights into our method's design choices.
B.1 Motion Primitive Window Length
The choice of motion primitive window length significantly affects the model's ability capture meaningful motion patterns. We evaluate different window lengths to determine the optimal temporal granularity for motion primitive extraction.
Table 3: Ablation study on motion primitive window length
PAMAP2
DSADS
Window Length
Acc
F1
Acc
Fl
25 50 (ours)
85.71
82.11
97.09
96.37
86.08
84.83
97.60
97.38
100
80.81
79.85
92.94
92.17
As shown in Table 3 window length of achieves optimal performance across both datasets. While shorter windows 25 demonstrate comparable performance, we select the length of 50 to ensure sufficient temporal context for comprehensive motion primitive analysis. This choice prioritizes cap- turing complete motion phases within each primitive, which is crucial for meaningful interpretability analysis. Longer windows 100 lead to substantial performance degradation due to the inclusion of multiple distinct motion phases within single primitive, confirming that our selected window length provides the appropriate temporal granularity without compromising motion primitive coherence.
B.2 Transformer Encoder Depth
We investigate the impact of transformer encoder depth on the model's representational capacity and its effect on motion primitive quality.
Table 4: Ablation study on transformer encoder layer number
PAMAP2
DSADS
Depth
Acc
F1
Acc
F1
2
85.01
84.59
96.92
96.56
5 (ours)
86.08
84.83
97.60
97.38
7
86.11
84.98
97.53
97.31
86.05
84.79
97.28
97.02
The results in Table 4 show that performance differences between 5, 7, and 10 layers minimal. However, we adopt layers as our configuration to ensure adequate modeling capacity for motion primitive extraction without introducing unnecessary complexity. This choice reflects preference for slightly conservative parameter settings that guarantee sufficient represttttonnn power for motion primitive analysis, while avoiding potential overfitting. The 2-layer configuration shows reduced performance, confirming that our selected depth provides the necessary capacity for high-quality motion primitive learning.
B.3 Hidden Embedding Dimensionality
The dimensionality of hidden embeddings critically determines the model's representational capacity for motion primitive encoding and subsequent interpretability analysis.
From Table we observe that performance reaches saturation around 128 dimensions, with 256 dimensions providing marginal improvements. We choose 256 dimensions to ensure comprehensive representational capacity that does not limit our motion primitive analysis. This decision follows principle of using slightly excessive parameters to guarantee that motion primitive quality and
14

---

Table 5: Ablation study on hidden embedding size
PAMAP2
DSADS
Embedding Size
Acc
F1
Acc
F1
32
82.18
81.70
94.81
93.88
64
85.90
85.86
96.93
96.31
128
86.45
85.50
97.28
97.35
256 (ours)
86.08
84.83
97.60
97.38
interpretability are not compromised by insufficient representational capacity. Lower dimensions (32- 64) show significant performance degradation, validating that our selected dimensionality provides adequate space for capturing the complexity and nuances of human motion patterns.
B.4 Vector Quantization Codebook Size
The VQ codebook size determines the discrete vocabulary available for motion primitive representa- tion, directly impacting both model performance and the richness of discoverable motion primitives.
Table 6: Ablation study on VQ codebook size
PAMAP2
DSADS
Codebook Size
Acc
F1
Acc
F1
16
78.81
76.89
90.05
88.93
64
84.31
83.79
95.55
95.43
256
85.62
84.87
97.52
97.18
1024 (ours)
86.08
84.83
97.60
97.38
As demonstrated in Table 6 performance steadily improves with larger codebook sizes and appears saturate around 256 codes for the current pretraining datasets. However, we adopt 1024 codes to ensure comprehensive coverage of motion primitive diversity, particularly to avoid any limitations that might affect our detailed motion primitive analysis. This choice exemplifies our approach of using generously-sized parameters to guarantee that the learned motion primitives fully capture the richness and subtlety human motion patterns without being constrained by codebook capacity. Smaller codebooks (16-64) exhibit substantial performance degradation, confirming the necessity of adequate representational diversity for meaningful motion primitive discovery.
B.5 Model Stability across Random Seeds
To validate the stability of MoPFormer's performance, we conducted additional experiments by fine-tuning our pre-trained model three times with different random seeds on the PAMAP2, DSADS, and RealWorld datasets. For each run, we followed the exact same fine-tuning protocol as in the main experiments (i.e., freezing the pre-trained encoder and training a linear classifier). Tablee 7 reports the mean and standard deviation of accuracy and F1-score across the three independent runs.
Table Comparison of original reported results with new results from multiple runs (Mean+Std).
Original Reported
New Results (MeantStd)
Dataset
Acc (%)
Fl (%)
Acc (%)
(%)
PAMAP2
86.08
84.83
87.49+4.10
85.74+3.04
DSADS
97.60
97.38
97.27+0.49
97.11+0.24
RealWorld
92.05
93.25
91.32+1.07
92.65+1.23
15

---

C
Additional Motion Primitive Analysis
To provide deeper insights into the interpretability of learned motion primitives, we analyze the distribution of motion primitive usage across different human activities. This analysis demonstrates how distinct activities exhibit characteristic motion primitive patterns, validating the semantic mean- ingfulness of our learned representations.Fiuur Figures 5 through [2] present the motion primitive usage distributions for eight representative activities from our dataset.
044
0.
302
322
714
350
1023
Figure 5: Motion primitive usage distribution for ascending stairs activity.
Figure 6: Motion primitive usage distribution for cycling activity.
102
Figure 7: Motion primitive usage distribution for walking activity.
16

---

Figure Motion primitive usage distribution for running activity.
Figure 9: Motion primitive usage distribution for jumping activity.
3.2
3.5
1.2
0.5
012
Figure 10: Motion primitive usage distribution for lying activity.
1.50
0.75
0:03
Figure Motion primitive usage distribution for sitting activity.
17

---

1.3
1.2
52
302
250
204
225
300
357
400
459
530
561
332
663
714
705
00.6
357
010
900
1023
Figure 12: Motion primitive usage distribution for standing activity.
18

---

Through careful observation of these distributions, we can identify distinct patterns that reflect the fundamental differences motion primitive composition across various activities. The distributions clearly separate into IWo categories: dynamic activities (Figures 5 through ( and static activities (Figures TO through 12). This contrast particularly striking in terms of primitive usage patterns. Dynamic activities demonstrate concentrated utilization of specific motion primitives, with certain primitives exhibiting significantly higher activation frequencies, indicating the presence of character- istic movement patterns essential for these activities. In contrast, static activities show more uniform distribution across motion primitives, suggesting these behaviors rely on broader range subtle motion components for postural maintenance and minor adjustments. This fundamental difference in primitive usage validates that our learned motion primitives capture meaningful biomechanical distinctions between different categories of human activities.
Furthermore, to analyze the temporal relationships between motion primitives, we construct and visualize the Markov transition probability matrix for all learned motion primitives, as illustrated in Figure
Markov Transition Probability Matrix (Top K Dynamic)
408
170
311
288
401
1005
Other
408
170
0.06
0:03
93
0.30
0.25
0.18
0.05
0.07
0.04
0.04
0.03
0.01
0.11
0.04
0.5
372
0.14
034
0.14
0.30
0.07
0.08
0.07
0.06
0.02
0.02
0.17
641
0.22
0.0
0.15
0.08
0.07
0:06
0.06
0.04
0.01
0.01
0.09
0.30
0.08
0.08
288
0.2
401
1005
0.06
0:07
Other
0.02
0.03
0.03
0.05
0.03
0.05
0.03
0.05
0.03
0.02
0.65
State
Figure 13: Markov transition probability matrix showing the temporal dependencies between motion primitives.
D Dataset Description
All datasets used in this paper adopt sliding window approach with window size 500 samples and step size of 500 samples the basic recognition unit. All sensor data is resampled to 100 Hz to ensure consistency across different datasets. Data segments shorter than the specified window size discarded from the analysis. For fine-tuning and evaluation purposes, allocate 20% of each
19

---

dataset for fine-tuning while the remaining data is used for testing. The datasets employed in this study are described follows:
PAMAP2 The PAMAP2 Physical Activity Monitoring dataset is comprehensive collection designed for activity recognition and intensity estimation research, containing recordings of 18 different physical activities such as walking, cycling, and playing soccer performed by 9 subjects. Each subject wore three Colibri wireless inertial measurement units (IMUs) positioned on the wrist of the dominant arm, chest, and ankle of the dominant side, sampling at 100 Hz, along with heart rate monitor operating at approximately Hz. The data collection protocol required each subject to perform 12 standardized activities, with some subjects also completing additional optional activities.
DSADS
This dataset captures activity recognition data from 19 different physical activities per- formed by 8 subjects (4 female, 4 male, aged 20-30). Activities include basic postures, locomotion, daily activities, and exercises, each performed for 5 minutes and segmented into 5-second intervals. Five sensor units were placed on different body locations (torso, arms, legs), with each unit containing accelerometer, gyroscope. and magnetometer sensors. Data was recorded at 25 Hz and organized hierarchically by activity, subject, and segment.
This dataset contains human activity recognition data from 12 physical activities (sta- tionary postures, locomotion, and exercises) performed by 10 volunteers in natural settings. Three wearable sensors placed on the chest, right wrist, and left ankle measure motion parameters (accelera- tion, rotation, magnetic orientation), with the chest sensor also capturing ECG data. All recordings were sampled at 50 Hz, providing comprehensive movement and physiological data for activity recognition research.
MHealth
Realworld This dataset captures human activity recognition data from 8 different physical activities (walking, running, sitting, standing, lying, stairs up. stairs down, jumping) performed by 15 subjects (8 males, females; age 11,4442.44 height 173.1=6.9 cm, weight 14.1 kg). Each activity was performed for approximately 10 minutes per subject (except jumping 1..7 minutes due to physical exertion), with data equally distributed between genders. The dataset includes IMU sensor readings (acceleration, gyroscope, magnetic field) collected at 50 Hz sampling rate simultancously from different body positions (chest, forearm, head, shin, thigh, upper arm, and waist).
UCI-HAR This dataset captures human activity recognition data from 6 different physical activities (walking, walking upstairs, walking downstairs, sitting, standing, lying) performed by 30 volunteers aged 19-48 years. The experiments were video-recorded for manual data labeling. The sensor signals were pre-processed with noise filters, while Butterworth low-pass filter with 0.3 Hz cutoff frequency was applied to separate body acceleration from gravity, with features extracted from both time and frequency domains for activity recognition analysis.
This dataset captures human motion data using MotionNode sensors operating at 100Hz (+6g accelerometer range, =500dps gyroscope range). It includes 12 different physical activities: Walking Forward/Leett/ightt Walking Upstains/ownsstirss Running Forward, Jumping Up, Sitting, Standing, Sleeping, and Flevator Up/Down.
USC-HAD
E
Limitations
Our approach introduces computational overhead through Vector Quantization and multi-layer Transformer processing, which could pose problem for real-time inference on lestttee-limited wearable devices.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the three main contributions: (1) the MoPFormer pre- training framework, (2) representation method that enables unified training across heterogeneous datasets, and (3) extraction of motion primitives with detailed interpretability analysis. These claims substantiated throughout the paper, particularly in the methodology, experiments, and analysis.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
21

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The paper provides comprehensive details on datasets (Section 4.1), prepro- cessing steps (resampling to 100 Hz, segmentation with 500-sample windows), architecture specifics (segment size, embedding dimensions, masking ratio), and training protocol. Addi- tionally, the methodology section fully describes all model components, loss functions, and training objectives. Additionally, we will provide further details in the appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
22

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Code will be provided soon.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.cc.
wwt.ssicynnooooooioooooooooooili for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wttps/cccppppgguuuaaaeaeae httpc//gguddeeeooeouuuuummibeae ottpee///oooeeeubmiiiiiimmmee assionoolcyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The paper details all necessary details about the training and testing processes.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The paper does not report error bars or other statistical significance measures.
Guidelines:
The answer NA means that the paper does not include experiments.
23

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errоrs is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't ' make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research focuses on analyzing public, anonymized sensor datasets for activity recognition, which presents minimal ethical concerns. The methodology and evaluations follow standard practices machine learning research without raising issues related to privacy, harm, or misrepresentation.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]

---

As HAR is a fundamental technique in healthcare, fitness tracking, and surveillance, our accurate and real-time model can reduce the negative societal impacts due to false predictions.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification:
We will include the usage guidelines or restrictions in code release.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cited the original paper that produced the datasets and our source code release follows CC BY 4.0 license.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
25

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper and the code release include the details about training, license, usage guidelines, etc.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve any research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve any research with human subjects.
Guidelines:
26

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification:
We only used an LLM for language polishing of the paper.
Guidelines:
In the Metadata Embedding Adapter module, we used Google's text-embeddiing-00
API to convert the original sensor information into embeddings.
Please refer to our LLM policy http:::///eurppss huttpp///eerrrppsccc///ofeeeeeee /atps///c///o.eeeeeeeee/2////// for what should or should not be described.
27


Reviews:
[{"review_id": "qyNwJTSCEG", "reviewer": "Reviewer_wYXz", "summary": "MoPFormer introduces a two-stage architecture: (1) Motion Primitive Module: Tokenizes raw IMU signals into discrete codewords via VQ-based quantization; and (2) Transformer Backbone with Context-Aware Embedding: Learns temporal representations using metadata-aware embeddings and a masked autoencoding objective. The model is evaluated on six public HAR benchmarks and shows SOTA performance with compelling ablation studies and qualitative analyses.", "questions": "Please refer to the Weakness part.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n- The paper clearly identifies key HAR challenges (interpretability, heterogeneity) and aligns contributions tightly to them.\n\n- Use of motion primitives learned via VQ in self-supervised HAR is a compelling idea, especially for generalization and interpretability.\n\n- The context-aware embedding module that integrates statistical features and sensor metadata is thoughtfully designed.\n\n- Consistent improvements over both contrastive and supervised baselines across six diverse datasets.\n\n- Ablations and visualizations demonstrate the importance and effect of each module.\n\n### Weaknesses\n\n- No error bars, standard deviations, or confidence intervals are reported, weakening the empirical rigor—especially important given the small dataset sizes and minor margins in accuracy.\n\n- While contrastive and transformer baselines are compared, the paper omits comparisons with interpretable models like decision trees or attention-based attribution techniques specifically adapted for HAR (e.g., segment attention). Are motion primitives strictly better?\n\n- Despite the motion primitive framework, the interpretability remains primarily quantitative (e.g., frequency and similarity). There’s no human evaluation of whether the discovered primitives correspond to semantically meaningful sub-activities.", "was_revised": true, "final_justification": "It is indeed a borderline work to me. I'm happy to see this work in NeurIPS but I'm also okay if we don't have room for it this time. The authors can incorporate all the comments and feedback for substantial revision to make the work more solid.", "rebuttal": "Thank you for your constructive feedback. Below, we provide point-by-point responses to your comments. Please don’t hesitate to let us know if any concerns remain.\n### [W1] No error bars, standard deviations, or confidence intervals are reported, weakening the empirical rigor—especially important given the small dataset sizes and minor margins in accuracy.\n\n> We thank the reviewer for highlighting this concern regarding empirical rigor. To address it, we fine-tuned our pre-trained model three additional times using different random seeds on the PAMAP2, DSADS, and RealWorld tasks. We then computed the mean and standard deviation for both accuracy and F1-score across these independent runs. The results, summarized below, demonstrate the stability of our model’s performance. And we will update these in the revised version.\n> \n>   |     | |   PAMAP2   |   DSADS    | Realworld  |\n>   |:---:|:---:|:---:|:---:|:---:|\n>   |   Original Reported |  Acc  | 86.08 |   97.60 | 92.05 |\n>   |    | F1 | 84.83 | 97.38 | 93.25 |\n>   |   New Results |  Acc | 87.49±4.10 | 97.27±0.49 | 91.32±1.07 |\n>   |    | F1 | 85.74±3.04 | 97.11±0.24 | 92.65±1.23 |\n\n### [W2] While contrastive and transformer baselines are compared, the paper omits comparisons with interpretable models like decision trees or attention-based attribution techniques specifically adapted for HAR (e.g., segment attention). Are motion primitives strictly better?\n\n> - Thank you for this valuable suggestion. To address the concern, we included comparisons with interpretable models, e.g., **Decision Trees** and **LightGBM** [r1]. For these baselines, we  followed standard practice and extracted 13 commonly used statistical features from each sensor channel: mean, standard deviation, minimum, maximum, median, first quartile, third quartile, variance, signal energy, zero crossing rate, peak-to-peak value, skewness, and kurtosis. The results are summarized in the table below. As shown, our proposed model (MoPFormer) significantly outperforms both interpretable baselines across all three datasets in terms of both accuracy and F1-score.\n> \n>   |            |         |   PAMAP2   |   DSADS    | Realworld  |\n>   |:---:|:---:|:---:|:---:|:---:|\n>   | Decision Trees | Acc |   65.16    |   81.91    |   77.70    |\n>   | |F1  |   61.29    |   83.68    |   73.21    |\n>   |    LightGBM |Acc    |   77.11    |   87.52    |   85.64    |\n>   |  |F1     |   76.74    |   86.23    |   83.11    |\n>   |      MoPFormer|Acc      |   **87.49**    |   **97.27**    |   **91.32**    |\n>   |      |F1       |   **85.74**   |   **97.11**    |   **92.65**    |\n>   \n> - While decision tree-based models provide a degree of post-hoc feature-based interpretability, they do not aim to uncover or model the compositional structure of human activities. In contrast, MoPFormer introduces an interpretable intermediate representation by explicitly decomposing activities into motion primitives. We believe that this inductive bias not only enhances the performance but also enables a more meaningful and structured path toward explainable human activity recognition.\n> \n>  [r1] LightGBM: A Highly Efficient Gradient Boosting Decision Tree. In NIPS 2017.\n\n\n\n### [W3] Despite the motion primitive framework, the interpretability remains primarily quantitative (e.g., frequency and similarity). There’s no human evaluation of whether the discovered primitives correspond to semantically meaningful sub-activities.\n\n> - We appreciate the reviewer’s insight and agree that evaluating the semantic meaning of the learned primitives is essential. While our current analysis is quantitative, we observed that certain motion primitives consistently correspond to specific patterns, such as increasing acceleration or rotation, suggesting potential alignment with meaningful sub-actions.\n> - Due to rebuttal constraints, we cannot include new visualizations, but our findings indicate that the VQ module captures semantically coherent patterns. As noted in the manuscript, our long-term goal is to map these primitives to a human-interpretable semantic space, enabling hierarchical decomposition (e.g., \"washing hands\" → “turning on tap”, “applying soap”) and compositional generalization to unseen activities. This also creates opportunities to incorporate LLM priors for semantic analysis and behavioral validation. While the lack of fine-grained sub-activity annotations remains a common challenge, MoPFormer offers a solid foundation for future progress in this direction."}, {"review_id": "Qph2JN6Z95", "reviewer": "Reviewer_MMnJ", "summary": "This paper tackles interpretability and generalization issues in Human Activity Recognition (HAR) using wearable sensors by introducing MoPFormer (Motion-Primitive Transformer). MoPFormer tokenizes inertial measurement unit (IMU) signals into \"motion primitive\" codewords, enhancing interpretability and utilizing a Transformer to learn temporal patterns. It operates in two stages: segmenting and quantizing sensor data, followed by enriching and processing with a Transformer encoder. Pre-trained with a masked motion-modeling objective, MoPFormer outperforms state-of-the-art methods across six HAR benchmarks (PAMAP2, DSADS, MHealth, Realworld, UCI-HAR, and USC-HAD) and generalizes well to new datasets by capturing consistent movement patterns.", "questions": "* While it does outperform all other models, I'm noticing that MoPFormer requires pertaining to achieve these results, whereas the TSLANet achieves better performance without pre-training. Are there any learnings here?\n* Tokenization: How do segment length and quantization affect performance? Is there a version of this where you could remove tokenization and run the model just on classification to understand the impact of VQ? Other ablation studies would also be valuable. \n* What is the impact of noise or perturbations on the sensor inputs on tokenization? Does minor perturbation every significantly change the token distribution?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n* The authors develop an approach to HAR using new VQ+LLM style modeling approaches. They pre-train using MAE and have a classifier head for each dataset. Overall it's a nice time-series variant of this paradigm. \n* They show some ways intermediates of the models can be interpreted. \n* Excels in cross-dataset performance and outperforms current methods on multiple benchmarks.\n\nWeaknesses\n* My understanding is that there are other Time-series foundation models in this direction. It would be nice to see more comparisons to these. \n* Tokenization: Needs more justification on how primitives capture patterns. Are these really \"motion primitives\" (i.e., can you interpret them in isolation?)\n* Missing Ablations: Lack of studies on component contributions weakens empirical claims.\n* I appreciate seeing someone release new work using VQ+LLM and adapting it to time-series problems. It seems a little too applied in a way? This isn't necessarily problematic, but the ideas aren't especially novel.", "was_revised": true, "final_justification": "They were able to mediate most of my concerns with the rebuttal so I am increasing my score.", "rebuttal": "We sincerely thank you for the valuable comments. Below, please find our responses to each concern, and let us know if any issues remain.\n### [Q1] While it does outperform all other models, I'm noticing that MoPFormer requires pertaining to achieve these results, whereas the TSLANet achieves better performance without pre-training. Are there any learnings here?\n- Thank you for your observation. As noted, MoPFormer is designed not only to achieve strong performance but also to introduce interpretable intermediate representations through motion primitives. These primitives enable the model to decompose complex behaviors into sequences of simple and human-understandable sub-actions, which facilitates compositional generalization to unseen activities. **This design naturally calls for a pretraining stage to learn a reusable and expressive primitive vocabulary**.\n- Regarding the comparison with TSLANet [r1], we would like to respectfully clarify that **TSLANet also involves a pretraining phase**. As stated in Appendix D of their paper:\n  > \"To train the classification experiments, we optimized TSLANet using AdamW with a learning rate of 1e-3 and a weight decay of 1e-4, **applied during both training and pretraining phases.** The experiments ran for 50 epochs for **pretraining** and 100 epochs for **fine-tuning**.\"\n  \n  As noted in our Appendix A.2, we follow the same settings. Thus, both MoPFormer and TSLANet benefit from the pretraining.\n\n[r1] TSLANet: Rethinking Transformers for Time Series Representation Learning. In ICML, 2024.\n\n### [Q2] Tokenization: How do segment length and quantization affect performance? Is there a version of this where you could remove tokenization and run the model just on classification to understand the impact of VQ? Other ablation studies would also be valuable.\nThank you for this insightful suggestion. We address these points as follows:\n- Segment length is indeed a critical hyperparameter. In Appendix B.1, Table 3, we conducted an ablation study by varying the **motion primitive window length**, and found that a length of 50 (i.e., 0.5s, given 100Hz resampling) provided the best results. This value was used consistently in all the experiments. We appreciate you highlighting its importance and will move this analysis into the main text in the revised version.\n\n- Following your suggestion, we conducted an additional ablation by **removing the VQ module**, effectively degrading the model into a standard Transformer Encoder (labeled \"w/o VQ\" in the table below). The observed performance drop across two tasks confirms that the tokenization step plays a critical role in overall model effectiveness.\n    |            |         |   PAMAP2       | Realworld  |\n    |:---:|:---:|:---:|:---:|\n    |    w/o VQ  |  Acc | 84.71  |  91.02   |\n    |      | F1  | 82.42  |   90.17   |\n    |    MoPFormer |  Acc   | 86.08  |   92.05   |\n    |      | F1  | 84.83  |   93.25   |\n\n- Additionally, we include an ablation on the **VQ codebook size** in Appendix B.4, Table 6, showing that the size of the codebook directly impacts both model performance and the expressiveness of the discovered motion primitives.\n\n\n### [Q3] What is the impact of noise or perturbations on the sensor inputs on tokenization? Does minor perturbation every significantly change the token distribution?\n- To assess the robustness of our tokenization process, we conducted a perturbation experiment by **adding Gaussian noise to the input signals**, with variance set to 10% of the original test data variance. We then evaluated our pre-trained model under this setting (denoted by \"Ours + Noise\"). As shown in the table below, the performance degrades only slightly, indicating that MoPFormer is robust to moderate input perturbations and that the learned motion primitives are stable under noise.\n     |  | | PAMAP2 | DSADS | Realworld |\n     |:----:|:---:|:---:|:---:|:---:|\n     |     Ours | Acc      | 86.08  | 97.60 |   92.05   |\n     |  | F1      | 84.83  | 97.38 |   93.25   |\n     | Ours + Noise | Acc | 85.21  | 97.03 |   92.51   |\n     |  | F1  | 83.13  | 96.87 |   92.87   |\n \n- To further assess the impact of perturbations on the **token distribution**, we analyzed the VQ index sequences before and after noise addition. For a representative 'walking' sample from the PAMAP2 dataset (z-axis, wrist sensor), the VQ index sequence remained completely unchanged:\n    > **Original:** `170→772→538→170→777→538→170→772→538→339`\n    >\n    > **With Noise:** `170→772→538→170→772→538→170→772→538→339`\n\n  In contrast, a 'standing' sample, representing more stationary activity, showed slight variation in a few tokens:\n    > **Original:** `853→29→271→401→853→853→72→452→853→29`\n    >\n    > **With Noise:** `29→29→271→401→853→853→72→452→853→29`\n\n  To quantify this effect, we analyzed 500 randomly sampled windows across the dataset and found that only **2.6%** exhibited some change in their VQ token sequence after noise was added. This demonstrates that the overall token distribution remains highly stable under minor perturbations, providing a strong basis for future work on assigning human-understandable semantic labels to motion primitives.\n\n### [W1] My understanding is that there are other Time-series foundation models in this direction. It would be nice to see more comparisons to these.\nThank you for the suggestion. In addition to the comparisons already included in the paper (TSLANet and CALANet), we have added TimesNet [r2], a strong time-series foundation model, as an additional baseline. We used the authors' default hyperparameters and adopted a training strategy consistent with the supervised baselines (80% train, 20% test). These results demonstrate that MoPFormer remains competitive with TimesNet, even though our self-supervised methods were evaluated under a more challenging 20% train, 80% test split.\n |  |  | PAMAP2 | DSADS | Realworld |\n |:---:|:---:|:---:|:---:|:---:|\n |   TimesNet | Acc    | 86.88  | 97.17 |   92.51   |\n | | F1 | 85.31  | 96.05 |   92.16   |\n | MoPFormer | Acc | 86.08  | 97.60 |   92.05   |\n |  |F1 | 84.83  | 97.38 |   93.25   |\n \n [r2] TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis. In ICLR, 2023.\n\n### [W2] Tokenization: Needs more justification on how primitives capture patterns. Are these really \"motion primitives\" (i.e., can you interpret them in isolation?)\nThank you for the insightful comment. While our current work does not assign semantic meaning to each primitive in isolation, we show that the learned tokens form distinct, repeatable patterns across similar activities (e.g., walking vs. standing), and remain stable under perturbations (the response to [Q3]).\n> **Walking**: `170→772→538→170→777→538→170→772→538→339`\n>\n> **Standing**: `853→29→271→401→853→853→72→452→853→29`\n\nWe view this as an important step toward interpretability. The primitives are designed to serve as compositional units that reflect structure in the data. Assigning human-understandable labels to them—potentially via LLM-based grounding—is a possible direction for future work. We will revise the manuscript to clarify this motivation.\n\n\n### [W3] Missing Ablations: Lack of studies on component contributions weakens empirical claims.\nThank you for pointing this out. In addition to the new ablation on removing the VQ module (discussed in our response to [Q2]), we have already **conducted and reported several key ablation studies** in the manuscript:\n- In Table 2 of the main text, we evaluate the contributions of three core components through:\n    - w/o Pretrain Stage\n    - w/o Stat Feature Embedding Layer\n    - w/o Metadata Embedding Adapter\n- In the Appendix, we further include supplementary ablations covering:\n    - Motion Primitive Window Length (Table 3)\n    - Transformer Encoder Depth (Table 4)\n    - Hidden Embedding Dimensionality (Table 5)\n    - Vector Quantization Codebook Size (Table 6)\n\nThese ablations provide a comprehensive understanding of how each module contributes to overall model performance.\n\n### [W4] I appreciate seeing someone release new work using VQ+LLM and adapting it to time-series problems. It seems a little too applied in a way? This isn't necessarily problematic, but the ideas aren't especially novel.\nThank you for your thoughtful feedback. While our work builds upon the VQ-based framework, we believe that its application to **decomposing complex human activities into interpretable motion primitives** represents a key and underexplored direction. Our primary innovation lies in enabling interpretable intermediate representations that move activity recognition toward a more human-aligned, compositional, and explainable paradigm. Achieving this required overcoming several domain-specific challenges:\n- To handle the wide-ranging values in raw sensor data, which would otherwise cause VQ to cluster primarily based on signal magnitude, we introduced **Instance Normalization** to standardize the inputs.\n- Since normalization removes intensity information, we reintroduced it via a **Stat Feature Embedding Layer**.\n- To generalize across datasets with inconsistent sensor configurations, we developed a **Metadata Embedding Adapter**, which explicitly encodes sensor attributes using Google's text-embedding-004 API.\n\nThese components are essential for adapting the VQ+LLM pipeline to the time-series domain while maintaining interpretability. We believe that they constitute practical but impactful advances toward explainable human activity recognition."}, {"review_id": "Tl5JZhZWES", "reviewer": "Reviewer_TS74", "summary": "MoPFormer is a self-supervised Transformer-based framework for human activity recognition using wearable IMU sensors, designed to improve interpretability and generalisation across datasets. Borrowing from techniques successful in natural language processing,  it tokenizes raw sensor data into discrete motion codewords using vector quantization, enriched via context embedding and is trained as masked reconstruction. The proposed method is compared across datasets with SOTA baselines, and ablation studies are performed on the module components", "questions": "- sec 3.1: what's the motivation for extracting features per channel and not from 6-dimensional vector representing 3D motion\n\n- Interpretability: Please provide more details and visualisations on interpretability. For example, appendix figures 5-7, while the activations are different, it seems the same motion primitives are employed in same ratios for multiple different activities. How do you ensure the codewords are used optimally?\n\n- Most datasets used contain multiple sensors, and I believe the proposed architecture can be extended to use more than one sensor, too.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "$\\textbf {Strengths} $\n- Use of codeword as motion feature embedding is novel to my knowledge.\n- The authors have ablated the architecture design and parameters in the main paper and the appendix\n\n\n$\\textbf {Weaknesses} $\n- Interpretability: The authors provide an intuitive explanation of interpretability in the introduction; however, I found the results and visualisation in the main paper and appendix insufficient to support this claim. \n- While normalising each input channel is reasonable, the rationale for independently quantising each feature channel was not clearly articulated. \n\n$\\textit {Minor weaknesses} $\n- Other than the type of GPU, the paper does not contain sufficient information on the experiments' compute resources\n- The learnable parameter sizes in the motion primitive module and Context-Aware Embedding Module are not stated.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns. I am not entirely convinced about the interpretability of motion primitives still. I do believe this work to be high impact and the problem to be useful for the AI community", "rebuttal": "Thank you for your insightful and constructive feedback. We sincerely appreciate the opportunity to clarify and improve our work. Some of your *Weaknesses* (W) and *Questions* (Q) touch upon overlapping themes, so we address them together for clarity and coherence. If you have any further comment, please feel free to let us know.\n\n### [W1 & Q2] Interpretability\n> [W1] The authors provide an intuitive explanation of interpretability in the introduction; however, I found the results and visualisation in the main paper and appendix insufficient to support this claim.\n> [Q2] Please provide more details and visualisations on interpretability. For example, appendix figures 5-7, while the activations are different, it seems the same motion primitives are employed in same ratios for multiple different activities. How do you ensure the codewords are used optimally?\n\nThank you for this comment. It helps us realize that our description in the paper was not sufficiently clear. We provide further justification below and will revise the manuscript to emphasize these ideas.\n- **The similarity in token usage is expected and reflects compositional structure.** Activities like 'ascending stairs', 'cycling', and 'walking' naturally share common motion elements. MoPFormer is designed to decompose complex activities into fundamental, reusable primitives. Many of these correspond to common local patterns (e.g., acceleration/deceleration along one axis). Due to instance normalization, similar patterns are mapped to the same index even if their original means or intensities differ. The model then relies on the embedded sensor metadata to distinguish whether a pattern originates from an arm accelerometer or a leg gyroscope.\n- **The sequence of tokens matters more than raw frequency.** As we demonstrate with the Markov transition matrix (line 581), the final activity classification depends critically on the sequence of primitives, not just their frequency. This is analogous to natural language, where the meaning of a sentence is determined by word order, not just word counts.\n- **The distributions in Figures 5–7 do exhibit subtle differences.** For example, Figure 7 shows a higher proportion of indices between 459-510 compared to Figures 5 and 6. Similarly, the proportion around index 765 in Figure 6 differs from the other two.\n\n### [W2 & Q1] Per-channel quantization design\n> [W2] While normalising each input channel is reasonable, the rationale for independently quantising each feature channel was not clearly articulated.\n> [Q1] sec 3.1: what's the motivation for extracting features per channel and not from 6-dimensional vector representing 3D motion\n\nWe clarify the motivation behind our design choices as follows.\n- **The core motivation is to capture fine-grained, universal primitives.** The most fundamental primitives, such as accelerating along a single axis, are often generalizable across different datasets and tasks, which promotes robustness. If we were to quantize multi-channel vectors directly, the resulting primitives would be composites of finer-grained actions. This would lead to a combinatorial explosion in the number of primitives, making it less likely that a primitive required for a target dataset was seen during pre-training.\n- **This aligns with recent findings in time-series research.** Both channel-independent (e.g., PatchTST) and channel-mixing (e.g., Autoformer [r1], FEDformer [r2]) approaches have been explored. A recent comprehensive benchmark [r3] suggests that channel-independent models tend to perform better on classification tasks. Further research [r4] indicates this phenomenon is closely related to the degree of correlation between channels.\n\n[r1] Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting. In NeurIPS, 2021.\n\n[r2] FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting. In ICML, 2022.\n\n[r3] Deep Time Series Models: A Comprehensive Survey and Benchmark. arXiv preprint arXiv:2407.13278.\n\n[r4] A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggle. In ICML, 2025.\n\n### [Q3] Most datasets used contain multiple sensors, and I believe the proposed architecture can be extended to use more than one sensor, too.\nYou are absolutely right that supporting multi-sensor data is essential. **Our model was explicitly designed with this** and has been evaluated on datasets that include multiple sensors. We apologize this was not sufficiently emphasized in the main text. To clarify, our architecture is inherently built to operate on multiple sensor channels, each of which is processed through motion primitive embedding layer and later integrated through metadata embeddings. The descriptions of the datasets—**all of which are multi-channel and most of which are multi-sensor**—were provided in Appendix D.\n\n### [Minor weaknesses] Missing details on compute resources and module parameter sizes\n- **Compute Resources**: All experiments were conducted on a single NVIDIA RTX 8000 GPU. The pre-training phase took approximately 30 hours, while fine-tuning times varied by dataset, ranging from about 1 to 2.5 hours. As noted in Appendix A.1, we employed PyTorch Lightning with gradient accumulation to support a batch size of 512 and optimize memory efficiency.\n- **Model Parameter Sizes**: As described in Appendix A.1, the MoPFormer architecture includes 5 standard Transformer encoder layers, each with 8 attention heads and an MLP ratio of 1. The total number of trainable parameters is approximately 2.8 million, with the Motion Encoder contributing 2.5M. The Motion Primitive module uses a codebook size of 1024 and an embedding dimension of 256. The MAE head and Classification head contribute 263K and 1.5K parameters, respectively. Additional ablation studies related to these architectural choices are provided in Appendix B.\n\nWe will ensure these details are explicitly included in the revised manuscript for completeness.\n\n### [Minor lauguage errors]\nThank you for pointing this out. We will correct the capitalization and ensure that all acronyms (VQ, MAE, CLS) are properly introduced upon first use."}, {"review_id": "7eE8RZl1AL", "reviewer": "Reviewer_nWUi", "summary": "In this paper, the authors aim to address 2 challenges in sensor-based human activity recognition, including interpretability and cross-dataset generalization capability. To address these challenges, the authors design a pre-training-based method call MoPFormer. The key idea is the pretraining will improve the performance. Evaluations are conducted on 6 HAR datasets, and results show the proposed MoPFormer have achieved better performance compared to other methods.", "questions": "I have questions regarding the experiments, the motivations, etc. Please see the comments above.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Pros: \n\n1.Overall ,the target problem is interesting. The focused two challenges are practical and could have broader impact. \n\n2.Most figures and tables are well-designed. \n\n3.I like the way the authors present the problem, also, I believe generalization is one of the most urgent challenges in the field. While I am not sure about the interpretabiltiy, it seems an interesting  point to consider with more justifications and analysis. \n\nConcerns: \n\n\na. Categorization and Limited Related Work. The reviewer suggests the authors to present related studies in better categorization. Also, there has been multiple cross-dataset, cross-domain human activity recognition papers, I would suggest the authors to carefully go through them. [1,2,3,4]\n\n\nb. The reviewer also has concerns regarding the baselines chosen for comparison. In the context of HAR, there are a variety of state-of-the-art methods, based on pretraining approach to address the cross-domain challenges. Currently, it seems most baseline models are general methods such as models designed for general time series problems. \n\nc. The experimental section might need more investigations. Correct me if I am wrong, for example, during the pretraining process, how different loss components contribution to the final performance remains unknown. \n\n\nd. There are some writing issues need to be addressed. For example, in line 104, there is a refnerecne missing. \n\ne. The reviewer is curious  about the motivation for studying the interpretability of HAR models. HAR seems like a basic tool that generates categories of activity that can be used by downstream applications. As a result, if the accuracy is good enough, it seems we do not need to understand why the model is doing good. \n\nf. The current analysis of interpretaability might be further improved, e.g., with more examples to justify the interpretability, or why this kind of interpretability if a good fit (given there are many kinds of ways to do interpretation). Also, could you explain more about how the proposed method actually \"interprets\" the HAR process. \n\n\nReferences:\n\n[1] Cross-Dataset Activity Recognition via Adaptive Spatial-Temporal Transfer Learning, IMWUT\n\n[2] CrossHAR: Generalizing Cross-dataset Human Activity Recognition via Hierarchical Self-Supervised Pretraining, IMWUT\n\n[3] Generalizable Low-Resource Activity Recognition with Diverse and Discriminative Representation Learning, KDD\n\n[4] Learning Generalizable Physiological Representations from Large-scale Wearable Data, NeurIPS", "was_revised": true, "final_justification": "The authors have addressed my concerns during the rebuttal process. I have increased the score.", "rebuttal": "We sincerely thank Reviewer nWUi for the thoughtful and constructive review. We strongly agree with your perspective that **generalization** is a critical challenge in wearable sensor-based human activity recognition. Our work aims to **advance the solution to this generalization problem by decomposing activities to make them more interpretable**. Below, we respond to the concerns raised, and let us know if any issues remain.\n\n### a. Categorization and Limited Related Work\nThank you for the valuable suggestion and for highlighting important related works. In the revised version, we will reorganize the *Related Work* section into four clearer categories to improve structure and clarity:\n- **Traditional HAR architectures and methods.** These approaches are often limited in their ability to generalize across datasets.\n- **Cross-domain and cross-dataset generalization methods.** We will expand our discussion of recent work in this area, including advanced representation learning and domain generalization techniques tailored to HAR, including the cited works [1–4].\n- **Multi-sensor composite activity recognition and compositional generalization.** This category includes methods that conceptually align with our motion primitive–based approach, particularly in their ability to handle complex and previously unseen activity compositions. We will also include relevant studies on compositional generalization from other domains, such as robotics and NLP, to provide a broader perspective.\n- **Interpretable HAR Methods.** We will compare our approach with prior work focused on interpretability in HAR (e.g., [29, 43, 44, 54] in the manuscript), highlighting both methodological differences.\n\n### b. The reviewer also has concerns regarding the baselines chosen for comparison\nIn our paper, we selected four baseline frameworks with distinct motivations:\n- **BYOL:** This choice was inspired by a summary work [r1]. **As contrastive learning is a classic framework for addressing cross-dataset challenges,** we chose a standard BYOL framework combined with three data augmentation methods shown to be effective in that work.\n- **ModCL:** This method [r2] designs specific data augmentations for the HAR task. Its authors claim that it \"outperforms existing state-of-the-art (SOTA) methods\" in extensive cross-dataset validation.\n  > **By extensive cross-data set validation**, experimental results show that ModCL outperforms existing  state-of-the-art (SOTA) methods on recognition accuracy metric, proving the superiority of ModCL in capturing meaningful features.\n- **TSLANet:** This work [r3] represents the rapid advancements in the general-purpose time-series domain. We included it to ensure our comparisons account for excellent research from this important upstream field.\n- **CALANet:** As a recent NeurIPS 2024 paper [r4] , we selected this model to compare our method against the latest and most advanced work specifically within the HAR domain.\n\nIn response to your comment, we are actively contacting the authors of the additional method you mentioned. Meanwhile, we have added TimesNet, a strong time-series baseline, to our comparisons:\n\n||PAMAP2|DSADS|Realworld|\n| :---: | :---: | :---: | :---: |\n|TimesNet (Acc)|86.88|97.17|92.51|\n|TimesNet (F1)|85.31|96.05|92.16|\n|Ours (Acc)|86.08|97.60|92.05|\n|Ours (F1)|84.83|97.38|93.25|\n\n[r1] What makes good contrastive learning on small-scale wearable-based tasks? SIGKDD 2022.\n\n[r2] Modality consistency-guided contrastive learning for wearable-based human activity recognition. IEEE Internet of Things.\n\n[r3] TSLANet: Rethinking Transformers for Time Series Representation Learning. ICML 2024.\n\n[r4] CALANet: Cheap All-Layer Aggregation for Human Activity Recognition. NeurIPS 2024.\n\n### c. The experimental section might need more investigations\n- In MoPFormer, the pretraining loss combines a Masked Auto-Encoding loss ($\\mathcal{L}\\_{mae}$) and a Vector Quantization loss ($\\mathcal{L}\\_{vq}$), with an auxiliary classification loss ($\\mathcal{L}\\_{cls}$):\n$$\n\\mathcal{L} = \\lambda\\_{mae}\\mathcal{L}\\_{mae} + \\lambda\\_{cls}\\mathcal{L}\\_{cls} + \\lambda\\_{vq}\\mathcal{L}\\_{vq}\n$$\nThe two core components, $\\mathcal{L}\\_{mae}$ and $\\mathcal{L}_{vq}$, are structurally interdependent: $\\mathcal{L}\\_{mae}$ predicts the quantized token indices generated by the VQ module trained with $\\mathcal{L}\\_{vq}$. Removing either would break the tokenization-reconstruction mechanism, making direct ablation of individual loss terms infeasible.\n\n- Instead, we indirectly assess their contribution by ablating the entire VQ module (denoted as w/o VQ), which disables both $\\mathcal{L}\\_{mae}$ and $\\mathcal{L}\\_{vq}$. The resulting performance drop confirms the tokenization mechanism is critical not only to effectiveness but also to interpretability:\n\n  |||PAMAP2|Realworld |\n  | :--: | :--: | :--: | :--: |\n  |w/o VQ| Acc|84.71|91.02|\n  ||F1| 82.42 |90.17|\n  | MoPFormer | Acc |86.08|92.05|\n  ||F1|84.83|93.25|\n  \n- Additionally, Table 2 in the main text presents ablation results on key architectural components, including:\n   - w/o Stat Feature Embedding Layer\n   - w/o Metadata Embedding Adapter\n\nThese experiments collectively highlight the necessity of each module and the tight integration between the loss components during pretraining.\n\n### d. There are some writing issues need to be addressed\nThank you again for your meticulous review. We will update it in the revised version.\n\n### e. The reviewer is curious about the motivation for studying the interpretability of HAR models \nThis is a fundamental and excellent question. We strongly agree that generalization is a core challenge for HAR. We argue that many of these challenges can be framed through the lens of compositional generalization—the ability of a model to recognize novel combinations of familiar components [r5]. To illustrate with practical challenges:\n  - Cross-Subject Generalization: Different individuals perform the same activity, like 'running', with unique styles. Our model would recognize that the core of 'running' lies in the leg motion primitives (e.g., alternating strides), while treating stylistic variations like different arm swings as separate, reusable components. This allows it to generalize to new users whose holistic running pattern has not been seen before.\n  - Cross-Dataset Generalization: Consider the activity 'ascending stairs'. In Dataset A, subjects might be instructed to climb stairs while holding the handrail. In Dataset B, subjects might perform the same activity but with their hands in their pockets. Semantically, both are 'ascending stairs', but the sensor data from the arms and torso will differ significantly. Previous methods trained only on Dataset A might fail on Dataset B because it has learned a holistic pattern that includes 'hand-on-rail' motion. In contrast, our method would learn to interpretably separate the core primitives of 'leg-lifting' and 'torso-stabilizing' from the context-specific arm primitives. This separation, when combined with prior knowledge, allows the model to correctly identify the core activity in Dataset B, even with different arm movements, demonstrating superior generalization.\n\nThis highlights why interpretability is not optional—it is the key to verifying whether the model learns meaningful, reusable primitives rather than spurious correlations. For example, seeing that the model has learned a clean representation of “leg swing” gives us confidence in its ability to recombine it across contexts. In this view, building inherently interpretable models is a direct path to solving HAR’s generalization challenges [r6].\n\n[r5] Human-level concept learning through probabilistic program induction. Science.\n\n[r6] Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence.\n\n### f. The current analysis of interpretaability might be further improved\nThank you for emphasising the need for clearer justification and explanation. Below we outline (1) why motion-primitive interpretability is well-suited to HAR and (2) how MoPFormer actually delivers it.\n- **Why motion-primitive interpretability is a good fit**\n  As argued in our response to (e), human activities are hierarchical and compositional: complex actions are built by combining simple, reusable sub-motions (e.g., a “leg-lift” appears in walking, running, and stair-climbing). Modeling these as interpretable motion primitives supports generalization across users, sensor placements, or datasets. This form of interpretability helps us understand how the model adapts to new contexts, rather than treating activities as monolithic classes.\n\n- **How MoPFormer interprets the HAR process**\n  Our model's interpretation process is conceptually analogous to how a Large Language Model (LLM) processes language. Just as sentences are composed of words, activities are composed of fine-grained motion primitives. Our model is explicitly designed to follow this logic:\n  - **Discovering motion primitives** (\"build a vocabulary\"): The first stage of our model uses a Vector Quantization (VQ) module with a Masked Autoencoder (MAE) objective, which is to construct a discrete codebook of effective, fundamental motion primitives from the raw, continuous sensor data. \n  - **Understanding primitive sequences** (\"learn the grammar\"): The subsequent Transformer module receives sequences of these discrete primitives as input. Its role is to learn the \"grammar\" how these primitives are combined in sequence to form complex, high-level activities.\n  - **Classification based on understanding**: Finally, the model classifies the activity based on its learned understanding of the entire primitive sequence.\n\nIn the revision, we will include visualizations (e.g., token usage patterns across activities) to better illustrate how different activities rely on distinct primitive combinations, reinforcing the interpretability argument."}]

Author rebuttal:

Meta-review: The paper proposes a self-supervised framework for human activity recognition that improves interpretability and cross-dataset generalization. It tokenizes sensor signals into discrete motion primitives and processes them with a Transformer, using masked motion modeling for robust pretraining. After the rebuttal stage and duscussions between the reviewers and authors, it recieved 4 BAs. The reviewers agreed that the proposed method makes a solid contribution by introducing motion primitives for HAR, showing strong performance across six benchmarks, and providing ablations and robustness tests. The main weaknesses that remain are limited interpretability evidence, since motion primitives are only quantitatively analyzed without strong human validation, and baseline coverage gaps, with comparisons to some SOTA HAR and interpretable models still missing or under-explored. The novelty of the method was also found to be incremental based on adapting VQ+Transformer pipelines.
Decision: Accept (poster)