Title: MoPFormer: Motion-Primitive Transformer for Wearable-Sensor Activity Recognition
Venue: NeurIPS 2025
Accepted: yes
Keywords: Human Activity Recognition

Wearable Sensors

Deep Learning
Abstract: Human Activity Recognition (HAR) with wearable sensors is challenged by limited interpretability, which significantly impacts cross-dataset generalization. To address this challenge, we propose Motion-Primitive Transformer (MoPFormer), a novel self-supervised framework that enhances interpretability by tokenizing inertial measurement unit signals into semantically meaningful motion primitives and leverages a Transformer architecture to learn rich temporal representations. MoPFormer comprises two stages. The first stage is to partition multi-channel sensor streams into short segments and quantize them into discrete ``motion primitive'' codewords, while the second stage enriches those tokenized sequences through a context-aware embedding module and then processes them with a Transformer encoder. The proposed MoPFormer can be pre-trained using a masked motion-modeling objective that reconstructs missing primitives, enabling it to develop robust representations across diverse sensor configurations. Experiments on six HAR benchmarks demonstrate that MoPFormer not only outperforms state-of-the-art methods but also successfully generalizes across multiple datasets. More importantly, the learned motion primitives significantly enhance both interpretability and cross-dataset performance by capturing fundamental movement patterns that remain consistent across similar activities, regardless of dataset origin.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors aim to address 2 challenges in sensor-based human activity recognition, including interpretability and cross-dataset generalization capability. To address these challenges, the authors design a pre-training-based method call MoPFormer. The key idea is the pretraining will improve the performance. Evaluations are conducted on 6 HAR datasets, and results show the proposed MoPFormer have achieved better performance compared to other methods.\n\nSTRENGTHS AND WEAKNESSES:\nPros: \n\n1.Overall ,the target problem is interesting. The focused two challenges are practical and could have broader impact. \n\n2.Most figures and tables are well-designed. \n\n3.I like the way the authors present the problem, also, I believe generalization is one of the most urgent challenges in the field. While I am not sure about the interpretabiltiy, it seems an interesting  point to consider with more justifications and analysis. \n\nConcerns: \n\n\na. Categorization and Limited Related Work. The reviewer suggests the authors to present related studies in better categorization. Also, there has been multiple cross-dataset, cross-domain human activity recognition papers, I would suggest the authors to carefully go through them. [1,2,3,4]\n\n\nb. The reviewer also has concerns regarding the baselines chosen for comparison. In the context of HAR, there are a variety of state-of-the-art methods, based on pretraining approach to address the cross-domain challenges. Currently, it seems most baseline models are general methods such as models designed for general time series problems. \n\nc. The experimental section might need more investigations. Correct me if I am wrong, for example, during the pretraining process, how different loss components contribution to the final performance remains unknown. \n\n\nd. There are some writing issues need to be addressed. For example, in line 104, there is a refnerecne missing. \n\ne. The reviewer is curious  about the motivation for studying the interpretability of HAR models. HAR seems like a basic tool that generates categories of activity that can be used by downstream applications. As a result, if the accuracy is good enough, it seems we do not need to understand why the model is doing good. \n\nf. The current analysis of interpretaability might be further improved, e.g., with more examples to justify the interpretability, or why this kind of interpretability if a good fit (given there are many kinds of ways to do interpretation). Also, could you explain more about how the proposed method actually \"interprets\" the HAR process. \n\n\nReferences:\n\n[1] Cross-Dataset Activity Recognition via Adaptive Spatial-Temporal Transfer Learning, IMWUT\n\n[2] CrossHAR: Generalizing Cross-dataset Human Activity Recognition via Hierarchical Self-Supervised Pretraining, IMWUT\n\n[3] Generalizable Low-Resource Activity Recognition with Diverse and Discriminative Representation Learning, KDD\n\n[4] Learning Generalizable Physiological Representations from Large-scale Wearable Data, NeurIPS\n\nQUESTIONS:\nI have questions regarding the experiments, the motivations, etc. Please see the comments above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMoPFormer is a self-supervised Transformer-based framework for human activity recognition using wearable IMU sensors, designed to improve interpretability and generalisation across datasets. Borrowing from techniques successful in natural language processing,  it tokenizes raw sensor data into discrete motion codewords using vector quantization, enriched via context embedding and is trained as masked reconstruction. The proposed method is compared across datasets with SOTA baselines, and ablation studies are performed on the module components\n\nSTRENGTHS AND WEAKNESSES:\n$\\textbf {Strengths} $\n- Use of codeword as motion feature embedding is novel to my knowledge.\n- The authors have ablated the architecture design and parameters in the main paper and the appendix\n\n\n$\\textbf {Weaknesses} $\n- Interpretability: The authors provide an intuitive explanation of interpretability in the introduction; however, I found the results and visualisation in the main paper and appendix insufficient to support this claim. \n- While normalising each input channel is reasonable, the rationale for independently quantising each feature channel was not clearly articulated. \n\n$\\textit {Minor weaknesses} $\n- Other than the type of GPU, the paper does not contain sufficient information on the experiments' compute resources\n- The learnable parameter sizes in the motion primitive module and Context-Aware Embedding Module are not stated.\n\nQUESTIONS:\n- sec 3.1: what's the motivation for extracting features per channel and not from 6-dimensional vector representing 3D motion\n\n- Interpretability: Please provide more details and visualisations on interpretability. For example, appendix figures 5-7, while the activations are different, it seems the same motion primitives are employed in same ratios for multiple different activities. How do you ensure the codewords are used optimally?\n\n- Most datasets used contain multiple sensors, and I believe the proposed architecture can be extended to use more than one sensor, too.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles interpretability and generalization issues in Human Activity Recognition (HAR) using wearable sensors by introducing MoPFormer (Motion-Primitive Transformer). MoPFormer tokenizes inertial measurement unit (IMU) signals into \"motion primitive\" codewords, enhancing interpretability and utilizing a Transformer to learn temporal patterns. It operates in two stages: segmenting and quantizing sensor data, followed by enriching and processing with a Transformer encoder. Pre-trained with a masked motion-modeling objective, MoPFormer outperforms state-of-the-art methods across six HAR benchmarks (PAMAP2, DSADS, MHealth, Realworld, UCI-HAR, and USC-HAD) and generalizes well to new datasets by capturing consistent movement patterns.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n* The authors develop an approach to HAR using new VQ+LLM style modeling approaches. They pre-train using MAE and have a classifier head for each dataset. Overall it's a nice time-series variant of this paradigm. \n* They show some ways intermediates of the models can be interpreted. \n* Excels in cross-dataset performance and outperforms current methods on multiple benchmarks.\n\nWeaknesses\n* My understanding is that there are other Time-series foundation models in this direction. It would be nice to see more comparisons to these. \n* Tokenization: Needs more justification on how primitives capture patterns. Are these really \"motion primitives\" (i.e., can you interpret them in isolation?)\n* Missing Ablations: Lack of studies on component contributions weakens empirical claims.\n* I appreciate seeing someone release new work using VQ+LLM and adapting it to time-series problems. It seems a little too applied in a way? This isn't necessarily problematic, but the ideas aren't especially novel.\n\nQUESTIONS:\n* While it does outperform all other models, I'm noticing that MoPFormer requires pertaining to achieve these results, whereas the TSLANet achieves better performance without pre-training. Are there any learnings here?\n* Tokenization: How do segment length and quantization affect performance? Is there a version of this where you could remove tokenization and run the model just on classification to understand the impact of VQ? Other ablation studies would also be valuable. \n* What is the impact of noise or perturbations on the sensor inputs on tokenization? Does minor perturbation every significantly change the token distribution?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMoPFormer introduces a two-stage architecture: (1) Motion Primitive Module: Tokenizes raw IMU signals into discrete codewords via VQ-based quantization; and (2) Transformer Backbone with Context-Aware Embedding: Learns temporal representations using metadata-aware embeddings and a masked autoencoding objective. The model is evaluated on six public HAR benchmarks and shows SOTA performance with compelling ablation studies and qualitative analyses.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- The paper clearly identifies key HAR challenges (interpretability, heterogeneity) and aligns contributions tightly to them.\n\n- Use of motion primitives learned via VQ in self-supervised HAR is a compelling idea, especially for generalization and interpretability.\n\n- The context-aware embedding module that integrates statistical features and sensor metadata is thoughtfully designed.\n\n- Consistent improvements over both contrastive and supervised baselines across six diverse datasets.\n\n- Ablations and visualizations demonstrate the importance and effect of each module.\n\n### Weaknesses\n\n- No error bars, standard deviations, or confidence intervals are reported, weakening the empirical rigor—especially important given the small dataset sizes and minor margins in accuracy.\n\n- While contrastive and transformer baselines are compared, the paper omits comparisons with interpretable models like decision trees or attention-based attribution techniques specifically adapted for HAR (e.g., segment attention). Are motion primitives strictly better?\n\n- Despite the motion primitive framework, the interpretability remains primarily quantitative (e.g., frequency and similarity). There’s no human evaluation of whether the discovered primitives correspond to semantically meaningful sub-activities.\n\nQUESTIONS:\nPlease refer to the Weakness part.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}