IMU2CLIP: Language-grounded Motion Sensor Translation with Multimodal Contrastive Learning
Authors: Seungwhan Moon
Andrea Madotto
Zhaojiang Lin
Aparajita Saraf
Amy L. Bearman
Babak Damavandi
Venue: EMNLP 2023 Findings (2023)
Keywords: Contrastive Learning
NLP Applications in Sensor Signals
TL;DR: We develop a new method to translate IMU motion sensor signals into text, allowing for novel applications in low-power use cases on wearable devices, etc.

Abstract: We present IMU2CLIP, a novel pre-training approach to align Inertial Measurement Unit (IMU) motion sensor recordings with text and video, by projecting them into the joint representation space of Contrastive Language-Image Pre-training (CLIP). The proposed approach allows IMU2CLIP to translate human motions (as measured by IMU sensors) into their corresponding textual descriptions and videos -- while preserving the transitivity across these modalities. We introduce several new IMU-based Wearable AI applications such as motion-based media search, or an LM-based multimodal reasoning with motion sensor data -- all using text as the grounding platform. In addition, we show that IMU2CLIP significantly improves downstream performances when fine-tuned for each application, demonstrating its universal usage as a new pre-trained resource. Our code and models will be released publicly.

IMU2CLIP: Language-grounded Motion Sensor Translation with Multimodal Contrastive Learning
Seungwhan Moon
Andrea Madotto" Zhaojiang Lin Amy Bearman
Aparajita Saraf
Babak Damavandi Meta Reality Labs & FAIR, Meta
Abstract
(x)
Video
We present IMU2CLIP, novel pre-training approach to align Inertial Measurement Unit (IMU) motion sensor recordings with text and video, by projecting them into the joint repre- sentation space of Contrastive Language-Iman Pre-training (CLIP). The proposed approach al- lows IMU2CLIP to translate human motions (as measured by IMU sensors) into their cor- responding textual descriptions and videos while preserving the transitivity across these modalities. We introduce several new IMU- based Wearable Al applications such motion- based media search, or an LM-hased multi- modal reasoning with motion sensor data all using text as the grounding platform. In addi- tion, we show that IMU2CLIP significantly improves downstream performances when fine- tuned for each application, demonstrating its universal usage new pre-trained resource. Our code and models will be released publicly.
Encuder
aiking
Encoder
6d Viloo-IMU--Tee Rocordings
IMU2CLIP
Joint Space
(0)
jumping
Text as query
IMU as quary
clamp
Figure 1: Illustration of IMU2CLIP (12C): (a) The model aligns the parallel Video=>IMeeTeen. data in the joint space. Once trained, IMU2CLIP is used retriever both (b) IMU and (c) videos, or as a text- based zeroshot classifier for downstream applications.
scale. Consequently, the utilization of IMU mod- els in real- world scenarios has been confined to relatively small number of use cases.
On the contrary, for the modalities that are widely studied (e.g. text, video), there are vast large-scale resources such BERT (Devlin et al., 2018) and GPT (Radford et al., 2018) for text, CLIP4Clip (Luo al., 2021) for videos. These powerful pre-trained resources have driven the de- velopment of many application--rreeteed models, showing significant improvements when fine-tuned for each respective task (Dodge et al., 2020). To our knowledge, however, the study on the equivalent resources for IMU signals has been lacking.
1 Introduction
With the growing popularity of smart glasses or new-generation wearable first-erro or egocentric videos have recently become prevalent (Grauman et al.. 2022; Damen et al., 2021; Lv et al., 2022). These egocentric videos are often accom- panied by the parallel head- mounted IMU sensor readings, which record devices' línear and rota- tional movements and accelerations.
Inspired by the recent works studied for other modalities, we present IMU2CLIP, a new ap- proach to pre-train an IMU encoder by aligning the parallel Video>>IMU data. Specifically, we propose to use CLIP (Radford et al., 202 la). which contains the video and language models pre- trained on the large image-text parallel data. The IMU encoder thus lcars semantic representations of various scenes transferred from other modalities.
Given its low power consumption level, IMU is regarded as an important modality for powering var- ious always-on on-device models that require un- derstanding of device wearer's movement patterns (e.g. exercise activity recognition for health appli- cations) which would not work with more battery- consuming camera sensors. The previous works IMU modeling typically focus the purpose- built datasets with manual annotations (Jiang et al., 2022: Chen al., 2021), which are limited in their "Joint First Authors. (shanemoon, andreamad8 @emeta. com
Note that anchoring onto the joint text-vision space essentially allows for translation across these modalities. This new model capability opens up

---

novel applications in the real-world Wearable AI, such as IMU-based media search and AM-based multimodal reasoning (Fig. 1) - serving as efficient low-power alternatives more power-consuming approach of video-based media retrieval, which can operate under the strict on-device hardware constraints of wearable devices. In our evaluation, we provide an in-depth analysis on these newly proposed applications, as well as its efficacy on the downstream applications when munet..ed.
activities from head-mounted devices. These include Ego4D (Grauman al., 2022), Epic- Kitchens (Damen et al., 2021), and Aria (Lv al., 2022). Using these datasets, we propose various sub-tasks that can effectively evaluate diverse ca- pabilities of IMU2CLIP, and demonstrate the fea- sibility of future applications. In addition, we im- plement a universal multimodal dataloader to allow for easy pross-modalitty and cross-domain studies.
IMU Modeling: IMU signals are used in various motion recognition tasks, such as pose estimation and walking speed estimation (Zihajehzadch and Park, 2017). Various architectures (Kim et al., 2021; Ashry et al., 2020) have been explored for modeling IMU in downstream tasks, including Ceaffomeer-NN based IMU models (Jiang et al., 2022). Our work proposes a new IMU model ar- chitecture, and conducts ablation studies over other models above. Different from prior works that train IMU models for specific tasks, however, our work focuses on learning multimodal IMU representa- tions by aligning IMU with text and image, which can enable a wider set of downstream applications.
2 Related Work
Contrastive Learning is as an efficient self- supervised framework applied across multiple do- mains, which learns similar/dissimilar represen- tations from paired data. For instance, SimCLR (Chen et al., 2020) is a unimodal application of con- trastive learning in the data augmentation setting, which proposes to learn a vision encoder given set of perturbed images. As an example in multimodal Pre-training
settings, Contrastive Language-Iman (CLIP) (Radford et al., 2021a) learns visual rep- resentations from natural language supervision us- ing image and text pairs, achieving competitive results in e.g. zero-shot image classification, im- age retrieval via text, and image/caption generation. Similarly, WAV2CLIP (Wu et al., 2022) proposes to learn audio representation by distilling it from CLIP. We extend this line of work to unique mul- timodal setting that utilizes IMU signals, which is specific to a new generation of devices (such as smart glasses) that are equipped with such sensors.
Methods
In a nutshell, we train an IMU encoder such that the IMU representation of given clip resembles the representation of its corresponding textual deserip- tions (narrations), or corresponding video frames. Fig. illustrates the overall approach.
Cross-modal Contrastive Learning. We consider a batch of B ground-truth IMU++Video++ TTxt parallel windows: {(h,, v1, t1),... (ig. VB. tB)}- where the embeddings of each modality lies on the unit hypersphere SP. Since they are unit- normalized, the similarity can be calculated as their inner products: sim(i,, vy) - (i,, v,) and sim(i,, t,) = (i;, ty)- We then train three fla- vors of IMU2CLIP: (a) aligning IMUx>>Text (b) IMU++ Video, and (c) VUUe--idoo+-Ter Specif- ically, we project the IMU representations into the joint CLIP space (Radford et al., 2021a) to lever- age the visual and textual knowledge already en- coded in CLIP. Similar to (Luo et al., 2021; Rad- ford et al., 2021a), we propose to minimize the symmetric cross-modal contrastive loss (ablations on loss choices in Appendix). For the IMU++Text alignment, we use the sum of IMU-to-Text (12t) and Text-to-IMU (t2i) cross-entropy losses:
Pre-training Resources: There are numerous pre- trained resources for well-studied modalities such text or image. Popular language models (LM) include BERT (Devlin al., 2018), GPT-2 (Rad- ford al., 2019), and GFT-i(Floridi and Chiriatti, 2020), which typically use celf-superivsion tech- niques such as next-word predictions or masked token predictions, thus not requiring any explicit task labels. Studies report that these pre-trained re- sources achieve competitive zhoo performance (Radford et al., 2021a), and when fine-tuned, of- ten outperform fully supervised models on several downstream tasks (Dodge et al., 2020).
To our knowledge, the equivalent resource for IMU is not made publicly available. We perform large-scale pre-training for the unique wearble sen- sor signals dataset, and show that such pre-training significantly improves downstream performances. Egocentric Datasets: We focus on egocentric (first-person) datasets, for understanding of users'
1
exp(sim(i, t,)
Cix
log Σ# k exp(sim(i;, (issmm(i,
B

---

Janging
Dataset
Statistics
Tra.
Val.
Tst.
Media files
1444
161
688
Total Media Durations
540h
60h
265h
Ego41>
IMU MMMUeTText/Vii Texe/Video Pairs
528K
68K
266K
Windows for Labels
1552
760
241
Media files
747
259
277
Aria
Total Media Durations
138h
43h
51h
Media Search via Tex Retrieval. Given free-form text query (e.g. "jumping"), (Left); IMU2CIIP? predictions of the semantically closest IMU signals from Ego4D (top-2). (Right): correspond- ing gold cold-parallel videas (as a reference). Retrieved re- sults match the semantics of imput queries.
Figure 2:
IMdeo Pairs
496K
157K
184K
Windows for Labels
25K
138K
162K
Table 1:
Dataset Statistics for Ego4D and Aria.
where List 1/2 (Lizt + Leai). with Liai de- fined symmetrically. The loss for MUdeo alignment (Citvv) can be defined similarly, and consequently Lievee Liev Liet.
Task 2. IMU-based Activity Recognition, where the goal is to predict a discrete motion-based ac- tivity label (e.g. hiking, running, walking, biking) given window of IMU signals, measured via FI.
IMU Encoder Architectures. We propose an ar- chitecture with a stack of ID-CNNs and a RNN, which performed the best in our ablation studies A.2). First, we perform Group- Norm to normalize the Accelerometer (3D) and the Gyroscope (3D) signals independently. We then add a stack of N = 3 ID-CNNs, a Max Pooling layer (kernel size 5), and another GroupNorm to normalize the output features. We use GRU to combine the CNN outputs as final representations.
Task 3. Question Answering on Sensor Logs, where the goal is to respond to users' memory re- call queries with natural language, based on the logs from ambient sensors (e.g. IMU, audio), as an exploratory task. We leave the quantitative evalua- tion of this task as future work.
(See Appendix.
4.2 Results
Table 2 shows the IMU++Text and IMU++ Video retrieval performance on the Ego4D test set, of IMU2CLIP trained via different combinations of modalities. Results on Aria are in Appendix A.3.
4 Experiments
Dataset. We use Ego4D (Grauman et al., 2022) and Aria (Lv al., 2022) as the main datasets for the experiments below, both of which feature large- scale parallel video and IMU signals. For Ego4D, a subset of the clips are also annotated with text narrations. We split the data into train, validation, and test sets (split by video IDs). The statistics of the datasets are provided in Table
IMU-based media search with textual queries: The TeX column in Table 2 shows perfor- mances on Task 1. Note that the CLIP embed- dings already exhibit the Video++ Text aligmment, and thus IMU2CLIP trained using IMU+ Video achieves a competitive zeroshot performance for IMU++Texi retrieval as well. When text narra- tions are used for pre-training, the model learns IMU representations that are better aligned with language, and thus achieves an even higher recall performance. See Figure 2 for visualizations
4.1 Tasks
Note that the proposed pre-training approach en- forees the alignment of the IMU, text, and video representations, which allows for new and unique cross-modal applications. We propose the fol- lowing real-world downstream applications as the novel tasks to evaluate the IMU encoders.
To help contextualize the performances Table 2, we also show (as a reference) the Textr Video retrieval performance of the CLIP video en- coder. The narrow margins (e.g. MRR=0,143 for Text->IMU vs. MRR=0. for Text-> -ideeo) show that the IMU encoder could serve as a power- efficient alternative for video encoder in many applications, where the memory and power usage is restricted (e.g. on wearable devices).
Task 1. Media Search via TM Retrieval, where the goal is retrieve a window of IMU sig- nals given free-form textual queries. Once IMU signals are retrieved, we can also retrieve their cor- responding videos, thus allowing for a new and power-efficient way of performing media retrieval or online action detection. We evaluate on the held- out test set (Recall@A and Mean Reciprocal Rank (MRR)), using the text narrations as queries and the IMU signals as the retrieval pool.
Fine-tuned IMU2CLIP significantly outper-
For better readability, we also provide the animated GIF visualizations in the supplementary materials.

---

Train Modalities
IMU - Text
Text
IMU
IMU
Video
IMU
Video
Text
R@1
R@ 10
R@50
MRR
R@1
R@ 10
R@50
MRR
R@]
R@ 10
R@50
MRR
4.86
18.75
48.26
0.104
4.17
15.62
43.06
0.084
43.13
78.75
0.2011
5.21
25.00
60.42
0.123
7.29
28.82
60.07
0.143
3.75
25.94
62.81
0.105
4.52
22.91
56.60
0.118
5.90
22.92
56.60
0.139
8.75
40.63
73.44
0.183
(Video + Text)
(Text
Video)
(CLIP)
V
6.94
32.29
64.24
0.150
8.33
33.68
65.28
0.168
Table 2: TeX and VI retrieval performances of the pre-trained IMU2CLIP models Ego4D, with different modalities used for training. The last row shows the video retrieval performance of OpenAl's CLIP model on the same test set.
Ego4D
Aria
[15:od] Motions no activity, Rodio: no activity
Models
[15:45] Mations walking, Audioz naise
F1
Acc.
F1
Acc.
[15:50] Motions aita dovn, Audios watching TV
[15152] Notions look azound, Nadioi kitchen sounda
[36:03]
Motlons walking, Audios activity
Vanilla IMU Encoder
23.23
49.92
56.35
76.1
lang did
watching TV?
Answex: Pron 15:50 16:03, for total minutas.
Zeroshot
19.39
23.08
18.46
21.52
IMU2CLIP
Figure Demonstration of LM-based multimodal rea- soning via IMU2CLIP, using two ambient sensors: IMU and audio. Given sensor logs (translated in text) and the question, LM generates responses grounded on the multimodal context (More examples in the Suppl.).
+ Probing
40.55
61.46
62.52
83.54
(ivv
Fine tuning
43.07
65,87
61.77
82.31
Zeroshot
31.89
36.38
IMU2CLIP
+ Probing
45.12
58.01
N/A
(1 t) + Fine tuning
45.15
63.14
Table 3: IMU-based activity recognition Ego4D and Aria datasets, comparing the randomly initialized model (vanilla) and the pre-trained IMU2CLIP models, with IMU+> and +ext, Bold denotes the best performance for each metric: F1 and Accuracy (Acc).
into text, we present the following demo (Figure 3). Specifically, we run IMU2CLIP (and an equiva- lent model trained for audio) as zeroshot tagging models on Ego4D, to predict textual descriptions of the ambient sensor readings (IMU+Audio). Using text as a grounding platform, we condition a large LM (GPT-3 (Floridi and Chiriatti, 2020)) on the sensor logs to answer summarization questions (e.g. "What can you tell me about the user activity?"" or memory recall prompts (e.g. "When did start biking?") The LM then generates response via causal language inference, demonstrating zeroshot reasoning capabilities. Unlike the similar approach by (Zeng et al., 2022), the proposed approach does not rely on video signals at all which incur much higher power consumption thus operating better under real-world on-device constraints.
forms the vanilla model with the same archi- tecture trained from seratch, on downstream tasks. Table 3 shows the activity recognition re- sults on Ego4D and Aria. For all experiments, use the same IMU architecture (Stacked CNN). For zeroshot experiments, we encode the surface names of each activity (e.g. hiking) with the CLIP text encoder, and use the nearest-neighbor classi- fier on the projected IMU embeddings (thus with- out any supervision labels). Probing adds linear layer on top of the IMU encoder while keeping its parameters frozen, and for fine-tuning we allow all parameters be trainable. The consistent im- provements in the fine-tuning performances (e.g. ~6 points absolute improvement in accuracy for Ego4D, comparing the randomly initialized model vs. fine-tuned model) show that IMU2CLIP can learn high quality representations for IMU signals.
Conclusions
With the growing popularity of wearable devices of diverse form factors (e.g. smart glasses), it is im- perative study ambient motion sensors such as IMU, We thus make the following contributions: 1) We propose large-scale pre-training method for IMU<-Texx and release the pre-trained multimodal encoders for future research. (2) We provide an in- depth empirical analysis for both upstream and fine- tuning tasks. (3) Most importantly, we demonstrate the feasibility of many multimodal NLP applica- tions for ambient sensors via their translation to text, which can spur future research.
Comparing the pre-trained models trained via various combinations of modalities again shows that IMU2CLIP preserves the transitivity among modalities (video ++ text ++ IMU).
Qualitative Analysis: Multimodal Reasoning with Ambient Sensors. Further exploring the ben- efit of IMU2CLIP that translates sensor signals

---

6 Limitations
Techniques. We train the proposed IMU2CLIP model leveraging other large-scale pretrained lan- guage and multimodal models (e.g. CLIP (Radford et al., 2021a), OPT (Zhang al., 2022). GPT-3 (Floridi and Chiriatti, 2020)), and thus any bias or contextual information from these models might be carried over to our model as well (de Vassi- mon Manela et al., 2021). Given the nature of the datasets we use, however, we do not anticipate that our models would produce any harmful outputs, es- pecially towards vulnerable populations, after the models are trained on our tasks.
We discuss the current limitations of our work:
(1) While we used the two of the largest egocen- tric multimodal datasets (Ego4d (Grauman et al., 2022) and Aria (Lv et al., 2022)) that are publicly available, the transferability of this work to an un- seen data (e.g. from other publicly available wear- able devices, or of different activity domains) re- mains unanswered. We do note, however, that the hardware specifications of most IMU sensors and video sensors are standardized across the industry, hence posing limited risks.
(2) Our demonstration of the multimodal rea- soning model (Figure 3) is currently at a qualita- tive level, due to the lack of benchmark dataset on this newly proposed task. While our analysis shows promising results, quantitative evaluation is required to further determine the feasibility of this task and to track progress in the field. How- ever, emphasize that the purpose of this work is to explore potential applications that the pro- posed approach brings forth, and to inspire future research in a fast-growing application domain that uses smart glasses.
References
Sara Ashry, Tetsuji Ogawa, and Walid Gomaa 2020. Charm-deep: Continuous human activity recognition model based on deep neural network using imu sen- sors of smartwatch, IEEE Sensors Journal.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for con- trastive learning of visual representations. In Inter- national Conference on Machine Learning (ICML).
Xinxing Chen, Kuangen Zhang. Haiyuan Liu. Yuquan Leng, and Chenglong Fu. 2021. A probability dis- tribution model-based approach for foot placement prediction in the early swing phase with wearable imu sensor, Neural Systems and Rehabilitation Engi- neering.
7 Ethics and Broader Impacts
We hereby acknowledge that all of the co-authors of this work are aware of the provided ACL Code of Ethics and honor the code of conduct. We state the ethical considerations and the potential impact the community as follows.
Dima Damen, Adriano Fragomeni, Jonathan Munro, Toby Perrett, Daniel Whettam, Michael Wray, An- tonino Furnari, Giovanni Maria Farinella, and Da- vide Moltisanti. 2021. kitthhes---0-22000 chal- lenges report.
Datasets. The datasets we use train our model (Ego4D, Aria) are the commissioned data that are released publicly. Both of the datasets note that the consent forms and/or release forms are collected for all videos, and that the majority of videos and data are de-identified pre-release.
Daniel de Vassimon Manela, David Errington, Thomas Fisher, Boris van Breugel, and Pasquale Minervini. 2021. Stereotype and skew: Quantifying gender bias in pre-trained and fine-tuned language models. In EACL 2021-16th Conference of the European Chap- ter of the Association for Computational Linguistics, Proceedings of the Conference.
In our code release, we provide the pre- processing scripts for the 3rd party datasets above (along with the links to download these datasets). We note that the pre-processed data (ready for train- ing) would not contain any of the meta information that is not needed for the purpose of this work such as date or location of the recordings.
Jacob Devlin, Ming-Was Wei Chang. Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training deep bidirectional transformers for language understand- ing, arXiv preprint 1rXiv:/810.04800)
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannanch Hajishirzi, and Noah Smith. 2020. Fine-tuning pretrained language models: Weight ini- tializations, data orders, and early stopping. arXiv preprint arXiv:2002.0633 06300
Applications. The models we train and the tasks that we propose are inherently made for on-device use cases where the model operates on the data that are already publicly released (such as the pub- lic datasets above) or with an explicit consent, thus posing minimal risks.
John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research.

---

Luciano Floridi and Massimo Chiriatti. 2020. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30(4):681-694.
Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison. Andreas Kopf, Edward Yang, 7achary DeVito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, perforrefrre deep learning library. In Neural Information Processing Systems.
Kristen Grauman, Andrew Westbury, Eugene Byme, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Nija Ra- dosavovic. Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Bric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra. Vincent Cartillier, Sean Crane, Tien Do Morrie Doulaty, Akshay Erapalli, Christoph Feicht- enhofer, Adriano Fragomeni, Qichen Fu, Christian Fuegen, Abrham Gebreselasie, Cristina Gonzalez, James Hillis, Xuhua Huang. Yifei Huang, Wenqi Jia, Weslie Khoo, Jachym Kolar, Satwik Kottur, Anurag Kumar, Federico Landini. Chao Li, Yanghao Li, Zhenqiang Li. Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ra- mazamova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang, Xindi Wu, Takuma Yagi, Yunyi Zhu, Pablo Arbelaez, David Crandall, Dima Damen, Gio- vanni Maria Farinella, Bernard Ghanem, Vamsi Kr- ishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Ki- tani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. 2022. Ego4d: Around the World in 3,000 Hours of Ego- centric Video. In JEEF/CVF Computer Vision and Pattern Recognition (CVPR).
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2 Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML).
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021b. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML).
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language under- standing by generative pre-training.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, lly Sutskever, et al. 2019. Language models are unsupervised multitask learners.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: State-of-the-an natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations.
Yujian Jiang, Lin Song, Junming Zhang, Yang Song, and Ming Yan. 2022. Multi-category gesture recognition modeling based on semg and imu signals. IEEE Sensors Journal.
Yeon-Wood Kim, Kyung-Lim Joa, Han- Young Jeong, and Sangmin Lee. 2021. Wearable imu-based human activity recognition algorithm for clinical balance assessment using Id-enn and gru ensemble model. IEEE Sensory Journal.
Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello. 2022. Wav2elip: Learning robust audio representations from clip. In IEEE In- lernational Conference on Acoustics, Speech, and Signal Processing (ICASSP).
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei. Nan Duan, and Tianrui Li. 2021. CLIP4Clip: An empirical study of clip for end to end video clip retrieval. arXiv preprint arXiv:2104 204.008800
Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vin- cent Vanhoucke, et al. 2022. Socratic models: Com- posing zero-shot multimodal reasoning with lan- guage. arXiv preprint 2204:2200000999
Zhaoyang Lv, Edward Miller, Jeff Meissner, Luis Pesqueira, Chris Sweeney, Jing Dong, Lingni Ma, Pratik Patel, Pierre Moulon, Kiran Somasundaram, Omkar Parkhi, Yuyang Zou, Nikhil Raina, Steve Saarinen, Yusuf M Mansour, Po-Kang Huang, Zi- jian Wang, Anton Troynikov, Raul Mur Artal, Daniel DeTone. Daniel Barnes, Elizabeth Argall, Andrey Lobanovskiy, David Jacyun Kim, Philippe Boutte- froy, Julian Straub, Jakob Julian Engel, Prince Gupta, Mingfei Yan, Renzo De Nardi, and Richard New- combe. 2022. Aria pilot dataset.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint 2rXh::205..01665
Shaghayegh Zihajehzadeh and Edward Park. 2017. A gaussian process regression model for walking speed estimation using head-wom imu. In International Conference of the IEEE Engineering in Medicine and Biology Society (FMBC).
Adam Paszke, Sam Gross. Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor

---

A Additional Experiments
Loss Function
MRR
R50
A.1 Ablation Studies over Loss Functions
InfoNCE loss
0.123
60.42
Triplet Loss
0.118
59.53
We survey the performance of the modality align- ment via different loss functions: InfoNCE con- trastive loss (used throughout our main experi- ments), Triplet loss with random sampling for neg- ative pairs, and Mean-Squared Error loss (MSE).
MSE Loss
0.101
48.55
Table 4: Ablation studies with varying loss functions on the IMU-Texx Retrieval task. Model: Stacked ID- CNN-RNN. Bold denotes the best performance.
A.2 Ablation Studies IMU Architecture
IMU Encoder Architecture
MRR
R50
We survey the performance of IMU encoders with varying architectures (see Section 2 for the list of IMU baselines in the litcrature), and use the best architecture throughout the rest of the experiments.
Stacked ID-CNN-RNN (proposed)
0.123
60.42
IxCNN+AAtentionRRNN
0.112
57.18
Patch Transformer
0.109
54.15
Patch Bi- Bi-RNN
0.112
56,21
Patch RNN
0.104
50.83
As seen Table 5, our proposed ID-CNN-RNN stacked model outperforms other baselines in the main target task (IMU to Text retrieval). See Sec- tion 3 for the detailed description the model.
Table 5: Ablation studies for varying IMU encoder archi- tectures the IMU~Text Retrieval task. Bold denotes the best performance.
A.3 I Video Retrieval
IMU Query
Vinoo Recival from DITISTLPP
We propose an auxiliary task of Video Retrieval based on IMU, specifically targeted for Aria data, which does not have text narrations annotated. The goal for the task is to retrieve videos based on IMU signals, allowing for intuitive way of analyzing motion signals data. We measure the performance the held-out test set, using the IMU signals as queries and the videos as the retrieval target.
RE
Figure 4: Video Retrieval based IMU (IMU~+Video). (Left): IMU signals and (Top): their corresponding ground-truth video. (Bottom): IMU2CLIP model prediction of their corresponding video from the Ego4D test set (top-1), given the IMU signals. can be seen that the videos retrieved based on IMU are visually and semantically similar to the gold video.
We search for videos, given IMU record- ings. Table 6 shows the performances on the IMU+> Video retrieval task on Ego4D and Aria datasets. We observe higher recall performances in general for both datasets, showing that the IMU sig- nals and videos have higher compatibility. When
the model is trained on all three modalities, we ob- serve competitive results across all tasks, while the best performances are from the bi-modal models aligned with each respective task.
we pre-process each media to have equal-sized par- allel windows (IMU <> Video < Text). The data module retrieves the parallel data of a requested window size a given timestamp, and caches them for faster training. In addition, to accommodate the memory constraints, we pool the negative sam- ples within the same batch (randomly shuffled), reducing the load on each GPU.
Fig. 4 shows illustrative examples, which demon- strate that the videos retrieved based on IMU are visually and semantically similar to the gold video.
B Implementation Details
B.1 Training Details
B.2 Hyperparameters
Model Freezing. To preserve the text-vision align- ment that CLIP already exhibits, we freeze the parameters of the image and text CLIP encoders during training. See Fig. 5 for an illustration.
Table 7a and 7b report the in this work for model training and their search bounds, respectively. We optimize the parameters with Adagrad (Duchi et al., 2011) 1) with epsilon 10 and decay 0.1.
used
Memory Constraints. To expedite the training,

---

Train Modalities
IMU
Video
Video
IMU
Ego4D
IMU
Video
Text
R@1
R@ 10
R@:0
MRR
R@1
R@ 10
R@:50
MRR
9,06
43.13
78.75
0.2011
12.19
45.31
80.00
0.226
3.75
25.94
62.81
0.105
3.75
24.06
56.88
0.098
8.75
40.63
73.44
0.183
11.56
42.19
75.94
0.213
Aria
7,58
40.62
83,92
0.181
7.58
45.08
83,92
0,194
Table 6: IMU retrieval performances of the pre-trained IMU2CLIP models Ego4D (top) and Aria (bottom), with different modalities used for training. The last row shows the video retrieval performance of OpenAl's CLIP model on the same test set.
IMU2CLIP (Video + IMU)
CLIP
IMU2CLIP (Text + IMU)
Contrastive Loes
Contrastive Loss
Contrastive Loss
IMU Encoder
Video Encoder
Video Encoder
Text Encoder
Text Encoder
IMU Encoder
Acc. Gyro VMA
Driving car on a road.
Moves the the table
Acc. Gyro MAYMN
VMA
WW
VM
WMA MMA
Figure 5: Illustration of the proposed multimodal contrastive learning for IMU2CLIP. CLIP (Radford et al., 202 is used align IMU+ Mideo (left), and IMU++Text (right). training.
the parameters of the encoder frozen during
Gradient Accu-
Models
Batch Size
Initial LR
# Epochs
# Params
mulation Steps
Stacked ID-CNN-RNC (IMU+Text) Stacked ID- AD-CNN-NE RNN (IMU+Vidco
16
1.7M
16
5 x 10
1.7M
Stacked ID-CNN-RNN (IMU+Text+ Video)
16
5 x 10
16
1.7M
(a) Hyperparameters Pre-training
Type
Batch Size
Initial I.R
Training Epochs
Gradient Accumulation Steps
Bound (lower-upper)
4-32
5
55 10-3
6-10
1-1
Number of Trials
5
(b) Search Bounds
Table 7: (a) Hyperparameters in this work: Initial LR denotes the initial learning rate. All the models trained with Adagrad optimizers (Duchi et al., 101 We include the number of learnalle parameters of each model in the column: porams. (b) Search bounds for the hyperparameters of all the models.
B.3 Code Base & Hardware
availability) on Ubuntu 20.04.2 operating system.
The implementations the transformer-based mod- els are extended from the HuggingFace? code
C Supplementary Materials
base (Wolf et al., 2020) and other cited authors' released code-bases. Our entire code-base is im- plemented PyTorch (Paszke et al., 2019). All models in this work are trained on a varying num- ber of Nvidia A100 3 GPUs (1-8 depending on the
We also provide the animated GIF visualizations (as a zipped file) for all of the illustrative tasks men- tioned in this manuscript, as part of the uploaded supplementary materials.


Reviews:
[{"review_id": "fJeSH7E0qE", "reviewer": "Reviewer_X66w", "summary": "", "questions": "N/A", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive comments and suggestions. We are appreciative of their acknowledgement of the novelty of the approach and its new application, and the strong performance on activity recognition.\n\n**[Exploration of other benchmarks.]**\n\nWe note that we report results on two datasets: Ego4D and Aria, which are the biggest & the latest datasets comprising IMU motion sensors (670+ hours total), covering diverse and unconstrained motions of multiple wearers (massive, unprecedented scale). On the other hand, the existing datasets are typically an order of magnitude smaller in size (<2 hrs of recording) and focused on constrained motion types. We therefore believe that the benchmark results we report on Ego4D and Aria can give a more generalized view on the performance than results on any other existing IMU datasets. To measure the performance improvement on focused studies, however, we will include results on the following publicly available IMU activity datasets [1-4] as well, in addition to Ego4D and Aria.\n\n- Ego4D: 923 participants, 540 hours, unconstrained motions\n- Aria: 138 hours\n- [1]: 8 participants, ~2 hours\n- [2]: 17 participants, ~2 hours \n- [3]: 114 participants, ~0.5 hour\n- [4]: 35 participants, ~1 hour\n\n[1] Kasebzadeh, Parinaz, et al. \"IMU dataset for motion and device mode classification.\" 2017 International Conference on Indoor Positioning and Indoor Navigation (IPIN). IEEE, 2017.\n[2] Casilari, Eduardo, Jose A. Santoyo-Ramón, and Jose M. Cano-García. \"Umafall: A multisensor dataset for the research on automatic fall detection.\" Procedia Computer Science 110 (2017): 32-39.\n[3] Nadeem, Adnan, Amir Mehmood, and Kashif Rizwan. \"A dataset build using wearable inertial measurement and ECG sensors for activity recognition, fall detection and basic heart anomaly detection system.\" Data in brief 27 (2019): 104717.\n[4] Ribeiro De Souza, Caroline, et al. \"A public data set of videos, inertial measurement unit, and clinical scales of freezing of gait in individuals with parkinson's disease during a turning-in-place task.\" Frontiers in Neuroscience 16 (2022): 832463.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Interesting idea to connect motion from IMU with text and video representations. \n2. Propose a low-resource way for media search. \n3. Demonstrate strong performance on activity recognition when utilizing the pre-trained IMU2CLIP representation. ", "reasons_to_reject": "1. IMU sensor information has been used in multiple other motion recognition tasks. It's underexplored whether the proposed IMU2CLIP can achieve better performance on other established benchmarks. ", "questions_to_authors": "N/A", "paper_topic_and_main_contributions": "This paper proposes IMU2CLIP, where they pre-train a text/video-aware motion representation. Specifically, this paper utilizes contrastive loss to align the motion from Inertial Measurement Unit to text and video data. Empirical results show that the IMU2CLIP representation can be used for media search with low resources. Besides, fine-tuning with IMU2CLIP significantly outperforms baseline models trained from scratch on activity recognition tasks.  ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "UOpEb072mQ", "reviewer": "Reviewer_JXWg", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive comments and suggestions. We are appreciative of their acknowledgement of the novelty of the application, and the strong performance on downstream tasks.\n\n**[Novelty of the proposed approach.]**\n\nWe would like to highlight that the innovation within this study resides in the investigation of aligning a novel modality (IMU) through contrastive learning. Our exploration encompasses:\n- The design of an effective IMU encoder architecture that demonstrates strong performance in downstream tasks. Our ablation study (Table 5) shows that the proposed architecture outperforms other baseline architectures from the literature.\n- The alignment of IMU data within a shared language-vision space.\n- Pioneering applications based on IMU-LLM, showcasing reasoning capabilities while upholding privacy preservation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. From an application perspective, aligning text, video and IMU signal is interesting and is relatively unexplored.\n2. Motivates using a lighter alternate to video modality for text-based cross-modal task.", "reasons_to_reject": "1. Pre-training method is not novel, its an extension of contrastive feature loss extended to three different modalities.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work presents a pre-training approach to align IMU sensor, video and text modality present in video data. The author optimizes contrastive loss between feature pairs from different modality, similar to the one used to pre-train CLIP, to align different modalities in a shared space. To show the effectiveness of their pre-training scheme in aligning representation from different modalities, the authors benchmark different cross-modal retrieval task.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pBYh5PJ3O0", "reviewer": "Reviewer_b6Gx", "summary": "", "questions": "1. If the task is to unify video and text modalities, why is CLIP, not any video-text pre-trained models being used? How is CLIP used here to extract video embedding?\n\n2. As you pointed out, LN184 \"For Ego4D, a subset of the clips are also annotated with text narrations\", how does the pre-training run differently on examples with and without text narrations? ", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive comments and suggestions. We are appreciative of their acknowledgement of the novelty of the application, and the strong performance on downstream tasks.\n\n**[Re: “It is unclear what specific challenges are being solved in the work”]**\n\nThis work addresses two crucial aspects for wearable device AI:\n\n- User privacy: Video recording raises privacy concerns as it's often intrusive and unavailable in sensitive situations, whereas IMU data remains non-intrusive.\n- Power consumption: In comparison to video recording, IMU signals are lightweight, resulting in significantly lower power consumption at inference time.\n\nOur exploration encompasses:\n- The design of an effective IMU encoder architecture that demonstrates strong performance in downstream tasks. Our ablation study (Table 5) shows that the proposed architecture outperforms other baseline architectures from the literature.\n- The alignment of IMU data within a shared language-vision space.\n- Pioneering applications based on IMU-LLM, showcasing reasoning capabilities while upholding privacy preservation.\n\n**[Cascaded retrieval.]**\n\nThe reviewer proposes comparison of our IMU → Text retrieval task with a “cascaded retrieval” approach of IMU → Video → Text.\n\nWhile we agree that it is an interesting approach to consider, the approach lacks practical merits in the context of Wearable on-device AI, where lightweight power computation level & privacy are of utmost consideration (as opposed to cloud-based computations).\n\n- The cascaded approach assumes availability of a pool of videos as retrieval targets, which faces constraints such as (1) the memory limit of a wearable device, (2) user privacy considerations (around storing and accessing visual information).\n- The computation of vision embeddings typically requires much higher power and memory consumption levels than computing IMU embeddings. \n\nWe will add more discussions and considerations on Wearable AI in the final manuscript as above, and also include new results comparing the cascaded approach.\n\n**[CLIP vs. Other Video-Text Pre-trained Encoders.]**\n\nWe base our rationale for using CLIP for getting video representations as follows:\n- We focus on using the lightest model possible to encode vision signals at inference time, to match the computational budget we have for on-device use cases of our model. \n- [1] reports competitive performance with the CLIP-based approach using just a single frame, compared to several strong video-text models.\n\nIn the final manuscript, we will include discussions on different choices of video encoders in terms of pre-training performance, and their respective computational requirement. \n\n[1] Portillo-Quintero et al., \"A straightforward framework for video retrieval using clip\", 2021.\n(https://arxiv.org/pdf/2102.12443.pdf)\n\n**[Pre-training schema.]**\n\nWe split each dataset for upstream (alignment training) and downstream tasks (activity recognition). For IMU training, we only use the subset of the Ego4D data (540 hours out of 3,670 hours) that have text narrations, IMU and video alignment.\n\n**[Question Answering on Sensor Logs.]**\nThank you for your suggestions. We will move this discussion to a new section, and add more qualitative analysis, page limit permitting.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Alignment across IMU and other modalities is relatively unexplored before and the proposed approach achieves good results in the main benchmarks, verifying the feasibility of unifying these modalities.", "reasons_to_reject": "Although the idea about aligning more different modalities is important, the proposed approach is lacking bit in novelty. It is unclear what specific challenges are being solved in the work, on top of existing contrastive learning frameworks. Especially I feel IMU input modality may only be considered as a small set of concepts (e.g., say movements) that can be covered by video and text. Results from Table-2 also verified that video-text alignment is already done in good by CLIP, now how would it compare, e.g., if you do IMU->video retrieval followed by plain video->text, to that IMU->text in Table-2?\n\nThe definition of pre-training can be made more clear here. Which datasets are used for pre-training. LN203 “… evaluate on the held-out test set … “, seems to say both pre-training and the evaluation are actually on the same datasets.\n", "questions_to_authors": "1. If the task is to unify video and text modalities, why is CLIP, not any video-text pre-trained models being used? How is CLIP used here to extract video embedding?\n\n2. As you pointed out, LN184 \"For Ego4D, a subset of the clips are also annotated with text narrations\", how does the pre-training run differently on examples with and without text narrations? ", "paper_topic_and_main_contributions": "This paper introduces a pre-training approach to align 3 modalities of data including IMU (Inertial Measurement Unit, used for capturing movements and orientations) and two common ones of video and text. Video and text inputs are encoded with pre-trained CLIP model, while the IMU input is encoded by customized stacks of CNN and RNN layers. All inputs, after being encoded, are unified by two contrastive losses conjugating the pair of IMU and video, as well as IMU and text. The authors evaluated the trained model on two main tasks (1) retrieve IMU from text (2) IMU to actions, and results show improvement from incorporating both video and text modality in the pre-training.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "LN210 Task3 seems to be kept out as a future work. Maybe it suits better in other sections instead of Experiments."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method to learn to map wearable IMU data to the same space as text and video data, and shows that the method can be used for retrieval tasks across text, IMU and video modalities.

Strengths:
- All reviewers highlighted that aligning IMU with other modalities is an interesting idea and not explored before.
- Most reviewers commended strong results on the benchmarks, which show that the method works.
- Most reviewers agree that the method is a low-resource way to do media search (the wearable device can be low-power and low bandwidth).

Weaknesses that may have been addressed in rebuttal:
- One reviewer cited lack of comparison to prior motion datasets. The authors mentioned that this was because prior datasets are too small (<2 hours activity data), but they will run the experiments and include these results anyway at the request of the reviewer.
- R b6Gx raised a number of concerns and did not elaborate which were addressed during rebuttal. From my reading, it seems the concerns are addressed.

Overall, there is consensus that the paper studies an interesting and novel idea, and proposes a method that works, as substantiated by the experiments. 

Overall: Sound and moderately exciting.
Decision: Accept-Findings