Unsupervised Modality Adaptation in Human Action Recognition via Cross-modal Representation Learning

Authors: abhi  kamboj
anh duy nguyen
minh  do
Venue: NeurIPS_cc_2024_Workshop_UniReps
Published: 2024-09-22 03:06:57

Abstract: Despite living in a multi-sensory world, most AI models are limited to textual and visual interpretations of human motion and behavior. In order to unlock the potential of diverse sensors, we investigate a method to transfer knowledge between modalities using the structure of a unified multimodal representation space for human action recognition (HAR). We introduce an understudied cross-modal transfer setting termed Unsupervised Modality Adaptation (UMA), where the modality used in testing is not used in supervised training. We develop three methods to perform UMA: Student-Teacher (ST), Contrastive Alignment (CA), and Cross-modal Transfer Through Time (C3T). Extensive experiments on various camera+IMU datasets demonstrate ST is effective on simple tasks, CA is the most modular and balanced method and C3T is the most robust through temporal noise. In particular, our C3T method introduces novel mechanics of aligning a signal across time-varying latent vectors, and we show that it demonstrates unique robustness to time-related noise, suggesting its potential for developing generalizable models for time-series sensor data.

Main paper: 
 introduction: 
 Motivation: Humans can naturally actuate a motion they have only seen before; however, transferring motion knowledge across sensors for machine learning models is nontrivial. Our interaction with computing has historically been centered around visual and textual modalities, which has provided these models an abundance of data. Thus, deep learning based human action recognition (HAR) systems often collapse 3D motion into related but imprecise modalities such as visual data [19, 35, 23, 40] or language models [32, 41, 37, 30, 11] . Sensing modalities in wearables (e.g. IMU, ECG, PPG, etc.) provide a salient signal to perform HAR, however, the data is less abundant and difficult to label.This raises the critical question of how to integrate new sensors with existing ones in the absence of labeled data. One promising solution is to leverage a well-documented modality to transfer knowledge to another modality, a process known as cross-modal transfer [28] , ideally without additional human annotation effort. Existing cross-modal learning techniques assume semi-supervised or fully supervised settings. Cross-modal learning has not thoroughly been investigated in a setting where one modality is completely unlabeled during training. We refer to this as Unsupervised Modality Adaptation (UMA), similar to the widely used setting of Unsupervised Domain Adaptation [24] where the domain shift is a new modality. Contributions: In order to perform UMA, we use the intuition that there exists some joint multimodal representation space for HAR, that can be leveraged to infer the same action across different modalities. We propose 3 methods for this to extract and leverage this latent space. The first is a student teacher (ST) method akin to existing knowledge distillation methods for other domain adaptation or semi-supervised settings (Figure 2a ). The second method performs contrastive alignment (CA) on latent representations of multimodal unlabeled data samples and uses a shared task head to perform transfer (Figure 2b ). The third method extracts a time-varying latent dimension, i.e. a set of t rec latent vectors, and performs Cross-modal Transfer Through Time (C3T) (Figure 2c ).We test these methods on Inertial Measurement Units (IMUs) and RGB video data on four datasets. Although the ST method works best on simple datasets, CA performs better in more difficult visual tasks. This indicates latent space alignment captures hidden correlations allowing the model to leverage one modality to infer a structure in the other. Furthermore, C3T consistently performs the best and is the most robust to time-shift, misalignment, and time-dialation noise as it accounts for the temporal information within each data sample. This investigation of UMA cross-modal transfer lies at the intersection of transfer learning, multimodal representation learning and holds significant implications for the applicability of machine learning in more diverse, underexplored, modalities. 
methods: 
 We construct the Unsupervised Modality Adaptation (UMA) setting with RGB videos as the source of labeled data and IMU data as the target unlabled modality. As shown in Figure 1 , a system can easily record synchronous data between these modalities, and then leverage an RGB model to perform HAR with only the IMU data. We mimic this setting by dividing 4 existing multimodal datasets into 4 splits as shown in Table 1 . Training for each method occurs in two phases: a) Supervised Learning with RGB data on D HAR and b) Unsupervised Alignment across both modalities on D Align . Inference can occur, with any combination of the input modalities (Table 2 ), however, we focus testing on IMU data (Table 3 ). D V al was used for hyperparameter search during training, and all tables report the average accuracy over three trials on D T est . We propose three methods for transferring knowledge to a new sensing modality without exposure to labels in that modality, as depicted in Figure 2 :(1) Student-Teacher (ST) Figure 2a : The ST method leverages an RGB video model trained in phase a) to produce psuedo-labels to train the IMU model in phase b). In this case, the latent space is the output logit space aligned using the cross-entropy loss, Equation (1) . Various student-teacher models have been proposed [21, 39, 3, 36] ; however, these models often assume the availability of student-teacher-labeled modality pairs during training to distill knowledge from the teacher to the student. Furthermore, we use only one student and teacher module distinguishing our method from Thoker and Gall [36] who require an ensemble to strengthen the model in a similar setting.(2) Contrastive Alignment (CA) Figure 2b : The CA method performs phase a) Supervised RGB training in the same fashion as the student teacher, however, it uses a model with 2 parts: An encoder f (1) to extract the latent variable z, and a task specific MLP head h. The extracted latent space Z allows for scalability and interoperability of adding different sensing modalities, types of encoders, and output task heads. Phase b) performs unsupervised contrastive alignment with the outputs of the the RGB encoder f (1) and the IMU encoder f (2) on unlabeled data using the symmetric contrastive loss formulation from [32, 27, 13] given by Equation ( 2), further detailed in appendix. The symmetric contrastive loss will cluster representations in Z by cosine similarity, which brings about the desired property of the latent space that vectors of the same class are near each other. A unified representation space for separate modalities allows the decision boundary trained on RGB representations, h, to be used to recognize actions on IMU representations. This latent space is visualized in Figure 3 .(3) Cross-modal Transfer Through Time (C3T) Figure 2c : ST and CA do not leverage the temporal information of the data, making them difficult to use in real-world settings. C3T removes the final linear layer from the feature encoders of CA and uses the output of the temporal convolutions directly. This temporal receptive field would have extracted the salient features of neighboring time steps of the data. Then during the alignment phase, each of these time vectors is aligned with the same time vector from the other modality, using the same contrastive loss CA uses (Equation ( 2)). When training the HAR model, we use self attention with a learned class token to predict the action, similar to the ViT architecture [9] , but instead of inputs being image chunks, they are temporal feature chunks. The intuition is that the encoder will learn which tokens over time are the most informative for the action class and predict accordingly.The cross-modal zero-shot transfer method is the main result of this paper and described above in Appendix C. Here the estimated latent vector is given by f (2) (x (2) i ) = ẑi. 3. Fusion (Sensor Fusion) When both modalities are present, the model estimates the latent vector ẑi from the outputs of modality-specific encoders assuming each estimate is equally as good as the other. ẑi = (2) i 
experiments: 
 How do we train the CA and C3T Architectures? We experimented with four ways to perform the two phases of training, as shown in Table 2 . 1) Align First: First aligns the representations We hypothesize that in method 2) training the HAR model first yields a latent space to capture the best HAR features for RGB data, which is not directly applicable to IMU data. Method 3), faced instability in training and was unable to converge. Method 4), performed better than 1) for CA potentially since one loss acted as a regularizer for the other pushing the latent space Z to the ideal balance for cross-modal transfer in HAR. The other experiments in this work use training method 1).Can UMA methods retain performance on the original modality they were trained on? Can they leverage both modalities? Table 2 shows the result of training in the UMA setting, but testing with all combinations of the modalites: 1. RGB uses the RGB encoder and HAR module, 2. IMU (Zero-Shot Transfer) uses the IMU encoder and the HAR module, and 3. Fusion performs feature-level fusion (or late fusion for the ST method), by averaging the outputs of each of the encoders. Surprisingly for C3T, fusing (highlighted in red) performs better than RGB, indicating that unlabeled IMU data may add structure to the shared latent space to boost performance.Can our methods perform well on diverse RGB-IMU datasets? We test on a small yet structured dataset (UTD-MHAD [8] ), a slightly larger dataset captured in a controlled environment with dense inertial data, i.e. 10 on-body sensors (CZU-MHAD [7] ), one very large dataset with viewpoint and occlusion challenges (MMACT [21] ), and one large egocentric camera dataset (MMEA-CL [43] ). More details on the baselines and datasets are given in Appendix D.2 and Appendix D.1 respectively. The results (Table 3 ) show large performance gaps indicating C3T is far superior to the other methods. Furthermore, we compare the UMA performance against the Supervised methods and show C3T even outperforms supervised RGB on some datasets where the IMU data is highly informative.How robust is our method to time shifts and noise? Table 2 demonstrates C3T's superiority in the presence of temporal noise. We test on three real life noisy scenarios: 1) Crop: Accounts for the continuous nature of real-time action recognition (i.e. there is no defined start and stop time) by randomly shifting and cropping the time sequence of both modalities by 60% 2) Misalign: Imitates slight hardware asynchrony or different device framerates of the multimodal system by performing crop but on only one of the modalities. 3) Dilation: Mimics an action being performed slower, by randomly cropping both modalities to 50% of their original size an then upsampling.Qualitative Results: We visualized the latent space outputs of the CA model using TSNE plots (Figure 3 ). These plots show training when the alignment phase (phase b in Figure 2 ) is performed first, and then labeled-RGB training (phase a) is performed. The model quickly segments classes during the align phase, even without labels, suggesting that the data's natural structure facilitates class distinction across different modalities. This implies that our methods could potentially adapt to new class labels during testing with just a few samples, as the latent structure would have already grouped similar classes. Furthermore, after alignment and HAR training, we notice how the model tends to misclassify points that are near the boundary between clusters. These visualizations support our initial hypothesis (Figure 1 ) on how a joint latent space could be leveraged to effectively perform UMA, by using a classification head trained only on RGB data.Interestingly, we observed that IMU data points consistently cluster towards the center of the plot, with RGB points surrounding them. This pattern persists even in early alignment stages, suggesting it's not solely due to labeled RGB HAR training. While this might indicate that RGB data is more informative, it contradicts our quantitative findings where supervised IMU models outperform RGB models for our given datasets. This phenomenon warrants further investigation as it may have implications for continual learning, test-time adaptation, or domain adaptation, where different modalities should be leveraged differently depending on their placement in the shared latent space.  
conclusion:: 
 In this paper, we motivated and explored the Unsupervised Modality Adaptation (UMA) framework for human action recognition which challenges models to perform inference with a modality that is unlabeled during training. We conduct experiments to determine how to construct a unified latent space between modalities, outline three methods to perform UMA with their constructed latent space and compare their strengths in various settings. We hope our results inspire others to exploit cross-modal latent spaces to integrate continuous time sensor signals into AI models for more robust human motion understanding. We conducted comprehensive ablation experiments on our model architecture (Table 5 ), comparing convolutional and attention modules for RGB (spatial and temporal) and IMU (temporal) encoders.Results generally favored convolutional architectures across various methods in our UMA setting. Notably, C3T's superior performance cannot be attributed solely to its attention head leveraging temporal information, as ST or CA models with temporal attention did not perform comparably well. Instead, C3T's effectiveness stems from its unique method of alignment in the temporal space.Further ablation on C3T head architectures (Table 4 ) compared the class token-based self-attention head with alternatives: concatenating output attention tokens and projecting, adding latent vectors Z 1 . . . Z trec and passing through an MLP, and concatenating vectors and using an MLP. The latter two methods do not use attention. While concatenating latent vectors and using an MLP performed best on clean data, the class token attention mechanism offered superior robustness to noise. The attention visualization in Figure 4 corroborates these findings, showing the class token approach's resilience to shift noise. In addition, we notice all C3T variants outperformed CA and ST in UMA performance (Table 3 ) on the UTD-MHAD dataset, emphasizing C3T's strength in temporal alignment, regardless of the classification head. 
 
 Appendix: We investigate the creation of a robust multi-modal latent space for human action recognition, denoted as Z. We assume that there exists a learnable projection f (k) from every modality k ∈ {1 . . . M } to this latent space Z, i.e. f (k) : X (k) → Z, and there exists a learnable mapping h from the latent space Z to the space of human actions Y, i.e. h : Z → Y.The critical intuition that drives our method is that for a point z i ∈ Z, any z j "near" z i is likely to map to the same class as z i , thus we can leverage the structure of Z to classify neighboring vectors regardless of which modality they are generated from, using the same decision boundary determined by h. In our experiments, we quantify closeness in terms of cosine similarity (and also perform some ablations with L2 distance metric).For simplicity, we experiment with 2 modalities M = 2 and assume n data points are split into 4 disjoint index sets I 1 ∪ I 2 ∪ I 3 ∪ I 4 ∈ {1 . . . n}. Under our cross-modal transfer setting, during training the model has access to 2 of these datasets. One contains labeled data for one modalityD HAR = {(x 1 i , y i )} I1 i=1and the other contains pairs of data between the modalities but these points are unlabeled:D Align = {(x 1 i , x 2 i )} I2 i=1. This is analogous to having an existing sensor with labeled data, and introducing a new sensor in which data can be synchronously collected, but there is no additional annotation effort. Now, if the sensor with labeled data is not present, can the model still perform inference with the unlabeled modality? The third and fourth sets are used for validation and testing and contain only labeled data from the second modality, i.e. D V al = {(x 2 i , y i )} I3 i=1 andD T est = {(x 2 i , y i )} I4 i=1 .We propose three methods for transferring knowledge to a new sensing modality without exposure to labels in that modality: (1) a student-teacher approach, (2) a contrastive alignment technique, and (3) a contrastive alignment through time method. Given our motivation of human action recognition, we experiment with RGB videos as the source of labeled data x (1) = x (RGB) and IMU data as the target datax (2) = x (IM U ) .Training for each method occurs in two phases: a) Supervised Learning with RGB data and b) Unsupervised alignment across both modalities. Inference can occur, with any combination of the input modalities or both, as shown in Table 2 , however, we mainly focus on the results of the IMU test, shown extensively in Table 6 . All tables report the Top-1 accuracy on the D T est for each method is given by: accuracy = 1 M M i=1 1 ŷi=yi To ensure the robustness and reliability of our empirical results, we conducted each experiment three times using different random seeds. The reported accuracies are averaged across these trials, providing a more rigorous and representative assessment of the model's performance.Each of the methods below uses a combination of the following three neural network modules: (i)Video Feature Encoder f (1) : X (1) → Z: This module applies a pretrained Resnet18 to every frame a video and then performs a single 3D convolution and a simple 2-layer feed forward network (FFN) with ReLU activations. (ii) IMU Feature Encoder f (2) : X (2) → Z: This module consists of a 1D CNN followed by a FFN. (iii) HAR Task Decoder h : Z → Y: This is a simple FFN. The student teacher method leverages a trained video model to produce psuedo-labels to train the IMU model. In this case there is no intermediate latent space and the model maps each modality's encoder directly to the output task. This essentially uses cross-entropy to align the models directly in the label space: Z = Y.Various student teacher models have been proposed [21, 39, 3, 36] . However, these models often assume the availability of student-teacher-labeled modality pairs during training to distill knowledge from the teacher to the student when updating the corresponding losses. Furthermore, we use only one student and teacher module distinguishing our method from Thoker and Gall [36] who require an ensemble of to strengthen the model in a similar setting.We denote the teacher network as f (1) : X (1) → Y and the student network as f (2) :X (2) → Y.First, we train the teacher f (2) on D HAR with labeled RGB data. Next, in order to train f (1) on D Align , we first useEQUATIONi ) = ŷ1 to generate psuedo-labels for every datapoint i ∈ I 2 . Then we use the augmented dataset DAlign = {(x 1 i , x 2 i , ŷi )} I2 i=1 to train f (1) . Both student and teacher networks were trained using cross-entropy loss:EQUATIONwhere ŷi = f (x i ) is the output of the ith sample in the batch of N samples, ŷi,j is the score for the jth class out of C classes, and P y represents the probability distribution produced by a given model's output logits. The teacher network minimizes L CE (P f 2 (x) , P y ) student network minimizes L CE (P f 1 (x) , P f 2 (x) ). Since the student approximates the teacher and the teacher approximates the true distribution, this implies that the student can only be as good as the teacher at approximating the true distribution: L CE (P f 1 (x) , P y ) > L CE (P f 2 (x) , P y ). Thus given noise in the teacher distribution the student also suffers in performance, as shown in our experiments section. Furthermore, given that the latent space is the label distribution, this method provides little flexibility in extending to various modalities and tasks.The contrastive alignment method performs phase a) Supervised RGB training in the same fashion as the student teacher, however, it uses an a model with 2 parts: An encoder f (1) to extract the latent variable z, and a task specific MLP head h. The extracted latent space Z allows for scalability and interoperability of adding different sensing modalities, types of encoders, and output task heads.Phase b) now performs unsupervised contrastive alignment with the outputs of the the RGB encoder f (1) and the IMU encoder f (2) on unlabeled data. To align different modalities in the feature space on D Align we use a symmetric contrastive loss formulation LCL [32, 27, 13] with temperature parameter τ :LCL = - 1 N N i=1 log exp(⟨ẑ (1) i , ẑ(2) i ⟩/τ ) N i=1 exp (⟨ẑ (1) i , ẑ(2) i ⟩/τ ), where ẑ(kEQUATIONThe symmetric contrastive loss will cluster representations in Z by cosine similarity, which brings about the desired property of the latent space that vectors of the same class are 'near' each other. This method clusters similar representations in the latent space despite being from separate modaliteis, and thus the decision boundary trained on RGB representations, h can be used to recognize actions on IMU represntations.The previous two discussed methods do not leverage the temporal information in time continuos data signals, which make them difficult to use in real-world settings, where there may be temporal noise (shift/misalignment) and the actions may occur over longer or shorter periods of time. We thus propose a Cross-modal Transfer Through Time (C3T) model that leverages the temporal information of sensing modalities when aligning and fusing their representations. C3T remove's the FFN from the feature encoders and use the output of the temporal convolutions directly. This temporal receptive field would have extracted the salient features of neighboring time steps of the data. We use each of these time steps as the a zt latent vector. Then during the alignment phase, we align each of these time vectors with the same time from the other modality. When training the HAR model, we use self attention with a learned class token to predict the action The intuition is that the encoder will learn which tokens over time are the most informative for the action class and predict accordingly. This is a common method to perform classification with transformers [9, 4, 26] . Our implemented HAR task head is most notably similar to the ViT architecture [9] , but instead of inputs being image chunks, they are temporal feature chunks.The updated modules are as follows:Video Feature Encoder f (1) : X (1) → Z trec : This module applies a pretrained Resnet18 to every frame a video and then performs a single 3D convolution. The resulting output is a set trec z:EQUATION1 . . . ẑ(1) trec ). IMU Feature Encoder f (2) : X (2) → Z trec : This is a 1D CNN that decreases the time dimension to trec, resulting in an output of Ẑ( 2) = (ẑ (2) 1 . . . ẑ(2) trec ). HAR Task Decoder h : Z trec → Y: This module is like a transformer encoder that uses self-attention on an input sequence of length trec vectors appended with a learned class token. The output class token of the self attention layer is then passed through a FFN and outputs a single action label yi.We present results on small yet structure dataset (UTD-MHAD), one larger dataset captured in a controlled environment (CZU-MHAD), one very large dataset with various challenges (MMACT), and one egocentric camera dataset (MMEA-CL). For each of these datasets we create an approximately 40-40-10-10 percent datasplit for the D Align , DHAR, D V al , and DT est splits respectively as shown in Table 7 .UTD-MHAD Most of the development and experiments were performed on the UTD-Multi-modal Human Action Dataset (UTD-MHAD) [8] . This dataset consists of roughly 861 sequences of RGB, skeletal, depth and an inertial sensor, with 27 different labeled action classes performed by 8 subjects 4 times. The inertial sensor provided 3-axis acceleration and 3-axis gyroscopic information, and all 6 channels were used for in our model as the IMU input. Given our motivation, we only use the video and inertial data; however, CA can easily be extended to multiple modalities.The Changzhhou MHAD [7] dataset provides about 1,170 sequences and includes depth information from a Kinect camera synchronized with 10 IMU sensors, each 6 channels, in a very controlled setting with a user directly facing the camera. We concatenate the IMU data to provide a 60-channel input as the IMU modality and use depth as the input modality. Given the controlled environment and dense IMU streams, the models performed the best on this dataset.MMACT The MMAct dataset [21] is a large scale dataset containing about 1,900 sequences of 35 action classes from 40 subjects on 7 modalities. This data is challenging because it provides data from 5 different scenes, including sitting a desk, or performing an action that is partially occluded by an object. Furthermore, the data was collected with the user facing random angles at random times. The dataset contains 4 different cameras at 4 corners of the room, and it measures acceleration on the user's watch and acceleration, gyroscope and orientation data from a user's phone in their pocket. We only use the cross-view camera 1 data, and again we concatenate the 4 3-axis inertial sensors into one 12 channel IMU modality.The multimodal egocentric activity recognition dataset for continual learning (MMEA-CL) is a recent dataset motivated by learning strong visual-IMU based representations that can be used for continual learning. It provides about 6,4000 samples of synchronized first-person video clips and 6-channel accelerometer and gyroscope data from a wrist worn IMU. The dataset's labels features realisitc daily actions in the wild, as opposed to recorded sequences in a lab. Due to issues with the data and technical constraints, we downsize the data proportionally and use about 1,000 samples. Nonethless, CA's superior performance shows how this method can generalize to a different camera view, and different types of activities.Many works deal with robustness to missing sensor data during training or testing, however, few works deal with zero-labeled training data from one modality. As a result, constructing baselines was tricky and most methods had to be modified or adopted to fit our our approach.Various student teacher models have been proposed [21, 39, 3] . However, these models often assume the availability of student-teacher labeled modality pairs during training to distill knowledge from the teacher to the student when updating the corresponding losses. Thus most of these architectures are not directly applicable. Nonetheless, we borrow the basic concept of a teacher modality distilling knowledge to the student modality through psuedo labels, most similar to [36] .We denote the teacher network as g (t) : X (1) → Y and the student network as g (s) : X (2) → Y. Our designated student-teacher baseline uses DHAR to train g (t) on the RGB data. Next, in order to train g (s) on D Align , we first use g (t) (x (1) 1 ) = ŷ1 to generate psuedo-labels for every datapoint i ∈ I2. Then we use the augmented dataset DAlign = {(x1 i , x 2 i , ŷi)} I 2i=1 to train g (s) . We note that g (t) and g (s) have similar architectures to h(f (1) (•)) and h(f (2) (•)), respectively for a fair performance comparison. This student teacher baseline presents a strong solution for our setup, and gives a competitive performance in Table 6 . However, the only method it outperforms CA was was in the relatively easy CZU-MHAD dataset. One drawback, with this model is that it requires labeled data from both modalities to improve it's performance, whereas CA can use unlabeled data to learn some correlation between modalities in Z. This representation space can be used for other purposes as well, such as dynamically adding modalities or task specific heads.Many IMU-RGB based sensor fusion models have the ability to train on partially available or corrupted data and are robust to missing modalities during inference [18, 17] . No works have attempted the extreme case where one modality is completely unlabeled during training. Existing esnsor fusion methods can be adapted to our setup using a psuedo-labeling technique, similar to the student-teacher model above. The difference here is that the model learns a joint distribution between the two modalities so hopefully it may be able to learn some correlation between the models. Nonetheless, we show that these methods cannot generalize to the scenario where there is zero-labeled training data for one modality.Let g(•, •) : (X (1) , X (2) ) → Y. Our approach uses DHAR, to train by passing in zeros for one modality, e.g. we train g(•, 0) : X (1) → Y. Then, with D Align we use g(•, 0) to generated psuedo-labels and then train g(0, •, ) with those labels.We reproduced the conventional sensor fusion models (early, feature, and late) from [42] and indicate the performance of the top model on 6. We further reproduce a self-attention based sensor fusion appraoch (HAMLET [17] ) and tested it on our setup. We selected these model due to their state-of-the-art performance on the UTD-MHAD dataset, making them ideal benchmarks for comparison with our model.Given that there is no open source implementation for the HAMLET attention based sensor fusion method [17] , we reproduce it from scratch. We follow a very similar architecture; however, extract spatio-temporal results using 3D convolution in the video as opposed to an LSTM and show similar results on the standard sensor fusion problem.Sensor fusion is often broken down into the following 3 methods based on where the data are combined [33, 25, 34] , also shown in Figure 7 : 1) Early or data-level fusion combines the raw sensor outputs before any processing. 2) Middle/intermediate or feature-level fusion combines each sensor modality after some preprocessing or feature extraction. 3) Late or decision-level fusion combines the raw output, essentially ensembling separate models.The UTD-MHAD dataset has no predefined splits and benchmarks, and little to no works have open sourced their code on it. As such, to ensure we were using the data correctly implemented a few sensor fusion models and compared to state-of-the-art reported methods and showed similar performance results. The results are given in Table 8 These experiments provide a standard of comparison for our results with other methods on the UTD-MHAD dataset, and these models are available in the code for this paper. a) 40% (X RGB , Y ) Train b) 40% (X RGB , X IM U ) Validation 10% (X RGB , X IM U , Y ) Test 10% (X RGB , X IM U , Y )Table 8 : SOTA Sensor Fusion Performance on UTD-MHADHAMLET [17] 95.12% WEI ET AL. [42] 95.6%REPRODUCED FROM [42] ImageBind [13] learns encoders for 6 modalities, (Images/Videos, Text, Audio, Depth, Thermal and IMU) by performing CLIP's training method [32] between each of those encoders and the Image/Video encoder. It was well tested for image, text and audio based alignment, retrieval and latent space generation tasks, however was not well test with IMU data and not used for specific tasks, such as HAR. In addition, one fundamental difference between Imagebind and CA is that Imagebind constructs a latent space amongst the sensing modalities and text and aligns between them. We hypothesize that this is vector space is more difficult and unnecessary to construct, for human action recognitoin using sensing modalities. The text modality, although sequential in nature, does not have a time dimension, thus it cannot leverage correlations between modalities in time like C3T.Let's denote the video, IMU and text encoders as g (1) : X (1) → Z, g (2) : X (2) → Z, and g (3) : X (3) → Z respectively. We perform two conventional task-specific adaptations for CLIP models. First, we attempt zero-shot transfer, in which we pass all the action labels through the text encoder. For a dataset with C classes, we haveEQUATIONi , yi) ∈ DT est, we pass x(2) i through the IMU encoder g (2) and retrieve ẑ(2) . Finally, we classify the point by looking at which points gives the highest cosine similarity score in the latent space, e.g. ŷi = argmaxj ⟨x(2) i ,ẑ (3) j ⟩ ||⟨x (2) i ,ẑ (3) j ⟩|| .Given that ImageBind is a large model trained on massive corpuses of data it becomes impractical to train the model from scratch on our smaller datasets collected from wearables and edge devices. Instead, we fine-tuned the ImageBind model using a linear projection head on the encoders, that can then be trained for a specific task. The results of this method are depicted in Table 6 .The results show a poor generalization of Imagebind to most experiments on our setup, and we hypothesize a few reasons. Firstly, ImageBind is a large model and may either overfit to small datasets, or not have enough training examples to learn strong enough representations. Second, ImageBind was pre-trained on Ego4D and Aria which contain egocentric videos to align noisy captions with the IMU data, whereas our datasets had fixed labels and were mostly 3rd person perspective. In fact ImageBind performed the best on the one egocentric dataset we used, MMEA-CL [43] . Lastly, Imagebind was trained on a IMU sequences of 10s length sampled at a much higher frequency, thus we zero-padded or upsampled the IMU data to fit into ImageBind's IMU encoder, and the sparse or repetitive signal may have been too weak for ImageBind's encoder to accurately interpret the data. We have shown that the CA method performs well on a new modality, however, the question remains whether it can still retain performance on the original modality it was trained on. Furthermore, if it is given multiple modalities during inference, can it leverage information from all of them? Through CA, for any data sample i given for inference, regardless of whether the sample contains data from X (1) , X (2) or both (X (1) , X (2) ), we can estimate the latent vector ẑi ∈ Z and thus predict the action using the HAR module, ŷ = h(ẑi)1. RGB (Supervised Learning) The typical supervised machine learning paradigm tests the model on different samples of the same distribution. In our case, this is testing CA on RGB data. Thus the estimated latent vector is given by f (1) EQUATIONi ) = ẑi.. Thus, for sensor fusion we average the outputs of each of the encoders.Cross-modal transfer learning is a method to transfer knowledge from a modality with abundant training data to one with limited data [28] . Cross modal transfer is a subset of domain adaptation. Domain adaptation allows a machine learning model trained in one domain to efficiently adapt to another related domain for the same output task with fewer data labels [29, 10] . Given this focus on scarcely labeled domains, adaptation is often performed through unsupervised [6] or semi-supervised [1] methods. In terms of human activity recognition, different data domains can imply adapting between different sensor inputs [2] , different positions of wearables on the human body [38, 6, 31] , different users [16, 12] or IMU device type [20, 45, 5] . In the context of this cross-modal learning our work may be considered domain adaptation involving different sensor inputs.IMU Virtualization: To overcome the difficulties associated with limited IMU data, many recent works have leveraged the ability to simulate IMU data from videos such as IMUTube [22] or ChromoSim [15] . This allows them to train an IMU model from camera data and perform zero-shot classification on IMU data with a model that has never seen real IMU data. However, these models are time and data-intensive and cannot easily be extended to other modalities.Contrastive Learning Methods: Contrastive learning-based multimodal models such as ImageBind [13] or IMU2CLIP [27] are relatively easy to extend to new modalities, however, they fail to fuse the sensor modalities when present and are designed for cross-modal retrieval or generation and perform poorly on specific tasks such as human action recognition.Cross-Modal Knowledge Distillation Knowledge distillation methods typically use an extra auxiliary modality during training to increase single modal performance during testing, however, they assume labeled training data from both modalities during training [44, 21, 39, 3] . Notably, [36] attempts to perform without knowledge distillation without labels for one modality using a student-teacher framework which CA can out-perform. Furthermore, they test transferring acrosss visual modalities which is likely more correlated that visual and inertial modalities.Unsupervised Modality Adaptation: Unsupervised Modality Adaptation is a new term we coined, however, previous methods have explored similar settings. Thoker and Gall [36] tests on a modality that doesn't seem labels during training, however, they work with RGB and skeletal modalities, both are which derived from visual modalities, making their task inherently easier. Their work is also limited to one dataset, tested with only an ensemble based student-teacher method, and their code and model is not opensourced. Furthermore, MMG-EGO-4D [14] provides a very similar zero-shot cross-modal transfer setting and they also work with IMU and RGB data. However, their work is limited to one custom-built egocentric dataset, and they only analyze one method (similar to our CA method). Their method also focuses more on 'few-shot' output generalization to new classes as opposed to latent space alignment techniques for cross-modal UMA. 


Decision: In light of the positive reviewers' feedback and relevancy of the submission, we are pleased to accept this paper for presentation at UniReps 2024. We kindly ask the authors to incorporate the reviewers' suggestions and feedback in the final camera-ready version of the manuscript.

Reviews: review: title: title: Unsupervised Modality adaptation in HAR via Cross-Modal Representation
main_review: main_review: The paper proposes an effective mechanism to align multiple modalities or sensors proposing three techniques. However unsupervised alignment does exist (perhaps not in HAR) so it would be good if the authors could perhaps frame the paper if this is specific to HAR or could be aligned to other modalities.



[1]Generalized Unsupervised Manifold Alignment
title: title: Review
main_review: main_review: ## Summary
The paper introduces a novel approach to human action recognition (HAR) called Unsupervised Modality Adaptation (UMA), which aims to transfer knowledge between modalities without labeled data for the target modality during training. The authors propose three methods for UMA: Student-Teacher (ST), Contrastive Alignment (CA), and Cross-modal Transfer Through Time (C3T). Each method leverages a unified multimodal representation space to enable cross-modal knowledge transfer. The effectiveness of these methods is demonstrated through extensive experiments on various datasets combining RGB video and Inertial Measurement Unit (IMU) data.

## Strengths
1. Three different methods (ST, CA and C3T) are proposed, and the advantages and disadvantages of these methods are compared and analyzed in detail.
2. The methods are tested on four different datasets, demonstrating their effectiveness across various conditions and data types.

## Weaknesses
1. This paper mainly focuses on RGB and IMU data. It would be better to discuss other types of multimodal data, such as image-text.
2. There is a lack of visualization and a lack of discussion of the successful and failure predictions.
3. The effect of hyperparameters on performance should be discussed.
title: title: C3T is novel for Unsupervised Modality Adaptation in action recognition
main_review: main_review: I. paper summary:

This paper conducted a thorough study on Unsupervised Modality Adaptation in action recognition by utilizing RGB+IMU multi-modality representation aligning.  Since generally the modality of video data are RGB information from camera, the other modalities like IMU lacks human annotations. The authors explored three different methods to transfer the action recognition capability from RGB modality to IMU modality, namely Student-Teacher (ST), Contrastive Alignment (CA), and Cross-modal Transfer Through Time (C3T). In particular, the novel C3T demonstrate unique robustness on dealing time-varying latent vectors for time series data.

II. pros:

1. The task of transferring knowledge from RGB modality to IMU modality seems interesting and practical in real world applications.  The proposed method on aligning multi-modalities is closely related to the topic of  UniReps workshop.

2. The proposed C3T contains novelty.  Built on the encoders trained by contrastive learning, C3T measures the frame-wise  similarity of RGB embeddings  and IMU embeddings.  Then a transformer structure is utilized to merge and learn from these two modalities. 

3.  The experiments are concrete and persuasive,  demonstrating the advantages and disadvantages of ST, CA, C3T.




III. cons and questions:

1. the ST methods is a variant of knowledge distillation with pseudo labels in action recognition domain,  which has been explored in a work 'Knowledge integration networks for action recognition' in AAAI 2020. It would be great if the authors could compare the differences of the proposed method and 'Knowledge integration networks for action recognition'.

2. The experiments have demonstrated that the proposed C3T is not always better than CA and ST.  While it is good to explore the different properties of these three methods, is it possible to improve the disadvantages of C3T?

3. The proposed C3T is capable of dealing with time-varying latents and robust to time noise. So what is your sampling strategy? Do you utilize video-level sampling strategy like TSN (ECCV 2016) and TFCNet (arxiv 2022), or clip-level sampling like 3D CNNs, eg. Nonlocal networks ( CVPR 2018), R2+1D (CVPR 2018) , Slowfast (ICCV 2019)? Or hybrid sampling like V4D (ICLR 2020)? Generally video-level sampling strategy and hybrid sampling will be more robust. Maybe the authors have tried these methods. I suggest the authors to compare these sampling methods in the experiment.
title: title: Evaluation of the paper "Unsupervised Modal Adaptation in Human Behavior Recognition through Cross Modal Representation Learning"
main_review: main_review: This article introduces the concept of Unsupervised Modal Adaptation (UMA) in Human Behavior Recognition (HAR) by using a unified multimodal representation space to transfer knowledge between modalities. The author proposes three UMA methods: Student Teacher (ST), Contrastive Alignment (CA), and Cross Modal Time Transfer (C3T). These methods were tested on multimodal datasets, including IMU and RGB data. This article emphasizes the unique ability of C3T to handle temporal noise and misalignment, demonstrating its effectiveness in noisy real-world scenarios.

Comments: comment: We would like to thank all the reviewers for their feedback and positive reviews. We have updated the paper with a few changes:

1. We replaced Table 2 with more robust experiments (each experiment was repeated 3 times and the average is displayed). The overall trends are relatively the same as before. We also compare performance in Unsupervised Modality Adaptation to the performance of the same modules in a Supervised setting. The original table with the sensor fusion and imagebind baseline are in the appendix for reference (Table 6), but believe that this new table 2 is more informative and the supervised baseline is more competitive/interesting.

2. We condensed the other tables into one Table 1, and similarly repeated the experiments thrice and averaged their results, for more robust empirical results.

3. We added some strong visualizations to support our method (Figures 3 and 4), some ablations (Table 4 and 5) and a computational efficiency analysis (Figure 5).
We deeply appreciate your positive review of this work!
We sincerely thank the reviewer for their in-depth analysis of our work and their positive outlook.  We would like to respond to the listed Cons/Questions and invite the reviewer to follow up further with any thoughts /questions.

III. Cons and Questions:
1. It seems the cited work indeed uses a similar student-teacher network, however, Knowledge Integration in that work refers to knowledge integration across auxiliary tasks as opposed to various modalities and that work is limited to video human action recognition (single modality). We provide a slightly more extensive related work in Appendix F where we discuss other cross-modal knowledge distillation methods and in particular methods that work with RGB and IMU data.
2. Yes. We indeed have updated our experiments (now we perform 3 trials and take the average) and we observed that on average C3T does indeed perform better. We also highlight circumstances where C3T always performs better (with time shift noise).
3. This is a great point! We are using a simple video-level sampling, and have not yet conducted a comprehensive comparison of different sampling strategies for C3T. Your suggestion to compare video-level sampling (like TSN and TFCNet), clip-level sampling (like NonLocal Networks, R2+1D, SlowFast), and hybrid sampling (like V4D) is excellent and would provide valuable insights. Although we did not have the capacity to perform these experiments at the moment, we will investigate this further for future iterations of this work.
Thank you for the insightful feedback. We would like to respond to the listed weaknesses and invite the reviewer to follow up further with any thoughts /questions.

Weaknesses:
1. We acknowledge this work is limited to RGB and IMU data. We believe that the C3T method might not bring any unique advantage in cross-modal alignment with Text as a modality, as text is discrete, not a time-continuous signal. Although, with some modification, this may be an interesting direction to pursue.
2. We have performed some visualizations using TSNE on the latent representations (Figure 3) and the C3T Attention head weights (Figure 4). These were added to the appendix, Figure 3.
3. We Have added some ablations in Tables 4 and 5  of the appendix, as well as an analysis of the latent vector "hidden_size" hyperparameter. This analysis shows convolutions are indeed the best architecture for the encoders and C3T maintains a strong performance even with smaller model sizes.
We sincerely appreciate your positive review. Your observation that connects our work to generalized unsupervised manifold alignment is particularly valuable. While space constraints and the specific focus of this paper precluded an in-depth discussion or reframing of our work in this context, we certainly intend to explore this connection more thoroughly in future iterations of our research!