MMTSA: Multi-Modal Temporal Segment Attention Network for Efficient Human Activity Recognition
Authors: Ziqi Gao
Jianguo Chen
Xin Liu
Yuntao Wang
Yuanchun Shi
Venue: Submitted to ICLR 2023 (2023)
Keywords: Multimodal Learning
Human Activity Recognition

Abstract: Multimodal sensors (e.g., visual, non-visual, and wearable) provide complementary information to develop robust perception systems for recognizing activities. However, most existing algorithms use dense sampling and heterogeneous sub-network to extract unimodal features and fuse them at the end of their framework, which causes data redundancy, lack of multimodal complementary information and high computational cost. In this paper, we propose a new novel multi-modal neural architecture based on RGB and IMU wearable sensors (e.g., accelerometer, gyroscope) for human activity recognition called Multimodal Temporal Segment Attention Network (MMTSA). MMTSA first employs a multimodal data isomorphism mechanism based on Gramian Angular Field (GAF) and then applies a novel multimodal sparse sampling method to reduce redundancy. Moreover, we propose an inter-segment attention module in MMTSA to fuse multimodal features effectively and efficiently. We demonstrate the importance of imu data imaging and attention mechanism in human activity recognition by rigours evaluation on three public datasets, and achieved  superior improvements ($11.13\%$ on the MMAct dataset) than the previous state-of-the-art methods. 

Under review as a conference paper at ICLR 2023
MMTSA: MULTIMODAL TEMPORAL SEGMENT AT- TENTION NETWORK FOR EFFICIENT HUMAN ACTIV- ITY RECOGNITION
Anonymous authors Paper under double-biee blind review
ABSTRACT
Multimodal sensors (e.g., visual, non-visual, and wearable) provide complemen- tary information to develop robust perception systems for recognizing activities. However, most existing algorithms use dense sampling and heterogeneous sub- network to extract unimodal features and fuse them at the end of their framework, which causes data redundancy, lack of multimodal complementary information and high computational cost. In this paper, we propose a novel multi-modal neu- ral architecture based on RGB and IMU wearable sensors (e.g., accelerometer, gyroscope) for human activity recognition called Multimodal Temporal Segment Attention Network (MMTSA). MMTSA first employs multimodal data isomor- phism mechanism based on Gramian Angular Field (GAF) and then applies a novel multimodal sparse sampling method reduce redundancy. Moreover, we propose an inter-segment attention module in MMTSA to fuse multimodal fea- tures effectively and efficiently. We demonstrate importance imu data imag- ing and attention mechanism in human activity recognition by rigorous evalua- tion on three public datasets, and achieved superior improvements (11.13% on the MMAct dataset) than the previous sate-of-thhcart methods.
INTRODUCTION
With the increased interests of wearable technologies and the development of deep learning, human activity recognition (HAR) has recently attracted widespread attention in human-computer inter- action, healthcare, and multimedia analysis. Uni-modal (e.g.. RGB video, audio, acceleration, frared sequence, etc.) HAR methods have been extensively investigated in the past years) Wang et al. (2016) [ii [ al. (2019); Starcfa-Hernnnde et al.) (2017); [Slim al. (2019); Akkla al. (2018). However, unimodal HAR methods limit generalization ability in real-world scenarios. The perfor- mance of a video-based HAR method is usually sensitive to illumination intensity, visual occlusion, complex background. Meanwhile, noisy missing data and movement variance between dif- ferent users can negatively affect a sensor- HAR method. In these cases, it is important to leverage both lision-based and sensor-based modalities to compensate for the weaknesses of single modality and improve the performance of HAR in multi-modal manners. For instance, the move- ment trajectories of hands are similar when people eat or drink, thus it is not easy distinguish the two actions only based on imu sensor data. However, when the vision modality is considered, the two actions can be distinguished based the visual characteristics of the objects held by the hands. Another example is outdoor activity recognition. Due to the similarity of the outdoor environment, is challenging for the aingle-oddall systems classify whether a user is walking or running by relying only on the visual data of the smart glasses. On the other hand, the imu sensor data (i.e., accelerometer data) of these two activities reveal significantly different feature representations.
In recent years, sensors have been widely equipped in smart glasses, smartwatches, and smartphones, making the available input modalities for HAR more abundant. Therefore, various multimodal learn- ing methods have been proposed to exploit complementary properties among modalities for fusion HAR. Existing multimodal learning methods fuse rision-sensor and imu sen- data have the following shortcomings: 1) Owing data heterogeneity. most existing methods feed uni-modal data into separate sub-networks with different structures to extract features and fuse them at the end stages. Obviously, there is huge structure divergence between imu sensor and
and no-leerning--asee

---

Under review as a conference paper at ICLR 2023
vision-sensor data. Since the imu sensor data one-dimensional time-series signals, most of the previous literature utilize ID-CNN or LSTM network to extract spatial and temporal features of raw imu sensor data Steven Eyobu & Han 2018); Panwar al. (2017); Wang et al. (2019a). The vision-sensor activity data, however, are usually images or videos with two or more dimensions, which is suitable for 2D-CNN or 3D-CNN to extract visual features Simonnyan & Zisserman (2014); Karpathy et al. (2014b); Sun et al. (017)) Thus, the form of input of existing multi-modal learn- ing models not only ignore the temporal synchronization correlation between multimodal data but also lose valuable complementary information. Additionally, it costs to add new sub-networks to extract the learning representations of new modality inputs. 2) Dense temporal sampling, which means sampling frames densely in video clip or sampling the entire series of sensor data in a time period, widely used in previous work to capture long-range temporal information in long-lasting activities. For example, those methods Tanberk et al. (2020); Wei et al. (2019) mainly rely on dense temporal sampling to improve the performance, which results in data redundancy and unnecessary computation since the adjacent frames in the video have negligible difference and the imu data of some activities are periodic. 3) Although some newly proposed attention based multi-modal learn- ing methods have improved the performance of HAR tasks, their complicated architectures lead to high computational overhead and make them challenging to be deployed on mobile and wearable devices. Islam & lqbal 20021 2022).
To address the challenges above, we propose a novel and efficient HAR multimodal network for vision and imu sensors called MMTSA. Our method includes three contributions:
We design MMTSA, a novel multi-modal neural architecture based on RGB videos and IMU sensor data for end-to-end human activity recognition application. MMTSA effi- ciently explores information from heterogeneous multi-modal data through an isomorphism mechanism and enables extensions for new modalities. Moreover, MMTSA leverages segment-basee sparse co-sampling scheme to enable a long-lasting activity recognition at low computational resources.
MMTSA first applies Gramian Augular Field (GAF) methods to encode the imu sensor data into multi-channel grayscale images in the multi-modal human activity recognition. The transformation arrows the structural gap among the RGB data and imu sensor data and enhances the resuability of the model's sub-networks, which bring the computation improvement.
To better mine the complementary information between synchronized data of different modalities and the correlation between different temporal stages, we design inter-segment modality attention mechanisms for spatiotemporal feature fusion. We compared the perfor- mance of our method to several state-of- the-art art HAR algorithms and traditional algorithms on three multi-modal activity datasets. MMTSA achieved an improvement of 1 11.13% and 2.59% (F1-score) on the MMAct dataset for the cross-subject and cross-session evalua- tions.
2 RELATED WORK
2.1 UNI-MODAL HUMAN ACTIVITY RECOGNITION
Single modality-based based Human Activity Recognition (HAR) has been extensively investigated in the past decades. Currently, many researchers are utilizing the deep neural network- based approaches to learn single modality feature representations for HAR.
With the success of CNN He et al. (016)): Zhang et al. (2018) in image tasks, researchers put great efforts on visual modality. The accuracy of Aideo-based HAR methods is highly related to video rep- resentation learning. Early works try to use 2D based methods Karpathy et al. (201 to learn video representations. Later, 3D convolutional networks Tran et al.) (2015) Xie et al. (2018) are explored and achieve excellent performance. However, 3D based methods always take several consecutive frames input, that huge complexity and computational costs make these methods expensive to be used, Recently, 2D convolutional networks with temporal aggregation WWang et al. (2016); Lin et al. (209) achieve significant improvements, which have much lower computational and memory
2

---

Under review as a conference paper at ICLR 2023
costs than 3D based methods. Although, the video-based HAR methods have good performance in most scenarios, it raises privacy concerns since visual modality contains rich appearance information of captured scene context.
Additionally, the robustness to lighting, occlusion, and subject angle are also key areas for im- provement in HAR. Researchers pay attention to other single modality data, such as audio, and imu wearable sensors. However, in recent years, compared to other single modalities, using audio is not very popular scheme for HAR. Only a few deep learning methods are proposed for HAR from audio alone Liang & Thomaz (011)) Moreover, wearable sensor data is good choice for HAR, due their robustness against viewpoint, occlusion and background variations, etc. Many works of literature WWang et al. (2019a) have proposed wearable sensor-based solutions for HAR. How- ever, the accuracy of wearable sensor-based methods is sensitive to the placement on the human body Mukhopadhyay (2014). Furthermore, most of current wearable sensor-based methods perform poorly in complex HAR scenarios.
Although these single modality methods have shown promising performances in many cases, these approaches have a significant weakness, which rely on high-quality sensor data. the single modal- ity data is noisy and missing, the uni-modal learning methods cannot extract robust features and have poor performance in human activity recognition.
2.2 MULTI-MODAL HUMAN ACTIVITY RECOGNITION
In order to overcome the shortcoming of single modality missing and occlusion, multimodal learning has been used HAR. By aggregating the advantages and capabilities of various data modalities, mul- timodal learning can provide more robust and accurate HAR. Moreover, most existing algorithms Islam & Iqbal (022) use each modality sensor data independently and fuse all modalities learning representations at the end of their framework for classification. Therefore. the final performance is highly related to salient feature representations of single modality. However, these architectures glect the intrinsic synchronous property among all modalities and assume all modalities contribute final performance equally.
To address these challenges, several works introduce new multi-modal HAR algorithms. Firstly, in- stead of aggregating different modalities at late stage, TBN architecture Kazakos et (2019) com- bined three modalities (RGB, flow and audio) with mid-level fusion at each time-step and showed that visual-audio modality fusion in egocentric action recognition tasks improved the performance of both the action and accompany object. However, the mid-level fusion method is only explored in video and audio modalities and has not been extended to other sensor data (e.g. imu sensors). Secondly, attention- tention mechanism allows the feature encoder to focus on specific parts of the representation while extracting the salient features of different modalities. For example. Long et al. (2018) proposed new kind of attention method called keyless to extract salient uni-modal features, which were com- bined to produce multi-modal features for video recognition. Morcover, Multi-GAT Islam & Iqbal (202) explored the possibilities of using graphical attention methods for multi-modal representation learning in HAR.
approaches have recently been applied in feature learning for HAR as at-
Although these multi-mode HAR methods have achieved good performance in various scenarios, several challenges still remain in mudti-modal HAR. For example, many of these methods encode the whole sensor data, which is redundant and highly computational. Therefore, a sparse and efficient sampling strategy would be more favorable and need to be designed. Furthermore, many existing frameworks do not allow motar-mydality interaction and may not learn complementary multi-modal features. Thus, we explore the modar-mydality and inter-segment attention mechanism and demon- strate that it improves the final result. Finally, we propose novel multimodal temporal segment attention network MMTSA, as described in detail in the next section.
METHOD
In this section, we present our proposed method: MMTSA, a multi-modal temporal segment at- tention network shown in Fig. [ MMTSA consists of three sequential learning modules: 1) Multimodal data isomorphism mechanism based on imu data imaging: The module is respon- sible for transforming imu sensor data into multi-channel grayscale images via Gramian Angular
3

---

Under review as a conference paper at ICLR 2023
Field (GAF), making visual-sensor data and imu sensor data representations be isomorphic. 2) Segment-based multimodal sparse sampling: We propose novel multimodal sparse sampling strategy in this module. It performs segmentation and random sampling RGB frame sequences and GAF images of IMU data, preserving the modal timing correlation while effectively reduc- ing data redundancy 3) Inter-segment attention for multimodal fusing: To better mine the spa- tiotemporal correlation and complementary information between modalities, we propose an efficient inter-segment attention method to fuse multimodal features, which improve HAR performance. We discuss how MMTSA works in greater detail.
2D-CNN
2D-ONN
2D-ONN
2D-CNN
(1)
2D-ONN
2D-ONN
2D-CNN
2D-ONN
2D-ONN
(2)
(3)
Figure 1: The architecture of MMTSA: 1) Multimodal data isomorphism mechanism. 2) Segment- based multimodal sparse sampling. Ieenee--ett attention modality fusing.
3.1 MULTIMODAL DATA ISOMORPHISM MECHANISM
Although deep learning has achieved great success in CV and NLP, techniques of it fail to have many comparable developments for time series. Most traditional deep learning methods for time series build models based RNN, LSTM, 1D-CNN. However, these approaches have been proved to have limitations Pascanu et al. (2013) 3); Wu et al. (2018)
To leverage the advanced 2D-CNNs and related techniques in computer vision, Wang & Oates (201)) first proposed a novel representation for encoding time series data as images via the Gramian Angular Fields (GAF) and hence used 2D-CNNs to improve classification and imputation perfor- mance. Since then, time series imaging methods have caught much attention. Inspired by the method proposed in Wang & Oates (0115), note that GAF based methods have great potential reduce structural differences in data from multiple modalities (e.g. RGB video and accelerometers). There- fore, we propose multimodal data isomorphism mechanism based on GAF, which can enhance the representation ability of the temporal correlation of imu sensor data and improve the reusability of different modal feature extraction networks. We will now briefly describe how our multimodal data isomorphism mechanism works.
1) IMU sensor series rescaling Let (st, St........ St,, a time series collected by imu sensor, where St, R represents the sampled value time T t, tj represents the sampling time duration of this time series. We rescale S onto | 1] by:
(St, max (S) + (st, - min(S))
max(S) min(S)
The max-min normalization step makes all values of fall in the definition domain of the arccos function, which satisfies the conditions for the coordinate system transformation.
4

---

Under review as a conference paper at ICLR 2023
2) Polar coordinate system transformation: In this step, transform the normalized Cartesian imm sensor series into polar coordinate system. For the time series S, the timestamp and the value of each sampled data point need to be considered during the coordinate transformation. Then use an inverse cosine function to encode each data point 8t, into polar coordinate by:
dr. arccos (si,) S st 1, 8t, S T22 - -..4 ET
where ot. and rt, represent the angle and the radius of sr in the polar coordinate, respectively. The encoding in equation[] has the following advantages. First, it is a composition of bijective functions [0, T.. which allows this transformation preserve the integrity of the original data. Second, it preserves absolute temporal relations, as the area of 6t, and ot, polar coordinates is dependent on not only the time interval and ty. but also the absolute value of them Wang & Oates (2015). The coordinate transformation above maps the 1D time series into a 2D) space, which imperative for calculating the Gramian Angular Field latter.
cos (Ø) is monotonic function when
Originl Seree
Normalization
Rescale
Penilee Tarns
Coordinate
Figure 2: IMU sensor data imaging via Gramian Angular Field in MMTSA
3) Image generation via Gramian Angular Field (GAF): The Gram Matrix G is the matrix of all inner products of can be seen calculating similarity between two vectors. However, in the polar coordinate mentioned above, the norm each vector causes a bias in the calculation of the inner product. Therefore, between each point in the
T2, where H; a vector. dot product Gram Matrix
exploit the angular perspective by considering the trigonometric polar coordinate to identify the temporal correlation within different time intervals, which is an inner product-like operation solely depending on the angle. The GAF is defined follows:
cos (dr, + din)
cos (Ør, + tr, cos (0is + Dt.))
Cos (012 + 11))
G(5)
(3)
cos (de,, + )ee))
cos (dr,, + srem))
where
cos (or, 01,)) = cos(coccoss(tt (87,) arccos((i,)
(4)
si.
S
The GAF representation in polar coordinate maintains the relationship with the original time- series data via exact inverse operations. Moreover, the time dimension is encoded into GAF since time increases as position moves from top-left to bottom right, preserving temporal dependencies.
Then we map each element in the GAF representation to a pixel of a grayscale image by:
cos(d,, + oh,)) min(G) 256, < i.j S G.
(5)
max(G') min
5

---

Under review as a conference paper at ICLR 2023
where G is grayscale image of size
Most wearable sensors (accelerometers, gyroscopes, magnetometers, etc.) are triaxial. Suppose given a time series data (S((), 5tw), ) sampled by a 3-axis accelerometer, we can generate three grayscale images {G(*), ((e), { for the æ, y, and z axes according to the above steps. After concatenating these three images, the time series sampled by each three-axis imu sensor can be uniquely converted to a multi-channe grayscale image of size 3 x 72 X n2.
3.2 SEGMENT-BASEE MULTIMODAL SPARSE SAMPLING
Segnert
Segnari
OAD ej
Segment
Figure 3: Multimodal sparse sampling strategy MMTSA
The two mainstream methods of traditional human activity recognition based on video understanding are 3D-CNN and two-stream CNN networks, but the limitations of these two schemes are that they only can capture short-range temporal dependencies in the video. order to capture long-distance temporal dependencies, these methods usually require densely sampled video clips. A video clip m consecutive frames sampled by a sliding window of size m in a period, and the whole video divided into several clips. However, the content changes relatively slowly between two adjacent frames in a clip, which means the sampled consecutive frames are highly redundant.
Similar to visual data, we observe redundancy in imu sensor data. As shown in Fig 2 the imu sensor data collected from the 3-axis accelerometer located on the smartwatch has obvious periodicity while the user was waving hands. We reasoned that this phenomenon should be common in imu sensor data. In most daily activities, human limb movements are regular and repetitive, so there is corresponding periodicity the data collected by wearable sensors. It means that local data features can represent the characteristics of the entire activity, thus the complete time series is redundant. However, traditional deep learning activity recognition models based on imu sensor data and newly proposed ones ignored this issue. In previous work, a common input method is to feed the complete collected data series into deep models Xu et al. (202T): Wang al. (2019b), such as CNN, RNN, LSTM, or Bert. Another widely-used method is dense sampling in fixed-width sliding windows and overlap Bianchi et al. (2019). These unnecessary dense sampling methods lead to larger memory overhead and longer inference time. In addition, the above dense sampling strategy will consume too much energy when deployed on the device.
Wang et al. (2016) proposed TSN framework to deal with the frame redundancy in video understand- ing by applying a sparse and global temporal sampling strategy. This strategy divides the video into fixed number of segments and snippet is randomly sampled from each segment. To overcome the aforementioned challenges of data redundancy in multimodal tasks, we leverage the segmenta- tion idea of Wang al. (2016) and propose sparse sampling method for multi-wearable sensor time series, as shown in Fig
We find that the some excellent properties. First, the diagonal of each grayscale image is made of the original value of the scaled time series (G,, 2s; 1). Second, the sampling along the diagonal direction has local temporal independence. Given two timestamps & and t, (i < j), we sample a square area in the grayscale image with diagonal extending from G.,i G The data in this square matrix only depends on the timestamps between t; and ty. and represents the temporal correlation of the original series in this period. Our proposed method firstly divides the entire imu sensor data
grayscale images generated by GAF based on imu sensor data have

---

Under review as a conference paper at ICLR 2023
into N segments of equal duration according timestamps. The dividing points of these segments correspond to equidistant points on the diagonal of the grayscale image generated based on GAF: (S.)).))) S)), ... (sx In each segment, we use a square window of size K for ran- dom multi-channel sampling:
GG.
G(AKKK1)
((S.)
(6)
GO+K-1.9
((4K-1(4--))
where S Si --1 S, For multi-axis sensors, random sampling of each segment is performed simultaneously multiple channels.
3.3
INTER-SEGMENT AND MODALITY ATTENTION MECHANISM
To fuse features of different modal data in each segment and extract more spatiotemporal informa- tion multimodal training, we propose an additive attention-base inter-segment modality fusion mechanism in MMTSA, which is effective and efficient. We first concatenate the features of differ- ent modalities each segment by:
Ys, = Concat (F (X3;: W F (X W²) ,
( W
Where Ys, is the output of the i-th segment, F (X3,; W))
represents ConvNet with parameters
W that operates on X and indicates the j-th modality. X and W represent the sampled data of the j-th modality in segment i and the shared parameters of modality j. respectively. Then we an output sequence of each segments: yout (Ys,, '...... ... Ysx) where Ys, R and d the feature dimension of each modality.
We utilize additive attention to calculate the attention weight of each segment,
exp (WWatt) T Ys.
EiEN exp (Watt)) T Ys.
where Wratt E M-d is learnable parameter and (Watt) Ys. is a score to evaluate the importance of each segment. The softmax function is used to regularize the scores that the sum of the scores of all segments 1, Here, the parameter Watt will weight the features of different modalities to compensate for the inter-modal information.
Next, the attention weights ag = {as, .... ,asy } will be used to fuse the outputs of each segment and get weighted global representation,
Ys.
Σ Ys. SEN
Finally, the global representation Ys, is fed into Feed Forward Neural Network with two fully- connected layers and a softmax to compute the probabilities for each class of human activities.
EXPERIMENT
4.1
DATASET
We evaluate our proposed model multimodal temporal segment attention network MMTSA for the human activity recognition task on three public datasets: Multimodal Egocentric Activity [Song et al. 2014), DataEgo Possas et al. (2018), and MMAct Kong et al. (0099)
4.2 EXPERIMENT SETTINGS
We compare our proposed model multimodal temporal segment attention network with the follow- ing state-of-the-art multimodal HAR training algorithms for comparison, such TSN Wang et al.
7

---

Under review as a conference paper at ICLR 2023
Table 1: cuoss- and cross-session performance comparison MMAct dataset
Cross-Suject
FI-Score (%)
Cross-Session
FI-Score (%)
SMD Hinton al.
2015
63.89
SVM+HOG Offi et al.
2013
46.52
Talii-Tecchers Kong et 2019
62.27
TSN (Fusion) Wang el 2016
77.09
Student Kong ei al.
2019
64.44
MMAD Kong et al. 2019
78.82
MMAD Kong ct al.
2019
66.45
Keyless Long al. 2018
81.11
Keyless Long et al.
(2018
71.83
HAMLET Islam Iqbal
22200
83.89
Multi-GAT Islam & Iqbal
[2021]
75.24
MuMu Islam & Iqbal (2022)
87.50
MuMu Islam & Iqbal 2022
76.28
Islam & Igbal 2021
91.48
MMTSA (our method)
87.41
MMTSA (our method)
94.07
2016), Keyless Long et al. (2018), HAMLET Islam & Iqbal (020)) MuMu Islam & Iqbbl (2022)
We use the mirco FI score to evaluate the performance of all methods.
Data preprocessing: The imu data of each sensor is imaged based on the method in Sec To achieve the efficiency of method, directly segment the imu series and randomly select timestamp within each segment. The window containing the timestamp is used to sample the orig- inal series, which equivalent strategies in Sec S.c The RGB video and its synchronous imu data series is divided into the same number of segments, and a frame is randomly sampled in each segment.
Training Details: We implemented our proposed method in Pytorch. We utilized Inception with Batch Normalization (BN-Inception) as a sub CNN to extract the unimodal features learning rep- resentations. Moreover, in the proposed model, we trained all the modalities simultaneously with N segments, SGD with momentum optimizer, and learning rate of 0.001. The convolutional weights for each modality are shared over the N segments, and hence reduce the model-size and memory cost.
5
RESULT
5.1 RESULTS COMPARISON
We evaluate our proposed model MMTSA performance and summarize all the results. For MMAct dataset, we follow originally proposed cross-subject and cross-session evaluation settings and report the results in Table The results show that MMTSA improves 11.13% and 2.59% in cross-subject and cross-session evaluation settings Kong et al. (2019). For Multimodal Egocentric Activity, we follow leave-one-subject-our cross validation. MMTSA outperforms all of the traditional methods and is close to the performance of MFV as shown in Table Compared to MFV that uses four types of sensor data, MMTSA only uses two types (accelerometer, gyroscope) input to make the model lightweight. Thus, small loss precision is acceptable. For DataEgo, we divide each 5-minute original data into 15 seconds clips with 5-second overlapping. We keep the train and test split size of each activity balance. We show the cross-subject performance in Table 3, which outperforms TSN by 17.45%.
Table 2: cross-subject performance comparison on Multimodal Egocentric Activity
Method
FI-Score (%)
SVM Kwapisz et al.
(2011)
47.75
Decision Tree Iree Kwapisz et (2011)
51.80
FVS Song et al. (2016b)
65.60
TFVS Song et al. (20165)
69.00
Multi-Stream with average pooling SSng et al. 201
76.50
FVV Song et al. (2016b)
78.44
FVV+FVS Song et (2016b)
80.45
Multi-Strean with maximum pooling Song et al. 2016a
80.50
MMTSA (our method)
81.00
MFV Song et al. (016b)
83.71
8

---

Under review as a conference paper at ICLR 2023
Table 3: cross-subject performance comparison DataEgo
Method
F1 F1-Score (%)
TSN (RGB) Wang et al. (2016)
65.77
MMTSA (our method)
83.22
5.2 ABLATION STUDY
This section is organized as follows. First, we show and discuss the performance of single modal- ities, and compare them with our proposed model MMTSA. Second, we compare direct feature concatenation and our additive inter-segment attention in feature fusion stage. Specifically, in this section, all the experiments are done on the MMact dataset and followed the original proposed cross-subject evaluation settings.
Single Modality vs Multi-Modality fusion:
We compare the performance of a single modality and different combinations of modalities in Table 4 The fusion of all modalities obtains the best results, which indicates that the multimodal isomorphism and fusion mechanism in MMTSA mines the complementary information among modalities more comprehensively.
Effectiveness of inter-segment attention modality fusion: We compare the simple concatenation method and our proposed inter-segment attention fusion method. The results shown in 5 suggest that our proposed method outperforms the concatenation method. It shows that the inter-segment attention modality fusion method has significant effect on the fusion of spatiotemporal features between modalities. Moreover, during training, we also find that the use of inter-segment attention leads to a faster convergence speed.
Table 4: modality ablation experiment on MMAct (cross-subject) Modality
FI-Score (%)
RGB
66.95
AccVacct-AccPhoneGGGn--oe
82.13
PhB-
84.47
87.41
Table 5: fusion method experiment on MMAct (cross-subject)
Fusion method
F1-Score (%)
Concat
84.02
Inter-segment attention
87.41
CONCLUSION
In this work, we presented MMTSA, new novel multi-modal neural architecture based on RGB and IMU wearable sensors for human activity recognition. MMTSA employs multimodal data isomor- phism mechanism by encoding the IMU sensor series into GAF images, which potentially reduces the structure differences of multiple modalities and enhances the reusability of the model's sub- networks. Additionally, our proposed segment-based sparse sampling method on multimodal input data is efficient and reduce data redundancy. Moreover, by adding the additive inter-segment atten- tion mechanism, our model enabled features interactions among different modalities and achieved better final results. As demonstrated on three public datasets, Multimodal Egocentric Activity, DataEgo, and MMAct, our method significantly improved state-of-the-an performance. Further- more, in the future. we plan deploy our proposed model MMTSA on device to enable safe and accurate human activity recognition in real-world setting.
REFERENCES
Aparna Akula, Anuj K Shah, and Ripul Ghosh. Deep learning approach for human action recogni- tion in infrared images. Cognitive Systems Research, 50:1 14511,4,, 2018.

---

Under review as a conference paper at ICLR 2023
Valentina Bianchi, Marco Bassoli, Gianfranco Lombardo, Paolo Fornacciari, Monica Mordonini, and Ilaria De Munari. Iot wearable sensor and deep learning: An integrated approach for person- alized human activity recognition in smart home environment. IEEE Internet of Things Journal, 6(5):8553-8562 2019.
Alejandra
Carlos E Galván-Tejada, Jorge I Galván-Tejada, José M Celaya- Padilla, Hamurabi Rosnbaa-Rosales Perla Velasco-Elizondo, and Rogelio Cárdenas- Vargas. A similarity analysis of audio signal to develop human activity recognition using similarity net- works. Sensors, 17(11):2688, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition.". CVPR, 2016.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint
2(7), 2015.
Md Mofijul Islam and Tariq Iqbal. Hamlet: hierarchical multimodal attention -rased human activ- ity recognition algorithm. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 10285-10292. IEEE, 2020.
Md Mofijul Islam and Tariq Iqbal. Multi-gat: A graphical attention-based hierarchical multimodal representation learning approach for human activity recognition. IEEE Robotics and Automation Letters, 6(2):1729-1738) 1736, 2021.
Md Mofijul Islam and Tariq Iqbal. Mumu: Cooperative multitask leamiin-bbaeed guided multimodal fusion," " AAAI, 2022.
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei- Fei. Large-scale video classification with convolutional nerual network,". CVPR, 2014a.
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei- Fei. Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 173-1732, 2014b.
Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen. Epic-fusion: Audio- visual temporal binding for egocentric action recognition. In Proceedings of the IEEE/CVE ternational Conference Computer Vision, pp. 5492-5501, 2019.
Quan Kong, Ziming Wu, Ziwei Deng, Martin Klinkigt, Bin Tong, and Tomokazu Murakami. Mmact: A large-scale dataset for cross modal human action understanding. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
Jennifer R Kwapisz, Gary M Weiss, and Samuel Moore. Activity recognition using cell phone accelerometers. ACM SigKDD Explorations Newsletter, 12(2):74-82, 2011.
Dawei Liang and Edison Thomaz. Audio- based activities daily living (adl) recognition with large- scale acoustic embeddings from online videos. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 3(1):17, 2019.
Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 7083-7093, 2019.
Xiang Long, Chuang Gan, Gerard Melo, Xiao Liu, Yandong Li, Fu Li, and Shilei Wen. Multi- modal keyless attention fusion for video classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
Subhas Chandra Mukhopadhyay. Wearable sensors for human activity monitoring: A review. IEEE sensors journal, 15(3):1321-133 2014.
Ferda Ofli, Rizwan Chaudhry, Gregorij Kurillo, René Vidal, and Ruzena Bajesy. Berkeley mhad: A comprehensive multimodal human action database. 2013 IEEE workshop on applications of computer vision (WACV), pp. 53-60. IEEE, 2013.
10

---

Under review as a conference paper at ICLR 2023
Madhuri Panwar, S Ram Dyuthi, K Chandra Prakash, Dwaipayan Biswas, Amit Acharyya, Koushik Maharatna, Arvind Gautam, and Ganesh R Naik. Can based approach for activity recognition using wrist-worn accelerometer. In 2017 39th Annual International Conference of the IEEE Engiineernng in Medicine and Biology Society (EMBC), pp. 2438-2441 IEEF, 2017.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pp. 1310-1318. PMLR, 2013.
Rafael Possas, Sheila Pinto Caceres, and Fabio Ramos. Egocentric activity recognition budget. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5967- 5976, 2018.
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. Advances in neural information processing systems, 27, 2014.
Salwa O Slim, Ayman Atia, Marwa MA Elfattah, and Mostafa-Sama M Mostafa. Survey on human activity recognition based acceleration data. International Journal of Advanced Computer Science and Applications, 10(3), 2019.
Sibo Song, Vijay Chandrasckhar, Ngai-Man Cheung, Sanath Narayan, Liyuan Li, and Joo-Hwee Lim. Activity recognition in egocentric life-logging videos. In Asian conference on computer vision, pp. 445-458. Springer, 2014.
Sibo Song, Vijay Chandrasckhar, Bappaditya Mandal, Liyuan Li, Joo-Hwee Lim, Giduthuri Sateesh Babu, Phyo Phyo San, and Ngai-Man Cheung. Multimodal multi-stream deep learning for egocentric activity recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 24-31, 2016a.
Sibo Song, Ngai-Man Cheung, Vijay Chandrasekhar, Bappaditya Mandal, and Jie Liri. Egocentric activity recognition with multimodal fisher vector. In 2016 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 2717-2721. IEEE, 2016b.
Odongo Steven Eyobu and Dong Seog Han. Feature representation and data augmentation for hu- man activity classification based on wearable imu sensor data using a deep Istm neural network. Sensors, 2889), 2018.
Lin Sun, Kui Jia, Kevin Chen, Dit- Yan Yeung, Bertram E Shi, and Silvio Savarese. Lattice long short-term memory for human action recognition. In Proceedings of the IEEE international ference on computer vision, pp. 2147-2156, 2017.
Senem Tanberk, Zeynep Hilal Kilimei, Dilek Bilgin Tükel, Mitat Uysal, and Selim Akyokuş. A hybrid deep model using deep learning and dense optical flow approaches for human activity recognition. IEEE Access, 8:19799-19809, 2020. doi: 10.1 109//OCSS 10.110//CCESS..20..2685 2020.2968529.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresano, and Manohar Paluti. Learning spa- tiotemporal features with 3d convolutional networks,". ICCV, 2015.
Jiahao Wang, Qiuling Long, Kexuan Liu, Yingzi Xie, et al. Human action recognition on cellphone using compositional bidir-lstm-ent networks. In 2019 International Conference Computer, Network, Communication and Information Systems (CNCI 2019), pp. 687-692, Atlantis Press, 2019a.
Jindong Wang, Yiqiang Chen, Shuji Hao, Xiaohui Peng. and Lisha Hu. Deep learning for sensor- based activity recognition: A survey. Pallern recognition letters, 119:3-11, 2019b.
Limin Wang, Yuanjun Xiong, Zhe Wang. Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In European conference on computer vision, pp. 20-36. Springer, 2016.
Zhiguang Wang and Tim Oates. Encoding time series images for visual inspection and classifica- tion using tiled convolutional neural networks. In Workshops at the twenty-ninth AAAI conference artificial intelligence, 2015.
11

---

Under review as a conference paper at ICLR 2023
Haoran Wei, Roozbeh Jafari, and Nasser Kehtarnavaz. Fusion of video and inertial sensing for deep learning- based human action recognition. Sensors, 19(17)33680, 2019.
Yunan Wu, Feng Yang, Ying Liu, Xuefan Zha, and Shaofeng Yuan. A comparison of 1-d and 2-d
deep convolutional neural networks in ecg classification. arXiv preprini arXiv:1810.07088. 2018.
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotem- poral feature learning: Speed-accuracy trade-offs in video classification,,, ECCV, 2018.
Huatao Xu, Pengfei Zhou, Rui Tan, Mo Li, and Guobin Shen. Limu-bert: Unleashing the potential of unlabeled data for imu sensing applications. In Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems, pp. 220-232 233, 2021.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices,". CVPR, 2018.
12


Reviews:
[{"review_id": "sZP6AGLlF8I", "reviewer": "Reviewer_hEtV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for multi-modal human activity recognition using RGB and IMU data. The main contributions include 1) modeling IMU data using several 2D images following the approach of Wang and Oats (2015) to build a Gramian Angular Field (GAF). The paper then proposes 2) extracting equal sized diagonal matrices from the GAF and then creates another multi-channel image that randomly subsamples this image several times to create a richer representation. This is the IMU feature representation multiscale input image that is then 3) combined with the RGB subscaled images through a learned attention mechanism to create a final representation that through a few NN layers and softmax returns the probability of each human activity class.\n\nThe paper provides results on three public datasets and provides comparable or better results than state-of-the-art. The paper also provides some ablation studies demonstrating better results with multi-modality using all the signals compared to other combinations with fewer signals and also demonstrates the benefits of using the attention module above just using concatenated features.", "summary_of_the_review": "As mentioned above, the main concerns I have are around minimal novelty and no deep insights into why the approach performs better than the previous works. The paper does have an ablation study showing attention improvements against simple concatenation. However as mentioned above there are some loose ends (why not use the correlations in the GAF directly) that reduce the impact of the paper's experiments. I would have loved to see more experimentation, demonstration of improvements on specific examples against state of the art and a discussion for what may be missing for the results that are still not performing well enough. In my opinion, this paper requires more empirical work as well as a discussion that could provide deeper insights into the area. It is not ready for publication. ", "strength_and_weaknesses": "Strengths:\n\n+ The approach is fairly well described and appears to be a reasonable application of attention to segments of GAF images.\n+ The results do show mild improvements above the state-of-the-art.\n\nWeaknesses:\n- The main weakness here is the lack of novelty in the paper. The paper takes about two pages just to describe GAFs which is previously published work by Wong and Oates (2015). To be clear, the paper cites their work correctly, however this seems to be the main technology used in the paper in addition to learned attention weighting of the different modalities. Attention mechanisms and transformers are also fairly well known works in the field. Therefore, this paper is more of an application paper than one that proposes novel ideas.\n- Given that the novelty in the paper is low, I would look for deep analysis on insights learned from the application to the datasets provided. The paper does provide results showing improvements with their proposed network but does not go deep into understanding why. Also there are no discussions on error modes or where the method works better or worse than state-of-the-art.\n- The authors proposed using attention for weighing multiple segment features across a large sequence. However the way the features were constructed using GAF, there are already correlations computed across longer time segment ranges. Why not build a feature with these correlations as opposed to using only the diagonals of the GAF? This kind of deeper analysis would have been helpful.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jiWrqNsLeML", "reviewer": "Reviewer_4Euk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors propose the MMTSA architecture, which stands for “Multimodal Temporal Segment Attention Network for Efficient Human Activity Recognition”. MMTSA is a network aimed to perform sensor fusion for imu and video data. By transforming the sensor data as GAF images, sampling them and using an attention mechanism, authors show an efficient approach for merging multi-modality data, modeling it as video data. The approach is tested on three public datasets, showing significant improvements and the overall feasibility of this architecture.", "summary_of_the_review": "The paper presents an interesting approach which I believe can be of potential interest for practitioners. However it would need to cover better the claims and initial statements to increase the value of the work. I think this work would benefit greatly from a more extensive and deeper discussion. I’d rather see less theory about the GAF images and more details about data syncing (the attention mechanism?), real life challenges when dealing with multimodal data,  and more insights regarding the experiments.", "strength_and_weaknesses": "This work indeed includes quite some strengths. This work presents an approach which I consider can be of interest for the community. It explores the representation of IMU signals as images, modeled by vision based approaches, completed with RGB data. The overall novelty of the work is not too strong, since it combines already known data representation with also known modeling on not novel datasets, however the combination of all I believe brings an interesting proposal. \n\nThe ablation study, although not extensive, I also think is quite interesting since it helps to illustrate the potential contribution of each data modality.\n\nHowever, the paper also presents some weaknesses. In my opinion the main flaw of the paper is the discrepancy between the technical challenges the authors claim to overcome and the final results and discussion.\n\n- An interesting point the authors raise regarding existing architectures is how “ … attention-based multi-modal learning methods [have] complicated architectures lead to high computational overhead and make them challenging to be deployed on mobile and wearable devices”, which I totally share. However I do not see how the proposed model simplifies such state of art to any degree. The approach is not characterized nor benchmarked. Results are not aimed toward measuring performance. I’d love to see how this proposal can bring a computation improvement and can work towards a feasible model running in a wearable device. Have the authors included any data in that regard that I may have missed?\n\n- Another claim is how this approach leverages the synchronous property among data modalities, however for me it’s not clear at all how this approach enables the synchronization of data from different modalities (which indeed need to be synced in order to be used by MMTSA). Can authors explain or maybe add further information regarding this point?\n\nI also believe that this work would greatly benefit from a deeper discussion, to come up as a much stronger work. \n- Some explanation about how the data synchronization among modalities i feasible\n- Some covering on the challenges of data labeling for this domain, which potentially may require to label video and sensor data. What are the options for getting training data? Would it be possible to train an independent network on independent datasets? Or finetune iteratively?\n- A more clear statement (and results) on how the attention mechanism is contributing to the model, is the key for syncing data? \n- How the model is simplifying (if any) current state of art\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WSAmeD9mYci", "reviewer": "Reviewer_6cv1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a method to perform human activity recognition using a novel multimodal temporal segment attention network leveraging a combination of RGB videos and IMU sensor (accelerometer and gyroscope) data from wearable sensors. To bring the IMU sensor data closer to the RGB data, the authors convert them to grayscale images using a Gramina Angular Field-based isomorphism. They perform sparse co-sampling of both the RGB and the transformed IMU sensor images to reduce the temporal redundancy and design attention networks to eventually predict the activity labels. The authors perform experiments to demonstrate the benefits of using their multimodal data and their attention mechanism.", "summary_of_the_review": "The paper presents an interesting method for human activity recognition using a multimodal attention mechanism on RGB video data and structurally similar grayscale images of transformed IMU sensor data. However, the components of the proposed method are all built on prior work and I am not clear on the technical limitations that the authors overcame by coming up with the design of their method. I invite the authors to respond to my concerns under \"weaknesses\" to get a clearer picture of their contributions before I can recommend acceptance.", "strength_and_weaknesses": "**Strengths**\n1. The proposed approach of transforming the IMU sensor data to grayscale images is sound and interesting. The transformation process is also clearly explained.\n2. The proposed attention mechanism provides appreciable performance improvements as per the authors' experiments.\n3. The paper is overall well-written and well-organized.\n\n\n**Weaknesses**\n1. While the paper nicely explains the different components of the proposed method, those components all appear to be based on prior work. I am not clear on the technical limitations this work overcomes compared to prior work. For example, what are the limitations in processing the 1D IMU sensor data using simpler recurrent networks and then combining the downstream features? How does \"converging\" the structural differences between the RGB and the IMU sensor data help with the temporal correlation that the authors mention in Sec. 1? Related to this question, are there any supporting works (or experiments performed by the authors) that show how and why existing multimodal methods ignore the temporal correlations between these types of data? On the other hand, why have the authors not considered optical flow alongside RGB video data, given that optical flow methods are popular for video-based activity recognition (such as Tanberk et al. 2020 and the seminal work of Simonyan and Zisserman 2014).\n\n2. The authors mention working with mobile devices and wearables, where the data can be highly noisy (shaky camera, general arm movements not related to any particular activities). How do the authors deal with the noise in their end-to-end pipeline? A simple experiment to demonstrate noise robustness is to artificially add noise into cleaned data and monitor the network performance as the noise level is increased.\n\n3. Is the proposed attention mechanism specifically designed for the RGB and the IMU sensor modalities or can it be reused and extended to other modalities such as audio and physiological signals such as heart rate?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WO3pu7P0n_4", "reviewer": "Reviewer_9pPc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a deep architecture based on RGB and IMU wearable sensors for multi-modal human activity recognition. The authors propose to encode the IMU sensor series into GAF images and then feed the RGB and GAF images into 2D CNNs for classification. Experiments are conducted on three public datasets.", "summary_of_the_review": "The paper is clear and easy to understand. However, there are many issues in the current manuscript, such as the novelty and experimental comparisons. Therefore, I would recommend rejecting this paper in the current form.", "strength_and_weaknesses": "Strengths:\n1. The paper is clear and easy to understand.\n2. The proposed model is straightforward and simple.\n\nWeaknesses:\n1. My biggest concern of this paper is novelty. All of the proposed three modules are not novel.\n(1) Multimodal data isomorphism mechanism. As the most significant contribution of this paper, it is obvious that the idea of encoding time series data as images via GAF is inspired by Wang & Oates (2015). It is ok if only the motivation and basic logic is similar. However, I find out that the encoding method, the usage of notations, the equations, and even the figures are all very similar with the original GAF. The only difference is the normalization function in Eq. 5. I would like the other reviewers and editors to notice this problem, and the authors to provide a reasonable explanation about this problem.\n(2) Multimodal sparse sampling. The sampling method proposed in the paper is just a simple extension of TSN (Wang et al. (2016)), where the authors additionally sample a GAF image from each segment. This extension is quite smooth and straightforward.\n(3) Inter-segment modality attention mechanisms. This attention-based fusion method is rather simple and has been widely used in previous works for activity recognition [a, b].\n[a] Meng, Lili, et al. \"Interpretable spatio-temporal attention for video action recognition.\" In Workshop on ICCV, 2019.\n[b] S. Sharma, R. Kiros, and R. Salakhutdinov, “Action recognition using visual attention.” In Workshop on ICLR, 2015.\n2. Deep analysis is missing in the paper. The authors should give a clear comparison with SOTA methods not only on experimental performance but also on theoretical technology.\n3. The performance comparisons are not satisfactory. The methods being compared in Tables 2 and 3 are out-of-date. The authors should compare with the state-of-the-art methods.\n4. The inference details, the introduction to the datasets, and the qualitative results are missing in the paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject