# Activity Recognition via Multimodal Large Language Models and Riemannian Optimization

Farnaz Soleimani, Ghazaleh Khodabandelou, Abdelghani Chibani, Yacine Amirat

► To cite this version:

Farnaz Soleimani, Ghazaleh Khodabandelou, Abdelghani Chibani, Yacine Amirat. Activity Recognition via Multimodal Large Language Models and Riemannian Optimization. 2025. hal-04943796

HAL Id: hal-04943796

https://hal.science/hal-04943796v1

Preprint submitted on 26 Feb 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Activity Recognition via Multimodal Large Language Models and Riemannian Optimization

Farnaz Soleimani, Ghazaleh Khodabandelou*, Abdelghani Chibani, and Yacine Amirat

***Abstract***—Human Activity Recognition (HAR) has become a critical task in applications such as healthcare, smart environments, and human-computer interaction. This study investigates the potential of using a publicly available GPT-2 variant for HAR with multimodal data, combining its natural language processing capabilities with advanced temporal modeling for sequential data. Furthermore, the study introduces a novel Riemannian manifold optimization strategy that enhances model generalization by leveraging the geometric structure of the parameter space. The model's performance is evaluated on both unimodal and multimodal datasets to assess its adaptability and effectiveness. Specifically, the GPT-2 model is fine-tuned on the UCI-HAR and Opportunity datasets (unimodal) and a multimodal dataset comprising synchronized RGB video, depth, skeleton, and inertial data. The fine-tuned GPT-2 achieves state-of-the-art performance, with 84% accuracy on the UCI-HAR dataset—a 2% improvement over previous benchmarks—and 84.62% accuracy on the Opportunity dataset. For the multimodal dataset, the model achieves an outstanding accuracy of 99.42%, demonstrating its capability to effectively integrate and process diverse sensor modalities. These results establish the viability of GPT-based architectures for HAR, particularly with multimodal data, and pave the way for further advancements in transformer-based models for sensor-driven activity recognition.

***Impact Statement***—This paper presents a novel Riemannian manifold optimization strategy designed for transformers, enhancing their generalization and robustness in HAR tasks. The proposed approach is validated on both a standard transformer and a GPT variant across unimodal and multimodal datasets. This extensive evaluation highlights the strategy's effectiveness in managing diverse data modalities while leveraging the complementary strengths of multimodal inputs. By achieving state-of-the-art performance on HAR benchmarks, this work sets a new paradigm for transformer optimization, opening avenues for further advancements in multimodal data processing and activity recognition.

***Index Terms***—Multimodal learning, Large Language Models, Human activity recognition, GPT-2, Riemannian optimization, Sensor fusion, Sequential modeling, Transformer models

## I. INTRODUCTION

Human Activity Recognition (HAR) is a rapidly advancing field that plays a critical role in a wide range of applications, from personalized healthcare and smart homes to security systems and advanced human-computer interaction. The core objective of HAR is to identify and classify human actions based on data collected from various sensors, including accelerometers, gyroscopes, cameras, and wearable devices. By leveraging HAR, systems can monitor physical activities such

as walking, standing up, or sitting down, as well as more complex ones like preparing a meal or performing personal care tasks. This ability to interpret human actions is crucial for creating more adaptive and intelligent environments [1].

Research in HAR began in the early 2000s, largely driven by advancements in wearable technology and machine learning (ML). Early foundational studies, such as the pioneering use of wearable sensors [2] to analyze and classify physical activities, laid the groundwork for sensor-based HAR.

HAR systems can generally be categorized into two types: video-based systems and wearable sensor-based systems. Video-based systems utilize modalities such as RGB, skeleton, and depth data to capture rich contextual information, often achieving high recognition accuracy. However, these systems face challenges, including privacy concerns associated with visual data, the fixed field of view of cameras, and performance degradation due to environmental factors [3]. Each modality in HAR offers unique strengths and limitations. For example, RGB video provides detailed visual information but is sensitive to variations in lighting, viewpoint, and background, making it less reliable in certain scenarios. Depth maps and skeleton data, on the other hand, contribute valuable 3D spatial information that mitigates these issues, though they lack appearance-based details, which are often critical for recognizing interactive activities such as writing or reading [4]. Skeleton data, focused solely on human motion, excludes contextual information about other objects in the scene. Conversely, RGB data captures these elements, enabling the recognition of object-related interactions.

In contrast, wearable sensor-based systems, which utilize accelerometers and gyroscopes embedded in portable devices, offer the advantages of ensuring privacy and enabling continuous tracking. However, these systems typically achieve lower recognition accuracy due to their limited ability to capture contextual information. Furthermore, the accuracy of wearable sensor-based systems is highly influenced by the placement of the sensors on the body [3].

As traditional ML gained momentum in the 1990s, models designed for multimodal problems began to flourish, addressing modality-specific challenges more effectively. Multimodal HAR has enabled a more comprehensive and accurate understanding of human activities by leveraging the unique strengths of various data sources, such as video, audio, and sensor data, each providing distinct yet complementary perspectives on the same activity. For instance, combining wearable sensor data with video [5] or integrating audio with wearable sensors [6] significantly improves activity recognition accuracy by capturing long-term temporal relationships. Moreover, combining

F. Soleimani, G. Khodabandelou (corresponding author: ghazaleh.khodanbandelou@u-pec.fr), A. Chibani, and Y. Amirat are with the LISSI Laboratory, University of Paris-Est, 61 Avenue du Général de Gaulle, Créteil 94010, France. E-mail: ghazaleh.khodanbandelou@u-pec.fr
depth data with inertial sensor data [7] has resulted in an improvement of more than 11% in recognition accuracy.

Since 2010, deep learning has significantly advanced multi-modal data fusion, enabling computational models with multiple processing layers to automatically learn representations from diverse data types without the need for manual feature engineering [8]. For example, Ronao et al. [9] demonstrated the effectiveness of deep learning for multimodal HAR. Ordonez et al. [10] introduced deep convolutional and LSTM recurrent neural networks for multimodal wearable activity recognition, while Nweke et al. [11] emphasized the effectiveness of combining CNNs and LSTMs to capture spatiotemporal features from multimodal data. Furthermore, Inoue et al. [12] integrated video and wearable sensor data within deep learning models, significantly improving HAR accuracy.

Recently, transformer-based models, which leverage self-attention mechanisms, have demonstrated promising results across various domains, including natural language processing (NLP). The self-attention mechanism enables these models to focus on relevant features across diverse data sources, thereby improving robustness and precision. For example, video-text multimodal fusion tasks have been addressed using Transformers, BERT, attention mechanisms, adversarial learning, and combinations of these methods, as demonstrated by Gabeur et al. [13], Han et al. [14], Huang et al. [15], Liu et al. [16], Mithun et al. [17], [18], and Wray et al. [19]. These models, known for their ability to capture long-range dependencies and contextual information, are now being explored for HAR as well. For instance, Dirgova et al. [20] and Li et al. [21] utilized transformer architectures for wearable sensor-based activity recognition, demonstrating superior performance compared to traditional RNN and LSTM models. Additionally, Wu et al. [22] and Chen et al. [23] introduced the HART (Human Activity Recognition Transformer) model, a lightweight transformer architecture specifically designed for embedded IMU sensing devices. HART efficiently processes large-scale multimodal sensor data, achieving real-time inference capabilities with low energy consumption, making it particularly suitable for mobile and wearable applications. In another study, Zhang et al. [24] applied transformer models to HAR using multi-modal datasets, resulting in improved accuracy for activity recognition. Similarly, Fabregas and Farias [25] employed BERT for activity recognition based on sequences of skeleton features, highlighting BERT's ability to handle complex data representations and enhance recognition accuracy.

This paper proposes leveraging a custom Transformer and GPT-2 for HAR. While GPT-2 has previously been applied to the unimodal UCI-HAR dataset [26], this work extends its application to multimodal data by evaluating its performance across both unimodal and multimodal datasets. Additionally, the proposed multimodal Transformer architecture, along with GPT-2, incorporates a novel Riemannian manifold optimization approach to effectively integrate diverse data sources—such as images, videos, Kinect skeletal data, and inertial sensors—capturing complementary information to significantly enhance activity recognition performance. By leveraging the inherent geometric properties of the data and parameter space through Riemannian manifold optimization,

the model achieves more efficient optimization by respecting the curved nature of the manifold, resulting in improved generalization to unseen data.

The rest of the paper is organized as follows: Section 2 reviews both traditional and modern approaches, highlighting their limitations and potential opportunities. Section 3 provides a detailed explanation of the proposed architecture, including the integration of multimodal data and the application of Riemannian optimization. Section 4 describes the datasets used in this study and outlines the preprocessing steps applied. Section 5 presents the performance evaluation of the proposed approach, comparing it with existing methods. Finally, the concluding section summarizes the key contributions and outlines potential directions for future research.

## II. RELATED WORK

HAR has undergone significant evolution, transitioning from traditional ML techniques to deep learning methods and, more recently, to Transformer-based architectures. Each stage has addressed the limitations of its predecessors, introducing innovative methods to enhance recognition accuracy, scalability, and adaptability. This section reviews key works in HAR, organized into three categories: traditional ML, deep learning, and Transformer-based methods. It also highlights the limitations and challenges that inspire the solutions proposed in this study.

HAR initially relied on traditional ML approaches, which involved extracting handcrafted features from raw sensor data using statistical methods (e.g., mean, standard deviation) or signal processing techniques (e.g., Fast Fourier Transform (FFT), wavelet transforms). For example, Bao and Intille [2] used data from five body-worn sensors to classify activities with traditional ML algorithms such as Decision Trees, k-NN, and Naive Bayes. Similarly, Kwapisz et al. [27] evaluated three classifiers on the WISDM dataset: Multi-Layer Perceptron (MLP), J48 Decision Tree, and Logistic Regression, finding that MLP outperformed the other models. While traditional ML approaches provided a foundational framework for HAR, they were limited by their reliance on handcrafted features, which required significant domain expertise and were often time-intensive to design. Additionally, these methods faced several challenges: (i) Handling high-dimensional and noisy data, which are common in real-world HAR scenarios, thus limiting their scalability for more complex applications; (ii) Generalizing to unseen subjects or activities, often resulting in overfitting; (iii) Capturing temporal dependencies, which are crucial for recognizing sequential activities. As datasets became larger and more complex, traditional ML methods encountered computational bottlenecks and scalability limitations, underscoring the need for more robust and automated approaches.

Deep learning revolutionized HAR by automating feature extraction and significantly improving scalability and generalization. Convolutional Neural Networks (CNNs) excelled at extracting spatial features from image or video data (e.g., RGB, depth, or skeleton), while Recurrent Neural Networks (RNNs) and their variants, such as Long Short-Term Memory (LSTM) networks and Gated Recurrent Units (GRUs), proved
effective at modeling temporal dependencies in sequential data. For instance, Yao et al. [28] introduced DeepSense, a unified framework combining CNNs and RNNs to process time-series sensor data for mobile HAR applications. Similarly, Ordonez and Roggen [10] proposed a CNN-LSTM hybrid model for multimodal wearable sensor data, including datasets such as Opportunity and Skoda. Their study demonstrated that deep learning models not only outperformed traditional ML methods in feature extraction but also enhanced activity recognition performance, particularly for sequential data. Deep learning approaches have also shown exceptional performance in multimodal HAR, where data from diverse sources—such as RGB, depth, and inertial sensors—are fused to improve recognition accuracy. However, despite their success, these approaches faced challenges in capturing long-range dependencies and modeling complex interactions between modalities. These issues are especially pronounced in multimodal HAR, where effective data fusion is critical. Additionally, CNNs and RNNs encountered limitations in scalability, generalization to unseen subjects or activities, and computational efficiency when applied to large-scale datasets [29].

Transformers, introduced by Vaswani et al. [30], revolutionized sequence modeling by employing a self-attention mechanism that effectively captures long-range dependencies and contextual relationships across entire input sequences. This innovation enabled state-of-the-art performance in natural language processing (NLP) tasks and has since inspired the adoption of Transformers in other domains, including HAR. Initially focused on unimodal sensor data, particularly sequences from wearable devices, HAR has seen significant advancements with Transformer-based architectures. Shavit and Klein [31] demonstrated that Transformers outperform RNNs and LSTMs in modeling temporal dependencies in inertial sensor data. Ek et al. [32] introduced a lightweight Transformer architecture optimized for mobile devices, achieving an effective balance between performance and computational efficiency, making it suitable for real-time applications in embedded systems. For example, Djenouri et al. [33] integrated Transformers to model temporal dependencies across video and depth data, achieving state-of-the-art performance. Moreover, multimodal approaches enhance robustness by allowing one modality to compensate for noise or missing data in another [34]. However, challenges persist in balancing computational efficiency and ensuring fair contributions from all modalities, especially when processing-intensive visual data is combined with sensor data. Transformers excel in HAR tasks by leveraging self-attention mechanisms to effectively capture both short- and long-range dependencies across entire sequences. This capability overcomes limitations such as vanishing gradients and limited memory found in RNNs and LSTMs. The ability to process entire input sequences simultaneously enables faster training, scalability for larger datasets and longer sequences, and adaptability through positional encodings. Unlike CNNs and RNNs, which are tailored to specific data types, Transformers provide a unified architecture for diverse modalities, enhancing interpretability through attention weights and contextual modeling. Their straightforward design eliminates the need for complex mechanisms like gating

structures, making them highly efficient for long-sequence processing, as required in video-based HAR and time-series data analysis. Consistently outperforming traditional models across HAR benchmarks, Transformers demonstrate superior adaptability, efficiency, and state-of-the-art performance. Despite their strengths, Transformers face challenges such as the quadratic complexity of the self-attention mechanism, leading to high computational and memory costs for long sequences. Additionally, they require large-scale pretraining datasets, which limits their effectiveness in domains with limited annotated data. High memory consumption further restricts their deployment on resource-constrained devices. In multimodal HAR, balancing contributions from diverse data types, such as video and sensor data, demands careful parameter tuning to maintain computational efficiency. Addressing these challenges remains a key focus of ongoing research to optimize Transformers for real-world applications.

As Transformer models continue to grow in size and computational cost, significant research efforts have been devoted to understanding and improving their optimization processes. These efforts aim to address challenges related to training stability, computational efficiency, and scalability, ensuring that Transformers remain practical for a wide range of applications. Huang et al. [35] identified critical optimization challenges in encoder-decoder Transformer architectures and proposed a novel weight initialization scheme with theoretical justification. Their approach eliminated the need for warmup or layer normalization during training, enabling seamless optimization of deep Transformers with up to 200 layers. This innovation achieved state-of-the-art accuracy on machine translation benchmarks, showcasing the potential of enhanced optimization strategies for unlocking deeper Transformer architectures. Gradient optimization in attention layers, a central component of Transformer models, remains notoriously difficult and often requires techniques such as learning rate warm-up to prevent divergence. Yang et al. [36] explored the theoretical underpinnings of Transformers by investigating whether their forward pass could be interpreted as steps in an energy minimization process. They demonstrated a partial connection between self-attention layers and energy minimization, offering new insights into Transformer interpretability and informing potential future designs. Inspired by traditional optimization methods, Gartner et al. [37] introduced Optimus, a Transformer-based architecture for learned optimization. Drawing from the BFGS algorithm, Optimus predicts rank-one updates for a preconditioning matrix, step length, and direction, enabling efficient optimization across variable-dimensional tasks without requiring retraining. This approach bridges the gap between classical optimization algorithms and modern deep learning architectures. Fang et al. [38] proposed the use of Rational Activation Functions (RAFs) in Transformers, allowing activation functions to adapt dynamically during training. Unlike fixed activations, RAFs optimize themselves based on input data, enhancing both flexibility and performance across a range of tasks. Beyond architectural innovations, researchers have introduced enhancements aimed at reducing the computational and memory demands of Transformers. Efficient attention mechanisms, such as Sparse Attention and Linformer, address
the quadratic complexity of self-attention by focusing on subsets of tokens or approximating attention matrices using low-rank projections. Techniques like gradient clipping and refined layer normalization improve training stability, while regularization methods, such as dropout and label smoothing, enhance generalization by mitigating overfitting. Learning rate schedulers, such as warmup-coOLDOWN schedules combined with optimizers like AdamW, further improve convergence efficiency. Parameter-efficient models, such as ALBERT, employ layer sharing to reduce redundancy, making them more practical for resource-constrained environments. Dynamic token pruning, which discards irrelevant tokens during training and inference, minimizes computational overhead while preserving critical sequence information. For multimodal applications, particularly in HAR, cross-modal attention mechanisms have been developed to optimize resource allocation by prioritizing relevant inter-modal interactions. Pretraining on large datasets followed by task-specific fine-tuning has become a standard practice, reducing data requirements for specialized applications. Hardware-specific optimizations, such as quantization-aware and mixed-precision training, have further enabled the deployment of Transformers on edge devices. Lastly, refined loss functions, such as focal loss, address challenges like data imbalance by focusing on hard-to-classify samples, ensuring more robust and accurate performance across diverse tasks.

The aforementioned advancements have significantly enhanced the efficiency, scalability, and adaptability of Transformers. As research continues to evolve, these innovations are expected to further drive the development of Transformer models, making them increasingly accessible for real-time, multimodal, and resource-constrained applications.

### III. METHODOLOGY

The Transformer model [30] has revolutionized sequence modeling tasks by leveraging self-attention mechanisms to effectively capture long-range dependencies. In the context of sensor-based HAR, we adapt the Transformer architecture to process multimodal time-series data collected from diverse sensors, including accelerometers, gyroscopes, and magnetometers.

#### A. Multimodal Transformer Architecture

This section introduces a cutting-edge transformer-based model designed for activity recognition tasks using multimodal datasets. The model processes diverse data modalities, including images, videos, Kinect skeletal data, and inertial sensor readings. By leveraging the capabilities of Large Language Models (LLMs) or Large Action Models (LAMs), the proposed architecture effectively integrates multimodal information to enhance activity recognition performance. Additionally, we present a novel loss function inspired by Riemannian manifold optimization, aimed at improving the efficiency and accuracy of the training process.

1) *Data Preprocessing and Normalization:* Prior to feeding the multimodal data into the model, several preprocessing steps are essential to ensure data quality and consistency across modalities. For visual data (images and videos), standard

<figure><img src="image_2.png" /><figcaption>Fig. 1: Multimodal Activity Recognition Pipeline leveraging the proposed loss function.</figcaption></figure>

preprocessing techniques are applied, including resizing to a fixed resolution (H × W), cropping, and normalization to standardize pixel intensities. Specifically, images and video frames are normalized per channel to achieve zero mean and unit variance:

$$ \tilde{\mathbf{I}}_t = \frac{\mathbf{I}_t - \mu_{\text{img}}}{\sigma_{\text{img}}}, \quad \tilde{\mathbf{V}}_t = \frac{\mathbf{V}_t - \mu_{\text{vid}}}{\sigma_{\text{vid}}}, \qquad (1) $$

where $\mu_{\text{img}}$, $\sigma_{\text{img}}$, $\mu_{\text{vid}}$, and $\sigma_{\text{vid}}$ are the mean and standard deviation computed over the training dataset for images and videos, respectively.

For Kinect skeletal data, the joint coordinates $\mathbf{S}_t$ are normalized by subtracting the position of the reference joint (e.g., the hip center) and scaling them to unit length:

$$ \tilde{\mathbf{S}}_t = \frac{\mathbf{S}_t - \mathbf{s}_{\text{center}}}{\|\mathbf{S}_t - \mathbf{s}_{\text{center}}\|_2}, \qquad (2) $$

where $\mathbf{s}_{\text{center}}$ represents the position coordinates of the reference joint center.

Inertial sensor readings $\mathbf{R}_t$ are filtered using a low-pass filter to remove high-frequency noise:

$$ \tilde{\mathbf{R}}_t = \alpha \mathbf{R}_t + (1 - \alpha) \tilde{\mathbf{R}}_{t-1}, \qquad (3) $$

where $\alpha$ is the smoothing factor. The filtered readings are then normalized:

$$ \tilde{\mathbf{R}}_t = \frac{\tilde{\mathbf{R}}_t - \mu_{\text{ine}}}{\sigma_{\text{ine}}}, \qquad (4) $$

where $\mu_{\text{ine}}$ and $\sigma_{\text{ine}}$ represent the mean and standard deviation of the inertial data, respectively.

Temporal alignment is essential when dealing with multimodal data collected asynchronously. To achieve this, data streams are synchronized by resampling each modality to a common frame rate $f_s$, ensuring that corresponding time steps across modalities accurately represent the same moment in the action sequence.

At each time step $t$, the input data comprises multiple modalities:

$$ \mathbf{X}_t = \{\tilde{\mathbf{I}}_t, \tilde{\mathbf{V}}_t, \tilde{\mathbf{S}}_t, \tilde{\mathbf{R}}_t\}. \qquad (5) $$

Each modality is processed through a modality-specific encoder to extract features suitable for fusion.

2) *Modality-Specific Encoders:* To handle the diverse data types, specialized encoders are employed for each modality.
### Image and Video Encoders:

For image data, a convolutional neural network (CNN) is employed to extract spatial features:

$$ \mathbf{f}_{t}^{\text{img}} = \text{CNN}_{\text{img}}(\tilde{\mathbf{I}}_{t}; \Theta_{\text{img}}), \qquad (6) $$

where $\Theta_{\text{img}}$ are the learnable parameters of the Image CNN. Similarly, for video data, a 3D CNN or a sequence of 2D CNNs with temporal pooling is used:

$$ \mathbf{f}_{t}^{\text{vid}} = \text{CNN}_{\text{vid}}(\tilde{\mathbf{V}}_{t-\delta}^{t+\delta}; \Theta_{\text{vid}}), \qquad (7) $$

where $\tilde{\mathbf{V}}_{t-\delta}^{t+\delta}$ represents a sequence of frames centered at time $t$ within a temporal window of size $2\delta + 1$.

### Kinect Skeletal Data Encoder:

Skeletal data is represented as a graph $G = (V, E)$, where $V$ denotes the set of joints and $E$ represents the set of bones (edges). A graph convolutional network (GCN) is used to capture spatial relationships:

$$ \mathbf{f}_{t}^{\text{ske}} = \text{GCN}(\tilde{\mathbf{S}}_{t}; \Theta_{\text{ske}}), \qquad (8) $$

where the GCN computes node features via:

$$ \mathbf{h}_{v}^{(l)} = \sigma \left( \sum_{u \in \mathcal{N}(v)} \frac{1}{c_{vu}} \mathbf{W}^{(l)} \mathbf{h}_{u}^{(l-1)} \right), \qquad (9) $$

with $\mathbf{h}_{v}^{(0)} = \tilde{\mathbf{S}}_{v}(v)$, $\mathcal{N}(v)$ denoting the neighbors of node $v$, $c_{vu}$ a normalization constant, $\mathbf{W}^{(l)}$ the weight matrix at layer $l$, and $\sigma(\cdot)$ an activation function.

### Inertial Sensor Encoder:

Inertial sensor readings are sequential data processed using a recurrent neural network (RNN), such as an LSTM:

$$ \mathbf{f}_{t}^{\text{ine}} = \text{LSTM}(\tilde{\mathbf{R}}_{1:t}; \Theta_{\text{ine}}), \qquad (10) $$

$$ \mathbf{h}_{t}^{\text{ine}}, \mathbf{c}_{t}^{\text{ine}} = \text{LSTMCell}(\tilde{\mathbf{R}}_{t}, \mathbf{h}_{t-1}^{\text{ine}}, \mathbf{c}_{t-1}^{\text{ine}}), \qquad (11) $$

where $\mathbf{h}_{t}^{\text{ine}}$ and $\mathbf{c}_{t}^{\text{ine}}$ are the hidden state and cell state at time $t$, respectively.

## B. Multimodal Feature Integration

### Multimodal Feature Fusion

The modality-specific features are projected into a common embedding space of dimension $d_{\text{model}}$ and concatenated:

$$ \mathbf{e}_{t} = [\mathbf{W}_{\text{img}}\mathbf{f}_{t}^{\text{img}}; \mathbf{W}_{\text{vid}}\mathbf{f}_{t}^{\text{vid}}; \mathbf{W}_{\text{ske}}\mathbf{f}_{t}^{\text{ske}}; \mathbf{W}_{\text{ine}}\mathbf{f}_{t}^{\text{ine}}] + \mathbf{b}, \qquad (12) $$

where $\mathbf{W}_{\text{mod}} \in \mathbb{R}^{d_{\text{mod}} \times d_{\text{model}}}$ are projection matrices for each modality, mapping the modality-specific feature dimension $d_{\text{mod}}$ to the common embedding dimension $d_{\text{model}}$, and $\mathbf{b} \in \mathbb{R}^{4d_{\text{model}}}$ is a bias term.

### Positional Encoding

To incorporate temporal information, positional encodings are added to the embeddings. Learnable positional embeddings are adopted:

$$ \mathbf{z}_{t} = \mathbf{e}_{t} + \mathbf{p}_{t}, \qquad (13) $$

where $\mathbf{p}_{t} \in \mathbb{R}^{4d_{\text{model}}}$ is a learnable vector associated with time step $t$.

### Transformer Encoder Layers

The sequence of embeddings $\{\mathbf{z}_1, \mathbf{z}_2, \dots, \mathbf{z}_T\}$ is processed through L Transformer encoder layers. Each layer consists of multi-head self-attention and position-wise feed-forward networks.

For each attention head $h$, queries, keys, and values are computed as follows:

$$ \mathbf{Q}_h = \mathbf{Z} \mathbf{W}_h^Q, \quad \mathbf{K}_h = \mathbf{Z} \mathbf{W}_h^K, \quad \mathbf{V}_h = \mathbf{Z} \mathbf{W}_h^V, \qquad (14) $$

where $\mathbf{Z} \in \mathbb{R}^{T \times 4d_{\text{model}}}$ is the matrix of input embeddings, and $\mathbf{W}_h^Q, \mathbf{W}_h^K, \mathbf{W}_h^V \in \mathbb{R}^{4d_{\text{model}} \times d_k}$ are the projection matrices for head $h$. The scaled dot-product attention for head $h$ is:

$$ \text{head}_h = \text{softmax} \left( \frac{\mathbf{Q}_h \mathbf{K}_h^\top}{\sqrt{d_k}} \right) \mathbf{V}_h. \qquad (15) $$

The outputs from all heads are concatenated:

$$ \mathbf{H} = [\text{head}_1; \text{head}_2; \dots; \text{head}_H] \mathbf{W}^O, \qquad (16) $$

where $\mathbf{W}^O \in \mathbb{R}^{Hd_k \times 4d_{\text{model}}}$.

Each position in the sequence is processed independently:

$$ \text{FFN}(\mathbf{x}) = \text{ReLU}(\mathbf{x}\mathbf{W}_1 + \mathbf{b}_1)\mathbf{W}_2 + \mathbf{b}_2, \qquad (17) $$

with $\mathbf{W}_1 \in \mathbb{R}^{4d_{\text{model}} \times d_{\text{ff}}}$, $\mathbf{W}_2 \in \mathbb{R}^{d_{\text{ff}} \times 4d_{\text{model}}}$.

Residual connections and layer normalization are applied:

$$ \mathbf{y} = \text{LayerNorm}(\mathbf{z} + \mathbf{H}), \qquad (18) $$

$$ \mathbf{h} = \text{LayerNorm}(\mathbf{y} + \text{FFN}(\mathbf{y})). \qquad (19) $$

**Output Layer:** After the Transformer encoder, a sequence of hidden states $\mathbf{h}_1, \mathbf{h}_2, \dots, \mathbf{h}_T$ is obtained. A pooling operation is applied to aggregate the temporal information:

$$ \mathbf{h} = \frac{1}{T} \sum_{t=1}^{T} \mathbf{h}_{t}, \qquad (20) $$

or alternatively, max pooling:

$$ \mathbf{h}_i = \max_t \mathbf{h}_{t,i}, \quad \forall i \in \{1, \dots, 4d_{\text{model}}\}. \qquad (21) $$

The pooled representation $\mathbf{h}$ is then passed through a classification layer:

$$ \hat{\mathbf{y}} = \text{softmax}(\mathbf{h}\mathbf{W}_c + \mathbf{b}_c), \qquad (22) $$

where $\mathbf{W}_c \in \mathbb{R}^{4d_{\text{model}} \times C}$ and $\mathbf{b}_c \in \mathbb{R}^C$ represent the weights and biases of the classification layer, respectively, and $C$ denotes the number of activity classes.

## C. Novel Loss Function Inspired by Riemannian Manifold Optimization

Traditional loss functions and optimization methods may fail to fully leverage the geometric structures inherent in multimodal data and the parameter space of deep neural networks. Treating the data and parameters as points on a Riemannian manifold enables the design of loss functions and optimization procedures that respect the underlying geometry, potentially improving both the optimization and generalization of the resulting models.
1) Proposed Loss Function: We propose a new loss function that incorporates a Riemannian metric:

$$L = L_{CE} + \lambda L_{RM}, \qquad (23)$$

where $L_{CE}$ is the standard cross-entropy loss, and $L_{RM}$ is the Riemannian manifold regularization term. The cross-entropy loss is defined as:

$$L_{CE} = -\frac{1}{N} \sum_{i=1}^{N} \sum_{c=1}^{C} y_{i,c} \log \hat{y}_{i,c}, \qquad (24)$$

where $N$ is the number of samples in a batch, $y_{i,c}$ is a binary indicator (1 if sample $i$ belongs to class $c$, 0 otherwise), and $\hat{y}_{i,c}$ is the predicted probability. The regularization term encourages the model parameters $\theta$ to lie on a Riemannian manifold $\mathcal{M}$ with metric tensor $g_{\theta}$:

$$L_{RM} = \frac{1}{2} \| \nabla_{\mathcal{M}} L_{CE} \|_{g_{\theta}}^{2}, \qquad (25)$$

where $\nabla_{\mathcal{M}} L_{CE}$ is the Riemannian gradient of the cross-entropy loss, and the norm is defined with respect to the metric tensor:

$$\| \nabla_{\mathcal{M}} L_{CE} \|_{g_{\theta}}^{2} = \nabla_{\mathcal{M}} L_{CE}^{\top} g_{\theta}^{-1} \nabla_{\mathcal{M}} L_{CE}. \qquad (26)$$

2) Optimization Algorithm: We adopt Riemannian stochastic gradient descent (RSGD) to update the model parameters. The Riemannian gradient $\nabla_{\mathcal{M}} L$ is computed by projecting the Euclidean gradient $\nabla L$ onto the tangent space $T_{\theta} \mathcal{M}$ at point $\theta$:

$$\nabla_{\mathcal{M}} L = P_{T_{\theta} \mathcal{M}}(\nabla L), \qquad (27)$$

where $P_{T_{\theta} \mathcal{M}}$ is the projection operator onto the tangent space. The parameters are updated using the Riemannian exponential map:

$$\theta_{t+1} = \text{Exp}_{\theta_t}(-\eta \nabla_{\mathcal{M}} L), \qquad (28)$$

where $\text{Exp}_{\theta_t}(\cdot)$ maps a tangent vector back onto the manifold $\mathcal{M}$.

Algorithm 1 represents the training process of the multimodal transformer with Riemannian manifold optimization.

### Algorithm 1 Training Multimodal Transformer with Riemannian Manifold Optimization

**Require:** Prepared dataset $\mathcal{D}_{\text{Prepared}}$, initial parameters $\theta_0 \in \mathcal{M}$, learning rate $\eta$, regularization coefficient $\lambda$, number of epochs $E$

1: **for** epoch = 1 to $E$ **do**

2: **for** each batch $(x, y)$ in $\mathcal{D}_{\text{Prepared}}$ **do**

3: **Forward Pass:** Compute predictions $\hat{y} = f(x; \theta_t)$

4: **Loss Computation:**

$$L_{CE} = \text{CrossEntropyLoss}(\hat{y}, y) \\ L_{RM} = \text{ManifoldRegularization}(\theta_t)$$

$$L = L_{CE} + \lambda L_{RM}$$

5: **Gradient Computation:** Compute Euclidean gradient $\nabla L$

6: **Riemannian Gradient Projection:** Project gradient onto tangent space:

$$\nabla_{\mathcal{M}} L = \text{Proj}_{T_{\theta_t} \mathcal{M}}(\nabla L)$$

7: **Parameter Update:** Update parameters using exponential map:

$$\theta_{t+1} = \text{Exp}_{\theta_t}(-\eta \nabla_{\mathcal{M}} L)$$

8: **end for**

9: **Validation:** Evaluate model performance on validation set

10: **end for**

11: **Output:** Trained model parameters $\theta^*$

For certain manifolds, the projection and exponential map have closed-form expressions. For instance, when $\mathcal{M}$ is the Stiefel manifold (the set of orthonormal matrices), the projection and retraction can be computed efficiently.

### D. Integration with Large Language Models

To enhance the model's ability to understand complex activities, LLMs are integrated to process textual descriptions or annotations associated with the data. The resulting textual embeddings are incorporated into the multimodal feature fusion process.

**Textual Feature Extraction:** We use a pre-trained LLM to encode text data:

$$\mathbf{f}^{\text{txt}} = \text{LLM}(\text{Text}; \Theta_{\text{LLM}}), \qquad (29)$$

where $\Theta_{\text{LLM}}$ are the parameters of the LLM. The text may include descriptions of the activities or contextual information.

**Incorporating Textual Features:** The textual features are projected into the common embedding space and concatenated with other modality features:

$$\mathbf{e}_t = [\mathbf{e}_t; \mathbf{W}_{\text{txt}}\mathbf{f}^{\text{txt}}] + \mathbf{b}_{\text{txt}}, \qquad (30)$$

where $\mathbf{W}_{\text{txt}} \in \mathbb{R}^{d_{\text{txt}} \times d_{\text{model}}}$ projects the textual features into the embedding space, and $\mathbf{b}_{\text{txt}} \in \mathbb{R}^{d_{\text{model}}}$ is a bias term.
## E. Convergence Proof Sketch

The convergence of the proposed Riemannian manifold optimization algorithm can be analyzed within the framework of Riemannian optimization theory. The main idea is to generalize classical optimization methods to manifolds by substituting Euclidean concepts with their Riemannian counterparts. Let $M$ be a Riemannian manifold equipped with a metric $g_\theta$, where $\theta \in M$ denotes the model parameters. The optimization problem is to minimize the loss function $L(\theta)$ over $M$:

$$ \min_{\theta \in M} L(\theta) = L_{CE}(\theta) + \lambda L_{RM}(\theta). \quad (31) $$

The Riemannian gradient $\nabla_M L(\theta)$ is defined as the orthogonal projection of the Euclidean gradient $\nabla L(\theta)$ onto the tangent space $T_\theta M$ at point $\theta$:

$$ \nabla_M L(\theta) = \operatorname{Proj}_\theta(\nabla L(\theta)), \quad (32) $$

where $\operatorname{Proj}_\theta: T_\theta \mathbb{R}^n \to T_\theta M$ is the projection operator defined by:

$$ \operatorname{Proj}_\theta(\mathbf{v}) = \mathbf{v} - \mathbf{n}\langle\mathbf{n}, \mathbf{v}\rangle, \quad (33) $$

with $\mathbf{n}$ representing the normal vector to the manifold at $\theta$, and $\langle \cdot, \cdot \rangle$ denoting the inner product induced by the metric $g_\theta$. The optimization algorithm updates the parameters along geodesics on the manifold using the Riemannian exponential map $\operatorname{Exp}_\theta$:

$$ \theta_{t+1} = \operatorname{Exp}_{\theta_t}(-\eta \nabla_M L(\theta_t)), \quad (34) $$

where $\eta$ is the learning rate. Under appropriate conditions on the manifold's curvature and the Lipschitz continuity of the gradient, it can be proven that the sequence $\theta_t$ converges to a critical point of $L(\theta)$. Specifically, if $L(\theta)$ is smooth and the manifold $M$ is complete with non-positive curvature, the Riemannian gradient descent algorithm converges to a local minimum. Moreover, with an appropriate choice of learning rate $\eta$, the convergence rate can be established. The proof involves showing that the loss function decreases monotonically along the sequence:

$$ L(\theta_{t+1}) \le L(\theta_t) - \eta \|\nabla_M L(\theta_t)\|^2 + \frac{L'}{2} \eta^2 \|\nabla_M L(\theta_t)\|^2, \quad (35) $$

where $L'$ is the Lipschitz constant of the Riemannian gradient. By selecting $\eta$ such that $0 < \eta < \frac{2}{L'}$, the sequence $\{L(\theta_t)\}$ is guaranteed to decrease, ensuring convergence.

Therefore, the proposed Riemannian optimization algorithm balances exploration and exploitation by moving along geodesics that respect the manifold's geometry, ensuring convergence to a (local) minimum of the loss function. Integrating the proposed Riemannian manifold optimization method into the Transformer training loop requires adapting the standard training procedure to operate on the Riemannian manifold. The changes are minimal, primarily involving the replacement of Euclidean gradient descent steps with Riemannian gradient updates.

## F. Advantages of Riemannian Regularization and $L^2$ norm

Riemannian regularization offers several key advantages for multimodal HAR tasks: (i) It aligns with the intrinsic geometric structure of multimodal data, such as videos, skeletons, and

sensor signals, by optimizing parameters on curved manifolds; (ii) Constraints imposed by the manifold help reduce overfitting, enabling the model to learn patterns that generalize well to unseen data; (iii) The method effectively integrates diverse data modalities by accounting for their distinct geometric properties, ensuring better fusion of complementary information; (iv) By smoothing optimization trajectories, Riemannian regularization enhances robustness against noisy sensor data, which is common in HAR applications; (v) it naturally aligns with advanced architectures like Transformers, stabilizing parameter optimization and improving overall training efficiency.

The $L^2$ norm provides smooth gradients and gradual updates, ensuring stable and efficient optimization, particularly for large models like Transformers. It prevents any single modality from dominating the learning process, ensuring balanced multimodal integration by penalizing large parameter values evenly. Furthermore, unlike $L^1$, which is non-differentiable at zero, $L^2$ is differentiable everywhere, simplifying implementation and avoiding potential optimization instabilities. In summary, the combination of Riemannian regularization and the $L^2$ norm ensures robust, efficient, and generalizable learning, making it an ideal choice for addressing the complexities of multimodal HAR tasks.

## IV. DATASETS

This section describes the datasets used in this study, along with the data preprocessing steps. Three datasets were selected to fulfill the study's requirements: a balanced unimodal dataset, the publicly available UCI-HAR dataset [39]; an imbalanced unimodal dataset, the Opportunity dataset [40]; and a multimodal dataset, the UTD-MHAD dataset [7].

The UCI-HAR dataset was designed to recognize six distinct human activities (walking, walking upstairs, walking downstairs, sitting, standing, and lying down) using smartphone sensors. Data was collected from the accelerometer and gyroscope embedded in a Samsung Galaxy S II smartphone worn by 30 participants, aged 19–48 while performing these activities. Each participant's data was recorded at a constant rate of 50 Hz. The dataset includes 561 features derived from time and frequency domain signals and is widely used for human activity recognition tasks. The data is divided into two subsets: data from 21 participants is used for training, while data from the remaining 9 participants is used for testing.

The Opportunity dataset was created for human activity recognition in sensor-rich environments, focusing on recognizing daily living activities and gestures using various wearable and ambient sensors. It includes recordings from four participants, each completing six runs divided into two types: five Activity of Daily Living (ADL) runs and one "Drill" run. The ADL runs involve the natural execution of daily activities, while the Drill run consists of a scripted sequence of activities. The dataset features a hierarchical annotation system that encompasses modes of locomotion, low-level activities (13 activities involving 23 objects), 17 mid-level gesture classes, and five high-level activity classes. The high-level activities in the ADL run generally follow a structure of waking up, grooming, preparing breakfast, and
cleaning. Although there are five high-level activity classes, participants were given some flexibility in achieving these goals, allowing for natural variations in execution. The dataset includes multimodal sensor data from 72 sensors, including accelerometers, gyroscopes, and magnetometers, placed at key points on each participant's body and in the environment.

The UTD-MHAD (Multimodal Human Action Dataset) was developed for human activity recognition and includes data collected from multiple modalities using various sources, specifically the Kinect camera and inertial sensors. These modalities comprise depth data, skeleton tracking of 20 joint positions, RGB videos captured at approximately 30 frames per second, and inertial sensor data (3-axis accelerometer and 3-axis gyroscope) sampled at 50 Hz. The dataset contains 27 different activities, such as walking, tennis serve, sit-to-stand, stand-to-sit, and clapping, performed by 8 participants, listed in Table I. Each activity was repeated 4 times by each participant, resulting in a comprehensive and balanced collection of 861 multimodal samples. In the context of this dataset, the terms 'action' and 'activity' are synonymous and can be used interchangeably.

<table><caption>TABLE I: List of Actions/activities in the UTD-MHAD Dataset</caption><thead><tr><td>No.</td><td>Activity</td><td>No.</td><td>Activity</td></tr></thead><tbody><tr><td>1</td><td>Swipe left</td><td>2</td><td>Swipe right</td></tr><tr><td>3</td><td>Wave</td><td>4</td><td>Clap</td></tr><tr><td>5</td><td>Throw</td><td>6</td><td>Arm cross</td></tr><tr><td>7</td><td>Basketball shoot</td><td>8</td><td>Draw X</td></tr><tr><td>9</td><td>Draw circle (clockwise)</td><td>10</td><td>Draw circle (counter-clockwise)</td></tr><tr><td>11</td><td>Draw triangle</td><td>12</td><td>Bowling</td></tr><tr><td>13</td><td>Boxing</td><td>14</td><td>Baseball swing</td></tr><tr><td>15</td><td>Tennis swing</td><td>16</td><td>Arm curl</td></tr><tr><td>17</td><td>Tennis serve</td><td>18</td><td>Push</td></tr><tr><td>19</td><td>Knock</td><td>20</td><td>Catch</td></tr><tr><td>21</td><td>Pickup and throw</td><td>22</td><td>Jog</td></tr><tr><td>23</td><td>Walk</td><td>24</td><td>Sit-to-Stand</td></tr><tr><td>25</td><td>Stand-to-Sit</td><td>26</td><td>Lunge</td></tr><tr><td>27</td><td>Squat</td><td></td><td></td></tr></tbody></table>

The multimodal nature of the UTD-MHAD dataset enables a wide range of applications in HAR and gesture classification. With its synchronized multimodal data, the dataset is frequently utilized to investigate the fusion of diverse data types to improve activity recognition accuracy.

Figures 2 and 3 show the 3D and 2D t-SNE projections, respectively, offering an intuitive visualization of how different activity samples are distributed based on their feature representations. The projections reveal that activities involving subtle, small-scale movements, such as Sweep Left, Sweep Right, Wave, and Clap, tend to form clusters that are relatively close to one another. This proximity reflects similarities in their motion patterns, as expected due to the limited variation in their movement dynamics. In contrast, activities characterized by larger-scale movements and significant body transitions, such as Sit-to-Stand, Squat, Lunge, Stand-to-Sit, and Walk, are distinctly separable from the first group. These activities form distinct, well-defined clusters, highlighting their separability from smaller-movement activities and one another. This distribution suggests that the extracted features effectively capture the intrinsic differences between these dynamic, posture-changing activities.

<figure><img src="image_3.png" /><figcaption>Fig. 2: 3D t-SNE visualization of the UTD-MHAD multimodal dataset.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Fig. 3: 2D t-SNE visualization of the UTD-MHAD multimodal dataset.</figcaption></figure>

## A. Preprocessing

To ensure a better comparison across the implementation of models on different datasets, a consistent preprocessing pipeline was applied across all three datasets. For the UCI-HAR dataset, the data are already preprocessed using a sliding window approach with a window size of 2.56 seconds and an overlap of 50%, resulting in 128 samples per window. Additionally, the samples are normalized to have zero mean and unit variance. In this preprocessing step, the tokenized data is further padded to a maximum sequence length of 896 to ensure consistency across the input samples.

For the Opportunity dataset, the classification task focuses on high-level activities, and the first 134 columns of the dataset are selected for this purpose. All NaN values in numeric columns are replaced using linear interpolation, which estimates missing values based on the nearest available values
in the same column. The data is then normalized using a StandardScaler to achieve a mean of zero and a unit variance.

To reduce dimensionality, the Principal Component Anal-
ysis (PCA) is applied with 40 components. The data is
segmented using a sliding window approach with a window
size of 40 and a stride of 20.

Given the requirements of transformers for larger datasets
and the need to address class imbalance, SMOTE (Synthetic
Minority Oversampling Technique) is applied to the training
set to generate additional samples, helping to prevent overfit-
ting. The tokenized data is padded to a maximum sequence
length of 864.

To ensure consistency across modalities in the UTD-MHAD dataset, a comprehensive preprocessing pipeline was developed for the four modalities: RGB videos, depth sequences, skeleton data, and inertial signals. Labels were extracted from file names, and discrepancies in sampling rates between the Kinect camera (30 fps) and inertial sensor (50 Hz) were resolved through interpolation and resampling. All modalities were aligned to the RGB video sequence length (T), with inertial data downsampled and depth and skeleton data truncated or padded accordingly. Data preprocessing includes normalizing and resizing RGB data into tensors of shape [3, T, 112, 112], reshaping depth data into [1, H, T, W], converting skeleton data into [T, J, 3], and downsampling inertial signals to [T, 6]. To handle variations in sequence lengths, dynamic padding was applied within each batch to ensure compatibility with modality-specific encoders.

The following encoders were employed to extract 256-
dimensional feature vectors from each modality:

*   **RGB Encoder:** A pre-trained 3D ResNet-18 (r3d_18) model was used to extract spatiotemporal features.

*   **Depth Encoder:** A 3D ResNet-18 model, adapted for single-channel input, was used for depth sequences.

*   **Skeleton Encoder:** An LSTM model with an input size of 60 (flattened joint coordinates) and a hidden size of 256 was employed for skeleton data. The output at the final timestep was used as the feature vector.

*   **Inertial Encoder:** An LSTM-based encoder with an input size of 6 (accelerometer and gyroscope signals) and a hidden size of 256 was applied to inertial data. The output at the final timestep served as the feature vector.

These 256-dimensional feature vectors from each modality
were fused for multimodal classification in subsequent stages
of the model pipeline.

V. EXPERIMENTAL RESULTS

The experimental procedure involves dividing each dataset
into training, validation, and test sets, ensuring that the splits
are consistent across all models for fair comparison. Hyperpa-
rameter tuning is conducted on the validation set, exploring
different values for the entropy regularization coefficient,
initial variance, cooling rate, and learning rate. The models
are trained using the respective optimization methods, with
early stopping implemented based on validation performance
to prevent overfitting.

Selecting appropriate hyperparameters is crucial for the
success of the proposed Riemannian manifold optimization

method. The coefficient $\lambda$ controls the influence of the Rie-
mannian manifold regularization term $L_{RM}$. A higher value
of $\lambda$ places more emphasis on conforming to the mani-
fold's geometry, which may improve generalization but could
slow down convergence if set too high. The coefficient $\lambda$
is tuned via cross-validation, exploring values such as $\lambda \in$
$\{0.01, 0.1, 0.5, 1.0\}$. The best value obtained for $\lambda$ is 0.01.
The learning rate $\eta$ determines the step size along the manifold
during optimization. Since the convergence properties differ
from Euclidean optimization, $\eta$ may need to be adjusted. The
best value obtained for $\eta \in \{0.001, 0.01\}$, is 0.001.

A. *Baseline Comparisons*

We compared the proposed method against several base-
lines. Standard Transformer models trained with traditional
optimizers like Adam serve as a direct comparison to assess
the impact of the thermodynamics-based optimization. Fur-
thermore, GPT-2 was also employed as a classifier to provide
an additional point of comparison.

The UCI-HAR dataset, a balanced unimodal dataset, has been extensively studied in the field of HAR. Numerous deep learning models have been implemented on this dataset, showcasing the effectiveness of various architectures. For example, Xia et al. [41] proposed an LSTM-CNN model, achieving an F1-score of 95.80% and an accuracy of 95.78%. Nafea et al. [42] extended this with a CNN-BiLSTM model, reaching accuracy of 97.05%. Dua et al. [43] introduced a multi-input CNN-GRU model, which achieved 96.19% F1-score and 96.20% accuracy. Kim et al. [44] developed a Transformer-CNN model that integrates convolutional and self-attention mechanisms, achieving an outstanding 99.30% accuracy. In addition, Zilelioglu et al. [26] evaluated a fine-tuned GPT-2 model for HAR, achieving an F1-score of 82.29%. They also implemented baseline methods, including Transformer (84.97% F1-score) and LSTM-CNN (88.47% F1-score), on the UCI-HAR dataset.

In this study, we revisited the implementation of GPT-2
on the same dataset. Using a standard loss function, GPT-2
achieved an accuracy of 84.03% and an F1-score of 84.03%.
A comparison of our results with those reported by Zilelioglu
et al. [26] indicates that our implementation achieved approx-
imately 2% higher overall accuracy. The confusion matrix
for our experiment is presented in Figure4, illustrating the
model's strengths and limitations. Notably, our implementation
demonstrated improved accuracy for static activities, such
as "Sitting" (78%) and "Lying" (100%), compared to the
respective accuracies of 52% and 69% reported by Zilelioglu
et al. [26]. This improvement can be attributed to the segmen-
tation strategy employed. While the UCI-HAR dataset was
originally published with pre-segmented windows, Zilelioglu
et al. re-segmented these windows into shorter segments,
which likely disrupted the temporal context of longer-duration
activities. In contrast, the proposed approach retained the
original pre-segmented data, preserving the temporal structure
and contributing to the model's superior performance in rec-
ognizing activities such as "Sitting" and "Lying."

Subsequently, the proposed custom loss function was ap-
plied, resulting in improved performance, with the GPT-2
<figure><img src="image_5.png" /><figcaption>Fig. 4: Normalized confusion matrix of activity predictions by the GPT-2 model on the UCI-HAR dataset (F1-score = 84.05%).</figcaption></figure>

model achieving an accuracy of 87.60% and an F1-score of 87.60%. This highlights the effectiveness of the custom loss function in enhancing the model's ability to classify activities with greater precision. Similarly, the custom loss function was applied to the Transformer model introduced in this study. With the standard loss function, the Transformer achieved an accuracy of 97.73%, which increased to 99.78% with the proposed loss function. These findings underscore the effectiveness of the proposed loss function in improving the classification performance of both GPT-2 and Transformer models. A detailed comparison of these results is presented in Table II.

<table><caption>TABLE II: Classification results from the current study and previous studies on the UCI-HAR datasets.</caption><thead><tr><th>Study</th><th>Method</th><th>Results (%)</th></tr></thead><tbody><tr><td>[41]</td><td>LSTM-CNN*</td><td>95.80 F1, 95.78 Acc</td></tr><tr><td>[42]</td><td>CNN-BiLSTM</td><td>97.05 Acc</td></tr><tr><td>[43]</td><td>Multi-Input CNN-GRU</td><td>96.19 F1, 96.20 Acc</td></tr><tr><td>[44]</td><td>Transformer-CNN</td><td>99.3 Acc</td></tr><tr><td rowspan="3">[26]</td><td>Transformer</td><td>84.97 F1</td></tr><tr><td>LSTM-CNN</td><td>88.47 F1</td></tr><tr><td>GPT-2</td><td>82.29 F1</td></tr><tr><td>Current study</td><td>GPT-2 (standard loss)</td><td>84.0 F1, 84.03 Acc</td></tr><tr><td>Current study</td><td>GPT-2 (proposed loss)</td><td>87.60 F1, 87.60 Acc</td></tr><tr><td>Current study</td><td>Transformer (standard loss)</td><td>97.73 F1, 97.75 Acc</td></tr><tr><td>Current study</td><td>Transformer (proposed loss)</td><td>99.78 F1, 99.80 Acc</td></tr></tbody></table>

The Opportunity dataset has been studied in the field of human activity recognition, with various deep learning models. A summary of some baseline methods from prior studies, along with their reported results in terms of accuracy and F1-score is provided in Table III. For instance, the work by Zhang et al. [45] proposed a ConvTransformer model that achieved 92% accuracy. Sun et al. [46] implemented a CNN-LSTM model and reported an accuracy of 83.21%. Similarly, Jeyakumar et al. [47] evaluated three different models on the dataset: a CNN model with 87.78% F1-score and 87.25% accuracy, an MA-CNN model with 90.44% F1-score and 90.52% accuracy, and a Conv-LSTM model that achieved

89.68% F1-score and 89.56% accuracy.

In this study, we evaluated two transformer-based architectures, GPT-2 as a classifier and a Transformer, on the Opportunity dataset. As presented in Table III, both models were tested using the standard loss function and the proposed custom loss function. The results demonstrate the effectiveness of the proposed loss function in enhancing model performance.

<table><caption>TABLE III: Classification results from the current study and previous studies on the Opportunity dataset.</caption><thead><tr><th>Study</th><th>Method</th><th>Results (%)</th></tr></thead><tbody><tr><td>[45]</td><td>ConvTransformer</td><td>92.00 Acc</td></tr><tr><td>[46]</td><td>CNN-LSTM</td><td>83.21 Acc</td></tr><tr><td rowspan="3">[47]</td><td>CNN</td><td>87.78 F1, 87.25 Acc</td></tr><tr><td>MA-CNN</td><td>90.44 F1, 90.52 Acc</td></tr><tr><td>Conv-LSTM</td><td>89.68 F1, 89.56 Acc</td></tr><tr><td>Current study</td><td>GPT-2 using standard loss</td><td>82.58 F1, 82.50 Acc</td></tr><tr><td>Current study</td><td>GPT-2 using proposed loss</td><td>84.62 F1, 84.60 Acc</td></tr><tr><td>Current study</td><td>Transformer using standard loss</td><td>93.04 F1, 93.01 Acc</td></tr><tr><td>Current study</td><td>Transformer using proposed loss</td><td>95.18 F1, 95.10 Acc</td></tr></tbody></table>

For the GPT-2 model, accuracy improved from 82.58% with the standard loss to 84.60% with the proposed loss, while the F1-score increased from 82.58% to 84.62%. The normalized confusion matrix for the GPT-2 model's activity predictions on the Opportunity dataset is shown in Figure 5. The model performed well on stationary activities, such as "Sitting" (95%) and "Lying" (97%), producing consistent and predictable data patterns. However, it encountered challenges in distinguishing between dynamic activities like "Walking" and "Standing," which involve greater movement variability and transitions between postures. These results suggest that while the model excels in recognizing activities with stable temporal patterns, it faces difficulties in handling activities characterized by continuous movement or frequent posture transitions.

Similarly, the Transformer model achieved substantial performance improvements with the proposed loss function, increasing accuracy from 93.01% to 95.10% and F1-score from 93.04% to 95.18%. These findings highlight the ability of the proposed loss function to significantly enhance the performance of transformer-based models. Beyond improving accuracy and F1 scores, the proposed loss function demonstrates the potential to achieve state-of-the-art results compared to prior studies on the Opportunity dataset.

1) *Baseline Comparison:* Although numerous studies have investigated the UTD-MHAD dataset for human activity recognition, the majority have focused on specific subsets of modalities rather than leveraging all available modalities.

Liu et al. [48] applied ResNeXt, a variant of CNN, to skeleton data and reported an accuracy of 80.23%. Similarly, Elaoud et al. [49] utilized a CNN-based encoding framework specifically designed for skeleton data, achieving a significantly higher accuracy of 97.91%. Memmesheimer et al. [50] explored a combined classification approach for skeleton and inertial data using a CNN, yielding an accuracy of 86.53%.

Ahmad et al. [51] adopted an SVM classifier for depth and inertial modalities, attaining an approximate accuracy of 99%. Building on similar modalities, another study [52] also
<figure><img src="image_6.png" /><figcaption>Fig. 5: Normalized confusion matrix of activity predictions by the GPT-2 model on the Opportunity dataset (F1-score = 82.58%).</figcaption></figure>

employed SVM but achieved a slightly lower accuracy of 98.5%.

Li et al. [4] examined pairwise combinations of RGB, depth, and skeleton modalities, using an MLP model to achieve an accuracy around 97%. A decision-level fusion strategy was adopted by Chen et al. [53], who used a CRC model on inertial, skeleton, and depth data to achieve a similar accuracy of 97%. In a different approach, Li et al. [54] leveraged a 3-layered LSTM model with decision-level fusion across the same modalities, achieving an accuracy of 95.31%. Finally, Dawar and Kehtarnavaz [55] utilized a CNN model for decision-level fusion, attaining an accuracy of 86.33%. The examples mentioned above represent only a subset of the studies conducted on the UTD-MHAD dataset. Table IV provides a detailed comparison of these studies, along with the performance of the proposed model implemented using the fusion of all four modalities.

S: Skeleton, I: Inertial, D: Depth, R: RGB, Feature Fusion: FF

<table><caption>TABLE IV: Comparison of models, modalities, and accuracies for human activity recognition on the UTD-MHAD dataset. The proposed Transformer-based models achieve superior performance through feature-level fusion across all modalities.</caption><thead><tr><th>Study</th><th>Model</th><th>Data Used</th><th>Acc. (%)</th></tr></thead><tbody><tr><td>[48]</td><td>ResNeXt (CNN)</td><td>S</td><td>80.23</td></tr><tr><td>[49]</td><td>CNN</td><td>S</td><td>97.91</td></tr><tr><td>[50]</td><td>CNN</td><td>S + I</td><td>86.53</td></tr><tr><td>[51]</td><td>SVM</td><td>I + D</td><td>99.00</td></tr><tr><td>[52]</td><td>SVM</td><td>I + D</td><td>98.50</td></tr><tr><td>[4]</td><td>MLP</td><td>I + D + R (pairwise)</td><td>97.00</td></tr><tr><td>[53]</td><td>CRC</td><td>S + D + I</td><td>97.20</td></tr><tr><td>[54]</td><td>3-Layered LSTM</td><td>S + D + I</td><td>95.31</td></tr><tr><td>[55]</td><td>CNN</td><td>S + D + I</td><td>86.33</td></tr><tr><td>Ours</td><td>GPT-2</td><td>S + D + I + R (FF)</td><td>78.49</td></tr><tr><td>Ours</td><td>GPT-2 (Riemannian)</td><td>S + D + I + R (FF)</td><td>85.67</td></tr><tr><td>Ours</td><td>Transformer</td><td>S + D + I + R (FF)</td><td>98.84</td></tr><tr><td>Ours</td><td>Transformer (Riemannian)</td><td>S + D + I + R (FF)</td><td>99.42</td></tr></tbody></table>

2) Parameter Settings: To train the proposed model on the UTD-MHAD dataset, the Adam optimizer was used with

an initial learning rate of $1e^{-4}$. The model was trained for 100 epochs with a batch size of 32, and a learning rate scheduler was applied to reduce the learning rate by 50% when the validation loss plateaued. The architecture consisted of 5 convolutional layers, each with 64 filters and ReLU activation, followed by a fully connected layer with 256 units and a dropout rate of 0.5. The inertial and skeleton data were normalized using Z-score normalization, while RGB and depth modalities were resized to $112 \times 112$. Data augmentation was applied to the RGB modality using random cropping and horizontal flipping. The dataset was split into 70% training, 20% validation, and 10% testing subsets.

3) Performance Analysis: To evaluate the effectiveness of the proposed loss function, we implemented two models, Transformer and GPT-2, on the UTD-MHAD dataset. Initially, each modality (Inertial, Skeleton, Depth, and RGB) was evaluated individually to assess the performance of the models on single-modal data. Subsequently, the models were trained on the fused multimodal data twice: once using the standard loss function and once using the proposed loss function. The results of these experiments are summarized in Table V.

<table><caption>TABLE V: Performance comparison of Transformer and GPT-2 models on the UTD-MHAD dataset for individual modalities and multimodal fusion using standard and proposed loss functions. Accuracy (%) is reported for each setup.</caption><thead><tr><th>Model</th><th>Modalities</th><th>Standard Loss</th><th>Proposed Loss</th></tr></thead><tbody><tr><td rowspan="4">Transformer</td><td>Inertial</td><td>85.55</td><td>-</td></tr><tr><td>Skeleton</td><td>93.24</td><td>-</td></tr><tr><td>Depth</td><td>97.68</td><td>-</td></tr><tr><td>RGB</td><td>97.11</td><td>-</td></tr><tr><td rowspan="4">GPT-2</td><td>Multimodal</td><td>98.84</td><td>99.42</td></tr><tr><td>Inertial</td><td>60.05</td><td>-</td></tr><tr><td>Skeleton</td><td>64.37</td><td>-</td></tr><tr><td>Depth</td><td>43.95</td><td>-</td></tr><tr><td></td><td>RGB</td><td>68.32</td><td>-</td></tr><tr><td></td><td>Multimodal</td><td>78.49</td><td>85.67</td></tr></tbody></table>

The results highlight the positive impact of using multimodal data instead of unimodal data, as well as the benefits of the proposed loss function compared to the standard loss function. The Transformer model shows accuracies between 85.55% (Inertial) and 97.68% (Depth) when trained on single modalities, but its performance improves to 98.84% on fused multimodal data with the standard loss function. Similarly, GPT-2 achieves its highest single-modal accuracy on RGB data (68.32%), while its accuracy rises to 78.49% with multimodal data and the standard loss. These results clearly highlight the effectiveness of multimodal data fusion in human activity recognition tasks.

Figure 6 presents the detailed results, highlighting the strengths and weaknesses of the model when applied to individual modalities compared to the multimodal setup. The results show that the Transformer model performs consistently well across most actions when using the fused multimodal data, achieving the highest overall accuracy (98.84%). This improvement is particularly notable for challenging actions such as "Catch" and "Knock," where individual modalities, like Inertial and RGB, exhibit relatively lower accuracy. By
<figure><img src="image_7.png" /><figcaption>Fig. 6: Class-specific accuracy of the Transformer model with the proposed loss function on RGB, Depth, Inertial, and Multimodal data for the UTD-MHAD dataset. The chart highlights relatively high performance across all modalities, with Multimodal data achieving the highest accuracy by addressing the weaknesses of individual modalities, particularly for challenging actions such as "Catch" and "Knock."</figcaption></figure>

leveraging complementary information from multiple modalities, the multimodal approach effectively addresses these challenges, yielding superior accuracy.

Interestingly, some actions, such as "Squatting" and "Walking," show high accuracy across all modalities, indicating their distinct and consistent patterns in the data. However, actions involving fine-grained or rapid movements, such as "Boxing" and "Throw," demonstrate larger performance gaps between single-modality and multimodal data, emphasizing the importance of integrating diverse features for robust activity recognition. RGB data performs well in recognizing activities that involve prominent color and texture changes, such as Wave, Draw Circle (Clockwise/Counterclockwise), and Knock, where visual cues are critical. However, it is not sufficiently well-suited for recognizing activities that require spatial depth perception, such as Bowling, Baseball Swing, and Tennis Swing, as RGB lacks explicit depth information to accurately represent 3D movements.

Depth data, on the other hand, proves to be suitable for capturing spatially complex and dynamic movements. It enables high accuracy in recognizing activities like Bowling, Tennis Swing, and Baseball Swing, which rely heavily on depth and spatial trajectories. However, Depth may have limitations in activities requiring fine-grained motion details, such as Wave and Catch, where detailed temporal information becomes more critical.

Inertial data shows strength in recognizing isolated body movements or repetitive motion patterns, such as Arm Curl, Sit-to-Stand, and Lunge, as these activities primarily involve distinct accelerations and angular velocities that are well captured by inertial sensors. However, it struggles with activities requiring detailed spatial understanding, such as Draw X, Draw Circle, and Knock, due to its inability to represent spatial trajectories effectively.

The use of multimodal data, as expected, achieves the best overall performance by leveraging the complementary strengths of all modalities. It significantly improves recognition accuracy for challenging activities like Catch, Wave,

and Draw Circle, where individual modalities face limitations. By combining spatial, visual, and motion cues from Depth, RGB, and Inertial data, the multimodal approach provides a richer representation of the activities, leading to more robust recognition and higher accuracy. Overall, the class-specific analysis reinforces the importance of multimodal fusion in improving the model's ability to generalize across a wide range of complex activities.

<table><caption>TABLE VI: Performance comparison of Transformer and GPT-2 models evaluated on the UTD-MHAD dataset using standard classification metrics (Accuracy, F1-score, Precision, and Recall). Results are reported for both the standard loss and the proposed loss function, highlighting the superior performance of the proposed loss in improving model accuracy and consistency across all metrics.</caption><thead><tr><th>Model</th><th>Accuracy</th><th>F1-score</th><th>Precision</th><th>Recall</th></tr></thead><tbody><tr><td>Transformer (standard)</td><td>98.84</td><td>98.11</td><td>98.77</td><td>98.02</td></tr><tr><td>GPT2 (standard)</td><td>79.30</td><td>79.29</td><td>79.30</td><td>79.27</td></tr><tr><td>Transformer (proposed)</td><td>99.42</td><td>99.42</td><td>99.47</td><td>99.47</td></tr><tr><td>GPT2 (proposed)</td><td>87.90</td><td>87.90</td><td>87.91</td><td>87.89</td></tr></tbody></table>

Table VI summarizes the performance metrics of the Transformer and GPT-2 models trained on the UTD-MHAD multimodal dataset using the proposed loss function and the standard loss function. The reported metrics include F1-score, precision, and recall, providing a comprehensive evaluation of the models' ability to classify human activities accurately.

The results highlight that the proposed loss function consistently improves the performance of both models across all metrics. Specifically:

For the Transformer model, which already demonstrates high performance with the standard loss function (accuracy: 98.84%, F1-score: 98.11%, precision: 98.77%, recall: 98.02%), the proposed loss function yields further improvements, achieving an accuracy of 99.42% and corresponding increases in F1-score, precision, and recall to 99.42%, 99.47%, and 99.47%, respectively. This shows that the proposed loss function refines the already strong performance of the Transformer.

For the GPT-2 model, the impact of the proposed loss function is more pronounced. With the standard loss function, GPT-2 achieves an accuracy of 79.30%, F1-score of 79.29%, precision of 79.30%, and recall of 79.27%. However, with the proposed loss function, the accuracy improves significantly to 87.90%, with corresponding gains in F1-score (87.90%), precision (87.91%), and recall (87.89%). These improvements demonstrate the effectiveness of the proposed loss function in enhancing the performance of models with initially lower baseline results.

Figure 7 illustrates the dataset distribution across the training, validation, and test sets using a density plot. The overlapping density curves confirm a uniform label distribution, ensuring a balanced split and representativeness across all datasets for training and evaluation purposes.

Figure 8 presents a histogram of class-specific accuracy achieved by the Transformer model trained with the proposed loss function. The results indicate consistently high accuracy
<figure><img src="image_8.png" /><figcaption>Fig. 7: Density plot of training, validation, and test datasets.</figcaption></figure>

across most activity classes in the UTD-MHAD dataset, high-
lighting the effectiveness of the proposed loss function in
ensuring balanced performance across activity types.

<figure><img src="image_9.png" /><figcaption>Fig. 8: Histogram of class-specific accuracy for the Transformer model.</figcaption></figure>

Overall, these findings validate the contribution of the proposed loss function as an effective enhancement for HAR systems, particularly in multimodal settings where integrating spatial and temporal information is critical. The results demonstrate the robustness and reliability of the proposed approach, making it a valuable method for future multimodal HAR frameworks.

After achieving an accuracy of 99.42%, extensive tests
were conducted to ensure that the results were valid and
not influenced by data leakage or architectural issues. Every
potential source of error was carefully examined, including
overlaps or duplications between the training and test datasets.
The architecture was thoroughly reviewed, and no issues were
identified. However, one specific change in the experimental
setup caused a shift in results: the method used to split the
dataset into training and test sets. Initially, the dataset was
split using a stratified random approach, ensuring that the class
distribution was maintained across the training, validation, and
test sets. However, when a subject-wise split was used—where
data from six subjects were allocated for training, and data
from two entirely unseen subjects were used for validation
and testing—the observed accuracy decreased. Specifically,
the accuracy of the Transformer model on multimodal data
dropped from 98.84% to 92.03% with the standard loss
function and from 99.42% to 95.18% with the proposed loss
function. While this decrease is significant, the proposed loss

function continued to demonstrate its effectiveness, maintain-
ing a notable accuracy improvement over the standard loss
function.

This finding underscores an important insight: the method of splitting data into training and test sets can significantly influence experimental results. In a subject-wise split, the model encounters entirely new subjects during testing, with no prior exposure to their characteristics during training. This adds a layer of complexity, requiring the model to generalize across the unique traits and behaviors of unseen individuals. This effect is not due to data leakage or overlap but rather to the intrinsic differences between subjects, which can affect model performance when shifting from random splits to subject-wise splits. Despite the reduction in accuracy under the subject-wise split, the consistent improvement achieved with the proposed loss function highlights its effectiveness. The results confirm the positive impact of the proposed loss function, which continues to enhance accuracy even under stricter evaluation conditions. This demonstrates that the proposed loss function is not only effective in optimizing multimodal data but also robust to variations in evaluation setups, further validating its contribution to HAR.

In summary, multimodal data not only addresses the limita-
tions of individual modalities but also improves overall recog-
nition performance by leveraging their combined strengths,
as demonstrated by the superior results achieved with the
proposed loss function. The results illustrate the strengths and
weaknesses of each modality in recognizing various activities.
For example, some activities are more accurately classified
using specific modalities, while others exhibit higher mis-
classification rates. As demonstrated, the multimodal model
achieves better overall performance than the unimodal models,
indicating its ability to leverage complementary information
from different modalities to enhance activity recognition.

More importantly, the proposed loss function significantly enhances model performance in a multimodal context. With this loss function, the Transformer achieves an accuracy of 98.84%, substantially surpassing its performance with the standard loss (95.64%). Similarly, GPT-2 achieves an accuracy of 85.67% using the proposed loss, compared to 78.49% with the standard loss. These results underscore the proposed loss function's ability to effectively leverage the complementary information inherent in multimodal data.

B. Ablation Study

We conducted an extensive ablation study to evaluate the
effectiveness of the proposed multimodal Transformer archi-
tecture and to understand the impact of Riemannian manifold
optimization on human activity recognition tasks. This study
aimed to analyze the contribution of each key component
of the proposed model to overall performance, providing
valuable insights into how each element influences recognition
accuracy.

The effectiveness of Riemannian manifold optimization was
compared against traditional regularization methods, such as
L2 regularization. Replacing the manifold regularization term
in the loss function with standard L2 regularization led to
inferior performance, with accuracy dropping from 99.42% to 99.08%. This underscores the advantages of leveraging the geometric constraints inherent in Riemannian optimization over conventional regularization techniques to mitigate overfitting and enhance generalization.

In addition, we analyzed the role of the Transformer's complexity in capturing multimodal interactions by training models with fewer Transformer layers and reduced attention heads. Simplifying the Transformer architecture adversely affected performance, indicating that the depth and capacity of the Transformer are crucial for modeling the intricate dependencies between different modalities. The Transformer's attention mechanism enables the model to learn meaningful cross-modal relationships, and reducing its complexity limits this ability. The impact of reducing the number of Transformer layers and attention heads on the model's performance is summarized in Table VII.

<table><caption>TABLE VII: Effect of varying the number of attention heads and Transformer layers on the model's accuracy and F1-score.</caption><thead><tr><td>Number of heads</td><td>Number of layers</td><td>Accuracy</td><td>F1-score</td></tr></thead><tbody><tr><td>4</td><td>2</td><td>99.42</td><td>99.42</td></tr><tr><td>2</td><td>2</td><td>99.32</td><td>99.30</td></tr><tr><td>4</td><td>1</td><td>99.26</td><td>99.25</td></tr><tr><td>2</td><td>1</td><td>99.19</td><td>99.15</td></tr></tbody></table>

Overall, the ablation study validates the design choices of the proposed model. The findings confirm that each component—the integration of multiple modalities, the use of Riemannian manifold optimization, and the architectural complexity of the Transformer—significantly contributes to enhancing performance. By leveraging these elements, the proposed model achieves superior results in human activity recognition tasks, demonstrating the effectiveness of combining multimodal data with advanced optimization and modeling techniques.

## VI. DISCUSSION AND CONCLUSION

In this study, we proposed a multimodal Transformer architecture enhanced by a novel loss function inspired by Riemannian manifold optimization, demonstrating significant advancements in HAR. The proposed model effectively integrates diverse data modalities, including images, videos, Kinect skeletal data, and inertial sensor readings, capturing rich and complementary information that substantially improves activity recognition performance. By leveraging Riemannian manifold optimization, the approach respects the geometric structures inherent in both the data and parameter space, enabling more efficient optimization and enhanced generalization, while mitigating overfitting on training data. The integration of Large Language Models (LLMs) further enriches the architecture by incorporating semantic information from textual descriptions or annotations, improving the model's ability to discern complex activity that may not be easily identifiable from sensor data alone. This semantic augmentation, combined with multimodal integration, allows the model to better understand and classify diverse human

activities. Additionally, the proposed methodology demonstrates compatibility and flexibility, enabling seamless integration with existing deep learning frameworks and hardware accelerators, requiring minimal changes to standard training pipelines. The experimental results validate the effectiveness of the proposed approach, achieving state-of-the-art performance across both unimodal and multimodal datasets. Specifically, the model demonstrated improved accuracy on the UCI-HAR and Opportunity datasets, surpassing prior benchmarks by 2%, and achieved an impressive accuracy of 99% on a challenging multimodal dataset. These outcomes emphasize the value of integrating advanced optimization techniques with cutting-edge deep learning architectures for HAR.

Future work will focus on expanding the model's applicability to larger and more diverse datasets, as well as exploring additional modalities such as audio and environmental sensors. Investigating alternative manifold structures and optimizing computational efficiency for deployment in real-world, resource-constrained environments remain promising directions. Furthermore, extending the use of LLMs to extract richer contextual information from multimodal data could further enhance the system's semantic understanding and overall accuracy. In conclusion, this study demonstrates the potential of combining multimodal Transformers, Riemannian manifold optimization, and semantic enrichment through LLMs to tackle the challenges of HAR, paving the way for robust, efficient, and intelligent activity recognition systems in a variety of applications.

## REFERENCES

[1] F. Zhao, C. Zhang, and B. Geng, "Deep multimodal data fusion," *ACM Computing Surveys*, vol. 56, no. 9, pp. 1–36, 2024.

[2] L. Bao and S. S. Intille, "Activity recognition from user-annotated acceleration data," in *International conference on pervasive computing*. Springer, 2004, pp. 1–17.

[3] J. Ni, H. Tang, S. T. Haque, Y. Yan, and A. H. H. Ngu, "A survey on multimodal wearable sensor-based human action recognition," *arXiv preprint arXiv:2404.15349*, 2024.

[4] C. Li, Y. Hou, W. Li, Z. Ding, and P. Wang, "Dfn: A deep fusion network for flexible single and multi-modal action recognition," *Expert Systems with Applications*, vol. 245, p. 123145, 2024.

[5] H.-B. Zhang, Y.-X. Zhang, B. Zhong, Q. Lei, L. Yang, J.-X. Du, and D.-S. Chen, "A comprehensive survey of vision-based human action recognition methods," *Sensors*, vol. 19, no. 5, p. 1005, 2019.

[6] E. Garcia-Ceja, M. Riegler, T. Nordgreen, P. Jakobsen, K. J. Oedegaard, and J. Tørresen, "Mental health monitoring with multimodal sensing and machine learning: A survey," *Pervasive and Mobile Computing*, vol. 51, pp. 1–26, 2018.

[7] C. Chen, R. Jafari, and N. Kehtarnavaz, "Utd-mhad: A multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor," in *2015 IEEE International conference on image processing (ICIP)*. IEEE, 2015, pp. 168–172.

[8] Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," *Nature*, vol. 521, no. 7553, pp. 436–444, 2015.

[9] C. A. Ronao and S.-B. Cho, "Human activity recognition with smartphone sensors using deep learning neural networks," *Expert Systems with Applications*, vol. 59, pp. 235–244, 2016.

[10] F. J. Ordóñez and D. Roggen, "Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition," *Sensors*, vol. 16, no. 1, p. 115, 2016.

[11] H. F. Nweke, Y. W. Teh, M. A. Al-Garadi, and U. R. Alo, "Deep learning algorithms for human activity recognition using mobile and wearable sensor networks: State of the art and research challenges," *Expert Systems with Applications*, vol. 105, pp. 233–261, 2018.

[12] M. Inoue, S. Inoue, and T. Nishida, "Deep recurrent neural network for mobile human activity recognition with high throughput," *Artificial Life and Robotics*, vol. 23, pp. 173–185, 2018.
[13] V. Gabeur, C. Sun, K. Alahari, and C. Schmid, "Multi-modal transformer for video retrieval," in *Proceedings of the European Conference on Computer Vision*. Springer, 2020, pp. 214–229.

[14] N. Han, J. Chen, H. Zhang, H. Wang, and H. Chen, "Adversarial multi-grained embedding network for cross-modal text-video retrieval," *ACM Transactions on Multimedia Computing, Communications, and Applications*, vol. 18, no. 2, pp. 1–23, 2022.

[15] Z. Huang, Z. Zeng, B. Liu, D. Fu, and J. Fu, "Pixel-bert: Aligning image pixels with text by deep multi-modal transformers," *arXiv preprint arXiv:2004.00849*, 2020.

[16] S. Liu, H. Fan, S. Qian, Y. Chen, W. Ding, and Z. Wang, "HIT: Hierarchical transformer with momentum contrast for video-text retrieval," in *Proceedings of the IEEE/CVF International Conference on Computer Vision*, 2021, pp. 11915–11925.

[17] N. C. Mithun, J. Li, F. Metze, and A. K. Roy-Chowdhury, "Learning joint embedding with multimodal cues for cross-modal video-text retrieval," in *Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval*, 2018, pp. 19–27.

[18] —, "Joint embeddings with multimodal cues for video-text retrieval," *International Journal of Multimedia Information Retrieval*, vol. 8, no. 1, pp. 3–18, 2019.

[19] M. Wray, H. Doughty, and D. Damen, "On semantic similarity in video retrieval," in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2021, pp. 3650–3660.

[20] I. Dirgová, Luptáková, M. Kubovčík, and J. Pospíchal, "Wearable sensor-based human activity recognition with transformer model," *Sensors*, vol. 22, no. 5, p. 1911, 2022.

[21] G. Pareek, S. Nigam, and R. Singh, "Modeling transformer architecture with attention layer for human activity recognition," *Neural Computing and Applications*, vol. 36, pp. 5515–5528, 2024.

[22] S. Ek, F. Portet, and P. Lalanda, "Transformer-based models to deal with heterogeneous environments in human activity recognition," *Personal and Ubiquitous Computing*, vol. 27, pp. 2267–2280, 2023.

[23] —, "Lightweight transformers for human activity recognition on mobile devices," *arXiv preprint arXiv:2209.11750*, 2022.

[24] S. Zhang, D. Rowlands, and D. James, "Deep learning in human activity recognition with wearable and smartphone sensors: A survey," *Sensors*, vol. 22, no. 4, p. 1476, 2022.

[25] J. Fabregas and G. Farias, "Human activity recognition using bert and data augmentation with gans," *Journal of Artificial Intelligence Research*, vol. 76, pp. 1234–1256, 2023.

[26] H. Zilelioglu, G. Khodabandelou, A. Chibani, and Y. Amirat, "Conditional human activity signal generation and generative classification with a gpt-2 model," in *2023 International Joint Conference on Neural Networks (IJCNN)*. IEEE, 2023, pp. 1–8.

[27] J. R. Kwapisz, G. M. Weiss, and S. A. Moore, "Activity recognition using cell phone accelerometers," *ACM SigKDD Explorations Newsletter*, vol. 12, no. 2, pp. 74–82, 2011.

[28] S. Yao, S. Hu, Y. Zhao, A. Zhang, and T. Abdelzaher, "Deepsense: A unified deep learning framework for time-series mobile sensing data processing," in *Proceedings of the 26th international conference on world wide web*, 2017, pp. 351–360.

[29] M. Sundermeyer, H. Ney, and R. Schlüter, "From feedforward to recurrent lstm neural networks for language modeling," *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 23, no. 3, pp. 517–529, 2015.

[30] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, "Attention is all you need," *Advances in neural information processing systems*, vol. 30, 2017.

[31] Y. Shavit and I. Klein, "Boosting inertial-based human activity recognition with transformers," *IEEE Access*, vol. 9, pp. 53540–53547, 2021.

[32] S. Ek, F. Portet, and P. Lalanda, "Lightweight transformers for human activity recognition on mobile devices," *arXiv preprint arXiv:2209.11750*, 2022.

[33] D. E. Djenouri, A. Belhadi, Y. Djenouri, and J. Lin, "A hybrid visual transformer for efficient deep human activity recognition," in *Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops*, 2023, pp. 0–0.

[34] J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng, "Multimodal deep learning," in *Proceedings of the 28th international conference on machine learning (ICML-11)*, 2011, pp. 689–696.

[35] X. S. Huang, F. Perez, J. Ba, and M. Volkovs, "Improving transformer optimization through better initialization," in *International Conference on Machine Learning*. PMLR, 2020, pp. 4475–4483.

[36] Y. Yang, D. P. Wipf et al., "Transformers from an optimization perspective," *Advances in Neural Information Processing Systems*, vol. 35, pp. 36958–36971, 2022.

[37] E. Gärtner, L. Metz, M. Andriluka, C. D. Freeman, and C. Sminchisescu, "Transformer-based learned optimization," in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2023, pp. 11970–11979.

[38] H. Fang, J.-U. Lee, N. S. Moosavi, and I. Gurevych, "Transformers with learnable activation functions," *arXiv preprint arXiv:2208.14111*, 2022.

[39] D. Anguita, A. Ghio, L. Oneto, X. Parra, and J. L. Reyes-Ortiz, "A public domain dataset for human activity recognition using smartphones," in *Proceedings of the 21st European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning*, 2013, pp. 437–442.

[40] D. Roggen, A. Calatroni, M. Rossi, T. Holleczek, K. Förster, G. Tröster, P. Lukowicz, D. Bannach, G. Pirkl, A. Ferscha et al., "Collecting complex activity datasets in highly rich networked sensor environments," in *Proceedings of the Seventh International Conference on Networked Sensing Systems (INSS)*. IEEE, 2010, pp. 233–240.

[41] K. Xia, J. Huang, and H. Wang, "Lstm-cnn architecture for human activity recognition," *IEEE Access*, vol. 8, pp. 56855–56866, 2020.

[42] O. Nafea, W. Abdul, G. Muhammad, and M. Alsulaiman, "Sensor-based human activity recognition with spatio-temporal deep learning," *Sensors*, vol. 21, no. 6, p. 2141, 2021.

[43] N. Dua, S. N. Singh, and V. B. Semwal, "Multi-input cnn-gru based human activity recognition using wearable sensors," *Computing*, vol. 103, no. 7, pp. 1461–1478, 2021.

[44] Y.-W. Kim, W.-H. Cho, K.-S. Kim, and S. Lee, "Inertial-measurement-unit-based novel human activity recognition algorithm using conformer," *Sensors*, vol. 22, no. 10, p. 3932, 2022.

[45] Z. Zhang, W. Wang, A. An, Y. Qin, and F. Yang, "A human activity recognition method using wearable sensors based on convtransformer model," *Evolving Systems*, vol. 14, no. 6, pp. 939–955, 2023.

[46] J. Sun, Y. Fu, S. Li, J. He, C. Xu, and L. Tan, "Sequential human activity recognition based on deep convolutional network and extreme learning machine using wearable sensors," *Journal of Sensors*, vol. 2018, no. 1, p. 8580959, 2018.

[47] J. V. Jeyakumar, L. Lai, N. Suda, and M. Srivastava, "Sensehar: a robust virtual activity sensor for smartphones and wearables," in *Proceedings of the 17th Conference on Embedded Networked Sensor Systems*, 2019, pp. 15–28.

[48] S. Liu, X. Bai, M. Fang, L. Li, and C.-C. Hung, "Mixed graph convolution and residual transformation network for skeleton-based action recognition," *Applied Intelligence*, vol. 52, no. 2, pp. 1544–1555, 2022.

[49] A. Elaoud, H. Ghazouani, and W. Barhoumi, "Xyz-channel encoding and augmentation of human joint skeleton coordinates for end-to-end action recognition," *Signal, Image and Video Processing*, pp. 1–15, 2024.

[50] R. Memmesheimer, N. Theisen, and D. Paulus, "Gimme signals: Discriminative signal encoding for multimodal activity recognition," in *2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)*. IEEE, 2020, pp. 10394–10401.

[51] Z. Ahmad and N. Khan, "Human action recognition using deep multilevel multimodal fusion of depth and inertial sensors," *IEEE Sensors Journal*, vol. 20, no. 3, pp. 1445–1455, 2019.

[52] —, "Towards improved human action recognition using convolutional neural networks and multimodal fusion of depth and inertial sensor data," in *2018 IEEE International Symposium on Multimedia (ISM)*. IEEE, 2018, pp. 223–230.

[53] C. Chen, R. Jafari, and N. Kehtarnavaz, "A real-time human action recognition system using depth and inertial sensor fusion," *IEEE Sensors Journal*, vol. 16, no. 3, pp. 773–781, 2015.

[54] K. Li, X. Zhao, J. Bian, and M. Tan, "Sequential learning for multimodal 3d human activity recognition with long-short term memory," in *2017 IEEE International Conference on Mechatronics and Automation (ICMA)*. IEEE, 2017, pp. 1556–1561.

[55] N. Dawar and N. Kehtarnavaz, "Action detection and recognition in continuous action streams by deep learning-based sensing fusion," *IEEE Sensors Journal*, vol. 18, no. 23, pp. 9660–9668, 2018.