Accurate and Steady Inertial Pose Estimation through Sequence Structure Learning and Modulation
Authors: Yinghao Wu
chaoran wang
Lu Yin
Shihui Guo
Yipeng Qin
Venue: NeurIPS 2024 poster (2024)
Keywords: Inertial Measurement Units (IMUs)
Pose Estimation
Transformer
Primary area: machine_learning_for_other_sciences_and_fields

Abstract: Transformer models excel at capturing long-range dependencies in sequential data, but lack explicit mechanisms to leverage structural patterns inherent in fixed-length input sequences. 
In this paper, we propose a novel sequence structure learning and modulation approach that endows Transformers with the ability to model and utilize such fixed-sequence structural properties for improved performance on inertial pose estimation tasks.
Specifically, our method introduces a Sequence Structure Module (SSM) that utilizes structural information of fixed-length inertial sensor readings to adjust the input features of transformers.
Such structural information can either be acquired by learning or specified based on users' prior knowledge.
To justify the prospect of our approach, we show that i) injecting spatial structural information of IMUs/joints learned from data improves accuracy, while ii) injecting temporal structural information based on smooth priors reduces jitter (i.e., improves steadiness), in a spatial-temporal transformer solution for inertial pose estimation.
Extensive experiments across multiple benchmark datasets demonstrate the superiority of our approach against state-of-the-art methods and has the potential to advance the design of the transformer architecture for fixed-length sequences.

Accurate and Steady Inertial Pose Estimation through Sequence Structure Learning and Modulation
Yinghao Wu!
Shihui Guo Schooo of Informatics, Xiamen University, China School of Computer Science Informatics, Cardiff University, UK
Chaoran Wang'
Lu Yin
Yipeng Qin
Abstract
Transformer models excel at capturing long-range dependencies in sequential data, but lack explicit mechanisms leverage structural patterns inherent fixed-length input sequences. In this paper, we propose novel sequence structure learning and modulation approach that endows Transformers with the ability to model and utilize such fixed-sequence structural properties for improved performance on inertial pose estimation tasks. Specifically, our method introduces a Sequence Structure Module (SSM) that utilizes structural information of fixed-length inertial sensor readings to adjust the input features of transformers. Such structural information either be acquired by learning or specified based users' prior knowledge. To justify the prospect of our approach, we show that i) injecting spatial structural information of IMUs/joints learned from data improves accuracy, while ii) injecting temporal structural information based on smooth priors reduces jitter (i.e., improves steadiness), in pptiialteemprrl transformer solution for inertial pose estimation. Extensive experiments across multiple benchmark datasets demonstrate the supe- riority of our approach against state-of-the-art methods and has the potential advance the design of the transformer architecture for fixed-length sequences.
Introduction
Estimating human pose long-standing and prominent task that underlies many computer vision and graphics applications, e.g., animation production, virtual reality. As an alternative to vision-based solutions, wearable device-based methods are gaining increasing interest as they are environment- free, occlusion-unawant and usability, most prior works [18, 55, 56, 20, 60] abandon densely placed configurations, instead leveraging sparse set of Inertial Measurement Units (IMUs) to reconstruct human motion. Since IMUs can provide continuous measurements of rotation and acceleration, we define pose estimation with sparse inertial sensors sequence learning task.
frrincy.,,endll... For ensuring high accuracy and maintaining portability
Recently, Transformer-basee based architectures have achieved tremendous success in various sequence learning tasks, and applying them to sparse inertial motion capture is a natural idea. However, empirically, we find that, directly using the native transformer to model IMU sequences results in unacceptable jitter and inaccurate postures. Through our analysis, we attribute this to the native Transformer architecture, whose self-attention mechanism was originally designed to flexibly handle seriable-length sequence inputs, thus lacks inductive bias for modeling fixed-length sequences that have clear structures. For instance, for an IMUs reading sequence, the length is usually fixed (e.g.. the number of observed past frames in time window or the number of IMUs) and each token in the sequence has a specific meaning (e.g., each temporal token denotes a frame and each spatial token
*Corresponding author (guoshihui @ xmm.edu.cn) Also with Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan, Ministry of Culture and Tourism.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Live Demo
Figure 1: Left: use only six IMUs to predict the full-body pose in real-time, which are fixed on left and right forearm, the left and right lower leg, the head, and the pelvis. Right: our system is capable of capturing wide range of daily motions as well as challenging movements.
represents IMU). In other words, representative type of fixed-length sequences, IMU readings have clear structures both in spatial and temporal dimensions. However, such structural information not explicitly modeled in native Transformers.
To bridge this gap, we present an innovative approach for learning and modulating sequence structure, which empowers transformers to effectively capture and leverage structural properties of fixed-length sequences, leading to enhanced performance inertial pose estimation tasks. Specifically, our method introduces novel Sequence Structure Module (SSM) designed to leverage the structural information fixed-length sequences to adjust the input features of Transformers. For inertial pose estimation tasks, devised two SSM variants: SSM-S and SSM-T, for injecting spatial and temporal structural information into spatial and temporal transformer features, respectively. Extensive experiments four public benchmarks demonstrate that method achieves superior performances than state-of-the-arr methods, where the average errors of the whole-body angles decreased by 13% and 38%, together with the lowest jitter, the DIP-IMU [18] and TotalCapture [46] datasets, respectively. In addition, we implemented a real-time pose estimation system to test the performance of our approach in real-world scenarios. In summary, our contributions include:
We identify a key limitation of the native transformer architecture: its lack of inductive biases for modeling fixed-length sequences with inherent structural properties. To address this shortcoming, we propose a novel Sequence Structure Module (SSM) that enables transformers to effectively capture and leverage the structural priors present in fixed-length sequential data.
For inertial motion capture tasks involving sequential IMU data, propose SSM variants: SSM-S and SSM-T, which incorporate structural inductive biases of the IMU sensor layout (spatial) and time frames (temporal), respectively, into transformer learning.
Extensive experiments demonstrate that our method outperforms state-of-the-art ones on the DIP-IMU and TotalCapture datasets by large margin. To further demonstrate the superiority of approach, implemented real-time motion capture system based six IMUs to evaluate the performance of our model in complex real-world scenarios.
2 Related Works
2.1 Human Pose Estimation
Human pose estimation (HPE) has been long-standing research topic, with numerous researchers exploring it using various types of sensors. As our research lies on IMU sensors, we roughly categorize sensors into two types: non-IMU and IMU.

---

HPE with non-IMU Sensors. In non-IMU human motion capture solutions, vision- methods still dominate the mainstream so far. Early methods [14, 23, 16, 42] used multiple cameras and marker points to capture human poses, which imposed significant constraints on the environment. With the popularity of deep learning, an increasing number of methods are using a single camera to capture human 2D/3D poses, such as CPN [8]. HRNet [44] and others [62, 22, 54, 51, 61, 45, 39, 28, 67, 58,588 25]. achieving significant success. In addition, there are also other types of sensors used for tracking human motion, such 6DOF trackers [43, 5, 13, 19, 631, flexible fabric sensors [7, 26, 68], wireless sensors [6, 53] and hybrid sensors [3, 35, 38]. Although each method has its own advantages, motion capture systems based on IMUs are emerging and gaining prominence, due to their wearability, portability, and ease of use.
HPE with IMU Sensors. The advantage of inertial motion capture systems compared to vision-based methods their resistance to extreme conditions such as bright lights and occlusion. Commercial systems like Xsens [41] and Noitom [36] place multiple IMUs on the user's body, while achieving accurate pose estimation, also restricting the user's movements and requiring lots of time for IMU attachment. To enhance user comfort and usability, an increasing number of studies [48, 18, 56, 55, 20, 60] are shifting towards sparse inertial motion capture and predict human postures using only few IMUs. As pioneering work in sparse inertial pose prediction, SIP [48] demonstrates that recovering full-body motion using only 6 IMUs is feasible for the first time, but it is an optimization- based approach which is computationally slow. Huang et al. [18], the first to introduce neural networks into this task, employ a bidirectional RNN to achieve real-time performance. After that, Transpose [56] proposes multt pose estimation framework, utilizing three sub netwoessrrs based on bidirectional RNN to predict pose, further enhancing accuracy. However, both of [18, 56] require future frames input, which adds additional latency. PIP [55], introduces physical constraints, improving the physical plausibility of motion without the need for future frames. Another work, TIP [20], utilizes Transformer to predict human body poses while simultaneously generating terrain, achieving prediction of human motion in non-planar environments. Contemporancous work, DynalP [60]. leverages pseudo-velocity learning to fully utilize acceleration and models the human body into three separate regions, each focusing on their unique characteristics. In addition, some studies [34, 69] have attempted to place IMUs in objects carried by the body for tracking human motion. For example. Zuo et al. [69] use a loose-wear jacket with 4 IMUs to capture the upper body motion, which provide the users with high comfort and freedom of movement.
However, existing methods only focus on modeling the temporal dimension (whether using RNNs or Transformers) while neglecting the spatial dimension. Unlike them, utilize two-stage Transformer-based spatial-temporal framework, independently capturing the dependencies of both space and time. Meanwhile, we have also designed two modules. SSM-S and SSM-T, enabling the Transformer to more effectively leverage structural information from fixed-length sequences.
2.2 Transformer Variants for Time Series
Transformer has seen a number of modifications to address the limitations of well-known works such as BERT [11] and ViT [12]. In the field of time-series data modeling, researchers have proposed various approaches, one common method being modifications in positional encoding [24, 59, 27, 64, 50, 65]. For example, Transformer-XL [10] introduced relative positional encoding, enabling the model to capture long-range dependencies, TCN--Tranformer [2] combines the characteristics of convolutional networks with relative positional encoding. Self-attenionn module is the central part of Transformer. However, for many long-sequence based tasks, the time complexity of self- attention module is a computational bottleneck. Various works [31, 64, 65, 49, 52] are proposed to address this issue. Longformer [4] employs sparse attention mechanism. specifying local and global attention, allowing the model to handle long sequences while maintaining computational efficiency. Linformer [49] approximates the original -igh--imensional attention matrix through low-rank projection, reducing both computational and memory requirements. Additionally, some researchers made structural modifications [52, 10, 64, 21] to Transformer for time series tasks. For example, Transformer-X XL [10] incorporates a segment-level recurrence mechanism in the encoder to handle longer contextual information. Reformer [21] utilizes sensittysee hashing (LSH) and reversible network structure, enabling the model to process extremely long sequences. Informer [64] introduces ProbSparse Sclf-Attention, reducing the computational complexity of self-attention.
3

---

Previous studies have modified the Transformer extensively, but they have largely overlooked its limitations in modeling fixed-length sequences and does not impose Transformer explicitly utilize the inherent structural information within it. Our work bridges this gap.
3 Method
3.1
Why is Sequence Structure Modeling Missing in Native Transformer Architecture?
The Transformer architecture [47] was originally designed to accomplish machine translation tasks in natural language processing, To handle variable-length textual inputs with different syntaxes, the native Transformer architecture does not make any inductive bias on their structures, but instead focuses on the content of the input text. Specifically, let X R xd be the embedded input sequence of length N and feature dimension d, the self- attentttenton mechanism is defined as:
Attention( (Q, K, V) aV = =oottmax(Q" /vav
where Q = XWQ. K XWx and V = XWy E RN Wq. WK. and Wv Rddd, E RNN, Among them, only the attention matrix a is modeling the relationships among the N iiipuu tokens. However, its element o't.)) calculated the product between the i-th query (Q) and the j-th key (K) pair, thus representing the relationship between individual tokens rather than the structure of the input sequence as whole. This enables it to handle sequences with different N that do not share common structure (e.g., sentences).
However, in many other domains (e.g.. pose estimation), the input sequences usually have fixed length (e.g., the number of observed past frames or body joints) and a clear structure (e.g., temporal continuity spatial relationship), which implies that the structural information of input sequences can facilitate learning. Motivated by this key insight, we propose a novel Sequence Structure Module (SSM) to endow the Transformer architecture with the ability model and utilize the structural information inherent in fixed-length input sequences.
3.2 Sequence Structure Module
Our Sequence Structure Module (SSM) aims to fully utilize the structured information of fixed- length sequence inputs to compensate for the lack of inductive bias in the transformer architecture. Specifically, as shown in Fig. 2 (d), given a sequence embedding X E IR Nxd, before entering the transformer encoder, we multiply X with structural matrix S NNN, followed by LayerNorm (LN) layer [1] and a MLP Block:
X = MLM((LN(SX))
(2)
where X Nxd LN(-) is used regularize the model and maintain gradient stability during training and MLP(.) is used to increase the capacity of the module. The key distinction between the structural matrix and the self-attenion matrix a is that the elements of S are independent of the input X, making S universally applicable to all inpul sequences regardless of their content. This allows S to effectively capture structural information. Then, feed the the subsequent transformer encoder for modeling the long-range dependencies using Eq. (1). The structural matrix S can be obtained in various ways. For example, it can come from prior knowledge in the specific domains, be entirely data-driven through learning, or be a combination of the two. Here, categorize the structure into the following three types: Explicit Structure S5, Implicit Structure 5,, and paplicit-Implice Hybrid Structure SEI-
X into
Explicit Structure (ES) is particularly useful when each token in the input sequence of a fixed length N has a clear meaning (c.g., each token in a time sequence represents a frame), and the structural relationships between the N tokens can be precisely captured based on prior knowledge. That is, each element Se (i.) j) in the structural matrix is provided by the user before training. allowing the user to impose prior knowledge of the sequence structure on learning. Notably, when Sg is the identity matrix I, no modifications are made and the sequence structure module degenerates. We provide two examples of how construct explicit structures S5 in Sec. 3.4.
Implicit Structure (IS) becomes appropriate when the meaning of each token in the sequence and/or the structural relationships between tokens are unclear. Unlike ES, which completely determined

---

IMU Measurement
Matrix Multiplication
Explicit
Temporal Encoder
pose
MLP
t
Implicit
Spatial Encoder
TE
Layer Norm
Sequence Structure Module
t
Hybrid
SSM
T
pose
TE
SE
N x
X
x
TE
SE
SSM-S
Input Sequence
Structural
Matrix
X
(a)
(b)
(d)
Figure 2: (a) Previous work only employ temporal encoders (RNN or transformer) to predict pose. (b) Our apatial-tmppore framework. (c) Our ppatial-tempora framework with SSM. (d) Our sequence structure module (SSM), simply consists of structural matrix S, LayerNorm [1] and MLP Block.
by the user, the establishment of IS relies on learnable matrix P. that learned in a data-driven manner. We define the implicit structure matrix S₁ using the following equation:
= P
where I, P RNN and denotes the identity matrix. Notably, another way define Sj is to solely use P without the identity matrix 1, namely S = P. But as pointed out in [17], in the extreme situation, it is easier optimize P into zero matrix rather than identity matrix.
Explicit-Impliie Hybrid Structure (EIHS) aims to strike a balance between ES and IS by allowing the user provide initial structure matrix based on prior knowledge, while also using data-driven methods to modify it. It can also be referred to as a mon-identity matrix initialization of Eq. (3):
SEr = SE + P
where Sg E N represents the explicit structure, and P E R NxN is a learnable matrix. In the extreme case, when the user-designed explicit structure Sg I. EIHS degenerates into IS.
3.3 SSM for Transformer-based Sparse Inertial Pose Estimation
Problem Formulation. Our task is to accomplish real-time human pose estimation using data acquired from 6 inertial sensors positioned on the wrists of both hands, ankles of both feet, waist, and head (Fig. 1). Each IMU can provide sequential acceleration A and orientation R signals on the body part is placed on, where A R is the linear acceleration and R Rax3 is the rotation matrix. Our goal is to learn a mapping f which reconstructs the joints rotations of the full body:
TT, = f(LA, R
(5)
where T denotes the number of observed frames from the past, denotes the number of predicted joints, N denotes the number of IMUs, and O E SO(3) is the rotation of body joints, representing the human pose with a certain skeleton (e.g. SMPL [29]).
Spatial-Temporal Transformer with SSM. To accomplish this task, unlike previous works that only employed temporal encoders (Fig. 2 (a)), we utilize a ppatial-tempora framework transformer network as our baseline model (Fig. (b)). where the spatial transformer models the local motion correlations among N IMUs/joints within frame, while the temporal transformer captures the global dependencies between T frames throughout the entire sequence. Since the values of N and T are typically fixed, (e.g. N and SSM-S and SSM-T, to leverage the structural information of fixed sequences in spatial and temporal dimensions, respectively (Fig. 2 (c)). As previously mentioned, each SSM has three different choices for the structure matrix S. After thorough experimental comparisons in Sec 4.4, our final choice is that, the structural matrix of SSM-S is derived from EIHS, and the structure of SSM-T ESS. Due to page limitations, we have included more network details in Appendix Sec. 7.
30), we introduce two variants of Sequence Structure Module,
5

---

3.4 Constructing Explicit Structures
The power SSM lies its large design space explicit structures SE. Here, we take the sparse inertial pose estimation task as an example, to demonstrate how to construct Sy in spatial and temporal dimensions based on prior knowledge.
A/and
6.900
03311
5.00))
62108
9,90
1001
01411
10
Avc.
6.500
06473
00433
43315
1000
Sg
S5 30)
Sx-r(a 16)
S5-r - 1)
Figure 3: The visualization of Sg-s, and Sg-r with different
Spatial Structure Construction. The motivation for constructing the spatial structure Sg-s is to explore relevance and consistency between body joints when the human body moves. "Relevance and consistency" refer to the tendency for certain joints to exhibit similar movement patterns across large sample human motions. For instance, when the left hand moves forward, the head, torso, and legs are likely move in certain coordinated manner.
We utilize statistical approach accomplish this task. Specifically, we utilize the AMASS [32] dataset to decompose the rotations of different joints in each frame into rotations along the r-axis, g-axis, and z-axis, representing the rotations in terms of Euler angles. By doing so, we obtain R", Ry, R TRx where f denotes the number of observed frames and J denotes the number of joints. We select subsets (sub) of the rotations consisting of the N joints where IMUs are placed (e.g.. N = 6) and have: R(s,sub) R(z,sub), R(z,,ub) Nx We independently calculate the correlation matrix C E RNNN, k = y.2 for each subset and sum up the results for average to obtain the spatial explicit structure Sg-s E N
cov (n((i,suub) R.h.oubb)
(*ii))
(6)
var (efh.sub) var R (k.sub))
Σ C*
Se kma.y.a
where cov denotes the covariance between two variables, and var denotes the variance of a variable. The resulting Sg-s shown Fig. 3. It can be observed that, the movements between the two hands and the two legs exhibit high correlation; but the movements of the head and the root (spine) are relatively independent; which aligns with our intuition.
Temporal Structure Construction. Unlike the use of statistical methods to calculate spatial explicit structures Sg-s. the construction of temporal explicit structures Sg-T is much more straightforward, which follows the "distance" between two frames within a time sequence. That is, when two frames are "close" enough, such adjacent frames, their correlation relatively high; when the frames "farther apart", such as the first and last frames in sequence, their correlation tends to decrease as the "distance" increases; akin smoothness prior. Mathematically, we define Sg-T E R using function that linearly decreases with increasing "distance":
0
if i j
SE-T(i.))
(8)
else
where a is hyperparameter, which represents the maximum "distance" between two frames that are considered correlated. Specifically, when the "distance" between i-th and j-th frame, namely jl 2 consider these two frames to be unrelated. Fig. 3 shows the visualization of Sg-T

---

under different values of Ø. In our experimental setup, T 30 and Ø = 10. For details on how to determine o, please refer Sec. 4.4.
Notably, in addition to the two methods mentioned above, there are many ways to construct explicit structures. In fact, any matrix with specific meaning can as explicit structure. While have presented two illustrative examples, the ongoing quest to design tailored SSM explicit structures for diverse tasks offers boundless opportunities for exploration and innovation.
4 Experiments
4.1 Datasets and Evaluation Metrics
Datasets. We use the following datasets in our experiments, which can divided into three categories:
1) Synthetic dataset: AMASS [32]. 2) Real datasets with SMPL [29] skeleton: DIP-IMU [18] and TotalCapture [46]. 3) Real datasets with Xsens [41] skeleton: AnDy [33], CIP [37], and Emokine [9]. We use them to train and evaluate our methods as follows: 1) Following [56, 55, 20], we first pre-train our model on AMASS and fine-tune it on the training set DIP-IMU, then test on the test set of DIP-IMU and the entire TotalCapture dataset; Following [60], train our model on the training sets of AnDy, CIP, and Emokine datasets and test on the test sets of AnDy and CIP datasets.
Evaluation Metrics. We use the following metrics to evaluate the mean global rotation error of upper arms and upper legs in degrees; 2) Angular error, measuring the mean global rotation error of all body joints in degrees; 3) Positional error, measuring the mean Euclidean distance error of all estimated joints in centimeters with the root joint (Spine) aligned; 4) Mesh error, measuring the mean Euclidean distance error of all vertices of the estimated body mesh with root joint (Spine) aligned. The vertex coordinates are calculated by applying the pose parameters to the SMPL [29] body model; 5) Jitter, measuring the mean jerk (time derivative of acceleration) of all body joints the global space, which reflects the smoothness of the motion [15].
method: 1) SIP error, measuring
4.2 Implementation Details
We implement our method using the PyTorch [40] framework one NVIDIA GeForce RTX 4090 GPU. PyTorch version is 2.0.0, and CUDA version is 11.8. During the training stage, we use the AdamW [30] optimizer to train
model with batch size of 4096. The learning rate is initialized to 0.0001 and decayed by 0.99 per epoch. We implement the live demo using a laptop equipped with an Intel@ TM 19-13900HX Processor CPU and an NVIDIA GeForce RTX 4060 GPU.
4.3 Comparisons with SOTA
Quantitative Results. We compare our method with state-of-the-arr ones, including TransPose [56]. TIP [20], PIP [55], DynalP [60], which also accomplish pose estimation from only 6 IMUs signals. All metrics are calculated the real-time setting and the best and runner-up results in each column marked in bold and underline respectively.
Table 1: Comparison with SOTA methods on DIP-IMU [18] and TotalCapture [46] datasets with
SMPL [29] skeleton. Bold indicates best and underline indicates runner-up results.
DIP-IMU
TotalCapture
SIP Eir
Ang
Pos Err
Mesh Err
Jitter
SIP Err
Ang Err
Pos
Mesh Err
Jitter
D8P[18]
17.10
15.16
7.33
8.96
18.62
17.22
9.42
11.22
3.62
[5a]spose[56]
17.03
8.86
6.03
7.14
1.08
16.40
12.77
6.42
7.20
1.83
[10[
16.92
9.07
5.63
6.62
1.53
13.20
12.24
5.68
6.78
1.57
51P[
15.02
8.72
5.01
6.02
0.14
12.93
12.04
5,61
6.51
0.18
Dyna
14.11
7.00
4.97
5.97
0.18
12.42
11.06
5.11
5.79
0.22
PNP[S7]
13.71
8.75
4.97
5.77
0.17
10.89
10.45
4.74
5.45
0.26
Ours
7.90
6.06
3.12
3.78
0.07
7.00
6.82
3.36
4.00
0.09
As shown in Table 1 and Table 2, the results indicate that our method has surpassed previous approaches by significant margin on both four benchmark datasets, achieving more accurate and steadier pose estimation. Specifically, SIP Err on four datasets is significantly ahead of other
7

---

Table 2: Comparison with SOTA methods on AnDy [33] and
Table 3: Ablation study of SSM-S
CIP [37] datasets with Xsens [41] skeleton.
and SSM-T.
AnDy
CIP
Models
Ang Err
Jitter
SIP Eir
Ang Fir
Pos Err
SIP
Ang Fir
Pos Err
Bascline
8.82
0,48
14,25
[56
4.91
20.06
6.86
[]][20]
10.11
4.55
3.56
13.05
5.67
4.30
SSM-S
7.83
0,43
12.04
PIP[55]
9.49
3.29
4.12
SSM-T
7.93
0.09
8.68
[60)
8.93
3:45
3,41
11.42
4.54
3.69
Ours
6.82
0.09
7,46
Ours
4.56
3.37
1.73
8.14
5.49
2.57
methods, outperforming the runner-up by 44%, 44%, 49% and 29% respectively. We attribute this spatio-temporal framework and the spatial structure information in SSM-S, which help the model better capture the motion correlation between body joints. Besides, on the DIP-IMU and TotalCapture datasets, our predicted motion sequences exhibit the least jitter (i.e., smoothest motion sequence). We attribute this to the temporal sequence structure information in our SSM-T, which provides more temporal prior knowledge between frames and smooths the input features generate more stable and consistent motion prediction results.
Qualitative Results. We also provide visual comparison between the estimated pose and the ground truth on the TotalCapture dataset. Compared with state-of-the- art methods, our method achieves precise predictions shown in Fig. 4. The comparison of the two actions (leaning forward and hending over) indicates that, our method can estimate the positions of arms and legs more accurately than previous methods. Additionally, the comparison of the two ambiguous actions (raising leg and raising both hands) in the second row shows that our model better identifies these ambiguous actions.
TIP
PIP
DynalP
Ours
Ground Truth
Figure 4: Qualitative comparisons with the state-of ootie--o--h--arr ahe-art methods on TotalCapture dataset.
Analysis for Error of Joints. As shown in Fig. 5, we also compare the angular errors of individual joints on the DIP-IMU and TotalCapture datasets. It can be observed that, the errors of previous methods are mostly concentrated
the joints of the hands (L/R elbows and shoulders) and legs (L/R hips and knees), which are the main sources of SIP Err. We believe this is because they only capture the fnter-frames dependencies in temporal dimension, while neglecting to model motion correlations between body joints in the spatial dimension. Unlike them, we utilize two-stage spatial-temporal framework, where the spatial encoder independently models the motion consistency of IMUs/joints within a frame, and SSM-S injects spatial structural information into the features. Their combined effect allows for more accurate estimation of each joint rotation.
4.4 Ablation Study
We conduct ablation experiments on the TotalCapture dataset, reporting two metrics: Ang Err and Jitter. Additionally, we observe a trade-off between Ang Err and Jitter, challenging the simultaneous achievement of the lowest values for both. For convenience, we introduce temporary metric = (Ang Err) E.p(( Jilter) as a reference for selecting the optimal model, where lower represents the balance between accuracy and stability, with Exp(-) denoting exponential operation.

---

Error of Different Joints DIP-IMU
Error of Different Joints on TotalCapture
20.0
Ours
16
Ours
17.5
TIP
PIP
14
TIP
PIP
15.0
Auguiar Error (*)
Augular Error
12.5
10-
10.0
7.5
0.0
Hip
Spine1
Knoe
Knee
Spine?
Spined
Heck
Collar
Collar
Shoulder
R Shouldar Elbow
Elbow
R Hip
Spinet
Kneee
R knee
Spine?
Spined
Heck
Collar
R Collar
Head Shoulder
Shoulder Elbowr Elbow
Figure 5: Error of different joints on DIP-IMU and TotalCapture datasets.
Design of Sequence Structure for SSM-S and SSM-T. We first explore the designs of sequence structure SSM-S and SSM-T. As previously mentioned, each SSM can have three different choices, resulting in a total of 3 3 9 combinations for the two SSMs. To simplify this, we first fix the type of SSM-S as and explore the best structure type for SSM-T. As shown in Tab. 4, comparing Settings 1, 2, and 3 demonstrates that ES is the best structure choice for SSM-T, as the other two structures result in unacceptable jitter. Based on this, we conduct experiments with Settings and By comparing Settings 4, and 6, we find that EIHS is the best choice for SSM-S, as it significantly reduces angular error without substantially increasing jitter (i.e., with lower 7). Thus, we conclude that, EIHS ES is the best combination for SSM-S and SSM-T.
Table 5: Performance under differ- ent selection for hyperparameter
Table 4: Ablation study on SSM design.
Setting
SSM-S
SSM-T
Ang Err
Jitter
T
Ang Err
Jitter
IS
IS
7.79
0.54
13.37
30
8.08
0.07
8.66
2
IS
ES
7.94
0.09
8.69
25
8.05
0.07
8.63
3
IS
EIHS
7.86
0.53
13.35
20
7.76
0.07
8.32
ES
ES
8.11
0.08
8.79
15
7.85
0.08
8.50
5
EIHS IS
8.13
0.34
11.42
10
6.82
0.09
7.46
5
7.33
0.13
8.34
6
EIHS
ES
6.82
0.09
7.46
7.25
0.38
10.60
Further Exploration for Sg-T. Fig. 3 shows the visualization of SE-T under different choices of a, and Tab. 5 shows their performance on TotalCapture dataset. It can be observed that the model achieves the best performance when 10. We hypothesize that this means the information from the adjacent 10 frames should be given more emphasis for IMU measurements in our case.
Contribution of Each Component. We conduct thorough ablation study when investigate the respective contributions SSM-S and SSM-T. We use the most basic spatial-temporal framework (Fig. (b)) the baseline model and sequentially add SSM-S and SSM-T. The results are shown in Tab. 3. It can be observed that the primary function of SSM-S is to reduce joint angle error to improve the accuracy of human pose prediction, while the role of SSM-T T is to reduce jitter to enhance the coherence of the posture and generate steadier motion sequence.
10, to
In-depth analysis of SSM-S. To look deeper into SSM-S, we visualized the spatial structure matrix Sg-s (before training), the learnable matrix Ps and the final spatial structure matrix SgI-s as shown in Fig. 6. It can be observed that:
The overall pattern of structure matrix remain the same before and after training (S and SFr-s). i.e., the movements between the two hands and the two legs still exhibit high correlation; and the movements of the head and the root (spine) are still negatively correlated. This demonstrates the effectiveness of our Sg-s as initialization/prior..

---

14
1103)
8.00
Laaad
comn
nEW
DE77 GriVE amN
Uid
B4603
L0000
634G2
04233
6044
2040E
6,045
0,3044
2004
GAI1> 40229
03001
00510
03354
04242
64655
CO100
B0437
60122
12000
03447
0.318 2012 GULLE
6014
03820
1,9g
(1003 400014
0003
Whidd F_hand
Uhp
Heed
(hind RLhand (igg
1JKg Hoad
Fuhand
Ukp
Heed
Hest
Sg-5 (Before Training)
Ps (Learnale Matrix)
SEI- (Afler Truining)
Lind
10030
SUNT
Lhadd
3074 BEONT 87771 amm
NAVE
34600
0,044) LO003
C5402
0.030
SONCE
4.1330
00915 4 42225
45072 44142 40000
0.2441
4.454 L0013
64150
C9007
L
00077
0M417
SENM 6X022
Lik
C.OTAA
401 300 02017 20077 EXI.
60113
61077
SONON
1J9g
RUag
10
Lindd FLhand Line
1e8d
1001
Ciidd FLhnd
Uig
Hoad
Fchind Uhp
Held
Hest
SE-3 (Before Trsining)
Ps (Learnable Matrix)
SEI- (After Training)
Figure 6: The visualization of Sg-s, Ps and SEI-S- First column: Sg-s obtained using the AMASS dataset before training. First row: results from our model trained on the AMASS, DIP-IMU dataset. Second row: results from our model trained on the AnDy, CIP, and Emokine datasets.
The learnable matrix Ps adds small offsets to the spatial structure matrix, i.c., slightly suppresses the correlation between two hands, two legs and head vs. root. We attribute this to the different motion distributions among datasets: i) in AMASS, daily actions (e.g., walking, jogging. running, sitting and stretching) are dominant, and the movements of both hands and legs show extremely high consistency; ii) in the DIP-IMU, although daily actions are also the majority, there are a large number of single-hand and single-leg movements, such as single-hand raising, grasping and swinging; single-leg lifting, etc., which weaken the movement consistency of both hands and legs; iii) in the Andy and CIP, there are numerous industry-oriented activities, which are very different from daily movements, resulting in a relatively large adjustment range of the learnable matrix Ps. This demonstrates the effectiveness of our Ps in adapting the structure matrix to different datasets, and maintains a high generalization ability.
4.5 Live Demo
We have implemented real-time pose estimation visualization system using Python and Unity. We select a variety of actions to evaluate the performance of our model in real-world scenarios. These include everyday actions like walking, sitting, kicking, stretching, sports and more. Additionally, we choose some challenging movements such as single-leg standing, rolling, Chinese Kung Fu and dance movements assess the generalization ability of our model. Through the live demo, can be observed that even under intense physical activity, our approach still maintains long-term stability and shows robust generalization capability. Please refer to the supplementary video for our demo.
Conclusion
In this paper, we propose novel sequence structure learning and modulation approach that empowers Transformers with the ability to model and utilize fixed-length sequence structural information. We present simple yet effective Sequence Structure Module (SSM) to accomplish this, achieving impressive performance in sparse inertial pose estimation tasks. This showcases the powerful potential of the SSM to generalize to other Transformer- based fixed-length sequence tasks.
10

---

Acknowledgments
This work is supported by National Natural Science Foundation China (62072383, 61702433), the Public Technology Service Platform Project of Xiamen City (No.3502220231042 Xiaomi Young Talents Program/Xiaomi Foundation, the Fundamental Research Funds for the Central Universities. This work is also partially supported by Royal Society (IECANSFC 2211022).
References
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization, arXiv preprint 16x.v:1607.06450, 2016.
[2] Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling, arXiv preprint 1803.0000.00122 2018.
[3] Yiming Bao, Xu Zhao, and Dahong Qian. Fusepose: Imu-vision sensor fusion in kinematic space for parametric human pose estimation. IEEE Transactions on Multimedia, 2022.
[4] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint 20Xiv:2004.05/ 2020.
[5] Angela Castillo, Maria Escobar, Guillaume Jeanneret, Albert Pumarola, Pablo Arbeláez, Ali Thabet, and Artsiom Sanakoyeu. Bodiffusion: Diffusing sparse observations for full-body hu- man motion synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1231,, 2023.
[6] Anjun Chen, Xiangyu Wang, Shaohao Zhu, Yanxu Li, Jiming Chen, and Qi Ye. mmbody bench- mark: body reconstruction dataset and analysis for millimeter wave radar. In Proceedings of the 30th ACM International Conference on Multimedia, pages 3501-3511 2022.
[7] Xiaowei Chen, Xiao Jiang, Jiawei Fang, Shihui Guo, Juncong Lin, Minghong Liao, Guoliang Luo, and Hongbo Fu. Dispad: Flexible on-body displacement of fabric sensors for robust joint- motion tracking. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 7(1):1-27, 2023.
[8] Yilun Chen, Zhicheng Wang. Yuxiang Peng. Zhiqiang Zhang, Gang Yu, and Jian Sun. Cascaded pyramid network for multi-person pose estimation. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7103-7112, 2018.
[9] Julia F. Christensen, Andrés Fernández, Rebecca A. Smith, Georgios Michalareas, Sina H. N. Yazdi, Fahima Farahi, Eva-Madeleine Schmidt, Nasimeh Bahmanian, and Gemma Roig. Emokinedataset, June 2023.
[10] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.022 02860, 2019.
[11] Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: /0.0: 040055 2018.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1999, 2020.
[13] Yuming Du, Robin Kips, Albert Pumarola, Sebastian Starke, Thabet, and Artsiom Sanakoyeu. Avatars grow legs: Generating smooth human motion from sparse tracking inputs with diffu- sion model. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 481-490, 2023.

---

[14] Patric Eichelberger, Matteo Ferraro, Ursina Minder, Trevor Denton, Angela Blasimann, Fabian Krause, and Heiner Baur. Analysis of accuracy in optical motion capture-a protocol for laboratory setup evaluation. Journal of biomechanics, 49(10):2885-2888 2016.
[15] Tamar Flash and Neville Hogan. The coordination of arm movements: an experimentally confirmed mathematical model. Journal of neuroscience, 5(7):1688-1703. 1985.
[16] Gutemberg Guerra-Filho. Optical motion capture: Theory and implementation. RITA, 12(2):61- 90, 2005.
[17] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[18] Yinghao Huang, Manuel Kaufmann, Emre Aksan, Michael Black, Otmar Hilliges, and Gerard Pons-Moll. Deep inertial poser: Learning to reconstruct human pose from sparse inertial measurements in real time. ACM Transactions Graphics (TOG), 37(6))1-15, 2018.
[19] Jiaxi Jiang, Paul Streli, Manuel Meier, Andreas Fender, and Christian Holz. Egoposer: Robust real-time ego-body pose estimation in large scenes. arXiv preprint arXiv:2308. X31423308.06493,, 06493, 2023.
[20] Yifeng Jiang. Yuting Ye, Deepak Gopinath, Jungdam Won, Alexander W Winkler, and C Karen Liu. Transformer inertial poser: Real-time human motion reconstruction from sparse imus with simultaneous terrain generation. In SIGGRAPH Asia 2022 Conference Papers, pages 1-9, 2022.
[21] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.0445), 2020.
[22] Muhammed Kocabas, Chun-Hao P Huang, Otmar Hilliges, and Michael J Black. Pare: Part attention regressor for 3d human body estimation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11127-11137, 2021.
[23] Kazutaka Kurihara, Shin ichiro Hoshino, Katsu Yamane, and Yoshihiko Nakamura. Optical motion capture system with pan-tilt camera tracking and real time data processing. In Proceed- ings 2002 IEEE international conference on robotics and automation (Cat. No. 02CH37292), volume 2, pages 1241-1248. IEEE, 2002.
[24] Shiyang Li, M Ziyang, Yingbo Zhou, Denny Wu, Qianxiao Li, Benjamin A Rozario, and Alexander Schwing. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. arXiv preprint arXiv: 1907.00255, 2019.
(25) Wenhao Li, Mengyuan Liu, Hong Liu, Pichao Wang, Jialun Cai, and Nicu Sebe. Hour- glass tokenizer for efficient transformer-based 3d human pose estimation. arXiv preprint arXiv:2311.12024 2023.
[26] Zhen Liang, Dongquan Zhang, Guanghua Xu, Fangting Xie, Hao Guo, Jingyuan Cheng, et al. Smart garment: A long-term feasible, whole-body textile pressure sensing system. IEEE Sensors Journal, 2023.
[27] Bryan Lim, Sercan Arik, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting, arXiv preprint (rXi::/912.09663, 2021.
[28] Qiuxia Lin, Kerui Gu, Linlin Yang, and Angela Yao. Synthetic-to-real pose estimation with geometric reconstruction. Advances in Neural Information Processing Systems, 36, 2024.
[29] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34(6):248:1-2488 16, October 2015.
[30] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, arXiv preprint axXvv1171.051001, 2017.
12

---

[31] Jianxin Ma, Peng Kang, Xiaojie Wang. Mengshuo Wang, Weinan Zhang, and Yong Yu. Logtrans: log sparse transformer network for sequential recommendation. Proceedings of the 43rd International ACM SIGIR Conference Research and Development in Information Reirieval, pages 2347-2356, 2020.
[32] Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black. Amass: Archive of motion capture as surface shapes. Proceedings of the IEEF/CVF international conference computer vision, pages 5442-5451, 2019.
[33] Pauline Maurice, Adrien Malaisé, Clélie Amiot, Nicolas Paris, Guy-Junior Richard, Olivier Rochel, and Serena Ivaldi. Human movement and ergonomies: An industry-oriented dataset for collaborative robotics. The International Journal of Robotics Research, 2019.
[34] Vimal Mollyn, Riku Arakawa, Mayank Goel, Chris Harrison, and Karan Ahuja. Imuposer: Full-body pose estimation using imus in phones, watches, and earbuds. In Proceedings of the 2023 CHI Conference on Human Factors Computing Systems, pages 1-12, 2023.
[35] Md Moniruzzaman, Zhaozheng Yin, Md Sanzid Bin Hossain, Hwan Choi, and Zhishan Guo. Wearable motion capture: Reconstructing and predicting 3d human poses from wearable sensors. IEEE Journal of Biomedical and Health Informatics, 2023.
[36]
Noitom. Perception neuron. 2017.
[37] Manuel Palermo, Sara M Cerqueira, João André, António Pereira, and Cristina P Santos. From raw measurements to human pose-a dataset with low-cost and high-end -eggetic sensor data. Scientific Data, 9(1):591, 2022.
[38] Shaohua Pan, Qi Ma, Xinyu Yi, Weifeng Hu,, Xiong Wang. Xingkang Zhou, Jijunnan Li, and Feng Xu. Fusing monocular images and sparse imu signals for real-time human motion capture. arXiv preprint arXiv:2309.00 00310, 2023.
[39] Hui En Pang, Zhongang Cai, Lei Yang, Qingyi Tao, Zhonghua Wu, Tianwei Zhang, and Ziwei Liu. Towards robust and expressive whole-body human pose and shape estimation. Advances in Neural Information Processing Systems, 36, 2024.
[40] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[41] Monique Paulich, Martin Schepers, Nina Rudigkeit, and Giovanni Bellusci. Xsens mtw awinda: Miniature wireless inertial-magneti motion tracker for highly accurate 3d kinematic applications. Xsens: Enschede, The Netherlands, pages 1-9,2015 2018.
[42] Natural Point. Optitrack. Natural Point, Inc, 2011.
[43] Jose Luis Ponton, Haoran Yun, Andreas Aristidou, Carlos Andujar, and Nuria Pelechano. Sparseposer: Real-time full-body motion reconstruction from sparse data. ACM Transactions on Graphics, 2023.
[44] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5693-5703, 2019.
[45] Zhenhua Tang, Zhaofan Qiu, Yanbin Hao, Richang Hong, and Ting Yao. human pose estima- tion with spatio-temporal criss-cross attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4790-4799, 2023.
[46] Matt Trumble, Andrew Gilbert, Charles Malleson, Adrian Hilton, and John Collomosse. Total capture: 3d human pose estimation fusing video and inertial sensors. In 2017 British Machine Vision Conference (BMVC), 2017.
13

---

[47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention all you need. In Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[48] Timo Von Marcard, Bodo Rosenhahn, Michael J Black, and Gerard Pons-Moll. Sparse inertial poser: Automatic 3d human pose estimation from sparse imus. In Computer graphics forum, volume 36, pages 349-360. Wiley Online Library, 2017.
[49] Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2000.04768, 2020.
[50] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition trans- formers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34:22419-22433 2021.
[51] Yufei Xu, Jing Zhang. Qiming Zhang, and Dacheng Tao. Vitpose: Simple vision transformer baselines for human pose estimation. Advances in Neural Information Processing Systems, 35:38571-38584, 2022.
[52] Haoyong Xue, Duanfei Zhang. Jie Cao, Chang Liu, Xincheng Yao. Xiyang Zhang, Yikang Shen, Yue Zhang, and Xinyu Dai. Pyraformer: cowpcempyy pyramidal attention for long-range 2021.
time series modeling and forecasting, arXiv preprint
[53] Hongfei Xue, Qiming Cao, Yan Ju. Haochen Hu, Haoyu Wang, Aidong Zhang, and Lu Su. M4esh: mmwave- mmwave-based 3d human mesh construction for multiple subjects. In Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems, pages 391-400 2022.
[54] Sen Yang, Zhibin Quan, Mu Nie, and Wankou Yang. Transpose: Keypoint localization via transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11802-11818 11812, 2021.
[55] Xinyu Yi, Yuxiao Zhou, Marc Habermann, Soshi Shimada, Vladislav Golyanik, Christian Theobalt, and Feng Xu. Physical inertial poser (pip): Physics-aware real-time human motion tracking from sparse inertial sensors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13167-13178, 2022.
[56] Xinyu Yi, Yuxiao Zhou, and Feng Xu. Transpose: Real-time 3d human translation and pose estimation with six inertial sensors. ACM Transactions on Graphics (TOG), 40(4):1-13, 2021.
[57] Xinyu Yi, Yuxiao Zhou, and Feng Xu. Physical non-inertial poser (pnp): Modeling non-inertial effects in sparse-inertial human motion capture. In SIGGRAPH 2024 Conference Pupers, 2024.
[58] Bruce XB Yu, Zhi Zhang, Yongxu Liu, Sheng-hua Zhong, Yan Liu, and Chang Wen Chen. Gla-gen: Global-local adaptive graph convolutional network for 3d human pose estimation from monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8818 8829,, 2023.
[59] George Zerveas, Sercan O Arik, Rajiv R Raitoharju, and Tomas Pfister. A transformer-based framework for multivariate time series representation learning, arXiv preprint arXiv:2106.01770. 2021.
[60] Yu Zhang, Songpengcheng Xia, Lei Chu, Jiarui Yang, Qi Wu, and Ling Pei. Dynamic inertial poser (dynaip): Part-based motion dynamics learning for enhanced human pose estimation with sparse inertial sensors. IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR), 2024.
[61] Qitao Zhao, Ce Zheng, Mengyuan Liu, and Chen Chen. A single 2d pose with context is worth hundreds for 3d human pose estimation. arXiv preprint vrxiv:2211.03331 2023.
[62] Jianan Zhen, Qi Fang, Jiaming Sun, Wentao Liu, Wei Jiang, Hujun Bao, and Xiaowei Zhou. Smap: Single-shot multi-person absolute 3d pose estimation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XV 16, pages 550-566. Springer, 2020.
14

---

[63] Xiaozheng Zheng, Zhuo Su, Chao Wen, Zhou Xue, and Xiaojie Jin. Realistic full-body tracking from sparse observations via joint-level modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14678-14688, 2023.
[64] Haoyi Zhou, Shanghang Zhang, Jieqi Peng. Shuai Zhang. Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. Proceedings of the AAAI Conference on Artificial Intelligence, 35(12):11106-1
2021.
[65] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. arXiv preprint M.Kv:220011127400 2022.
[66] Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5745 51555,57,311 2019.
[67] Wentao Zhu, Xiaoxuan Ma, Zhaoyang Liu, Libin Liu, Wayne Wu, and Yizhou Wang. Motion- bert: A unified perspective on learning human motion representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15085-15099, 2023.
[68] Chengxu Zuo, Jiawei Fang, Shihui Guo, and Yipeng Qin. Self-adaptive motion tracking against on-body displacement of flexible sensors. In Thirtt-seventh Conference on Neural Information Processing Systems, 2023.
[69] Chengxu Zuo, Yiming Wang, Lishuang Zhan, Shihui Guo, Xinyu Yi, Feng Xu, and Yipeng Qin. Loose inertial poser: Motion capture with imu-attached loose-wear jacket. IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR), 2024.
15

---

Appendix
7 Detailed Network Architecture
Linear Ensbedding
Joint
Mensurement
Kotation
Regression Hend
Spatial Tokeas
Temporal Tokras
Poudtion Emblodding
Temporal Transformer Encoder
x
Temporal
pose
Embedding
SSM
Spatial Trausformer Encoder
SSM-T
In Detail
2
SE
Sputial Posilion
SSM-S S
Embudding
SSM-S
Figure 7: Our network architecture in detail.
As shown in Fig. 7, network mainly consists of Linear Embedding, SSM-S, Spatial Transformer Encoder, SSM-T, Temporal Transformer Encoder and Regression Head Module. the main text, we introduce SSM-S and SSM-T. Here, provide introduction to the remaining parts.
Linear Embedding. Our network takes M E RT*Nx12 the measurements from N IMUs over the past T frames as input. Firstly, we linearly map each IMU measurement m(..) E RR1 into an embedding vector =(.)) E P by means of learnable matrix E' 122DD;
((.)) m(a)) E After that, the input M E TxNN12 becomes F, E RxNND The spatial feature F, is fed into SSM-S to inject spatial structure, and the resulting F, is then send into the Spatial Transformer Encoder to model the motion correlation between body joints.
(9)
Spatial Transformer Encoder. Given F., we first add learnable spatial position embedding P. E R to each token for maintaining spatial position information. The resulting joint sequence of features z, are fed into a spatial encoder consisting sequence of transformer layers. Each layer comprises of Multi-Head Self-Attention [47], LayerNorm [1], and MLP blocks:
2. F. + P.
(10)
1 = MSA(IN(()) + 4
(11)
MLP(LN(( +y-
(12)
The output of the last transformer layer is = E RTNND which is sent into SSM-T where temporal structural information incorporated.
Temporal Transformer Encoder. We treat the output of SSM-T, F, RTXNx as the input of temporal transformer encoder, to further extract the global dependencies across frames in the entire sequence. We first reshape F; into F; E ( Before the temporal transformer encoder, we add learnable temporal positional embedding Pt E (NN-DD to retain frame position information. The resulting frame sequence of features 2 are fed into temporal encoder, which has the same architecture as the spatial transformer encoder. The procedure can be formulated as:
24 Fi P.PPP
(13)
16

---

D. = MSA(LN(Z.)) +
(14)
MLP(LN(V)) +y{
(15)
The output of the last transformer layer is 24' E (NX(N-D), a compact spatial-temporal feature representation, which is sent into regression head module.
Regression Head Module. We map of into the whole body joint rotations Ô E TRxx using Layer Normalization [1] and MLP block:
O MLP(LN(2)))
(16)
where J denotes the number of joints and 6 denotes 6D rotation representation [66]. The whole network is optimized by minimizing the Mean Squared Error (MSE) between Ô and the ground-truth O as:
C= o o
(17)
During the training stage, we compute the loss using the predictions Ô and ground-truth O of T frames. During the inference stage, only utilize the last frame the output.
8 Additional Experiments
Result Another Definition for Sg-T-
(a 10)
(a 10)"
(a= 20)
(a 20)
Figure 8: The visualization SE-T of different definition when and a
We further explore an alternative definition of Sg-T:
0
if 1 2
SF-T(i.))
(18)
else
T
Differing from Eq. (8), in Eq. (18), has been replaced by T. We compare the cases with o = 10 and 20 to the previous scenario, denoted *. The visualization for Sg-T are shown in Fig. 8 and the performance on TotalCapture dataset is shown in Tab. 6. It is easily discernile that the definition of Eq. (8) is significantly superior to that of Eq. (18), both terms of accuracy and stability.
Table The performance under different definition for S on TotalCapture dataset.
Ang Err
Jitter
(o 10)
6.82
0.09
7.46
(o = 10)*
7.86
0.08
8.51
(o 20)
7.76
0.07
8.32
(a 20)*
8.33
0.07
8.93
Ablation Study of the Order of SE/TE.
We provide the results of switching the order from "SE-TE" to "TE-SE" as Tab. 7. The results show that our "SE-TE" order outperforms its "TE-SE" variant by a large margin, which is consistent with the common practice in many spatio-temporal frameworks.
17

---

Table The performance under different definition for Sg-T on TotalCapture dataset.
Method
Ang Err
Jitter
SE-TE (ours)
6.82
0.09
7.46
TE-SE
8.67
0.11
9.68
Generalization of Module under More Sensors.
Although we focused on the more challenging "sparse" settings (6 IMUs) in our paper, showing addi- tional results of applying our modules more sensors could further demonstrate their generalization ability. Specifically, we increase the number of IMUs from to 10 (with the additional 4 IMUs placed the left and right shoulders and thighs), and conduct experiments and ablation studies the DIP-IMU dataset.
Table 8: The performance under more sensors on DIP-IMU dataset.
IMUs
Ang Eir
Jitter
(ours)
6.06
0.07
6.49
10 (ours)
4.40
0.03
4.53
10 (ours w/o SSM-S) 10 (ours w/o SSM-T) 10 (ours w/o SSM-S, w/o SSM-T)
5.69
0.03
5.86
4.56
0.13
5.19
5.58
0.14
6.42
As shown in Tab. 8, after adding IMUs, our proposed method still works effectively (higher accuracy and lower jitter). Additionally, the ablation study results show that the roles of SSM-S and SSM-T remain consistent with their performance when using IMUs. That is, SSM-S improves accuracy of motion prediction, while SSM-T reduces jitter to enhance the coherence of the posture. Based on the above experimental results, it can concluded that our proposed method maintains strong generalization ability the number of sensors increases.
Performance on Different Users with Different Physiques. To study the performance on different users with different physiques, we conduct experiments on DIP-IMU dataset. To demonstrate this more clearly, we computed the BMI values (BMI Mass(kg) / Height? for each individual, categorize them into three groups, and report our model's performance across these three categories as Tab.
Table 9: The performance on different users with different physiques.
BMI
ID
Number
Ang Err
Jitter
C1: BMI<21.7 C2: 21.7<=BMI<=249
$2/S6
2
7.33
0.08
7.94
$1/$3/55/5//88/9
6
7.63
0.09
8.35
C3: BMIs24.9
$4/S10
2
8.07
0.09
8.83
average
10
7.66
0.09
8.35
The experimental results demonstrate that our method robust and performs well across users with different physiques. For reference, also include the results for each individual Tab. 10.
More Comparisons with Other NN Structures. As shown Tab. 11, we construct spatio-temporal framework using GCN layers as the spatial encoder and Conv I layers as the temporal encoder. The results show that our method significantly outperforms the GCN + Conv 1d implementation.
Discussion
The Design of r. In our experiments, we observed that it is difficult for Ang and Jitter to simultancously reach their minima, forming trade-off in-between. For example, in Tab. 6, with
18

---

Table 10: The performance for each individual on different users with different physiques.
ID
Ang Err
Jitter
Mass(kg)
Height(em)
BMI
Categorya
sl
7.74
0.05
8.13
86
186
24.85
s2
7.47
0.05
7.85
65
178
20.51
1
s3
7.95
0.11
8.87
87
187
24.87
s4
7.63
0.11
8.51
78
170
26.98
3
s5
7.64
0.12
8.61
80
180
24.69
s6
7.20
0.11
8.03
58
172
19.60
S7
6.89
0.09
7.53
70
178
22.09
S8
7.13
0.09
7.80
80
180
24.69
2
89
8.48
0.06
9.00
187
24.30
810
8.51
0.07
9.12
87
181
26.55
3
Table 11: More Comparisons with Other NN Structures.
Method
Ang Err
Jitter
GCN + Convld
14.31
0.28
18.93
Transformer + Transformer(ours)
6.82
0.09
7.46
20, Ang Err is 7.76 and Jilter is 0.07; with 10, Ang Err decreases 6.82 while Jilter increases to 0.09. To strike a balance in this trade-off, we introduce = (Ang (AngErr) * Exp(Jiiter) to combine Ang Err and Jitter into single measure. The reason for using Exp that have observed that the impact of the Jitter on viewing experience is non-linear:
When Jitter is relatively high jitter 0.3), the visual quality is unacceptable.
When Jitter is relatively low (e.g., 0 jiller < 0.2), the visual experience is good and insensitive to changes in Jitter.
Therefore, we use Exp to measure the impact of Jitter on the viewing experience.
10 Limitation
Lack of Global Translation. Although our method has made significant progress in predicting human pose from sparse inertial sensor data, lacks global tracking of human motion trajectories. We believe that relying solely on IMU for accurate global translation prediction is challenging, as IMUs may drift, resulting in unreliable measurements of acceleration. Combining other types of sensors with IMUs is promising solution.
The Construction for Explicit Structure. As mentioned earlier, any matrix can serve as explicit structure Sg. We provide two examples demonstrating how to construct an explicit structure solely for illustrative purposes. Constructing Sg more effectively for specific tasks requires further exploration.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, discussed them in Sec. 10 of the supplementary materials.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification:
We did not make theoretical contributions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
21

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Yes, we will release the code/data later.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips) cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ttpps.ccppublll/ http:ccccppbcccccaccceeecccccccc guides/ CCodeSubmiieeeee
S issi ssisssoololooooooo oonooiiyyyylolooooooooooomiiimii for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, we specified all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: No, we did not do that.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
22

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, For each experiment, we provided sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https / /twuureurip.../ cc httpppllic/cchhcccuuieeeeeleeeee
Answer: [Yes]
Justification: Yes, the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http:///eeuu.peeeeeee cc httpppplc/ttiistuuiiiiiae
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work does not have direct potential positive societal impacts and negative socictal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
23

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not have such issues.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification; Yes, all the creators or original owners of assets (e.g., code, data, models), used in the paper, are properly credited and are the license and terms of use explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not have such issues this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: Yes, we included relevant details.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our paper does not have such issues.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
25

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "22FGskgAFq", "reviewer": "Reviewer_bDZ5", "summary": "This paper proposes a novel sequence structure learning and modulation approach that endows Transformers with the ability to model and utilize such fixed-sequence structural properties for improved performance on inertial pose estimation tasks. Specifically, this paper introduces a Sequence Structure Module (SSM) that utilizes structural information of fixed-length inertial sensor readings to adjust the input features of transformers. Further, this paper proposes two SSM variants: SSM-S and SSM-T. These variants incorporate the structural inductive biases of the IMU sensor layout (spatial) and time frames (temporal) into transformer learning.\nExtensive experiments across multiple benchmark datasets demonstrate the superiority of the approach against state-of-the-art methods and has the potential to advance the design of the transformer architecture for fixed-length sequences.\n\nI have read the responses of the authors and the comments of other reviewers, I would keep my original score.", "questions": "1. In the field of pose estimation, integrating spatio-temporal information is not new. Please explain the innovation and difference of the method in this paper compared with previous methods. \n2. In the model structure design, whether the order of encoders such as SE\\TE will affect the model performance.", "limitations": "This paper has no obvious negative social impact.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper identifies a key limitation of the native transformer architecture: its lack of inductivebiases for modeling fixed-length sequences with inherent structural properties. To address this shortcoming, we propose a novel Sequence Structure Module (SSM) that enables transformers to effectively capture and leverage the structural priors present in fixed-length  sequential data.\n2. For inertial motion capture tasks involving sequential IMU data, we propose two SSM variants: SSM-S and SSM-T, which incorporate structural inductive biases of the IMU sensor layout (spatial) and time frames (temporal), respectively, into transformer learning.\n3. Extensive experiments demonstrate that the method outperforms state-of-the-art ones on the DIP-IMU and TotalCapture datasets by a large margin. To further demonstrate the superiority of the approach, this paper implemented a real-time motion capture system based on six IMUs to evaluate the performance of the model in complex real-world scenarios.", "weaknesses": "1. In the field of pose estimation, integrating spatio-temporal information is not new. Please explain the innovation and difference of the method in this paper compared with previous methods. \n2. The claim regarding the limitations of the Transformer architecture in handling fixed-length input sequences and the extent of these limitations is not sufficiently supported by evidence or theory.\n3. The number of comparison methods in the experiments is insufficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Clarifications on Innovations**\n\nWe believe there is a misunderstanding and hope to clarify that: our technical innovations and contributions **lie not in** the integration of spatio-temporal information, **but in** addressing an inherent shortcoming of the native transformer architecture. Specifically,\n\n- We identify a key limitation of the native transformer architecture: its lack of inductive biases for modeling fixed-length sequences with inherent structural properties. To address this shortcoming, we propose a novel Sequence Structure Module (SSM) that enables transformers to effectively capture and leverage the structural priors present in fixed-length sequential data.\n\n- For inertial motion capture tasks involving sequential IMU data, we propose two SSM variants: SSM-S and SSM-T, which incorporate structural inductive biases of the IMU sensor layout (spatial) and time frames (temporal), respectively, into transformer learning.\n\nTherefore, our contributions are orthogonal to spatial-temporal integration. Please see our summary of contributions at the end of Sec. 1 and Sec. 3 for more details.\n\n**Q2: The claim regarding the limitations of the Transformer architecture in handling fixed-length input sequences and the extent of these limitations is not sufficiently supported by evidence or theory.**\n\nWe believe this is a misunderstanding as well. As discussed in **Sec. 3.1** \"Why is Sequence Structure Modeling Missing in Native Transformer Architecture?\", we **analysed the native transformer architecture** and **provided clear evidence** that its attention matrix only captures the relationship among input tokens rather than the structural information of input sequences. Please see **Sec. 3.1** for more details.\n\n**Q3: More Comparisons with SOTA.**\n\nThank you for your suggestion! As shown in the tables below, we further included comparisons with the classic DIP [1] and the most recent SOTA method PNP [2] (**please note that PNP has not yet been published at the time of submission of this paper: SIGGRAPH 2024 was just held several days ago between 28th July and 1st August**).\n\n**Comparison with SOTAs on DIP-IMU Dataset:**\n| $\\quad \\quad \\quad \\quad$Method | SIP Err| Ang Err| Pos Err| Mesh Err| Jitter |\n| :----: | :----: | :----: | :----: | :----: | :----: |\n| DIP (SIGGRAPH Asia'2018) | 17.10| 15.16  | 7.33 | 8.96  | 3.01  |\n| PNP (SIGGRAPH'2024)| 13.71| 8.75  | 4.97  | 5.77  | 0.17  |\n| **ours**  | **7.90**  | **6.06** | **3.12**  | **3.78** | **0.07** \n\n**Comparison with SOTAs on TotalCapture Dataset:**\n| $\\quad \\quad \\quad \\quad$ Method | SIP Err| Ang Err | Pos Err | Mesh Err | Jitter  |\n| :----: | :----: | :----: | :----: | :----: | :----: |\n| DIP (SIGGRAPH Asia'2018) | 18.62 | 17.22 | 9.42 | 11.22         | 3.62    |\n| PNP (SIGGRAPH'2024) | 10.89  | 10.45| 4.74  | 5.45     | 0.26    |\n| **ours**  | **7.00**  | **6.82**  | **3.36**   | **4.00** | **0.09** \n\nIt can be observed that our method not only outperforms the classic DIP published in 2018 but also the latest PNP published several days ago well after the submission of this paper, which further demonstrates the superiority of our method.\n\nWe will incorporate the results and detailed discussions in our final version.\n\n**References:**\n\n[1] Deep Inertial Poser: Learning to Reconstruct Human Pose from Sparse Inertial Measurements in Real Time. SIGGRAPH Asia 2018.\n\n[2] Physical Non-inertial Poser (PNP): Modeling Non-inertial Effects in Sparse-inertial Human Motion Capture. SIGGRAPH 2024.\n\n**Q4: Ablation study of the order of SE/TE.**\n\nThank you for your suggestion, but as clarified in our response to Q1, our contributions are orthogonal to spatial-temporal integration. Therefore, we followed the common practice [1-5] of arranging the encoders in the \"SE-TE\" order, where i) SE considers the correlation of body joints and returns a latent feature representation for that frame; ii) TE analyzes global dependencies between each spatial feature representation, and generates an accurate pose estimation; which makes more sense as it follows an intra-frame and inter-frame order. Thus, we believe this ablation study is a bit deviated from the focus of our paper.\n\nNevertheless, we provide the results of switching the order from \"SE-TE\" to \"TE-SE\" as follows:\n\n| Method | Ang Err | Jitter | $\\enspace \\tau$ |\n| :----: | :----: | :----: | :----: |\n| SE-TE(ours) | **6.82** | **0.09** | **7.46** |\n| TE-SE | 8.67 | 0.11 | 9.68 |\n\nThe results show that our \"SE-TE\" order outperforms its \"TE-SE\" variant by a large margin, which is consistent with the common practice in many spatio-temporal frameworks.\n\n**References:**\n\n[1] 3D Human Pose Estimation with Spatial and Temporal Transformers. ICCV 2021.\n\n[2] Spatial-Temporal Transformer for Dynamic Scene Graph Generation. ICCV 2021.\n\n[3] ViViT: A Video Vision Transformer. ICCV 2021.\n\n[4] BSTT: A Bayesian Spatial-Temporal Transformer for Sleep Staging. ICLR 2023.\n\n[5] VideoTrack: Learning to Track Objects via Video Transformer. CVPR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CcWjCjUYBN", "reviewer": "Reviewer_8LBr", "summary": "Existing transformers have shown great promise in modeling temporal data. However, in the field of inertial pose estimation from inertial measurement units (IMUs),\nthe lengths of time series are often fixed. This is a property that transformers can take advantage of. Thus, this paper proposes a Sequence Structure Module \nthat explicitly accounts for fixed-length input but also factors in the spatial dependencies of the IMUs worn at various body placements. \n\nThis paper deals with an emerging paradigm for inertial pose estimation that relies on IMUs instead of video. The experiments showed \nthat the proposed architecture can improve the task performance on several metrics including rotation errors, distance errors and motion smoothness.", "questions": "1. Can you show a few examples of what your structure matrices look like before and after training? It will help understand what has changed.\n2. How will your model perform on different users with different physiques? In your demo, most users have similar body shapes?", "limitations": "NA", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper improves IMU-based inertial pose estimation with fewer sensors, which is preferable in their real world applications \n2. The Sequence Structure Module trick is well-motivated. The authors identified the unique aspects of inertial post estimation: (1) fix-length input sequence (2) spatially rich. The proposed module can incorporate the spatial and temporal structure within the data with different priors and optimisation techniques. \n3. The demo provided was much appreciated. The pose estimation seems to be realistic and better than other benchmark techniques.", "weaknesses": "1. Given that the proposed sequence structure module could work with data of fixed lengths with rich spatial information, I am curious how this module could also be applied with more or sensors that either make the structural correlation stronger or weaker. It will help to understand the generasibility of the proposed module.\n2. In your evaluations, how did you tune other models?\n3. Your SSM-T for temporal dependency modelling seems to be very effective in bringing down the jitter from 0.48->0.09 with pre-defined structure. Are there cases where SSM-T will benefit from training at all? If not, maybe there is no need for SSM-T after all but some sort weighted averaging with neighbouring timestamps?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The generalization of the module under more sensors**\n\nThank you for your suggestion! Although we focused on the more challenging \"sparse\" settings (6 IMUs) in our paper, showing additional results of applying our modules to more sensors could further demonstrate their generalization ability. \n\nSpecifically, we increase the number of IMUs from **6** to **10** (with the additional 4 IMUs placed on the left and right shoulders and thighs), and conduct experiments and ablation studies on the DIP-IMU dataset:\n|$\\quad$IMUs|Ang Err|Jitter|$\\enspace\\tau$|\n|:----:|:----:|:----:|:----:|\n|6(ours)|6.06|0.07|6.49|\n|10(ours)|**4.40**|**0.03**|**4.53**|\n\n**Ablation Study**\n|$\\hspace{4.5em}$IMUs|Ang Err|Jitter|$\\enspace\\tau$|\n|:----:|:----:|:----:|:----:|\n|10(ours)|**4.40**|**0.03**|**4.53**|\n|10(w/o SSM-S)|5.69|0.03|5.86|\n|10(w/o SSM-T)|4.56|0.13|5.19|\n|10(w/o SSM-S, w/o SSM-T)|5.58|0.14|6.42|\n\nIt can be observed that, after adding 4 IMUs, our proposed method still works effectively (higher accuracy and lower jitter). Additionally, the ablation study results show that the roles of SSM-S and SSM-T **remain consistent with their performance when using 6 IMUs**. That is, SSM-S improves the accuracy of motion prediction, while SSM-T reduces jitter to enhance the coherence of the posture. Based on the above experimental results, it can be concluded that our proposed method maintains strong generalization ability as the number of sensors increases.\n\n**Q2: Implementation details for other SOTA methods**\n\nThank you for your comments! For Transpose and PIP, since the official open-source repositories do not include any training-related code, we evaluated them on four benchmarks using the provided models and weight files. For TIP and DynaIP, we retrain their models and evaluate them on the same four benchmarks with minimal modifications according to the guidance provided in the official code. We will include more details in our revision.\n\n**Q3: Are there cases where SSM-T will benefit from training at all?**\n\nYes, there are. For example, as the table below shows, in the combination of \"SSM-S(EIHS) && SSM-T(IS)\", we can obtain results of $Ang Err=8.13$, $jitter=0.34$ and $\\tau = 11.42$ with **trainable SSM-T** on TotalCapture dataset, outperforming the baseline model. Therefore, SSM-T could benefit from training, although for our specific task, SSM-T(ES) works the best.\n\n|$\\qquad\\qquad$Method|Ang Err|Jitter|$\\quad\\tau$|\n|:----:|:----:|:----:|:----:|\n|Baseline|8.82|0.48|14.25|\n|SSM-S(EIHS)&&SSM-T(IS)|**8.13**|**0.34**|**11.42**|\n\nWe will include more combinations in Table 4 in our revision.\n\n**Q4: What do structure matrices look like before and after training?**\n\nThank you for your suggestion! We visualized the spatial structure matrix $S_{E-S}$ (before training), the learnable matrix $P_{S}$ and the final spatial structure matrix $S_{EI-S}$ (please refer to .pdf file). It can be observed that: \n- **The overall pattern of the structure matrix remain the same** before and after training ($S_{E-S}$ and $S_{EI-S}$), i.e., the movements between the two hands and the two legs still exhibit high correlation; and the movements of the head and the root (spine) are still negatively correlated. This demonstrates the effectiveness of our $S_{E-S}$ as initialization/prior.\n\n- **The learnable matrix $P_{S}$ adds small offsets** to the spatial structure matrix, i.e., slightly suppresses the correlation between two hands, two legs and head vs. root. We attribute this to the different motion distributions among datasets: i) in AMASS, **daily actions** (e.g., walking, jogging, running, sitting and stretching) are dominant, and the movements of both hands and legs show extremely high consistency; ii) in the DIP-IMU, although daily actions are also the majority, there are a large number of **single-hand and single-leg movements**, such as single-hand raising, grasping and swinging; single-leg lifting, etc., which weaken the movement consistency of both hands and legs; iii) in the Andy and CIP, there are numerous **industry-oriented activities**, which are very different from daily movements, resulting in a relatively large adjustment range of the learnable matrix $P_{S}$. This demonstrates the effectiveness of our $P_{S}$ in adapting the structure matrix to different datasets, and maintains a high generalization ability.\n\nWe will include this discussion in our revision.\n\n**Q5: The performance on different users with different physiques**\n\nThank you for your suggestion! The diversity of physiques among users has already been considered in the DIP-IMU dataset, i.e., it contains body shape information for its 10 participants (e.g., mass, height). To demonstrate this more clearly, we computed the BMI values (BMI = Mass(kg) / Height(m)^2) for each individual, categorize them into three groups, and report our model's performance across these three categories as follows:\n\n| Category | $\\qquad \\enspace$ BMI | $\\quad\\enspace$Subject ID | Number | Ang Err | Jitter | $\\enspace\\tau$ |\n| :----: | :----: | :----: | :----: | :----: | :----: | :----: |\n| C1 | BMI < 21.7 | S2/S6 | 2 | 7.33 | 0.08 | 7.94 |\n| C2 | 21.7 <= BMI <= 24.9 | S1/S3/S5/S7/S8/S9 | 6 | 7.63 | 0.09 | 8.35 |\n| C3 | BMI > 24.9 | S4/S10 | 2 | 8.07 | 0.09 | 8.83 |\n| Average | / | / | 10 | 7.66 | 0.09 | 8.35 |\n\nThe experimental results demonstrate that our method is robust and performs well across users with different physiques. For reference, we also include the results for each individual as follows:\n|ID|Ang Err|Jitter|$\\enspace\\tau$|Mass(kg)|Height(cm)|BMI|Category|\n|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n|S1|7.74|0.05|8.13|86|186|24.85|2|\n|S2|7.47|0.05|7.85|65|178|20.51|1|\n|S3|7.95|0.11|8.87|87|187|24.87|2|\n|S4|7.63|0.11|8.51|78|170|26.98|3|\n|S5|7.64|0.12|8.61|80|180|24.69|2|\n|S6|7.20|0.11|8.03|58|172|19.60|1|\n|S7|6.89|0.09|7.53|70|178|22.09|2|\n|S8|7.13|0.09|7.80|80|180|24.69|2|\n|S9|8.48|0.06|9.00|85|187|24.30|2|\n|S10|8.51|0.07|9.12|87|181|26.55|3|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GYU71Tg3w2", "reviewer": "Reviewer_eYY8", "summary": "In this paper, the authors study the inertial pose estimation problem and propose to add Sequence Structure Modules (SSM) to the spatial-temporal transformer architecture. The proposed SSM carries prior structural information of both spatial and temporal domain, and is shown to outperform multiple baselines and a plain spatial-temporal transformer.", "questions": "1. It is not clear whether the spatial structural information calculated by (6) and (7) really works. According to setting 4 in Table 4, using ES for SSM-S does not show significant improvement. The best performance is achieved with EIHS for SSM-S (setting 5). However, since it contains learnable parameters, the final attention matrix might look very different from the pre-computed $E_{E-S}$ shown in Figure 3. Maybe it is better to show the learned spatial matrix in the paper as well.\n\n2. It is also not clear how the SSM-T module works. The authors mention that the aim of the SSM-T module is to enforce a high correlation between adjacent frames. However, since SSM-T is only applied to the inputs of the temporal encoder, it is not guaranteed that such constraint still exists within the temporal encoder. It would be great if the authors could provide some more explanations.", "limitations": "Tthe authors have addressed the limitations", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed method achieves significant improvement compared to the baselines across multiple datasets.", "weaknesses": "1. Though the proposed method achieves great performance, it is unclear why the proposed SSM structure works. The proposed SSM modules are essentially heuristically designed attention modules. Self-attention modules should be able to learn those structural patterns as well.\n\n2. The motivation for this approach is to add inductive bias to the original transformer structure. However, this paper lacks comparisons with other NN structures that contain such inductive bias by their design. For example, one could use GNNs as the spatial encoder and CNNs as the temporal encoder.\n\n3. The novelty of this paper is somewhat limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Why does SSM structure work && The difference between attention and SSM.**\n\nWe believe there is a misunderstanding. In short, our SSM is significantly different from attention modules as **its values is independent from input tokens**; and it can be used **with or without heuristics**, i.e., its ES/IS/EIHS versions (Sec. 3.2). Specifically, as discussed in Sec 3.1, the native attention modules does not make any inductive bias on sequence structures, but instead focuses on **the content of the input**. Every element $\\alpha_{(i,j)}$ in attention map $\\alpha$ is calculated as the product between the $i$-th query ($Q$) and the $j$-th key ($K$) pair, thus representing the relationship between individual tokens rather than **the structure of the input sequence** as a whole. \n\nIntuitively, our SSM remains the same across all input sequences, thus capturing the structural patterns shared by them; the attention module, on the other hand, changes with each input sequence and therefore cannot learn such patterns.\n\n**Q2: More Comparisons with other NN structures.**\n\nThank you for your suggestion. As shown in the table below, we construct a spatio-temporal framework using GCN layers as the spatial encoder and Conv1d layers as the temporal encoder.\n\n|$\\qquad\\qquad\\qquad$ Method | Ang Err | Jitter | $\\quad\\tau$ |\n| :----: | :----: | :----: | :----: |\n| GCN + Conv1d | 14.31 | 0.28 | 18.93 |\n| Transformer + Transformer（ours） | **6.82** | **0.09** | **7.46** |\n\nThe results show that our method significantly outperforms the GCN + Conv1d implementation. We attribute this improvement to the fact that our work provides, for the first time, a new approach to injecting different types of structural inductive bias into the transformer structure, thus inheriting its advantages while addressing its drawbacks to achieve SOTA results.\n\n**Q3: Clarifications on novelty**\n\nAs discussed in our response to Q1, our SSM is very novel and significantly different from attention modules. The values of the attention matrix depend on input tokens and changes with **individual input sequences** while our SSM captures the structural information shared by **all input sequences** and remain the same across different instances, thereby **filling an important gap** of the native transformer architecture in handling fixed-length sequences of rich structure information (e.g., inertial pose estimation).\n\n**Q4: Does the final structure matrix $S_{EI-S}$ look very different from the pre-computed $S_{E-S}$?**\n\nNo, it doesn't, but the overall pattern **remains consistent**. Specifically, we visualized the spatial structure matrix $S_{E-S}$ (before training), the learnable matrix $P_{S}$ and the final spatial structure matrix $S_{EI-S}$ (please refer to .pdf file). It can be observed that: \n\n- The overall pattern of the structure matrix **remain the same** before and after training ($S_{E-S}$ and $S_{EI-S}$), i.e., the movements between the two hands and the two legs still exhibit high correlation; and the movements of the head and the root (spine) are still negatively correlated. This demonstrates the effectiveness of our $S_{E-S}$ calculated by Eq. (6) and (7) as initialization/prior.\n\n- The learnable matrix $P_{S}$ adds **small offsets** to the spatial structure matrix, i.e., slightly suppresses the correlation between two hands, two legs and head vs. root. We attribute this to the different motion distributions among datasets: **i)** in AMASS, **daily actions** (e.g., walking, jogging, running, sitting and stretching) are dominant, and the movements of both hands and legs show extremely high consistency; **ii)** in the DIP-IMU, although daily actions are also the majority, there are a large number of **single-hand** and **single-leg movements**, such as single-hand raising, grasping and swinging; single-leg lifting, etc., which weaken the movement consistency of both hands and legs; **iii)** in the Andy and CIP, there are numerous **industry-oriented activities**, which are very different from daily movements, resulting in a relatively large adjustment range of the learnable matrix $P_{S}$. This demonstrates the effectiveness of our $P_{S}$ in adapting the structure matrix to different datasets, and maintains a high generalization ability, which accounts for the improvement between Settings 4 and 5 in Table 4.\n\nIn conclusion, the role of the learnable matrix $P_{S}$ is to make **slight adjustments** to the motion distribution sampled from AMASS, and the overall pattern of the spatial structure matrix **remain consistent** before and after training. We will include this in the revision as it will further enhance our contribution. Thank you for your suggestion!\n\n**Q5: Explanation on how the SSM-T module works and why applied to only the inputs of the temporal encoder.**\n\nThanks for such an insightful question! Yes, the motivation of our SSM-T(ES) is to enforce a high correlation between adjacent frames. However, enforcing such correlation may sacrifice the transformer's ability to learn long-term dependencies, thus forming a trade-off. \n\nAs the table below shows, empirically, we observed that applying the proposed SSM-T(ES) module to only the input of the temporal encoder (consisting of 4 transformer layers) works the best (smallest $\\tau$) and uses it as our final choice. In other words, we consider which layer to apply as a hyperparameter choice of our proposed modules, which does not affect our contributions. We will include more detailed discussions in our revision.\n\n| $\\qquad$Method | Ang Err | Jitter | $\\enspace\\tau$ |\n| :----: | :----: | :----: | :----: |\n| SSM-T x 1 (ours) | 6.82 | 0.09 | **7.46** |\n| SSM-T x 2 | 7.41 | 0.06 | 7.87 |\n| SSM-T x 3 | 7.79 | 0.05 | 8.19 |\n| SSM-T x 4 | 7.78 | 0.04 | 8.09 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FdwJWrL9h2", "reviewer": "Reviewer_2q9D", "summary": "This paper addresses the human pose estimation task with signals of 6 IMUs on the body. Because there are spatial correspondence in multiple IMUs across the body and temporal correspondence in the signal, this paper proposes to model the spatial relation of the 6 devices and the temporal relation in the signal by using learnable weight matrix. The weight matrix is initialized by distribution in the dataset and the transformer model first model the spatial info and then the temporal info. The proposed model achieves the SOTA accuracy compared to previous methods.", "questions": "Figure 3 and 7 show the explicitly-set weight matrix which show obvious patterns in spatial and temporal dimensions. Do you observe any explainable patterns in the learned weight matrix (EIHS)?", "limitations": "This paper has listed some limitations and there is no obvious negative societal impact of their work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* This paper identifies a key character of IMU signals, the structural relation in the spatial and temporal dimensions, especially between devices.\n* The experiment accuracy is clearly better than previous methods.\n* This paper is well written and the structure is clear.", "weaknesses": "1. The authors attribute the performance increment to 1) spatio-temporal framework 2) SSM-S 3) SSM-T, but they only conduct ablation studies for 2) and 3). From table 3, we can see that the 1) as the baseline already outperforms all the previous methods significantly. The readers may expect more analysis and ablation experiments on the reason why 1) already achieves the SOTA accuracy.\n\n2. Figure 3 shows that there is an obvious relationship between the left and right hands and the left and right legs. This data distribution is specific to particular datasets. Although good results have been achieved with the current datasets, if such a distribution pattern does not exist in a new dataset, using the same parameters to model multiple devices may not yield good results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Why does the baseline already outperform SOTA?**\n\nThank you for your comments! We hope to clarify that our baseline is a \"strong\" one that applies the powerful spatial-temporal framework [1-5] to inertial pose estimation task, thus outperforming SOTA. Specifically, as elaborated in [1-5], the key insights of spatial-temporal framework in motion-related works are: i) spatial encoders can model local relationships between human body joints; ii) temporal encoders can capture the global dependencies across frames in the entire sequence. Although to our knowledge, we are the first to apply spatial-temporal framework to inertial pose estimation, we think it is **somewhat straightforward** and suggest that it should only be **used as a baseline** from which to contribute. Therefore, our contributions (SSMs) are orthogonal to it and we think experiments on it may deviate a bit from the main focus of our work.\n\nNevertheless, we provide ablation study results of our spatial-temporal framework on the TotalCapture dataset as follows.\n\n|$\\qquad\\qquad\\qquad$ Method | Ang Err | Jitter | $\\quad \\tau$ |\n| :----: | :----: | :----: | :----: |\n| Spatial Transformer Encoder **only** | 13.61 | 0.56 | 23.83 |\n| Temporal Transformer Encoder **only** | 10.20 | 0.53 | 17.33 |\n| Spatial-Temporal Transformer Encoder | 8.82 | 0.48 | 14.25 |\n\nIt can be observed that both spatial and temporal transformer encoders play crucial roles in reducing angular error and jitter, especially the temporal encoder, justifying their effectiveness in inertial pose estimation. We will include more detailed explanation in our revision.\n\n**Q2: Generalization of $S_{E-S}$ (Fig. 3) across datasets**\n\nGreat question! We hope to clarify that we did not rely solely on $S_{E-S}$ but introduced a learnable matrix $P_{S}$ to model the differences between specific datasets and $S_{E-S}$ constructed using the AMASS dataset (Eq. 4). The reason we used AMASS to construct $S_{E-S}$ is that it is a large-scale human motion dataset consisting of a variety of human motion data, which can be used to construct a representative $S_{E-S}$ as a useful starting point/initialization for learning.\n\nIn addition, we provided the method (Sec. 3.4, Eq.6, 7) to construct $S_{E-S}$ so that one can also compute a dedicated $S_{E-S}$ for each new dataset used easily.\n\n**Q3: Are there any explainable patterns observed in the learned weight matrix (EIHS)?**\n\nYes, there are. To facilitate explanation, we visualized the spatial structure matrix $S_{E-S}$ (before training), the learnable matrix $P_{S}$ and the final spatial structure matrix $S_{EI-S}$ (please refer to .pdf file). It can be observed that: \n\n- **The overall pattern of the structure matrix remain the same** before and after training ($S_{E-S}$ and $S_{EI-S}$), i.e., the movements between the two hands and the two legs still exhibit high correlation; and the movements of the head and the root (spine) are still negatively correlated. This demonstrates the effectiveness of our $S_{E-S}$ as initialization/prior.\n\n- **The learnable matrix $P_{S}$ adds small offsets** to the spatial structure matrix, i.e., slightly suppresses the correlation between two hands, two legs and head vs. root. We attribute this to the different motion distributions among datasets: i) in AMASS, **daily actions** (e.g., walking, jogging, running, sitting and stretching) are dominant, and the movements of both hands and legs show extremely high consistency; ii) in the DIP-IMU, although daily actions are also the majority, there are a large number of **single-hand** and **single-leg movements**, such as single-hand raising, grasping and swinging; single-leg lifting, etc., which weaken the movement consistency of both hands and legs; iii) in the Andy and CIP, there are numerous **industry-oriented activities**, which are very different from daily movements, resulting in a relatively large adjustment range of the learnable matrix $P_{S}$. This demonstrates the effectiveness of our $P_{S}$ in adapting the structure matrix to different datasets, and maintains a high generalization ability.\n\nWe will include this in the revision as it will further enhance our contribution. Thank you for your suggestion!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers for providing detailed and constructive comments that have helped to improve the quality of our manuscript. 

- We have provided rebuttals to the comments of each reviewer. 

- We have also attached a pdf file containing figures that were requested by the reviewers (8LBr, eYY8, 2q9D). 

We hope these resolve the concerns raised and we are happy to answer any additional questions!

Meta-review: This paper proposes a novel approach to enhance Transformers for inertial pose estimation by introducing a Sequence Structure Module (SSM). The SSM utilizes structural information from fixed-length inertial sensor readings to adjust Transformer input features, with two variants, SSM-S and SSM-T, focusing on spatial and temporal inductive biases, respectively. Experiments across multiple benchmark datasets show that this approach outperforms state-of-the-art methods, improving metrics like rotation and distance errors, as well as motion smoothness. The paper highlights that existing Transformers are effective in modeling temporal data but don't fully exploit the fixed-length nature of time series in inertial pose estimation from IMUs. By accounting for this fixed-length input and spatial dependencies of IMUs, the proposed SSM enhances model performance. The SSMs, integrated into a spatial-temporal Transformer architecture, incorporate prior structural information from spatial and temporal domains, leading to superior results compared to baselines and standard Transformers. For human pose estimation using six body-mounted IMUs, the model leverages a learnable weight matrix to capture spatial and temporal relationships, achieving state-of-the-art accuracy.

The paper is well written, structured, and motivated. The proposed method clearly outperforms existing methods on publicly available datasets. In my opinion there is still some room for discussion on the novelty of the approach, in particular under comparison of other architectures that apply inductive biases. I recommend to accept the paper and encourage the authors to use the feedback provided to improve the paper for its final version, and to add some discussion on the raised open questions.
Decision: Accept (poster)