HUMOF: Human Motion Forecasting in Interactive Social Scenes
Authors: Caiyi Sun
Yujing Sun
Xiao Han
Zemin Yang
Jiawei Liu
Xinge Zhu
SM Yiu
Yuexin Ma
Venue: ICLR 2026 Poster (2026)
Keywords: scene-aware human motion forecasting
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Human motion prediction considering human-scene and human-human interactions

Abstract: Complex dynamic scenes present significant challenges for predicting human behavior due to the abundance of interaction information, such as human-human and human-environment interactions. These factors complicate the analysis and understanding of human behavior, thereby increasing the uncertainty in forecasting human motions. Existing motion prediction methods thus struggle in these complex scenarios. In this paper, we propose an effective method for human motion forecasting in dynamic scenes. To achieve a comprehensive representation of  interactions, we design a hierarchical interaction feature representation so that high-level features capture the overall context of the interactions, while low-level features focus on fine-grained details. Besides, we propose a coarse-to-fine interaction reasoning module that leverages both spatial and frequency perspectives to efficiently utilize hierarchical features, thereby enhancing the accuracy of motion predictions. Our method achieves state-of-the-art performance across four public datasets. The source code will be available at https://github.com/scy639/HUMOF.

Published as a conference paper at ICLR 2026
HUMOF: HUMAN MOTION FORECASTING IN INTER- ACTIVE SCENES
Caiyi Sun 1:2 Yujing Sun Xiao Han', Zemin Yang',
Jiawei Liu', Xinge Zhu", Siu-Ming Yiu Yuexin Ma
ShanghaiTech University
"The University of Hong Kong
3 Digital Trust Centre, Nanyang Technological University
'Sun Yat-sen University
Thee Chinese University Hong Kong
caiyisun. gmail 1.. com, mayaex_n@@Stannnaattcc.eedd
ABSTRACT
Complex dynamic scenes present significant challenges for predicting human be- havior due the abundance of interaction information, such human-human and human environment interactions. These factors complicate the analysis and understanding of human behavior, thereby increasing the uncertainty in forecasting human motions. Existing motion prediction methods thus struggle in these complex scenarios. In this paper, we propose an effective method for human motion forecast- ing in dynamic scenes. To achieve comprehensive representation of interactions, we design a hierarchical interaction feature representation so that high-level fea- tures capture the overall context of the interactions, while low-level features focus on fine- ane-ggrined details. Besides, we propose a coarse-to to-se- interaction reasoning module that leverages both spatial and frequency perspectives to efficiently utilize hierarchical features, thereby enhancing the accuracy of motion predictions. Our method achieves state-of-the-arr performance across four public datasets. The source code will be available at
com faom/scy639 9/HUMOF)
INTRODUCTION
Human motion forecasting is essential across wide range of applications, including surveillance, healthcare, autonomous driving, and human-robot interaction. The ability to accurately anticipate human behavior in dynamic environments is key to enhancing system safety, operational efficiency, and user experience. However, this task presents significant challenges, including the inherent complexity and variability of human motion, well as the impact of diverse environmental factors.
In early times, many works predominantly addressed the task of human motion prediction by using simple representations of environmental states. For example, some methods (Zhang et al. [2023] Xu et al.] 2023b Ma et al. 2022. Xu et al. 2023a Gao et al., 2023 Aksan et al. 2021] Wang et al. [0244] Su et al. 2021 Tang al. 2023) rely solely past human actions predict their future motions,
while others (Cao et al., 2020 Mao et al.) 2022 Scofano et al. 2023 Zheng et al. [022 Xing et al.)
2025) integrate static scene features into the network all once. However, these approaches struggle to adapt to real-world applications, where dynamic environmental constraints play crucial role. Actually, to better predict how humans respond dynamic environments. is essential consider
the interaction influence. Some works (Wang et al., 2021 Guo et al.) 2022b; Vendrow et al. 2022
Saadatnejad et 2024 Gao et al. 2024a Xu et al. 2023c] Peng et al.) 2023 Xiao et al. 2025) have started addressing motion prediction in challenging multi-person scenarios, using attention
mechanisms implicitly model the human-humm interaction. However, these works overlook the dynamic relationship between humans and the nonhuman environment, which is equally critical for accurate motion forecasting in real scenes.
" "Equal contribution.
I Work done during internship at ShanghaiTech University. 1 "orrespondiig authors.

---

Published as a conference paper at ICLR 2026
In fact, real-world environments inherently complex and dynamic, where existing frequent human- human interactions, c.g., engaging in conversation, approaching others, or avoiding collisions, as well as human-scene interactions, e.g., sitting on stairs, lying on a bed, shown in Figure [ It is important to model all human-related interactions in one framework for more accurate human motion forecasting. Although (Mueller al. 2024) made the first attempt to address the problem under this setting, it decouples feature extraction for interacting humans and scenes, fails to fully capture the interaction features, and relies on predefined semantic labels for the scene. As result, its prediction performance is limited, and it is not practical for real-world applications. The main challenge for forecasting human motion in a realistic and dynamic environment is twofold. Given the vast array of diverse, multi-level interactions between humans and their surroundings, as well as between individuals, how can we design effective representations to capture these complex interactions? Moreover, even with well-encoded interaction representations, how can we leverage them effectively to enhance prediction accuracy?
In this paper, we have addressed the above two chal- lenges and propose a novel method, named HUMOF, for HUman MOtion Forecasting in complex dynamic scenes. It effectively models human kinematics and dynamics, spatial environment states, temporal in- formation, and the most crucial interaction features, offering significant potential as a world model for human motion. In particular, we introduce a Hier- archical Interaction Representation to effectively capture complex and valuable interaction features. The hierarchical representation manifests in several dimensions: (1) It includes both human-human inter- action modeling and human-scene interaction mod- eling; (2) It captures interactions through explicit representations, i..., interactive distances, and implic- itly learns interaction features through the network; (3) It integrates both high-level semantic interaction features and low-level geometric interaction features. Furthermore, to fully utilize the hierarchical represen- tation for enhancing human motion prediction, we design Coarse-to-Finn Fine Interaction Reasoning Module. Specifically, to encourage the model to focus on global environmental understanding while minimizing interference from noisy low-level environmental information and high- gig-frreuency details in the earlier stages, and refine fine-grained details in the later stages, we implement the coarse-to- fine mechanism from two perspectives: (1) spatial perspective, through our coarse-to- to-ffne injection strategy, high-level features injected into early Transformer layers for semantic understanding of human actions, while low-level features are introduced in later Transformer layers to perceive geo- metric details; (2) In frequency perspective, our DCT rescaling mechanism suppresses the updating of high-frequency components of human motion in earlier layers, and progressively encourages the model to focus on low-frequency details in later stages. Extensive experimental results demonstrate that our method achieves state-of-the-art performance on four public datasets, and ablation studies show the effectiveness of our detailed designs. Our contributions are summarized follows:
Figure I: Real dynamic scenes involve com- plex numann-human and human-scene interac- tions. We propose to predict human motions under such challenging settings, where exist- ing methods struggled.
We present an effective method for human motion prediction in dynamic environments, involving both human-human and human-scene interactions, achieving state-of-the-art performance in various dynamic scenarios.
We introduce hierarchical interaction feature representation to achieve a comprehensive understanding of human-human and human-scene interactions.
We propose coarse-to- coarse-to-fine interaction reasoning module to fully leverage hierarchical interactive features to enhance prediction accuracy.
2 RELATED WORK
Single-Persee Human Motion Prediction Early works mainly consider the own kinematic and dynamic influence future human motions and predict the motion for single person (Zhang et al.
2

---

Published as a conference paper at ICLR 2026
2023:
Xu et al. 2023b Ma et al.
2022 Xu et al. 2023a Gao et al. 2023 Aksan et al. 2021 Wang lang et al. 2023). Many approaches (Fragkiadaki et al. [011]] Jain Jain et al.
et al. 2024| Su et al.
2021
2016
Martinez et al. 2017: Liu et al. 2022) relied on Recurrent Neural Networks (RNNs) to capture
temporal dependencies, overlooking spatial relationships. More recent methods have shifted towards
Graph Convolutional Networks [Li et al. (2022); [Chen al. (2020); Dang al. (202T), ) Temporal
Convolutional Networks (Sofianos et al. (021). and Transformers Mao et al. (2020); Cai et al. (2020);
Aksan et al. (20TT; Xu et al. (2023a), aiming to capture complex spatial-temporal relationships. However, these methods primarily concern personal situations to predict future motions, limiting the
application in real-world scenarios.
Scene-Aware Human Motion Prediction Recent advancements (Cao et al. 2020 Mao et al. 2022
Scofano et al., 2023; Zheng et al. 2022, Xing et al., (025) have started incorporating scene context into human motion prediction tasks. Some approaches (Cao et al. [2020) represented scenes as 21) images, but struggled when handling occlusions and failed to maintain consistency between local and global motion. GIMO (Zheng al. 2022) attempted to enhance prediction accuracy by incorporating eye gaze; ContactAware (Mao et al. 2022) leveraged a contact map to encode human- scene re relationships STAG (Scofano al. (023) proposed a three-stage approach that sequentially processes contact points, trajectories, and poses. MutualDistance (Xing et al., (025) offered an explicit human-scene interaction model using mutual distance. Although these methods have effectively modeled human-scene interactions, they focus on static scenes, neglecting dynamic social interactions between humans.
Social-Aware Human Motion Prediction Recent studies ( (Adeli et al. 2020; 2021 Wang et al.
2021 Guo al., 2022b Vendrow et al.
2022
Tanke al. 2023b Saadatnejad al. [2024] Gao et al.
2024a bi Xu et al. 2023cc] Peng et al.
2023
Jeong et al.) 2024 Xiao et al., 2025 in multi-person
human pose forecasting focus mainly on modeling human interactions in group scenarios. Most
Guo et al. 2022b; Vendrow et al. 2022; Saadatnejad al., 2024] Gao et al., 2024a Xu et al. 2023c Peng et al.) 2023] Xiao et al. 2025) are popular for this task due to their strong learning capabilities: T2P (Jeong et al. (024) sequentially predicts global trajectory and local pose; IAFormer (Xiao et al. 2025) proposed to learn amplitude- baseed interactions and prior knowledge. However, methods in this category overlook the importance of scene information. A recent work (Mueller et al. 2024) uses the diffusion model for long-term motion generation considering both static scene and motion of other individuals. However, it only implicitly encodes the scene and other individuals, without explicit modeling of human-to to-un-tt--cenee and human-tor human interaction. Additionally, it treats the scene set of discrete objects with semantic tags, relying on ground-truth segmentation results, which limits its applicability in real-world scenarios involving raw sensor data.
recently, Transformers (Wang et al.] 2021
3 METHODOLOOGY
The key challenge in forecasting human motion within complex dynamic environments lies in effec- tively encoding and leveraging the involved human-human and human-scene interactions. Hence, we, on one hand, propose hierarchical approach comprehensively encode these representations (Fig- ure Bbb)), and on the other hand, present a Coarse-to-Fin Interaction Reasoning Module (Figure 3p) to fully leverage the representations.
Problem Definition. The task is to predict person's s future motion given their past motion, the point cloud of static scene elements, and the past motion of other individuals in the vicinity.
Inturaativ
The input of our model includes three parts: 1). A historical motion sequence of the target person X 1:
Ang jive
where x, = {x].- .x'' } E R represents the motion of jin. joint, with each x corresponding to the 3D coordinates of a joint at Le, frame; 2). The scene's 3D point cloud S (s..... sN}} with N points; And 3). the historical motion sequence yi(k) [y(*) .... .y (k)) E xxxxx3 of the kes (k E K) interactive person in the scene, which also consists
Coerse
Finz
[c] Canrect-faa flas inserictiee Reasening Modalo
Tarprpernon
Probelas
Do Mimeentoattee th www.eetaaaaattit...aaaao Regeeseiiiee
3
Figure 2: HUMOF Overview.

---

Published as a conference paper at ICLR 2026
(a) Hicrarchical Human- Human Interaction (HHH)) Repecscutation
Transtemer Laper
Brietirs Leoding
Sell Lauydiieerr.....
you
Matie Focader
PCT Buscalag
Anmoaade
FTN
Juhca
Folan
BCT Rescalog
Acteation
Crow Alteake
Mua
0000
BCT Rocalig
MLP
Salf Afteetion
Kedy-imn Jnken
(c) Coars-do-llte Ressoning Modalr
Target
Prediction
GCN
Trandernet
Tranferner
Trandienet
Trandaneer
Tranderner
OCN
Medium Decoder
Targe
gu
Seanc
ICT
Dberde Coaite: Tranions
BNCT
Insen DC1
Pariional
Add
mudiipiiiztion
Elonectiin wiler
Figure 3: Detailed architecture of HUMOF. Our method takes inputs from three aspects: the past mo- tions of the target person, a 3D point cloud for the scene, and motion sequences interactive persons. The interactions sentation and (b) Hierarchical Human-Sean -Senne Interaction Representation, respectively. Thereafter, the hierarchical representations
comprehensively encoded by (a) Hierarchical Human- Human Interaction Repre-
leveraged by (c), a Coarse-to- Fine Interaction Reasoning Module, to predict future motions for the target person. Details of the Interaction-Perreppii Transformer layer are shown on the top right.
of body joints, each with H consecutive poses. Similarly, y (y. -y (K)H) resents the motion of j-th joint of the kis interactive
(k)1 ;...
R rep-
person, with each y; goal is to predict the motion #+ of the target person for the future time steps.
(k)) corresponding to the 3D coordinates of one of his joint at 4r frame. Our
3.1 MOTION ENCODER
Follow prior works (Mao el al. 2022 Scotano et al. 2023 Xiao et al. 2025: Xing et al., 2025), we first pad the sequence X of length H by repeating the last historical pose X for 7 additional frames, to make a padded sequence of length H + T. For simplicity, we still call the padded sequence X. Providing that DCT effective in handling temporal information in human motion prediction (Mao et al. 2022 Scofano et al. 2023) Xing et al. 2025: Xiao et al. (005).. and that GCN excels at uncovering spatial dependencies between human joints Mao et al. [019 Xing et al.) 2025 Li et al. 2020 2022), we combine Discrete Cosine Transform layer (DCT) and Graph Convolutional Network (GCN) (Mao al. 2019 to extract both spatial and temporal representations in the motion encoder (Figure 3 c)left). To help the model identify different joints, learnable
position embedding P E Jx is added to each joint. Here C' C x 3, where C' 20 is the number of DCT coefficients and corresponds to the three directions: x, y, and Finally, the encoding E JxC for a person can be formulated as
X GCN((CT(XX) +
(I)
which encodes features in the frequency domain for each joint x, over the entire motion sequence.
4

---

Published as a conference paper at ICLR 2026
3.2 HIERARCHICAL INTERACTION REPRESENTATION
Complex dynamic scenes involve interactions between humans (Section. 3.2.1 as well as between humans and their environment (Section. B.2.2). To achieve a comprehensive representation of both human-human and human-scent interactions, we incorporate hierarchical features so that high-level features capture the overall context of the interactions, while low-level features focus on fine-grained details. This multi-level approach ensures thorough and nuanced understanding of the interactions.
3.2. HIERARCHICAL REPRESENTATION FOR TUMAN-HUMAN INTERACTION
Regarding human-level interactions (Figure B). when person engages with others, they are involved in two types of motion: independent motion, such as walking, and interactive motion, such as approaching person converse adjusting one's 's path to avoid a collision. Therefore, we introduce a self- self-ennoding submodule (Figure 3 right) to describe their independent motions and a relation- encoding submodule (Figure 3a left) model their interdependencies.
Self Encoding, For each interactive person, we encode their motion sequence independently, captur- ing semantic information specific his motion, shown in Figure (aa) right. This self-encoding step enables each person's 's motion to contribute meaningful social cues. Specifically, the motion sequence (*()) of ben interactive person is first processed through a motion encoder as described in Section 3.1
(k) obtaining joint-level features in the frequency domain 50(4) (y(H), ......,, ..) E ''C'' Then two-layer Transformer processes a learnable body-level feature (kk
body together with the joint-level
feature 51() Note that body-level feature (kk bodyy contains the information from all joints after being passed by the Transformer, serving as the body-level self encoding. While the updated joint tokens (k) 55(k) [y1 .... , } constitute the joint-level self encoding.
Relation Encoding. We observe that despite of the various types of interactions, different interactions always lead distinct distance patterns. Therefore, it is effective and efficient to model interactions with "distances". Hence we model the interactions explicitly to capture their dependencies via defining interactive distances shown in Figure 3(a) left. First, for the juh joint of the kes interactive person, we calculate the interactive distance between this joint and the closest joint of the target person for each of the H frames as the joint-level relation encoding. Specifically, at fth frame, the (k)) joint-level interactive distance D D is computed as:
D D/ o( min
[E(1,J]
where o(-) is mapping function such that closer joints have higher values than more distant ones. Then, we convert the time series of interactive distances {D (k)1
.... (A)H into frequency domain via DCT to get the joint-level relationship encoding D (*) E BCC Second, for the in joint of the target person, we similarly calculate its joint-level relationship encoding with the kes interactive person, denoted as Dx(k) Thereafter, we obtained the body-level relation encoding body (L 1i6)) (,.)),,..D"())),
(())
Human Interaction Tokens. Finally, we concatenate the Self Encoding and Relation Encoding on their respective levels to obtain the Human-Huumal Interaction (HHI) token. To be clear, the Kiy interactive person's 's body-level HHI token is Ō concat( body"
(k) ((k) body)) and joint-level
HHT token is Joint (ō) ((8)) }. where of concat(y (K) (k)).
(%h)
3.2.2 HIBRARCHICAL REPRESENTATION FOR HUMAN-SCER INTERACTION
Considering the vast number of points in the 3D point cloud scene, it is impractical and inefficient to enumerate the target person's interactions with every point. Recalling that a centre point is frequently used to represent its neighbouring points as an approximation in geometric processing, hope to progressively approximate neighbouring points through central points, reducing the total number of points while retaining essential scene information. In this way, we can construct different levels of point approximations with gradually decreasing number of points, ensuring to maintain
5

---

Published as a conference paper at ICLR 2026
rich interaction features across different spatial scales. Meanwhile, noting that most raw 3D scene point clouds lack object-level annotations, our method does not rely on predefined semantic labels as required by SAST (Mueller et al. 2024).
As illustrated in Figure (b), to obtain hicrarchical point approximations, we employ a series of set abstraction layers from PointNet++ (Qi et al. 2017). ). denoted as ,...,.... g j. b [o, 3]. At each level of abstraction, we apply Farthest Point Sampling following PointNet++ to obtain point subsets. Each set abstraction operation processes and refines the point set to create new set with fewer points, preserving efficiency and structure within the point cloud.
Notice that the abstraction layer g(D) takes an interactive feature matrix F())
{f1 (11
Noo)
as imput, where the interactive feature ft of point 8n computed as a collection of interactive distances in the frequency domain. Specifically, for point sn and a joint Xj, we firstly calculate their interactive distance each frame, constituting a time series my
mj [o[[|> x) 12)...... "....,,,((s. - -x"133) ER""
where Φ(-) is a mapping function that closer scene points have higher values than more distant ones (Mao et al. 2022). Next, we convert my into frequency domain and obtain my E RC Finally, we concatenate m, from all joints, along with the coordinates of the scene point 8n, forming s, interactive feature / E /=x+'3+
Such a feature matrix F(() is iteratively computed across subsequent set abstraction layers (Fig- ure 3 b)), where f(6) = ( To further enhance the positional information, we add a position encoding derived from 3D spatial coordinates of each point to the corresponding feature at each abstraction level b E [1. Finally, Fi(b serves as the Human-Scenn Interaction (HSI) tokens.
3.3 COARSE-TO-FING FINE INTERACTION REASONING MODULE
Accurate human motion prediction requires capturing kinetics and dynamics, involving inherent correlations among joints, across the temporal dimension, and with the surrounding environment. To simultaneously leverage these three types of correlations, we present coarse-to- interaction reasoning module. We take the target person's 's representation X and all the interaction features in the frequency domain including human-to-hume interactions (HHI) tokens (Öbody and Öjoint) and human-to-scent interactions (HSI) tokens (+) input, the model reasons about the motion of the target person through all interaction perecpppi-eeeeettivee Transformer layers using coarse-to-fie strategy.
3.3.1 COARSE-TO-FINE INJECTION STRATEGY
With the obtained hierarchical representations for interactions both between human and human (Section 3.2.I) and between humans and scenes (Section 3.2.2 -we establish a strategy to fully leverage this information.
Different from crudely injecting features from multiple levels of the hierarchical representation into each interaction layer of the model, we sequentially inject hierarchical interaction features in coarse-to- fine manner. We assign high-level features to early layers and progressively incorporate low-level features at deeper layers, shown in Figure c). For example, the first layer, high-level HSI tokens (3) and HHI tokens Ösadg are concatenated along token dimension are injected, totaling (3) + K interaction tokens. At the last layer, we inject low-level HSI tokens F(I) and HHI tokens Ojoine. totaling (1) + K x tokens. It allows the model to begin with a global understanding of high-level semantics and gradually narrow its focus local geometry, improving prediction accuracy.
3.3.2 ENTERACTION-PERCERR
TRANSFORMER LAYER
As depicted in the upper right of Figure 3 our Transformer layer begins with processing the target person's joint tokens x via self-attention (SA) designed capture long-range dependencies among joints. To incorporate interactions, we employ a cross-attention (CA) where joint tokens
x()) serve queries, while interaction tokens act keys and values. A feed forward network

---

Published as a conference paper at ICLR 2026
(FFN) (Vaswani al. 2017) follows CA enhance joint tokens. More importantly, to better align with our coarse-to- fine strategy, an adaptive DCT rescaling mechanism is performed on x after each SA, CA and FFN.
Adaptive DCT Rescaling Mechanism. Recall that our coarse-to-fine injection strategy focuses on capturing coarse-level human motions in the early stages, it would be helpful if we could control the influence of finer details. Observing that each primary joint token is constructed from the DCT coefficients of joint motion, with each channel corresponding to specific DCT coefficient, we introduce a learnable DCT rescaling mechanism from the frequency domain to suppress high- frequency details. This mechanism further supports coarse-to-fine strategy by regulating the impact of high-frequency details and noise the Transformer layer through the rescaling vector v' v(X)(") e R which applied on joint tokens
x after each SA, CA, and FFN in element-wiss ((X)(()) where
multiplication manner, formulated as x
~(X)== oa(() ( MLP J==
Here, O is element-wise multiplication, and v(0 is a pre-defined rescaling vector shared across all joint tokens. We design y(!) such that values are close to 1.0 for low-frequency components and progressively decrease for higher frequencies in early layers, effectively suppressing high-frequency components. The suppression is most prominent in the first layer (1 = 1) and gradually weakens in deeper layers, with the rescaling vector eventually having all values equal to 1.0 in the last layer (I = 6). Moreover, for that different types of action may have varied optimal rescaling, a shared rescaling v(() applied uniformly across all input samples is inadequate. We thus further include sample-adaptive vector (x)) to capture such variations, which is computed by applying average pooling across all J joint tokens, followed by MLP to acquire sample-specific information.
3.4 MOTION DECODER AND LOSS
As shown in Figure 3 c)) right, the updated joint tokens ( from the 6. Transformer layer is passed into a GCN decoder and Inverse Discrete Cosine Transform (IDCT) (Mao et al. 2019) to get predicted motion sequence X, which is formulated by x IDCT(GCN(X ())) E (x(#+)))
Loss is computed the L2 distance between the predicted path and pose and the ground-truth Details can be found in Appendix A
4 EXPERIMENTS
4.1 SETUPS
Implementation Details cab be found in Appendix
Datasets.
We conduct experiments on 2 datasets with human-human and human- scene interactions, HIK TTake al. 2023a) and HOIMM (Zhang et 2024), well on datasets with human-
scene interaction scenes, GTA-IM (Cao et al. (020) and HUMANISE (Wang et al. 2022).
Please
refer Appendix B for more dataset details.
Baselines. 3 Scene-Aware Methods, ContactAware (Mao et al., 2022 GIMO (Zheng et al.
2022), STAG (Scofano al. 2023), and MutualDistance (Xing et al. (2255) To evaluate them on dynamic datasets, we introduce the multi-person context by concatenating all persons' information to the original input of their motion decoder. 2 Social-Awaa -ware Methods, T2P Jeong et al. (2024) and IAFormer (Xiao et al. 2025). Since they are highly dedicated to pure multi-person input, we introduce the human-scene-ine interacttor human-sceen-innteracioon features generated by our HUMOF. And Povial-Scene-Awan Method, SAST (Mueller et al.] 2024). As it requires instance segmentation of the scene input, provide it with the ground-trum truth segmentations, except on GTA-IM where we use the segmentation predicted by (Shin et al. 2024) due to the absence of ground truth.
7

---

Published as a conference paper at ICLR 2026
(mmm)
Datsett
Mathed
147.8
106.8
GIMO (o
121.6
199.3
SIAG
239-7
1109
Mutus
Aunrec
246.
138.5
105.9
T2P [eonggat. 2 Parol
33.6
199.5
318.8
487.1
208.7
74.2
108.6
127.5
142.6
96.9
FA Maad
195.0
311.1
71.5
106.5
125.9
137.7
95.0
SAST Mutii:: m0 4024
56.7
187.4
234.9
298.1
189.0
72.3
101.4
118.0
128.6
93.2
Ours
78.8
1774
278,8
388.4
180.7
71.2
100L5
116.9
127.1
90L2
AartArgm Monei 2005
125.6
230.9
285.4
432.0
236.9
106-2
152.8
174.3
197
137-5
GIMO (hehrass Su1ed
131.4
247.7
300.9
454.4
255.2
107.9
155.9
182.6
207
141.0
1101 M²
STAG (Scoiii (118 kurs]
128.1
234.4
289.5
438.1
239.7
102.5
145.0
167.1
185.6
131.2
Mutuy Diitine Nilie F71 2025
53.6
169.7
278.5
402.5
189.9
94.4
137.1
158.2
181.3
125.3
T2P [eoug 2044
74,2
108.8
296.9
429.2
194.1
88.0
135.8
16019
183.2
124.6
JARamer E300 2012
69.0
166.5
200.
423.5
186.1
86.1
135.0
165.9
180.7
121.6
SAST Mudite
75.0
166.2
280.4
403.9
184.8
89.2
133.8
167.0
182.9
122.3
67.1
156.6
268.4
393
176.6
86.3
129.6
155.0
172.1
117.9
Table 1: Comparisons on datasets with dynamic scenes. We compare with scene-aware methods,
ContectAware, GIMO, STAG, and MutuallDistance, social-aware method, IAFormer and T2P, and social- wwwee-cwwaaaeeeee method SAST.
HUMANISE Dataset Wang et al. 2022
GTA-IM Dataset Cao et al. 2020
Parh Error (mmm)
Pose Emro (mm)
Eirur
Pose
0.55
1.0s
1.0s
0.5
1.bx
1.5x
2.0c
0.5
1.0k
1.5x
2.0x
CHE Amcre
52.8
121.1
57.8
97.6
141.4
92.9
53.1
124.0
58.7
94.0
196.1
903
44.5
82.6
125.6
182.9
87.1
40.1
$4.1
65.2
77.2
51.8
GIMO
70.1
129.2
72.0
141.4
150.5
140.2
77.7
144.0
80.2
146.3
192.4
146.5
$2.7
97.8
162.5
241.7
110.3
47.9
60.7
71.1
62.7
59.9
STAG
55.2
134.6
60.7
88.3
131.4
83.0
57.0
131.5
63.2
80.9
131.7
85.6
43.2
79.8
119.9
176.4
83.4
35.4
88.7
59.8
73.5
47.0
SAST
56.2
122.4
62.1
86.0
111.6
S018
57.2
129.
63.7
90.6
124.7
86.7
41.0
77.8
123.1
181.8
85.2
28.2
41.0
53.6
66.8
41.5
MunuiiDisaace
41.5
93.5
45.6
K3.7
130.9
3000
46.7
100.2
50.1
54.3
131.8
80.6
14.
65.9
104.0
158.0
72.0
31.0
48.8
58.9
70.7
44.6
Ours
36.9
874
41.7
69.0
102.0
64.1
39.5
88.1
43.4
69.8
109.3
66.2
2944
55.9
91.7
139.2
62.9
27.0
40.3
30.7
61.5
38.7
Table 2: Comparisons on datasets with static scenes. Results with are from MutanlDDtance.
Histerical mation for Interactive Persons
Historccal moticn for Tirget Person
Predictions for Target Person
(a) Ground-Truit
(b) Ours
(c) SAST
(d) MutualDistance
(e) IAFormer
Visualization of motion prediction results on dynamic scenes in HOI-M More visual results are in the Supplementary Video and Appendix Section
Figure 4:
Xing et al.) (025). evaluate all methods using path error and pose error, which are computed in the same way as C and posse defined in Eq.
Metrics. Following prior works (Mao al. 2022; Scofano al. 2023
4.2 COMPARISONS
Evaluations on Human-human and Human-scene Interaction Scenes We first quantitatively evaluate our approach on two real-worke datasets with dynamic social scenes, HIK (Tanke al.) 2023a) and H (Zhang et al. 2024). As demonstrated in Table I our approach achieves outstanding superior performance to other methods in three categories, highlighting our strong capability in forecasting human motion in real scenarios with complex human-humas and human- scene dynamics. Visual comparisons on dynamic scenes in H are shown in Figure 4 In the first example, the target person exhibits tendency to turn toward the interactive person. Only our method captures this intent and correctly predicts the direction. In the second example, SAST and IAFormer mistakenly infer some poses to be underneath the floor (marked in red boxes). In contrast, our result shows the best physical plausibility and provides the most accurate path and pose prediction. Evaluations long-term predictions be found in Appendix H
8

---

Published as a conference paper at ICLR 2026
Evaluations Human-Scene Interaction Scenes shown in Table. We demonstrate signifi-
cantly superior performance over SOTA scene-aware methods (Mao et al. 2022 2 Scofano et al.) [023)
Xing al. 22025 Zheng et al. 2022) on both the HUMANISE (Wang al. 2022 and GTA-IM (Cao
et al. 2020) datasets. Meanwhile, our method also greatly outperforms the social-scene-aware method
SAST (Mueller et al.) 2024). Note that we do not rely ground truth instance segmentation, which
is required by SAST.
4.3 DISCUSSIONS
Joint Purti- Inference Our method is readily scalable joint multi-person motion forecasting by treating each individual as separate target in data sample and then inferring the batch (batch size = + K). visual result is shown in Figure 5
Handling of Dynamic Scene Elements Currently, few datasets tain a significant number of dynamic scene elements. But note that our model architecture can natively handle dynamic scene elements without any structural changes. Our HSI module computes time series of distances between the target person and scene points. If scene point were dynamic, its coordinates would become time-dependent (p, ps (4)). The distance calculation naturally extends to this dynamic case, and no architectural modifications would be required. Therefore, given a dataset with significant dynamic objects, framework could be trained to handle such scenarios directly. We provide a preliminary validation on a subset of dynamic scenes in Appendix M showing the potential ability to handle dynamic furniture objects despite limited data.
Figure 5: Joint forecasting.
Scalability and Generality Our framework is designed with flexibility and scalability. The core principle- encoding environmental information into unified hierarchical representation and pro- cessing it with a coarse-to- to-fiee mechanism is not fundamentally tied to motion and point cloud data. To incorporate new modalities, such video audio streams, one could employ standard pre-trained encoders (e.g., ViT for images) to produce hierarchical feature tokens. These new tokens could then be seamlessly concatenated with our existing HSI and HHI tokens and processed by the interaction reasoning module. This adaptability allows the framework to be extended to a wider range of scenarios and input types without requiring significant redesign, pointing a path toward more general and scalable models for motion forecasting.
More Discussions on Design Choices are included in Appendix E covering interpretability of cross- attention weights, and choice of holistic point cloud rather than object-level scene representation.
Runtime Analysis Our model takes similar or shorter time to perform inference than other methods. Specifically, our model has 9.6M parameters and achieves an inference time of 43ms on HOI-M². See Appendix Section for more detailed comparisons with baselines.
4.4 ABLATION STUDIES ON THE HOI-M DATASET
All ablated variants share the same network depth, and the parameter counts of all variants are basically the same, ranging from 9M to 11M. More ablations on variants of Coarse-to-Fin Interaction, different number of sampled points, different number of transformer layers, robustness to incomplete interaction information can be found in Appendix Section D
Hierarchical Interaction Representation. Table a) shows the effectiveness of our human-to-scene and human-to-hune tha-tttt-huannn (including self-encoding and relation-encoding) interaction representations. Results indicate a decline in our motion prediction performance when any of these modules are removed, underscoring their individual contributions.
Coarse-to-Fine Interaction Feature Injection. We assess the effectiveness of the coarse-to- interaction feature injection strategy in Table (b). The experiment shows that multi-level interaction feature injection outperforms single-level approaches (coarse-only and fine-only). But our coarse-to- fine strategy utilizes multi-level features in more effective way and further boosts the performance.
Adaptive DCT Rescaling, Table 3(c) validates the effectiveness of the adaptive DCT rescaling mechanism. By suppressing higuufrryyencc updates of joint motion in early stages encourage the

---

Published as a conference paper at ICLR 2026
focus coarse and low-frequency updates, the shared static rescaling vector v() itself improves prediction accuracy. Besides, the performance is further enhanced after combining the sample- adaptive vector (XX), as allows the frequency rescalings to adapt to different input samples.
(a) Hierarchical Representations
(b) Injection Strategy
(c) Adaptive DCT Rescaling
HaHl
Pala Ervor (aua)
Pose Erve (mee)
DCT resciliis
Pose
strangy
167.0
476.4
187A
1347
181.8
1252
164.1
415.7
183.7
133.2
175.5
1309
Coase
4110
1827
135.
1996
1753
158.2
430.
175.
137.0
131.4
175.6
175.0
199.
Matrolivel
Pio-odly
158. 3589
163.5
4093
1771
1824
1338
1333
1367
1789
1212
x
1580
187.2
4n1 177.
5964 196.3
1306 198. 1192
1746
1564
2931
1766
1398
172.1
1150
174.
Table 3: Ablations studies.
CONCLUSIONS
In conclusion, we present an effective approach for human motion forecasting in interactive environ- ments. By representing hierarchical interactive features and employing the coarse-to-fing interaction reasoning module, our method achieves mate-of-theear performance across four public datasets, demonstrating the potential to construct world model for human motion. This approach holds signif- icant promise for real-world applications, such as enhancing closed-loop simulations for autonomous driving and improving the understanding and interaction capabilities of robots.
ETHICS STATEMENT
Our work presents no direct ethical concerns. The primary application of our method is for human motion prediction.
REPRODUCIBILITY STATEMENT
To ensure reproducibility: (1) While not included with this submission, our full project will be released on GitHub upon publication. (2) All experimental details are included in Appendix and (3) Usage of datasets explained in Appendix B
ACKNOWLEDGEMENTS
This work was supported by MoE Key Laboratory of Intelligent Perception and Human-Machine Collaboration Mulloo, Shanghai Frontiers Science Center of Intelligence (ShangHAD), HKRGC Theme-based research scheme project T35-710/20-BB and sz- HK-Macau Technology Research Programme #SCDXX202108222105333700
Artificial
REFERENCES
Vida Adeli, Ehsan Adeli, Ian Reid, Juan Carlos Niebles, and Hamid Rezatofighi. Socially and contextually aware human motion and pose forecasting. IEEE Robotics and Automation Letters, (4):6033-604p 2020.
Vida Adeli, Mahsa Ehsanpour, Ian Reid, Juan Carlos Niebles, Silvio Savarese, Ehsan Adeli, and Hamid Rezatofighi. Tripod: Human trajectory and pose dynamics forecasting in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13390-13400, 2021.
Emre Aksan, Manuel Kaufmann, Peng Cao, and Otmar Hilliges. A ppatio-temporal transformer for 3d human motion prediction. In 2021 International Conference on Vision (3DV), pp. 565-574. IEEE, 2021.
Yujun Cai. Lin Huang, Yiwei Wang. Tat-Jen Cham, Jianfei Cai, Junsong Yuan, Jun Liu, Xu Yang, Yiheng Zhu, Xiaohui Shen, et al. Learning progressive joint propagation for human motion prediction. In ECCV, pp. 226-242. Springer, 2020.
10

---

Published as a conference paper at ICLR 2026
Zhe Cao, Hang Gao, Karttikeya Mangalam, Qi-Zhi Cai, Minh Vo, and Jitendra Malik. Long- human motion prediction with scene context. In ECCV, pp. 387-404. Springer, 2020.
Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In International conference on machine learning, pp. 1725-1735 PMLR, 2020.
Lingwei Dang, Yongwei Nie, Chengjiang Long, Qing Zhang, and Guiqing Li. Msr-gen: Multi-scale residual graph convolution networks for human motion prediction. In Proceedings of the IEEF/CVF international conference on computer vision, pp- 11467-11477 2021.
Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. Recurrent network models for human dynamics. ICCV, pp. 4346-4354, 2015.
Xuehao Gao, Shaoyi Du, Yang Wu, and Yang Yang. Decompose more and aggregate better: Two closer looks at frequency representation learning for human motion prediction. In Proceedings of the IEEF/CVE conference computer vision and pattern recognition, pp. 6451-6460, 2023.
Xuehao Gao, Yang Yang, Yang Wu, Shaoyi Du, and Guo-Jun Qi. Multi-condition latent diffusion network for scene-aware neural human motion prediction. IEEE Transactions on Image Processing. 2024a.
Yang Gao, Po-Chien Luan, and Alexandre Alahi. Aulti-transmotion: Pre-trained model for human motion prediction. arXiv preprint arXiv:2411.02673 2024b.
Anand Gopalakrishnan, Ankur Mali, Dan Kifer, Lee Giles, and Alexander G Ororbia. A neural temporal model for human motion prediction. In CVPR, pp. 12116-12125, 2019.
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5152-5161, 2022a.
Wen Guo, Xiaoyu Bie, Xavier Alameda-Pineee and Francesc Moreno-Noe -ogguer Multi-person extreme motion prediction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 33053-13066 2022b.
Ashesh Jain, Amir Zamir, Silvio Savarese, and Ashutosh Saxena. Structural-rnn: Deep learning on spatio-temporal graphs. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 5308-5317, 2016.
Jaewoo Jeong. Daehee Park, and Kuk-Jin Yoon. Multi-agent long-term 3d human pose forecasting via interaction-aware trajectory conditioning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 1617-1628, 2024.
Maosen Li, Siheng Chen, Yangheng Zhao, Ya Zhang, Yanfeng Wang, and Qi Tian. Dynamic multiscale graph neural networks for 3d skeleton based human motion prediction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 214-223, 2020.
Maosen Li, Siheng Chen, Zijing Zhang, Lingxi Xie. Qi Tian, and Ya Zhang. Skeleton-pant graph scattering networks for human motion prediction. European conference on computer vision, pp. 18 36-36.660 Springer, 2022.
Zhenguang Liu, Shuang Wu, Shuyuan Jin, Shouling Ji, Qi Liu, Shijian Lu, and Li Cheng. Investigating pose representations and motion contexts modeling for 3d motion prediction. IEEE transactions on pattern analysis and machine intelligence, 45(1):681-697) 2022.
Tiezheng Ma. Yongwei Nie, Chengjiang Long, Qing Zhang, and Guiqing Li. Progressively generating better initial guesses towards next stages for high-quality human motion prediction. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 6437-6446, 2022.
Wei Mao, Miaomiao Liu, Mathieu Salzmann, and Hongdong Li. Learning trajectory dependencies for human motion prediction. In ICCV, pp. 9489-9497, 2019.
11

---

Published as a conference paper at ICLR 2026
Wei Mao, Miaomiao Liu, and Mathieu Salzmann. History repeats itself: Human motion prediction via motion attention. In ECCV, pp. 474-489, Springer, 2020.
Wei Mao, Richard Hartley, Mathieu Salzmann, et al. Contact-aware human motion forecasting. Advances in Neural Information Processing Systems, 0937356-7367,, 2022.
Julieta Martinez, Michael J Black, and Javier Romero. On human motion prediction using recurrent neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2891-29000 2017.
Takeru Miyato, Bernhard Jaeger, Max Welling, and Andreas Geiger. Gta: A geometry-aware attention mechanism for multi-view transformers. In International Conference Learning Representations (ICLR), 2024.
Felix B Mueller, Julian Tanke, and Juergen Gall. Massively multi-person 3d human motion forecasting with scene context, arXiv preprint arXiv:2409. 12189, 2024.
Xiaogang Peng, Siyuan Mao, and Zizhao Wu. Trajectory-aware body interaction transformer for multi-person pose forecasting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 17121-17130, 2023.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet++: Deep hierarchical feature learning on point sets metric space. Advances in neural information processing systems, 30, 2017.
Saced Saadatnejdd, Yang Gao, Kaouther Messaoud, and Alexandre Alahi.
Promptable human trajectory prediction. In International Conference on Learning Representations (ICLR), 2024.
Luca Scofano, Alessio Sampieri, Elisabeth Schiele, Edoardo De Matteis, Laura Leal- Taixé, and Fabio Galasso. Staged contact-aware global human motion forecasting. In BMVC, 2023.
Sangyun Shin, Kaichen Zhou, Madhu Vankadari, Andrew Markham, and Niki Trigoni. Spherical mask: Coarse-to-fine 3d point cloud instance segmentation with spherical representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1060-4069, 2024.
Theodoros Sofianos, Alessio Sampieri, Luca Franco, and Fabio Galasso. Space-time-separable graph convolutional network for pose forecasting. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 11209-11218, 2021.
Pengxiang Su, Zhenguang Liu, Shuang Wu, Lei Zhu, Yifang Yin, and Xuanjing Shen. Motion prediction via joint dependency modeling in phase space. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 713-721, 2021.
Jianwei Tang, Jieming Wang, and Jian-Fang Hu. Predicting human poses via recurrent attention network. Visual Intelligence, 1(1):18, 2023.
Julian Tanke, Oh-Hun Kwon, Felix В Mueller, Andreas Doering, and Juergen Gall. Humans in kitchens: a dataset for multi-person human motion forecasting with scene context, Advances in Neural Information Processing Systems, 36:10184-10196 2023a.
Julian Tanke, Linguang Zhang, Amy Zhao, Chengcheng Tang, Yujun Cai, Lezi Wang, Po-Chen Wu, Juergen Gall, and Cem Keskin. Social diffusion: Long-tern multiple human motion anticipation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9601-9611, 2023b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pp. 5998-6008, 2017.
Edward Vendrow, Satyajit Kumar, Ehsan Adeli, and Hamid Rezatofighi. Somoformer: Multi-person pose forecasting with transformers. arXiv preprint arXiv:2208. /4023, 2022.
12

---

Published as a conference paper at ICLR 2026
Jiashun Wang, Huazhe Xu, Medhini Narasimhan, and Xiaolong Wang. Multi pulti-person 3d motion prediction with multi-range transformers. Advances in Neural Information Processing Systems, 34: 6036 6036-6049, 2021.
Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang. Humanise: Language- conditioned human motion generation in 3d scenes. Advances in Neural Information Processing Systems, 35:14959-14971 2022.
Zhihao Wang, Yulin Zhou, Ningyu Zhang. Xiaosong Yang, Jun Xiao, and Zhao Wang. Existence is chaos: Enhancing 3d human motion prediction with uncertainty consideration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 5841-5849, 2024.
Peng Xiao, Yi Xie, Xuemiao Xu, Weihong Chen, and Huaidong Zhang. Multi-person pose forecasting with individual interaction perceptron and prior learning. In European Conference on Computer Vision, pp. 402-419. Springer, 2025.
Chaoyue Xing, Wei Mao, and Miaomiao Liu. Scene-aware human motion forecasting via mutual distance prediction. In ECCV, 2025.
Chenxin Xu, Robby T Tan, Yuhong Tan, Siheng Chen, Xinchao Wang, and Yanfeng Wang. Auxiliary tasks benefit 3d skeleton-based human motion prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9509-9520, 2023a.
Chenxin Xu, Robby T Tan, Yuhong Tan, Siheng Chen, Yu Guang Wang, Xinchao Wang, and Yanfeng Wang. Eqmotion: Equivariant multi-agent motion prediction with invariant interaction reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
1410014200 2023b.
Liang Xu, Xintao Lv, Yichao Yan, Xin Jin, Shuwen Wu, Congsheng Xu, Yifan Liu, Yizhou Zhou, Fengyun Rao, Xingdong Sheng, et al. Inter-x: Towards versatile numan-human interaction analysis. 2024.
In CVPR, pp.
Qingyao Xu, Weibo Mao, Jingze Gong, Chenxin Xu, Siheng Chen, Weidi Xie, Ya Zhang, and Yanfeng Wang. Joint-relation transformer for multi-person motion prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9816 9826, 2023c.
Juze Zhang, Jingyan Zhang, Zining Song, Zhanhe Shi, Chengfeng Zhao, Ye Shi, Jingyi Yu, Lan Xu, and Jingya Wang. Hoi-m 3: Capture multiple humans and objects interaction within contextual environment. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 516-526, 2024.
Wanying Zhang, Shen Zhao, Fanyang Meng, Songtao Wu, and Mengyuan Liu. Dynamic com- positional graph convolutional network for efficient composite human motion prediction. In Proceedings of the 31st ACM International Conference on Multimedia, pp. 2856-2864, 2023.
Yang Zheng, Yanchao Yang. Kaichun Mo, Jiaman Li, Tao Yu, Yebin Liu, C Karen Liu, and Leonidas J Guibas. Gimo: Gaze-informed human motion prediction in context. In European Conference on Computer Vision, pp. 676-694, Springer, 2022.
APPENDIX
We organize our appendix as follows.
In Section
we provide more implementation details.
In Section
we provide more dataset details.
In Section
we present the runtime analysis.
In Section
present more ablation results.
In Section
provide more discussions on our model design choices.
In Section
we present more visual comparisons on GTA-IM Cao et al. 2020) dataset and
H
Zhang et al. (024) dataset.
13

---

Published as a conference paper at ICLR 2026
In Section
show performance under closer interactions.
In Section
we present extended evaluation with NPSS metric and long-term prediction.
In Section
we present extended evaluation with FID metric.
In Section
we report human-object and human human penetration.
In Section
we discuss the limitations and future works.
In Section
we investigate SE(3) representations.
In Section M we present preliminary validation on scenes with dynamic objects.
A IMPLEMENTATION DETAILS
For each motion sequence, we crop the 3D scene to a region that is within 2.5 meters of the root joint of the last observed pose, and the root joint is used the origin of the cropped scene, following prior works (Ma et al. 2022 Scofano et al. 2023). Then we obtain S E R2 x3 by randomly sampling
de
N 1000 points in the cropped scene. The mapping function ((-) ) is defined as 6(.) : -d-+ + 20 where = 0.2.
Network Architecture. The model consists of 6 Transformer layers with hierarchical dimensions:
early layers (1-3) dimension 512, while late layers use dimension 256 (layers 4-5) or dimension 128 (layer 6). The feed-forward dimension
times the hidden dimension. All Transformer layers self-attention heads and cross-attention heads. The Graph Convolutional Network (GCN) encoder uses 5 residual stages. A learnable position encoding is applied to joint tokens with dimension 3 C 60. Object nodes are augmented with 8-dimensional position encoding: 3D
Jei
center coordinates, exponential encoding of center (e 2o ). distance to origin, and exponential encoding of distance. Object point clouds normalized by subtracting their center.
Loss Function. Loss is computed as = Epath + pose- The path loss path and local pose loss Gicall are defined as
H+T Σ
l
f path
X root
X Y0o0
T
(=#+1
(5)
H+T J-1 Σ
Local
Xheeat
X
T(J-1) - t= 11 j=1
Here, X 323 and & foo 3 are ground-truth and the predicted global path the root joint at time I. ] E R and local,) E R&3 are ground-truth and predicted local pose of the jth non-root
joint at time L.
More Training Details. We build our network on PyTorch 1.12.0 and CUDA 12.4. We follow previous work Mao et al. (2022) use the Adam optimizer with a linear learning rate schedule from 0.0005 to 0. The initial learning rate is 0.0005. Models are trained over 80 epochs. Weight decay set to 1 10-6, and Adam epsilon is 1 10-6 We dropout of 0.1. For single-person datasets, the batch size is 16. For multi-person datasets, dynamic batch sampling is used with maximum sum other persons set 256 (in one batch, the number of other persons for each sample should be the same. Thus we use custom batch sampler such that in each batch sample, the number of other individuals is the same across each sample). Rotation augmentation is applied during training so that the model can learn the sirection-agnostic representation of the inputs.
More Dataset Details. Following prior works Mao et al. (002)); Scofano et al. (023): Xing
et al. (2025): Mueller et al.] (2024), 4. the FPS of four datasets are: GTA-IM: 30 Mao et al. (022); Scofano et al. (2023); Xing et al. (2025). j, HUMANISE: 30 Xing et al. 2025). HOI-M 30, and HIK: Mueller et (2024), We H 30 motion frames predict 7 60 future steps for datasets
HOI-MP (Zhang et al. 2024) and GTA-IM (Cao et al. 2020),
25 and T = 50 for HIK Tanke
al. 2023a), H 15 and 7 30 for HUMANISE (Wang et al. 2022
14

---

Published as a conference paper at ICLR 2026
B DATASET
A. Datasets with Klmann-huma and Human-scene Interaction Scenes. 1) HIK (Tanke al. 2023a is a multi-person interaction datasets in real kitchen environments. We follow the dataset split used in (Tanke al. 2023a Mueller et al., 2024), using the recordings A-C training data and evaluating the recording D. 2) HOI-M Zhang et al. 2024) captures a rich collection of interactions involving multiple humans and objects across 46 diverse scenes in the real world. We randomly allocate 1/5 of these scenes for the test set and utilize the remaining for training. For datasets, we filter out sequences with few social interactions and human movements, and retain those with significant interactions and motion displacement.
B. Datasets with Human-Scene Interaction Scenes. 1) GTA-IM (Cao et al. 2020 is a synthetic human-scena interaction dataset, comprising 3D human motions for 50 distinct characters across diverse scenes. We adopt the same dataset setting as |Mo et al. (2022); Scofano et al. (2023); Xing
ctt al. 2025). 2) HUMANISE (Wang et al. 2022 is a synthetic human-scene interaction dataset.
All methods adopt the dataset setting used in MutualDistance (Xing al. 2025) which ensures motions in the test set are entirely unseen during training. The test set scenes are further divided into seen and unseen scenes, with about 6,000 sub-sequences for testing.
C RUNTIME ANALYSIS
An analysis of inference time and model size is shown in Table Overall, our model takes a reasonable time to perform inference while achieving higher accuracy than baselines. It is worth noting that though SAST is designed to solve the problem under a similar setting as us, it is not practical for real-world applications as it adopts diffusion mechanism, which makes it slow.
Table 4: Runtime analysis on 3OI-M Dataset.
Method
# Param.
Inference Time
ContAware Mao et (2022)
15.9 M
41 ms
STAG Scofano et al. (2023)
16.4 M
38 ms
MutualDistance Xing al.
2025
8.6 M
1 114 ms
IA FA Xiao et al. 22025
9.2 M
69 ms
SAST Mueller et al.
2024
15.4 M
2 s
Ours
9.6 M
43 ms
D MORE ABLATION RESULTS
D.1 VARIANTS OF COARSE-TO- -FIEE INTERACTION
For the coarse-to-fine injection of interaction features at different levels, there are dozens of potential variants. In our main paper, we adopt relatively symmetric injection strategy, where high-level HSI and HHI features are injected into the first three layers, while low-level or mid-level features are injected into the later three layers. Here, we conduct an ablation study on different variants of the injection strategy.
For simplicity, we fix the injection strategy of HHI features i.e., body-level tokens are injected into the first three layers and joint-level tokens into the last three layers. We only apply variations to the hicrarchical HSI tokens F(11, F(2), and (33) Specifically, we evaluate four distinct injection variants depicted in Table 5
As shown in T the four variants exhibit similar prediction performance, indicating that the model is insensitive the specific injection variant chosen. The prediction errors consistently lower than those of the non-coarse-to -ine methods presented in Table This experiment demonstrates the robustness and effectiveness of the coarse-to-fine interaction strategy.
15

---

Published as a conference paper at ICLR 2026
Variant FF(2
Mean Path Error (mm)
Mean Pose
Method
F(1)
Error (mm)
Multi-level
1~6
1~6
1~6
177.1
120.5
1,2,3
4,5
174.6
117.9
1,2,3
5,6
175.2
118.1
Coarse-to-Fine
1,2
3,4
5,6
174.8
117.9
1,2
4,5,6
175.0
118.2
Table 5: Impact of different variants of coarse-tor to-fine injection. We report the metrics on the HOI- M Zhang et al. 2024) dataset. The Variant column specifies which Transformer layers receive HSI features each level. For instance, the last row means that F33 is injected into Transformer layers and 2, F(2) into Transformer layer 3, and F(I) into Transformer layers and 6. This experiment indicates that we are insensitive to different variants of coarse-to- to-fine injection, demonstrating the robustness and effectiveness of the coarse-to-fine injection.
D.2 ABLATION STUDY ON DIFFERENT NUMBER OF SAMPLED POINTS
As shown in Table 6 the performance differences between 1000 and 4000 sampled points negligible. However, reducing the number of points to 250 leads to degradation both path and pose accuracy. To balance computational efficiency with performance, select 1000 points the default configuration in our experiments.
Path Error (mm)
Pose Error (mm)
#points
0.5s
1.0s
1.58
2.0s
mean
0.5s
1.Os
1.5s
2.0s
mean
250
29.9
57.2
94.2
148.7
65.2
27.8
42.6
53.9
64.1
40.5
1000(default)
29.4
55.9
91.7
139.2
62.9
27.0
40.3
50.7
61.5
38.7
4000
29.2
55.9
91.8
139.7
63.1
27.0
40.1
50.3
60.8
38.4
Table 6: Ablation study on different number of sampled point of the static scene GTA-IM dataset.
D.3 ABLATION STUDY ON DIFFERENT NUMBER OF TRANSFORMER LAYERS
We ablate on the number of Transformer layers of the Coarse-to- to--Fne Interaction Reasoning Module. For feature injection, adopt symmetric strategy: high-level HSI (scene) and HHI (human huuma))human) features are injected into the first half of the layers, while low- and mid-level features are injected into the latter half. Table Table|Jreveat reveals that while deeper architectures (more layers) generally achieve better accuracy, the rate of improvement decreases as we add more layers. To balance computational efficiency with performance, we adopt layers as our default configuration.
Path Error (mm)
Pose Error (mm)
# #rranfformer layers
0.5s
1.0s
1.5s
2.0s
mean
0.5s
1.0s
1.5s
2.0s
mean
4
30.3
58.1
93.3
140.4
64.0
28.2
42.2
52.8
63.7
40.2
6(default)
29.4
55.9
91.7
139.2
62.9
27.0
40.3
50.7
61.5
38.7
8
28.9
55.4
90.4
135.2
61.9
26.3
39.5
49.9
61.0
38.3
Table 7: Ablation study on different number of Transformer layers GTA-IM dataset.
D.4 ROBUSTNESS TO INCOMPLETE INTERACTION INFORMATION
To evaluate the robustness of our model against incomplete information, we conduct experiments where parts of the scene and some interacting individuals randomly occluded. For the scene,
16

---

Published as a conference paper at ICLR 2026
simulate occlusion by randomly removing points within 6 cones originating from the target person. For other individuals, we randomly remove 0-2 persons from the scene. As shown in Table & while method's performance sees slight degradation expected, it still outperforms other methods, demonstrating its robustness.
Table 8: Evaluation with incomplete interaction information the HOI-M3 dataset. We report mean Path Error (mm) and Pose Error (mm). Lower better.
Method
Path Error
Pose Error
MutualDistance
190.7
125.3
IAFormer
186.4
121.6
SAST
185.9
122.7
Ours
176.6
118.1
Ours (w/o occlusion)
174.6
117.9
We further analyze the impact of varying scene occlusion levels on model performance. By increasing the number of occlusion cones from 2 to 16, simulate progressively severe scene incompleteness. As shown in Figure 6 while prediction error naturally increases with occlusion severity, our method maintains consistent performance advantage over baselines.
(a) Path Error vs. Occlusion
(b) Pose Error vs. Occlusion
Wormmr
126
190.0
Path Erter (mm)
Pore Error (mmm)
187.5
322
182.5
180.0
120
1.
12
14
2.0
10
12
14
10
Number of Cones
Number Of Conee
Figure 6: Performance under varying levels of scene occlusion. The x-axis represents the number of occlusion cones (indicating occlusion severity), and the y-axis represents the Mean Error.
D.5 ROBUSTNESS TO NOISY INPUTS
We conducted experiments using noisy inputs (injecting Gaussian noise into the joints of other individuals and the scene point cloud). As illustrated in Figure 7 while performance naturally degrades for all methods noise levels increase, HUMOF consistently outperforms the baselines. This demonstrates that our approach remains effective and robust even with imperfect inputs.
FURTHER DISCUSSIONS ON MODEL DESIGN AND CAPABILITIES
E RATIONALE FOR NOT PREDICTING A SPECIFIC INTERACTIVE TARGET
Our decision not build in an inductive bias for predicting an explicit interaction partner deliberate design choice, motivated by the complex and fluid nature of real-world human interactions. In the real world, human motion is rarely governed by a singular environment element. Instead, often results from blend of multiple surrounding entities. For instance, person might navigate around a table while simultaneously turning their head to speak to a friend. The final motion is synthesis of these coneurrent spatial and social cues. Therefore, forcing the model to explicitly predict a specific interaction target would be an ill-posed simplification, failing to capture the rich, blended nature of these influences.
17

---

Published as a conference paper at ICLR 2026
(a) Path Error vs. Noise
(b) Pose Error vs. Noise
Ours
126
SA57
Woroen
Poth Eror (mrm)
1:0
Prose Error [mmm)
1.55
120
175
0,000
6,025 0.060 6,055 0.100 0.125 0.150 Noise Stid (m)
0.200
0.000 0.025
2.075
Noise Std (m)
Figure 7: Robustness analysis under noisy inputs. The plot shows performance degradation with increasing Gaussian noise.
Instead, we build interaction features for all nearby individuals and scene points and allow a cross- attention mechanism to dynamically weigh the influence of each. This approach is more flexible and better reflects the complex nature of social dynamics.
E.1.1 INTERPRETABILIT OF CROSS-ATTENTION WEIGHTS
Our decision not to explicitly predict specific interaction target does not mean that our model is black box. The cross- attentionno mechanism in our framework offers a way to interpret the model's s focus. To demonstrate this, provide quantitative analysis of the attention weights. On the Humanise dataset, which provides ground-trun labels for the primary object a person interacts with, find that the HSI tokens with the highest attention scores often correspond to that ground-truth object. Specifically, we identify the top-3 HSI tokens with the highest average attention scores for each sample and check if any of them correspond the ground-trute truth interaction object. We formalize this as:
Σn I (Tip-33 corresponds to OGT) 83.85%
Accuracy
N
3
Baseline (Random Chance) Accuracy
nam ob) 10.56%
As shown in Table our attention based identification achieves an accuracy of 83.85% significantly outperforming a random-chance baseline. This result quantitatively validates that our model learns to focus relevant objects and is not simply overfitting, providing interpretable insights into its decision-making process.
Table 9: Interpretability of the cross- roos-atteenionn mechanism. We report Top-3 accuracy for identifying the ground-trute interactive object on the Humanise dataset. Higher is better.
Method
Top-3 Accuracy
Random Chance
10.56%
Via Attention Map of Our Model
83.85 %
E.2 SCENE REPRESENIATION: HOLISTIC POINT CLOUD VS. OBJECT-LEVEL MODELING
In our framework, we model the scene as a holistic point cloud rather than segmenting it into individual objects. This design choice is motivated by two primary factors: efficiency and practical applicability. Modeling every object individually, especially in complex scenes with numerous objects, would introduce significant computational overhead. More importantly, it would create dependency on accurate and readily available instance segmentation, which is often not the case in real-world scenarios that rely raw sensor data. Our approach avoids this dependency. The strong
18

---

Published as a conference paper at ICLR 2026
performance of our method across multiple datasets validates the effectiveness of this modeling strategy.
F MORE VISUAL COMPARISONS
We provide additional visualization results on the GTA-IM Cao et al. (2020) dataset in Figure X and the HOI-M dataset in Figure 9 Our method demonstrates superior accuracy in predicting human motion, including both global trajectories and local poses.
(a) Ground Truth
(b) Ours
(c) SAST
(d) MufualDistance
(c) STAG
Figure 8: Visual comparisons on the GTA-IM Cao et al. (202 dataset. Our method produces the best predictions. For instance, the 1st row, SAST Mueller al. (2024) predicts poses that intersect with the space beneath the stairs, likely due to its lack of explicit modeling of human-scent interactions. While MutualDistance Xing al. (2025) and STAG [cofaao al. (2023) avoid this issue, they also STAGSScoanno
produce inaccurate predictions. Our method generates predictions closest to the ground truth.
G PERFORMANCE UNDER CLOSER INTERACTIONS
To evaluate the model's s capability in handling intense interactions, we conducted an evaluation on specific subset of the " dataset focusing close interactions, where the target person is in close proximity (< 15cm) to scene objects or other individuals.
As shown in Table 10 the DCT-based method does not limit interaction quality. The results on the subset are even slightly better compared to the average result on the full test set. This is likely because when there are closer scene objects other individuals, they provide stronger geometric constraints on the target motion. Thus, becomes easier for the model to give better prediction compared to samples where there very close scene objects other persons.
19

---

Published as a conference paper at ICLR 2026
Target
Targel Permin
Ground Truth
Ours
(d) MutualDistance
IAFormer
Figure 9: More visual comparisons on HOI-M dataset.
Table 10: Comparison of performance between the full test set and the close-interaction subset on the HOI-M dataset.
Test Set
Mean Path Error (mm) 4
Mean Pose Error (mm)
Ours (Full Test Set) Ours (Close Interaction Subset)
174.6
117.9 1 116.8
172.5
We also provide visual examples of these close human-human (InterX dataset Xu et al. (2024)) and human-scene (HUMANISE dataset) interactions in the Supplementary Video (1:23 to 1:59) to further demonstrate the high interaction quality.
H EXTENDED EVALUATION: NPSS METRIC AND LONG-TERM PREDICTION
In the main paper, we follow prior works Mao al. (0022: Scofano et al. (2023); Xing et (2025)
and report Path Error and Pose Error metrics. However, noted in Gopala Kopplakrishnan al. (2019),
for prediction horizons longer than one second, the inherent stochasticity of human motion can make traditional geometric error metrics less informative. To provide more comprehensive evaluation, additionally add the normalized power spectrum similarity (NPSS) metric, which offers a statistical evaluation of motion quality.
The NPSS calculated as the Euclidean distance between the power spectra of the prediction and the ground truth. The formula is:
NPSS 1 D
Σ Σ (Pak- Pa,n))
where D the number of dimensions (joints coordinates), is the number of DCT coeffi- cients, P is the power spectrum of the ground-truth sequence, and f corresponds to the predicted sequence Gopalakrishnan et al. (2019).
While our primary focus on short-term prediction (up to 2 seconds), we also evaluated our method on a challenging long-term prediction task (10 seconds) on the 3OI-M² dataset to provide reference. As shown in Table 12 the performance horizon increases, which is expected due to the compounding uncertainty in long-term forecasting. Nevertheless, our method consistently outperforms the baselines across all metrics, demonstrating its robustness for longer-term predictions.
all methods degrades significantly as the prediction
20

---

Published as a conference paper at ICLR 2026
Table 11: Quantitative results with the NPSS metric for 2-second prediction on the HIK and HOI-M datasets. This table extends Table 1 from the main paper by including the NPSS metric.
Dataset
Method
Mean Path Error (mm) 1
Mean Pose Error (mm)
NPSS
MutualDistance Xing al.
2025
246.0
105.9
0.00778
IAFormer Xiao
2025
200.1
95.0
0.00718
HIK
SAST Mueller 2024
189.0
93.2
0.00711
Ours
180.7
90.2
0.00703
MutualDistance Xing et al.
2025
189.9
125.3
0.0195
H
IAFormer Xiao et al. SAST Mueller et al.
2025
186.3
121.6
0.0172
2024
184.8
122.3
0.0179
Ours
174.6
117.9
0.0169
Table 12: Long-term (10s) prediction performance on the HOI-M dataset.
Method
Mean Path Error (mm)
Mean Pose Error (mm) 4
NPSS
MusualDistance Xing al. 2025
824.5
191.9
0.308
IAFormer Xiao al. 2025
768.3
203.
0.246
SAST Mueller et al. 2024
814.7
191.3
0.320
Ours
747.8
188.7
0.227
MORE EVALUATION METRIC: FID
To further evaluate the quality of the generated motions distribution level, we calculate the Fréchet Inception Distance (FID). We utilized the pre-trained motion encoder from T2M Guo et al. (2022a) the feature extractor to map motion sequences into a 512-dimensional feature space and then calculate FID on the HOI-M dataset. The results are summarized in Table [3 Our method achieves the lowest FID score, demonstrating that our generated motions match the ground-truth distribution better than the baselines.
Table 13: FID scores on H dataset.
Method
FID ())
Mueller SAST-Muller
SAST
al. 2024
0.0278
Mulual Xing al.
2025
0,0233
IAFormer Xiao et al.
2025
0.0170
Ours
0.0164
J HUMAN-OBJECT AND HUMAN-HUMAN PENETRATION
To measure human-object and Homan-humalt penetration, we calculated the mean penetration rate and penetration depth the H dataset, which provides the high-quality scene meshes and human body models (SMPL-X) necessary for this analysis. The metrics are defined as follows:
Penetration depth at the f-th frame (in meters) is defined as the sum of absolute signed distance field (SDF) values for all joints of the target person that penetrate the scene or other persons:
Σ (x 1=1
where (-) denotes the signed distance field (SDF) of the scene or interactive persons, (-) clips positive distances zero, and X the 3D position of the j-th joint at time
Penetration rate frame is the ratio of joints with negative SDF value to the total number of joints:
Number of joints with (X))
21

---

Published as a conference paper at ICLR 2026
We then take the average over the frame and sample dimensions to obtain the final mean metrics.
Table 14: Penetration Results on the HOI-M dataset.
Human-to-Scene
Human-to-Human
Method
Mean Pen. Rate Mean Pen.
Depth (mm)
Mean Pen. Rate Mean Pen. Depth (mm)
Ground-Truth Truth (GT)
1.84%
11.26
0.070%
0.61
SAST
1,54%
11.07
0.063%
0.45
TAFormer
1.62%
10.89
0.068%
0.59
MutualDistance
1.45 %
10.68
0.057%
0,41
Ours
1.49%
10.77
0.052%
0.39
As shown in Table [4] MutualDistancc achieves the best HSI penetration scores, likely due to its of mesh-basee modeling for both the target person and the scene, providing more explicit surface information to avoid penetration. However, we note that penetration metrics alone, without considering motion accuracy, should be interpreted with caution. Our case analysis reveals that higher penetration rates can sometimes result from accurately predicting dynamic motion, whereas lower penetration may occur when method predicts static incorrect motion (e.g., the person remaining stationary). This is also supported by the fact that the Ground-Trute Truth (GT) motion itself registers the highest penetration, due small misalignments between the motion capture data and scanned scene geometry in the dataset, whereas the four prediction methods may inadvertently avoid penetration by under-pun under-prediee predicting movement (e.g. remaining static). Our method strikes strong balance, achieving high motion accuracy while maintaining penetration scores comparable to or better than most baselines.
To further reduce penetration while preserving accuracy, we could also adopt mesh- based modeling e.g., after computing noint-to-point/jon distance, we then adjust these by subtracting the point-to- mesh surface distance to get mesh-to-mesh distances. In our current method we did not adopt this design as this would introduce dependencies on scene meshes and SMPL-X parameters, increasing complexity and reducing practical applicability.
K LIMITATIONS AND FUTURE WORKS
Failure cases Our method occasionally struggles with accurately predicting abrupt motion changes, as illustrated in Figure TO. the upper example Figure T0] our method fails to predict the bending-over action, it is difficult to infer from the past motion. In the lower example, our method incorrectly predicts that the target person will continue standing by the desk, while the person unexpectedly starts walking away. Note that all methods fail in these challenging cases. These issues could potentially be mitigated by incorporating additional modalities, such as human gaze Zheng et al. ) to provide richer contextual information. We show the failure cases in the HIK dataset in the supplementary video.
Internetive
Historical
Targrt
(8) Ground Truth
(b) Ours
(e)
(d) MutualDistance
(e) IAFormer
Figure 10: Some failure cases. In the upper example, all methods fail to predict the bending-over action with abrupt motion changes. In the lower example, all methods incorrectly predict that the target person will continue standing by the desk, while the person unexpectedly starts walking away.
The Monotonous Interaction Modeling Paradigm In our work mainly adopt monotonous modeling paradigm i.c., the distance- based modeling, where we concatenate distance- features with intrinsic scene and human features (such self-encoding) serve interaction features.
22

---

Published as a conference paper at ICLR 2026
Euler distance is natural way to model interaction, different types of interactions follow distinct distance patterns, even in complex motions like dance. For example, in waltz or tango, the distance between dancers' bodies exhibits repetitive patterns of upproaching and retreating, Distance-based modeling captures these patterns, enabling action recognition and future motion prediction. While this modeling paradigm is effective and efficient, exploring alternative paradigms and integrating diverse approaches could lead to more robust and generalized performance.
Improvement of Joint Multi-person Predictions We have demonstrated our scalability in joint multi-person forecasting in the paper. However, in our current approach, the model is not explicitly aware of future interactions between individuals, as the input to the HHI module consists only the historical motion sequence y(k) In the future, we could extend the method an iterative approach where, in each iteration, the HHI module can take the historical motion sequence y/(k) concatenated with the predicted motion from the previous iteration. This way, when inferring each individual, the model is aware of the predicted future motion of other individuals from the prior iteration via HHI. We leave this extension for future exploration.
L INVESTIGATION ON SE(3) REPRESENTATIONS
To investigate whether explicit geometric transformations could enhance our model, we implemented the SE(3) relative encoding similar to Miyato et al. (2024) and compared it with our original distance- based encoding. Specifically, we constructed a local coordinate system for each individual in every frame. We defined the root joint (pelvis) as the origin, the vector from the pelvis to the neck as the vertical axis, and the vector between the left and right hips determine the lateral axis. The forward direction was then derived via the cross product to complete the orthonormal basis. Based on this local frame, we calculated the relative SE(3) transformation between the target and interactive persons. We encoded the rotation using 6D continuous representation and concatenated it with the original relative XYZ translation coordinates form the interaction feature. As shown in Table [5] the comparison reveals that the SE(3) encoding achieves performance par with our original distedce-based method. We hypothesize that the lack of improvement suggests that our model already capable of implicitly learning the necessary geometric relationships and orientations from the temporal patterns of interactive distances. The sequence of distances over time contains rich information about relative motion and heading, which our hierarchical interaction reasoning module effectively captures.
Table 15: Comparison between SE(3) encoding and our distance- based encoding the HOI-M dataset.
Variant
Mean Path Error (mm) +
Mean Pose Error (mm)
Ours
174.6
117.9
Ours SF(3)
174.6
118.1
M PRELIMINARY VALIDATION ON SUBSET OF DYNAMIC SCENES
As stated in Line 435, current mainstream datasets lack significant and diverse dynamic scene elements, which limits large-scale benchmarking. However, to validate our architectural claim (that our HSI module can natively handle time-dependent scene coordinates), we conducted an additional experiment on a specific subset of the HOI dataset that contains dynamic objects (e.g., passing or moving an object). Since these samples are rare, we upsampled them during training.
Table 16: Quantitative results on a subset of dynamic objects from the HOI dataset.
Method
Mean Path Error (mm) 1
Mean Pose Error (mm)
MutualDistance
285.0
187.9
IAFormer
288.3
176.0
SAST
281.2
189.1
Ours
244.8
159.4
23

---

Published as a conference paper at ICLR 2026
As shown in Table 16 static scenes (mainly due to the scarcity of training samples) as expected, our method consistently outperforms baselines in this challenging setting. We demonstrate end of the supplementary video. This confirms our potential to handle dynamic scene elements and suggests that performance would likely improve significantly given dataset with significant and diverse dynamic scene elements,
while the quantitative errors significantly higher compared to completely
successful example the
USE OF LARGE LANGUAGE MODELS
During the preparation of this manuscript, large language models (LLMs) were used exclusively as writing-assistance tool. particular, LLMs assisted in:
Language Polishing:
Improving grammar, sentence structure, and readability while pre- serving the technical accuracy of the content.
Terminology and Style Consistency: Ensuring consistent usage of technical terms and
notation across sections.
LLMs were not involved in the conception of the research problem, the design of the HUMOF framework, or the analysis of results. All scientific contributions are solely the original work of the authors. The LLM was employed only to improve the clarity and presentation of the manuscript's text.
24


Reviews:
[{"review_id": "Rm1gVLU3n5", "reviewer": "Reviewer_7peC", "summary": "This paper proposes HUMOF, a method for human motion forecasting in complex dynamic scenes that involve both human–human and human–scene interactions.\nThe approach introduces a hierarchical interaction representation that separately encodes high-level contextual and low-level geometric information, and a coarse-to-fine interaction reasoning module that injects these features into Transformer layers from semantic to geometric levels.\nThe framework combines Discrete Cosine Transform, Graph Convolutional Networks, and Transformer architectures.\nExperiments on four public datasets (HIK, HOI-M3, GTA-IM, and HUMANISE) show that HUMOF achieves state-of-the-art performance.", "questions": "Please see the weakness section", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tComprehensive interaction modeling.\n\n- Unlike prior works that focus on either human–scene or human–human interactions, this paper successfully integrates both within a unified framework. This joint modeling is meaningful and leads to noticeable performance improvements on dynamic scenes.\n\n2.\tStrong empirical performance.\n\n- The method achieves consistent improvements across multiple datasets and evaluation metrics. The results demonstrate that the hierarchical representation and coarse-to-fine reasoning work effectively together.\n\n3.\tPractical contribution as a baseline.\n\n- If the authors release the code as promised in the abstract, HUMOF can become a valuable benchmark for future research on interactive human motion prediction.", "weaknesses": "1.\tLimited novelty in components.\n\nMost elements of the architecture come from existing works (coarse-to-fine approach, distance-based interaction modeling, abstraction, ...). The contribution lies primarily in how these components are integrated, rather than in introducing a fundamentally new modeling paradigm.\n\n2.\tLack of quantitative evaluation for multi-person inference.\n\nThe paper briefly demonstrates qualitative results for joint multi-person inference in Figure 4 but does not provide quantitative metrics or runtime analysis. Since multi-person prediction is highly relevant for real-world applications such as social robotics or crowd simulation, the absence of measurable evaluation limits the practical significance of this claim.\n\n3.\tHigh computational complexity for multi-agent prediction.\n\nThe proposed framework models each target person independently and computes pairwise interactions with every surrounding person. \nThis design implies that when there are K individuals, the method requires roughly K² human–human interaction computations. \nFurthermore, because the scene abstraction (HSI) is recomputed for each target, the total cost scales linearly with K, resulting in O(K³) level complexity when both factors are considered. \nThis raises concerns about the scalability and feasibility of real-time multi-agent forecasting.\n\n4.\tIssue in references.\n\nThe paper repeatedly cites prior works using informal author patterns such as “Jeong & etc., 2024” or “Xing & etc., 2025.” This format is inappropriate for a scientific paper and must be corrected to proper citation styles.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZlGCvQY6H4", "reviewer": "Reviewer_ztD4", "summary": "HUMOF presents a human motion forecasting method in social environments that takes into account both human-human as well as human-scene interactions. A DCT rescaling allows for controllability of the coarseness of the signal processing. The authors compare on two social scene datasets and on two human-scene interaction datasets and consistently outperform SOTA.", "questions": "Did the authors measure human-object and human-human penetration? This feels like a natural form of evaluating this task.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The problem of social human motion forecasting is highly relevant but relevantly under-explored. HUMOF is a complex method with many moving parts. However, the authors ablate the model relatively well. Utilizing DCT is a commonly used technique in motion forecasting. Using the rescaling for coarse-to-fine prediction in the context of human motion fc is novel and clever. The method description is mostly clear experiments have been conducted on two social scene datasets and two human-scene interaction datasets, demonstrating the methods effectiveness, while being parameter and inference time efficient.", "weaknesses": "The method is complex (as the task itself is complex) but I have some concerns about some of the model specifics:\n(1) The utilized relative encoding is overly simplistic: I wonder if instead of just utilizing the point distance the method could utilize the geometric transformations in SE(3), for example as has been utilized in [1]. \n(2) While DCT works well for-single person motion, I wonder if it is limiting the human-object and human-human interaction quality. In the transformer, the tokens do not directly correspond to frames anymore, so temporal alignment over longer time frames might be hindered. The authors should evaluate this by showing either closer human-human (or human-object) interaction.\nIf the current datasets do not contain sufficient close person-to-person data, the authors should utilize other dyadic datasets, i.e Inter-X [2].\n\nMy second concern is with regards to the evaluation: the authors only evaluate directly comparing to GT (path error, pose error) - however, due to the complexity of the scene, multiple “answers” could be correct - I wonder if the authors have considered utilizing methods to compare the generated sequences on a distribution level, i.e. by utilizing FID, i.e. the authors could use the combined input-output sequence ($X^{1:H} \\oplus \\hat{X}^{H+1:H+T}$) to compare the distributions. \n\n[1] GTA: Geometric Transform Attention (ICLR 2024)\n\n[2] Inter-X: Towards Versatile Human-Human Interaction Analysis (CVPR 2024)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qcf1BsK2Qw", "reviewer": "Reviewer_sVzv", "summary": "The paper introduces HUMOF, a novel framework for human motion forecasting in complex social environments. HUMOF effectively integrates human kinematics, dynamics, spatial–temporal context, and interaction cues into a unified predictive model.\n\nA key contribution is the Hierarchical Interaction Representation, which jointly captures human–human and human–scene interactions using both explicit distance features and learned semantic–geometric representations. To exploit this representation, the authors design a Coarse-to-Fine Interaction Reasoning Module, which improves motion prediction through two mechanisms:\n\nSpatial hierarchy: High-level semantic features are introduced in early Transformer layers, while fine-grained geometric cues are progressively refined in later layers.\n\nFrequency control: A DCT-based frequency modulation strategy prioritizes high-frequency motion components early in training and focuses on low-frequency refinements in later stages.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is logically organized and easy to follow, with a coherent narrative from motivation to methodology and results.\n\n2. The evaluation covers both multi-person interactive and single-person forecasting scenarios, including tests on unseen environments. The proposed method consistently outperforms baselines across all benchmarks.\n\n3. The visualizations and supplementary videos are convincing — showing smooth, stable, and realistic motions with minimal jitter and drift compared to prior works.", "weaknesses": "No major weakness", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MxN0Mxe39Z", "reviewer": "Reviewer_9FNW", "summary": "The paper proposes a novel approach to address the problem of human motion forecasting in interactive environments.  It builds a hierarchical interaction representation, modeling interactions at multiple levels by combining explicit cues (e.g. inter-object distances) with learned features. Specifically, HUMOF captures high-level context and low-level details for both social interactions and scene contacts. A key innovation is the coarse-to-fine reasoning module, a multi-layer Transformer that progressively incorporates interaction features: early layers use high-level features, while later layers integrate fine-grained features. An adaptive DCT‐based rescaling further suppresses high-frequency components in early layers, encouraging a coarse-to-fine refinement of the motion prediction.\n\nHUMOF obtained state-of-the-art results on two datasets with human-human and human-scene interactions (HIK. HOI-M3) as well as two datasets with human-scene interactions (GTA-IM, HUMANISE).", "questions": "1. The paper uses a function $\\phi (\\cdot)$ to map distances to higher values for closer points. How sensitive is the performance to the choice of this mapping? Could a learned function improve results?\n2. HUMOF 3D uses poses for all humans and a 3D point cloud of the scene. In real-world settings, these could be noisy or incomplete. Would the approach still work with noisy or incomplete inputs?\n3. The runtime is around 43ms per inference for HOI-M3. Does this runtime change on other datasets? Have the authors tested HUMOF in an online setting? Is the latency competitive for real-time applications?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written and explains deeply the architecture and how it works.\n2. The paper models both human-human and human-scene interactions in a single framework, addressing a realistic scenario of interactive environments.\n3. The use of multi-level representations (body-level vs joint-level for social cues, and multi-scale point clouds for scene context) is a strong idea. It balances global context and local detail effectively.\n4. The injection of high-level features in early Transformer layers and finer details later (along with the DCT rescaling mechanism) is novel and well motivated by the ablation studies.\n5. The authors compare their approach to a large set of baselines and include visualizations to support their claims.", "weaknesses": "1. HUMOF’s architecture is quite elaborate (multiple modules, Transformer layers, DCT processing). This complexity might make it hard to reproduce or tune and the paper should give more details in the appendices.\n2. The authors note that existing datasets have few moving scene objects. Thus, HUMOF’s performance on truly dynamic environments (e.g. moving furniture or vehicles) is untested.\n3. The baselines weren’t supposed to work with all the tested datasets, hence they had to be adapted. This raises a question of fairness in comparisons: it’s possible some methods could be improved with similar context. However, the large gaps suggest HUMOF’s advantage is likely genuine.\n4. The runtime analysis in the Appendix was performed on only one dataset (HOI-M3).\n5. The supplementary video lacks failure cases. Although some failure scenarios are described in the Appendix, it would be helpful to include them in the video as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)