Gaze Beyond the Frame: Forecasting Egocentric 3D Visual Span
Authors: Heeseung Yun
Joonil Na
Jaeyeon Kim
Calvin Murdock
Gunhee Kim
Venue: NeurIPS 2025 spotlight ()
Keywords: Egocentric vision
Gaze anticipation
2D to 3D lifting
Primary area: applications
TL;DR: We address the challenge of predicting where a person's visual perception will focus next within their three-dimensional environment.

Abstract: People continuously perceive and interact with their surroundings based on underlying intentions that drive their exploration and behaviors.
While research in egocentric user and scene understanding has focused primarily on motion and contact-based interaction, forecasting human visual perception itself remains less explored despite its fundamental role in guiding human actions and its implications for AR/VR and assistive technologies.
We address the challenge of egocentric 3D visual span forecasting, predicting where a person's visual perception will focus next within their three-dimensional environment.
To this end, we propose EgoSpanLift, a novel method that transforms egocentric visual span forecasting from 2D image planes to 3D scenes.
EgoSpanLift converts SLAM-derived keypoints into gaze-compatible geometry and extracts volumetric visual span regions.
We further combine EgoSpanLift with 3D U-Net and unidirectional transformers, enabling spatio-temporal fusion to efficiently predict future visual span in the 3D grid.
In addition, we curate a comprehensive benchmark from raw egocentric multisensory data, creating a testbed with 364.6K samples for 3D visual span forecasting.
Our approach outperforms competitive baselines for egocentric gaze anticipation and 3D localization, while achieving comparable results even when projected back onto 2D image planes without additional 2D-specific training.

Gaze Beyond the Frame: Forecasting Egocentric 3D Visual Span
Heeseung Yun', Joonil Na', Jaeyeon Kim?, Calvin Murdock Gunhee Kim
Seoul National University, CCannegie Mellon University, RReality Labs Research at Meta https: / /hs- hhsssnn..... www.giithu.b.... watthub..o/aazeeeyoooooooo hto//azeBBeyondFramee
TIll watk the kitchen and mroth nome snack.
EgoSpanLift
3D Visual Span Forecasting
Visaal
Prediction
Future (1 (ineeen)
Previous Visual Span
Predicted Visual Span
Figure 1: Can we forecast our gaze beyond the frame? We aim to predict person's 's future visual focus in 3D surrounding environment by lifting egocentric 2D gaze history to 3D regions and forecasting future 3D visual spans from previous observations.
Abstract
People continuously perceive and interact with their surroundings based on un- derlying intentions that drive their exploration and behaviors. While research in egocentric user and scene understanding has focused primarily on motion and contact-based interaction, forecasting human visual perception itself remains less explored despite its fundamental role in guiding human actions and its implications for AR/VR and assistive technologies. We address the challenge of egocentric 3D visual span forecasting, predicting where a person's 's visual perception will focus next within their dineesionnii environment. To this end, we propose EgoSpan- Lift, novel method that transforms egocentric visual span forecasting from 2D image planes to 3D scenes. EgoSpanLift converts SLA.M-deriver keypoints into cuze-compatible geometry and extracts volumetric visual span regions. We further combine EgoSpanl with 3D U-Net and unidirectional transformers, enabling spatio-temporal fusion to efficiently predict future visual span in the 3D grid. In addition, we curate a comprehensive henchmark from raw egocentric multisensory data, creating testbed with 364.6K samples for 3D visual span forecasting, Our approach outperforms competitive baselines for egocentric 2D gaze anticipation and 3D localization while achieving comparable results even when projected back onto 2D image planes without additional 2D-specific training.
Introduction
People continuously perceive and interact with their surrounding environments in everyday life. Underlying these interactions are their intentions, which drive them actively explore their surround-
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

ings and engage in various behaviors. Understanding how individuals will perceive contexts and take actions in advance becomes crucial element in comprehending their overall behavior patterns. The ability to accurately anticipate behavior not only reduces latency in real-time egocentric applications but also serves as a foundation for proactively delivering information and services in our daily lives, e.g., immersive VR/AR, ambient computing, and assisting individuals with impairments.
To better understand person's intentions, research in egocentric user and scene understanding has been widely explored, primarily involving action and contact-based interaction. For instance, when given egocentric visual or multimodal context, studies have focused on predicting subsequent human actions 2 3 or localizing regions in 2D images where interaction will occur 4 Recent studies have actively investigated human pose or motion forecasting 3D space [5] 6 7 as well as contact location prediction during object interactions 8 9. However, attempts to forecast human visual perception itself remain less explored. Studies in vision science and cognitive psychology suggest that perceptual exploration significantly influences human motion [10] 11 12 [3], and intuitively, most of our daily interactions follow perception, i.e., we look before we leap. Therefore, forecasting visual perception is essential for proactively understanding and anticipating human behavior.
In this work, we address the novel challenge of egocentric 3D visual span forecasting- where a person's visual perception will be focused in the surrounding environment. We draw inspiration from the vision science literature regarding text reading
14 15] and object/scene per- ception [6] 17 18), where the visual span often refers to the fixated region of human vision from peripheral awareness to precise foveal gaze fixations. In our context, we adopt this term as 3D Visual Span to refer to egocentric visual focus in 3D surroundings for addressing daily and casual behaviors and interactions. While previous research has shown impressive results in predicting egocentric future gaze fixations on 2D image frames [19] [2] forecasting gaze for dynamic scenarios in 2D remains unclear. Gaze anticipation requires jointly modeling the user's self-motion and attention-both of which are naturally directed toward specific locations in 3D space rather than arbitrary regions in 2D projections. That is, modeling visual span as 3D regions offers a more accurate and consistent repre- sentation of perceptual focus even beyond our current observations, unlocking promising egocentric applications that call for robust and proactive content rendering (11 22).
Our main contribution comprises three key components: (i) method for bridging egocentric visual spans and 3D scenes, (ii) framework for forecasting future spans, and (iii) a newly curated benchmark from raw egocentric multisensory data. First, we propose EgoSpanl.itt. a novel method that lifts visual spans in 2D image frames into structured 3D volumetric regions, as illustrated in Fig. [ Unlike existing 3D egocentric localization approaches method uniquely transforms SLAM-derived keypoints into gaze-compatible geometry to precisely extract volumetric regions corresponding to 3D visual span. EgoSpanLift encodes multi-level span information grounded in vision science [r]n-nggg from wide soaad--rinnttion-baae
motion trajectories object interactions, our
regions to fine-grained foveal fixations- ekations-enabling semantically rich understanding of where and how people look in space. Building on this, we introduce an autoregressive forecasting framework that combines EgoSpanLift with 3D U-Net for spatial representation using unidirectional transformer for temporal modeling, effectively capturing the evolving dynamics of egocentric visual attention.
To establish a testbed at scale, we rigorously curate raw egocentric multisensory data streams 23 24 for benchmarking 3D visual span forecasting under daily and skilled activity scenarios, namely FoVS-Aria and FoVS-EgoExo, covering total of 364.6K samples. Our framework outperforms several competitive baselines, including frameworks for 3D egocentric localization and 2D gaze prediction models equipped with our EgoSpanLift. We conduct extensive analysis across varying spatio-temporal windows and activity categories. Notably, when our 3D visual span predictions are back- projected onto 2D image planes, they achieve accuracy on par with 2D-specific models even without 2D- -peciffi training, demonstrating the versatility and robustness of modeling gaze in 3D.
2 Related Work
Egocentric Anticipation of User Behavior. Predicting user behavior from egocentric observations represents one of the core challenges in egocentric user and scene understanding. The EPIC-Kitchenl dataset [1] introduced the challenge of predicting future action classes within procedural kitchen activities. Various approaches have been proposed to address this problem, including dual LSTTM architectures for past and future modeling (2), and contrastive learning with RNNs for future visual
2

---

feature learning [3. Beyond semantic action classification, studies have also proposed methods for
predicting locomotion trajectories from current user observations 4 25 26]. For more detailed user posture prediction beyond trajectories [27 recent work has utilized reinforcement learning- based recurrent control 5 MI.P-based residual modeling [6], and video-pose bimodal transformers [7].
Egocentric Gaze Prediction Understanding and predicting human gaze behavior is crucial for modeling various aspects of perception and interaction. Many studies primarily aim to estimate gaze direction from facial/head images [28 |2 30), predict the object of visual attention [3] 32 33
34 35
focus on VR/mobile scenarios 36
37
38
40), and model the interplay with language
Estimating and predicting gaze from egocentric perspectives pose additional challenges due to complex egocentric scenes and combination of dynamic gaze shifts with frequent head and body movements, i.e., self self-motion. Egocentric gaze estimation often involves bottom-u up saliency [3 joint gaze-action prediction 44, and global-local correlation
On the other hand, the frontier of gaze anticipation remains relatively unexplored due to insufficient input context for predicting futuure frames. Zhang et al. [19] employ GAN synthesize future frames and forecast gaze locations, while Lai et al. [20] also introduce a multimodal anticipation model that uses contrastive spatial/temporal separable fusion capture audio-visual correlations and improve future gaze prediction. However, predicting gaze in 2D images is often ill-posed in dynamic scenarios, it requires jointly modeling the user's self-motion and attention, which are inherently directed toward specific locations in 3D rather than arbitrary regions in 2D frames. In this work, we address these issues by directly predicting the user's gaze and visual focus in 3D scenes through spatio-temporal fusion of volumetric representations.
Egocentric 3D Interaction. Research on egocentric interaction has evolved from 2D 3D under-
standing and from contact-based to intention- analysis. Earlier work on 2D image interaction
analysis primarily addresses hand detection, segmentation, and pose estimation 46 47 48 49 50).
Moving beyond 2D approaches, 3D-based frameworks leverage hand poses and object interactions
involving physical contacts from RGB(-D) inputs [5] 52 53
55 56 57 allowing for precise
prediction of hand trajectories and action targets [58 59 60 Recent line of work interprets potential
3D interaction regions as spatial affordances, learned from either synthesized geometry and motion [9] or intentional cues in 2D interaction images 88.
Concurrent research with ours, FICTION 61 focuses on 4D) human-object interaction. FICTION jointly predicts the 3D bounding box of objects involved in physical interactions along with the user's spatial location and body poses where contact occurs at future time steps. A key distinction of our work is that it address the forecasting problem for 3D regions where the user's visual perception focused and precedes these everyday actions and interactions. Our predictions take the form of multi-level 3D volumetric regions of the visual span, enabling more fine-grained forecasting of potential interaction hotspots than bounding boxes.
3 EgoSpanLift: Lifting Egocentric Gaze Prediction from 2D to 3D
3.1 Preliminary: Simultaneous Localization and Mapping
Simultaneous Localization and Mapping (SLAM) refers to the problem of jointly estimating an agent's ' position and reconstructing the surrounding 3D environment from sequential observa- tions. It is often formulated probabilistic framework that maximizes the posterior distribution P(x;, m C1.,t-1, 01.,-1), where X, denotes the pose at time t, m is the 3D mapping information, and CI.,-11, o are the control and observation history [62]. SLAM algorithms have been devel- oped across sensing modalities, including LiDAR and RGB-D, with visual SLAM offering solutions based on feature matching (63] direct photometric tracking 664 65 and visual-inertial fusion (66]. With growing efficiency and robustness, these methods output semidense 3D keypoints and time- aligned camera poses, providing structurally faithful and computationally efficient information about 3D scenes that users perceive and interact with from an egocentric perspective.
3.2 Keypoint Selection and Classification
Observation-based Keypoint Selection. An overview of EgoSpanl is illustrated in Fig. 2 Our method initiates with a set of 3D semidense keypoinns/ P and localization information E, typically obtained through visual SLAM [67]. Specifically, each pi E P consists of (p., o,, t.), where P; E R
3

---

Semidense Keypoints
Multi-level 3D Visual Span
Osientation
0 30
2 (Cental (Foveal Span)
8
(Porpheal Spea)
(b) Classified Keypoints
(c) Volumetric Regions
(a) Observed Keypoints
Filtering
Perimberal
Fovell
Spatiall bound
Temporal window
Statistical outlier removal
Figure 2: Overview of EgoSpanL.ft. Using 3D semidense keypoints from egocentric observations, e.g., SLAM, filter observed keypoints given time window and leverage multi-level human visual span to compute volumetric regions in 3D scenes.
the 3D keypoint in global coordinates, d; represents the confidence of estimation the form the variance of inverse distance, and f; denotes the time at which the point was observed. Additionally, E, E represents an SE(3) element that transforms from the local egocentric frame (i.e., central
R t, pupil frame) at time to the global coordinate system, i.e., E, 07 1
where the 2-axis in the
local coordinates denotes forward. From this, we obtain the set of keypoints observed at time t:
P. = (p& E E = I, |p. t.ll <
Pi) =},
where the first term serves and the second term acts as a spatial filter that retains only points within a cubic boundary of length D around the user. The last term, I, (p;; P). an indicator function returning the result of neighbor-basee statistical outlier filtering f. where indicates validity and 0 indicates invalidity.
filter that preserves visually observed keypoints at the given time,
We establish the rationale for each filtering term as follows. For temporal filtering, we consider only a limited context within a temporal window spanning few seconds before the current time rather than all keypoints observed from beginning to end. This enables operation without requiring an offline algorithm and prevents the inclusion of points that were visible from different viewpoints but are currently occluded (e.g, items inside the fridge). For spatial filtering, prevent gaze from erroneously overshooting to distant locations and to focus regions within the user's vicinity that will be effectively perceived and interacted with. In most experiments, we used D = 3.2m, though we discuss experiments extending beyond 6m in Sec. Scc.5.2 For outlier removal, we use neighhor-ban based statistical filtering instead of confidence- based filtering, ie., d; which typically used when extracting static 3D scenes from SLAM outputs. This approach can retain important dynamic elements in egocentric scenes, such as moving people, the user's hands, and other dynamic objects in the space, while effectively removing invalid points.
restrict the area to
Gaze-based Keypoint Classification. Using keypoints from Pr, we classify points that fall within a visual span defined around the user's gaze direction after transforming each point to the local coordinates as E, Pp. To determine whether points are included in the visual span, we utilize a 3D gaze representation defined have employed egocentric 3D representations the form of directions or cones, e.g., intersection between gaze vectors and triangulated meshes of static objects (68] overlap of multiple users' gaze cones [69]. our approach features key distinction as it covers more general use cases by addressing which local regions of 3D scene capture an individual user's visual attention during daily activities without discriminating between dynamic and static components.
E.'
cone extending from the user. While some previous works
When the aforementioned gaze cone and keypoints the local coordinates are projected onto the 2 = 1 plane, they are represented as green ellipses and black dots in the lower left of Fig. (b). We select the dots that fall within these cllipses using the angular distance threshold 0 (i.e., eccentricity):
E,
'P'P:.
g: gd
Ps P
cos 8
(2)
E,

---

where < <.. denotes inner product, and gi denotes the gaze direction in local frame at time t. Through this, we can classify a set of points that correspond to the egocentric 3D visual span.
While employing SLAM for visual spans may pose challenges in modeling regions between semidense keypoints, our approach remains effective for two main reasons. First, human visual span or peripheral vision is known to be significantly influenced by contrast as well as eccentricity from the gaze direction [70] 711. Since people overwhelmingly interact with visually salient objects rather than empty white walls, this approach is highly compatible with keypoints obtained through SLAM. Indeed, >99% of the 3D visual spans in our curated data samples from raw egocentric multisensory observations overlapped with SLAM keypoints. Furthermore, rather than relying on keypoints as inputs or outputs, we represent visual spans as volumetric regions derived from them, e.g., e.g.EEq.3) which provides good coverage of the given 3D scene and helps mitigate the aforementioned challenges.
3.3 Multi-level Volumetric Region Localization
Using classified keypoints Qf in a cube of length D, we obtain regions corresponding to visual spans by computing their occupancy in a 3D Cartesian grid with resolution R and duration [to, te]:
& [t..,.] (1 k) [p E U g. 0 S (pr L,, + D
S 1} > 0), (3)
where I is an indicator function and ||| represents the cardinality of a set. This approach indirectly covers the regions not captured by semidense keypoints while maintaining constant space complexity regardless of the increasing number of points with longer durations. Also, since V/ is represented as a binary occupancy grid, computing the similarity between overlapped visual spans can be trivial.
Inspired by taxonomy defined in vision science literature 15]. we categorize 3D volumetric regions of vision spans into four levels, as illustrated in Fig. 2 First, foveal localization v corresponds to the conventional 2D gaze localization area, covering a highly localized region with an eccentricity of 0, 2". While foveal span exists in most regions, i.e., around 80% in our curated dataset, there may be instances where it is absent due to our use of semidense keypoints. To address this limitation, wider eccentricities: the central span V with 0_ 8 and the near peripheral span &d with 8p- = 30 Finally. regions observed in spans defined beyond these are significantly influenced by head orientation much as the gaze, with a complex interplay between the two (72 773. Therefore, we employ the most broadly defined span as the region within the field-of-view centered on head orientation, i.e., 7 which is analogous to the view frustum capturing visual input in 2D setups (e.g., O. = 55 in our experiments). Note that while continuous distributions such as Gaussian could used, we analyze through the lens of this well-established taxonomy for intuitive evaluation and level-by-level integration across multiple timesteps.
we utilize spans corresponding
As result, for given temporal window, we obtain multi-level volumetric regions that represent varying degrees of user visual attention focus in the surrounding environment. Since EgoSpanLiit does not rely on time-consun consuming algorithms, when combined with existing platforms capable of real-time SLAM and gaze processing 74 [7 76),, it enables the acquisition of 3D information about human visual attention with trivial latency. Additionally, its representation as points or volumetric regions in 3D Cartesian grid facilitates adaptation to existing 3D frameworks.
4 Forecasting Network
One of the most representative problems when considering a user's visual span in 3D involves predicting future visual attention based on past contextual focus patterns. To this end, we introduce a straightforward framework to forecast 3D visual spans for future Ty frames, given the history of observations from past Tp frames, illustrated in Fig, (a). Note that our primary interest lies not in precisely matching what was seen in each respective frame but rather in understanding the trends and coverage areas of the user's attention over a given period. To perform precise evaluation while minimizing the effects of exploration order gaze, we use the union of visual spans over Ty frames as our prediction target.
Autoregressive Encoder. Given localization information Eprev and observed semidense keypoints Peee for the past 7, seconds, we obtain multi-level volumetric regions representing previous visual spans through EgoSpanl as described in Sec. 3 Specifically, we utilize grid of size Tр x (4 + 1) x R. R. x R input, where 4 represents visual spans ranging from orientation to foveal span,
5

---

Fnceder
3D
Fi
Trunh
Transfomer
Unidirectionll
LoIcE
EgoSpanLift
0%.5
Forecast
PTp
3D
Decoder
(a) Our proposed framework
(b) Installing
(c) Cooking
(d) Repairing
Figure 3: (a) Illustration of our framework and (b-d) diverse scenario examples in our curated dataset.
and represents the complete scene in the given space regardless of span inclusion. To extract spatial features from this input grid, we initially compress them using the encoder a 3D U-Net [77]. By reducing the spatial dimension by factor of R through pooling, we obtain encoded features of size Tp x C, where C is the feature dimension. Note that performing spatial reduction by factor smaller than R showed negligible impact performance.
While using Tp embeddings encodes how visual span has evolved over time, we separately incorporate global embedding to serve prediction head. Thus, utilize embedding that encodes all visual spans within the duration as a global embedding, serving as our prediction head. By appending this embedding after the other embeddings, obtain features of size (Tp + 1) C, i.e., U1. Upp Unen- To learn temporal dependencies among these features and increase the model's expressiveness, we employ transformer with a unidirectional attention mask [8 77),, ensuring that information about temporal dynamics is integrated toward the final global embedding.
Decoder. Using the output embedding Uhead that corresponds to thend from the transformer, we perform upsampling via U-Net decoder. During this process, we utilize intermediary features from the U-Net encoder through residual connections. By applying sigmoid, we ultimately obtain an output Yuf& of size 4 x R x R x R representing 0-1 soft occupancy, which corresponds to aggregated forecast of the user's 's visual span over the future Tr frames.
Learning Objective. Since visual spans occupy only very narrow region of the space surrounding the user, learning meaningful signals through conventional cross-entropy losses could be challenging. For instance, in the case of foveal span, almost all samples occupy less than 1 % of the entire grid region. Therefore, we train our model using the dice loss with the ground truth forecast Yijk- where @ is elementwise product:
2 Eiy @ Yijk
L C=1-
(4)
Yiis + EYss 115x++11 +1
Latency Analysis. Our framework has two primary sources of latency: (i) extracting the relevant set of points from gaze and SLAM keypoints (performed every 100ms), and (ii) performing model inference every second on set of points spanning two seconds. Since the point extraction can be pre-computed and stored at 10 fps for continuous use, we only need to consider the computation time for processing the final observation when calculating inference latency.
Table 1: Latency analysis of our framework.
Operation
Latency
Point preprocessing
4.541=1.999mm
3D visual span localization
1.844mm
Voxelization
45.406+26.223ms
Model inference
19.483#8.255ms
Average latency
1l.241
We measured this in resource-constrainge environ- ment compared our training setup, using CPU cores and GPU with 12GB VRAM.
The results summarized in Table I The first stage, which handles outlier removal, axis-aligned bounding box cropping for keypoints, and selection of points within certain degree of eccentricity from the gaze, can be processed within 10ms. the second stage, we identified that the primary bottleneck lies in voxelization rather than in the model itself. This occurs because the large number of keypoints from the previous stage should voxelized, whereas the model operates efficiently once receives the 3D voxelized representation. Consequently, the average inference latency is 71.241 ms, yielding a real-time factor of 0.036, confirming our claim on fast processing capability. However, actual AR/VR environments typically operate with even fewer computational resources, and thus

---

Table 2: Comparison of forecasting accuracy the FoVS-Aria test split. Higher the better.
Orientation
Peripheralaor
Centralg
Fovealze
Methods
IoU
Fl
IoU
F1
IoU
F1
IoU
F1
2D Center Prior EgoSSpan jitt
80
0,4061
0.5583
0.1621
0.2497
0.0685
0.1049
GLC [45
EgaSpanLift
80
0,4553
0,6064
0.2227
0.3267
0.1321
0,1873
CSTS
20
EgoSpanLift
80
0,4567
0.6076
0.2342
0,3423
0,1388
0,1948
OccForrmr 81
0.1395
0.2352
0.1106
0.1846
0.0429
0.0632
0.0158
0.0289
VoxFormer
82
0.2192
0.3093
0.1847
0,2580
0.0915
0.1279
0.0453
0.0704
IAG 8
0,3250
0.3669
0.2154
0.2542
0,1250
0,1851
0.0747
0.1050
EgoChoir
0,4959
0.6579
0,4302
0,5581
0,2612
0,3608
0.1987
0.2311
Global Prior
0.1359
0.2314
0.1048
0.1825
0.0331
0.0618
0.0146
0.0280
Ours (w/o previous span)
0.3424
0.4906
0.2616
0.3928
0.1071
0.1739
0.0594
0.0828
Ours (CBCE)
0,5730
0.7134
0.4594
0.6017
0,2836
0.3879
0.2059
0.2609
Ours (Gaze-only+ (80)
0,4723
0,6214
0.2666
0.3791
0.2494
0.3193
Ours (Single-task)
0,5832
0.7238
0,4721
0.6154
0.3351
0.4485
0.2494
0.3193
Ours (w/o global cmbedding)
0,5602
0.7042
0.4647
0.6128
0.3241
0.4476
0.2624
0.3487
Ours (full)
0.5838
0.7247
0.4886
0.6350
0.3513
0,4762
0.2836
0,3709
additional optimization techniques such as model quantization more efficient voxelization could be considered for further performance improvements.
5 Benchmarking Egocentric 3D Visual Span Forecasting
5.1 Forecasting Egocentric Daily Activities
Curation of FoVS-Aria. Due to the lack of an existing testbed for egocentric 3D visual span forecasting, we curate dataset by processing the raw data streams from existing egocentric multisensory dataset. Aria Everyday Activities dataset [23] encompasses diverse scenarios of people engaging in daily activities across different environments and interacting with others, comprising 143 recordings with a total duration of 7.3 hours. Initially, we inspect the SLAM keypoints and gaze scenarios of all recordings, manually filtering out cases with insufficient keypoints those limited to stationary viewing of phones/TVs. Following recent research in 2D gaze anticipation [20). we define sample as a 2-second prediction task based 2-second previous observation with sliding window of I second. For spatial parameters, we use resolution A = 16 and cube length D = 3 3.22m, indicating that accurately matching a cell corresponds to precision within a D/R 20cm error margin. We construct the test split using all recordings from location 4 to enable the evaluation of unseen locations and the validation split by randomly stratifying the rest. Consequently, our constructed Fo Fo 3D Visual Span)-Aria consists of 23 3.22 samples in total, with 19.3k, 1.9k, and 1k samples for train, validation, and test splits, respectively.
For multi-level 3D visual span, we evaluate each level separately. We primarily utilize 3D IoU as primary metric since our main concern is the overlap of volumetric regions. Additionally, we report F1 scores, commonly used in 2D gaze evaluation [45] (20), while precision and recall are reported the Appendix. Finally, since the foveal span is defined within highly narrow region, overlap-based metrics cannot fully capture its distribution. Therefore, we also examine the distribution of the (metric) distance between the ground truth region and the predicted region. Analysis on saiiency-based metrics is deferred to Appendix.
Evaluation Protocol.
Regarding baselines, due to lack of well-established frameworks for task configuration, construct applicable models from various domains. First, we adapt 3D localization methods like OccFormer [8]] and VoxFormer ( to predict visual span, which are known for effectively inferring semantic labels for voxels in outdoor driving scenarios. We also employ IAG [ and EgoChoir (2), models designed to predict affordances or interaction hotspots in 3D from egocentric user-object interaction perspective. Another framework includes state-of-the-art methods for 2D gaze anticipation, such as GLC (45) and CSTS [20]. Thanks to our EgoSpanLift, we can project these models' 2D inference results to 3D, extrapolating head pose information from past context using a multi-task Gaussian process model [80].
7

---

Comparison with Prior Arts. Table 2 presents compar- ative results among various methods. Overall, there is substantial performance gap between existing methods and our approach, with the disparity becoming more pro- nounced when predicting more localized regions in 3D (e.g., foveal span), where our method outperforms oth- ers by more than 50%. EgoChoir [9] displays the most promising performance among baselines, due to its capa- bility of predicting 3D interaction hotspots that could be in line with user intentions. Compared to the other 3D localization methods, frameworks that consider the gaze dynamics show improved performance, despite falling significantly short on foveal span forecasting. This performance gap is further evidenced in Table where baseline frameworks produce errors approximately twice as high as our approach on average.
Table 3: Distribution of metric errors in 3D foveal span localization.
Distance (cm)
min.
avg.
max.
GLC 45
59.65
73.47
87.20
CSTS [20]
59.71
73.79
87.68
w/o prev. span
66.94
83.07
98.98
Single-task
36.54
52.90
69.18
Ours
19.04
34.85
51.23
display Ablation Studies. The last six rows in Table display the results of ablation studies our methodol- ogy. The global prior-predicting answers based on the forecast distribution of the train split-yields considerably low numbers, implying that our FoVS-Aria encompasses diverse visual spans driven by various user intentions. Additionally, our model's performance significantly deteriorates when attempting prediction without previous span information; this indicates that knowledge of where the user previously focused is crucial for accurate forecasting, as visual span can vary substantially according to intention. Using binary cross-entropy (BCE) performance degradation up to the central span level, but shows marked decline in foveal span performance. Similarly, jointly solving multi-level spans consistently outperforms the single-task approach, particularly benefiting foveal span improvement.
loss function results in marginal
Multi-level interpretation carries significant implications due to uncertainties in self-motion fore- casting and geometric correspondence between 2D and 3D spaces, as exemplified in suboptimal performance of predicting only gaze in 2D or 3D and applying postprocessing [80]. Two key con- ceptual differences between our framework and the postprocessing variant are (i) learning from multi-level representation and (ii) mitigating uncertainties in self-motion anticipation through end-to- end prediction. Leveraging the interconnection between gaze and periphery [15] allows us to capture cues about future gaze from previous periphery and forecast future periphery in light of previous gaze, which can be observed in several qualitative examples. Moreover, given the nature of the viewing frustum, extending 3D gaze predictions to broader spans necessitates the forecasting of ego- centric 6DoF pose trajectories with a separate postprocessing stage, which propagates uncertainties in self-motion forecasting and geometric correspondence between 2D and 3D spaces. In contrast, our end-to-end framework does not assume a specific forecast trajectory and predicts plausible 3D spans within the scene while implicitly learning to mitigate such uncertainties.
5.2 Forecasting Skilled Activites at Scale
Curation of FoVS-EgoExo. To analyze the effectiveness of visual span forecasting in skilled activities at a larger scale, we utilize Ego-Exo4D (24] as our source data, comprising hundreds of hours of observations. Ego-Exo4D consists of eight activity categories, from which we excluded soccer, basketball, and dance, as these activities involve overly large or dynamic scenes where clear fixations in visual spans are unidentifiable in the majority of cases. Instead, we focus on five categories: cooking, music, health, repair, and bouldering. After collecting only the footage corresponding to actual task execution, we refine the data and calculate volumetric regions following the same procedure used for FoVa-Aria. Considering the increased scale and the focus on specific tasks, we establish a 4-second prediction the default setting. Thus, we collect a total of 341.4k samples, divided into 274.7k, 29.6k, and 37.0k samples for train, validation, and test splits, respectively.
Performance Analysis. Table 4 coopprre baseline model performance on PoVS-EgoEBe While our approach shows substantial performance gains over ---ra, presumably due to the nature of skilled activities and an order of magnitude larger sample size, the corresponding improvements in baseline methods are relatively modest. Unlike V- the change of numbers regarding the orientation span likely stems from the inherent characteristics of skilled activities in FoVS- -ogoox,, where participants tend to maintain more sustained focus on specific tasks rather than engaging in frequent, diverse head movements. Given that visual span forecasting prioritizes accurate prediction
8

---

Table 4: Comparison of forecasting accuracy the FoVS-EgoExo test split. Higher the better.
Orientation
Peripheralgg
Centralge
Fovealge
Methods
loU
F1
IoU
F1
IoU
F1
IoU
F1
CSTS 20 EgoSpanl.ift
0.4978
0.6398
0.2867
0.4010
0.1556
0.2107
OccHormmr
0,1287
0.2280
0.0920
0.1685
0.0251
0.0490
0.0084
0.0167
VoxFormer 82
0,1896
0,3188
0.1475
0,2571
0.0620
0.1168
0.0179
0.0350
EgoChoir 9
0.3287
0.4948
0,2851
0,4437
0.1976
0,3300
0.1266
0.2247
Global Prior
0.2329
0.3621
0.2478
0.3776
0.1245
0.2119
0.0658
0.1188
Ours (w/o prev. span)
0,4091
0.5665
0.3876
0.5296
0.2855
0.4107
0.2255
0.3152
Ours (CCCE)
0,5112
0.6621
0.49015
0.6338
0.3722
0.4867
0.2870
0.3578
Ours (w/o glohal emhedding)
0.4998
0.6542
0.4892
0.6381
0.3954
0.5294
0.3475
0.4500
Ours (full)
0.5230
0.6743
0.5108
0.6569
0.4212
0.5541
0.3692
0.4702
D = 3.2, R = 32 (1 0cm)
0,4443
0.6040
0,4362
0,5893
0.3319
0,4656
0.2493
0.3497
D 6.4, R = 32 (20em)
0.4920
0.6462
0.4902
0.6394
0,4121
0.5464
0.3640
0,4689
D 3.2, R 16 (20em)
0.5230
0.6743
0.5108
0.6569
0,4212
0.5541
0.3692
0.4702
Cooking
Repair
Music
Health
Bouldering
Orientation
Periphoral
Central
Foveal
C
n
M
B
C
H
B
0.51 45000 4590449
0,3650.300
0.330
0.50
0.65
0.522 0.522 0.1760.4610.00
0.391 0.3960.313..300 0.368
0.45
0.60
& 0.4F10. 2.77 0.4.0.1..00011133030 0.1130.3870.007
0:202 0.261 0.291
0.237
0:226
8.596
2.530 0 0.0.0..0..0044 02.440.4425
n:424 0.410.......... 0.36
0.00
420 0.330.330
0.40
0.55
8.55 (.4790.533 0.3.0.5.0..553....
n.486
0.510.55
0.35
0.50
B 0.366 2370 211 0.222
0.081 0.045 0.039 0.500.0
0.30
0.45
1s
25
35
1s
26
35
46
(a) Cross- category transfer of peripheral/fovell forecast (loU)
(b)
Influence of temporal window length
Figure 4: Analysis of our proposed framework on the FoVS-EgoExo test split.
of narrower regions, it is particularly noteworthy that substantial performance gaps exist between our method and baselines for Central and Foveal areas. These results indicate that skilled activity forecasting in FoVSS Exoooo not inherently casier than daily activity forecasting in FoVS-Ari while demonstrating that our model achieves robust performance across multiple scales and domains compared to prior arts.
We further analyze the impact of scale, category-specific training, and spatio-temporal granularity. According experiments conducted with varying spatial configurations in Table the impact performance marginal if the grid range is proportionally increased to maintain the same precision of 20cm despite an increased resolution of R 32. However, for our model trained 10cm precision with It = 32, performance decreases to levels similar to those observed in F-Araa Similarly, as demonstrated in Fig- (b), performance tends to decline as we attempt cover visual span further into the future, which is particularly significant when predicting foveal span.
Finally, cross-category transfer results are summarized in Fig. (a). Despite significant variations in patterns across task categories, training an integrated model at sufficient scale achieves virtually identical performance maintaining separate models for each category. Transfer among static tasks (e.g., cooking, music, and health) is relatively effective, but less so for more dynamic tasks such as repair or bouldering. Additionally, peripheral span, which covers wider areas visual exploration, preserves performance better in cross-category scenarios compared to foveal span, which targets more localized regions. Qualitative examples can be found in Fig. (b) and the Appendix.
5.3 Extension 2D Gaze Anticipation
For completeness in our experimental analysis. we explore whether the reverse direction -from 3D 2D-is also possible using our framework, given that we successfully extended egocentric visual span from 2D to 3D. To this end, we utilize samples from FoVS-Aria to compare 2D gaze anticipation performance. While our previous experiments Table separate procedures like EgoSpanl ift to bring 2D gaze models into 3D, deflating our model's inference results to 2D can be accomplished trivially. We select the cell with maximum logit value from our model's foveal span prediction and project it onto the 2D image plane using the user's current head pose. Through

---

Past
Future
Previous Span
Our Forecast
Ground Truth
(a) 3D Visual Span Forecasting on FoVa-Aria
(b) 3D Visual Span Forecasting on FoVS-EgoExo
Previous Span
3D Forccast
2D Forccast (T
(c) Egocentric 2D Gaze Anticipation
Figure 5: Qualitative examples. Our framework effectively forecasts various scenarios, such as (a) closing fridge and turning around or (b) deciding which rocks to grab and navigate.
interpolation between the user's current and projected gaze, we can easily obtain gaze anticipation results for the future two seconds.
The performance comparison results be found in Ta- ble Interestingly, while 2D methodologies struggled to accurately predict foveal span when transferred 3D, approach achieves on-par performance to models trained specifically for 2D despite not conducting any 21D-specific training. The baseline performance we obtained is slightly lower than that the referenced paper (20), which we at- tribute to our exclusion of trivial recordings from FoVS- Aria like watching TV or smartphones.
Table 5: Comparison of egocentric 2D gaze anticipation.
Pr
Re
GLC 45
0.505
0.453
0.571
CSTS 20
0.515
0.497
0.535
Ours (Single)
0.505
0.432
0.608
Ours
0.515
0.440
0.619
6 Conclusion
We addressed the challenge of egocentric 3D visual span forecasting by designing a method that lifts multi-level visual span from 2D to 3D semidense keypoints and introducing an end-to-end framework for forecasting volumetrie regions corresponding each visual span. Furthermore. curated testbed for this problem by processing two datasets with raw multisensory observations, where we consistently outperformed wide range of prior arts across all metries. We anticipate that this framework will play crucial role in proactively understanding human intent captured through perception that precedes interaction, enabling preemptive delivery of various latency-sensitive services. Future extensions of this work could incorporate forecasting problems for non-visual perception, such as auditory or proprioceptive inputs, or focus on forecasting highly precise attention tracking on dense scenes, which would represent particularly intriguing directions for further research.
10

---

Acknowledgment. This work was supported by Samsung Advanced Institute of Technology. Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Ko- rea government (MSIT) (No. RS-2021 1343, No. RS-2022-11220156 No. 14-2025--55422330 and National Research Foundation of Korea funded by the Ministry of Sonccaatin(RS22222 00274280). This research was conducted as part of the Sovereign Al Foundation Model Project( Track), organized by the Ministry of Science and ICT(MSIT) and supported by the National Informa- tion Society Agency(NIA), Korore. Gunhee Kim is the corresponding author.
References
[1] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, al. Scaling egocentric vision: The epic-kitchens datasct. In ECCV, 2018.
[2] Antonino Furnari and Giovanni Maria Farinella. What would you expect? anticipating egocentric actions with unnolling Istms and modality attention. ICCV, 2019.
Yu Wu, Linchao Zhu, Xiaohan Wang, Yi Yang, and Fei Wu. Learning to anticipate egocentric actions by imagination. IEEE TIP, 2020.
Kristen Grauman, Andrew Westhury, Eugene Byme, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR, 2022.
[5] Ye Yuan and Kris Kitani. Ego-pose estimation and forecasting as real-time pd control. In ICCV, 2019.
[6] Wen Guo, Yuming Du, Shen, Vincent Lepetit, Xavier Alameda-Pinee and Francese Moreno-Noguer. Back to mip: A simple baseline for human motion prediction. In WACV, 2023.
[7] Maria Escobar, Juanita Puentes, Cristhian Forigua, Jordi Pont-Tusee Arbelacz. Egocast: Forecasting egocentric human pose the wild. In WACV, 2025.
Maninis, and Pablo
[8] Yuhang Yang, Wei Zhai, Hongehen Luo, Yang Cao, Jiebo Luo, and Zheng- Jun Zha. Grounding 3d object affordance from 2d interactions in images. In /CCV, 2023.
[9] Yuhang Yang, Wei Zhai, Chengfeng Wang, Chengjun Yu, Yang Cao, and Zheng-Jun Zha. Egochoir: Capturing 3d human-object interaction regions from egocentric views. In NeurIPS, 2024.
[10] Maurizio Corbetta and Gordon Shulman. Control of goal-directed and stimulus-driven driven attention in the brain. Nature Reviews Neuroscience, 2002.
[11] Samuel Tuhkanen, Jami Pekkanen, Paavo Rinkkala, Callum Mole, Richard M Wilkic, and Otto Lappi. Humans use predictive gaze strategies to target waypoints for steering. Nature Scientific Reports, 2019.
[12] Soukayna Bekkali, George J Youssef, Peter Donaldson, Jason He, Michael Do, Christian Hyde, Pamela Barhoun, and Peter G Enticott. Do gaze behaviours during action observation predict interpersonal motor resonance? Social Cognitive and Affective Neuroscience, 2022.
[13] Charlie Burlingham, Naveen Sendhilnathan, Oleg Komogortsev, Scott Murdison, and Michael Proulx. Motor "laziness" constrains fixation selection in real-world tasks. PNAS, 2024.
[14] Keith Rayner. The perceptual span and peripheral cues in reading. Cognitive Psychology, 1975.
[15] Hans Strasburger, Ingo Rentschler, and Martin Jüttner. Peripheral vision and pattern recognition: A review. Journal of Vision, 2011.
[16] Pieter Blignaut. Visual span and other parameters for the generation of heatmaps. ETRA, 2010.
[17] Antje Nuthmann. On the visual span during object search in real-world scenes. Visual Cognition, 2013.
[18] Freek Van Ede, Sammi R Chekroud, and Anna C Nobre. Human gaze tracks attentional focusing in memorized visual space. Nature human behaviour, 2019.
[19] Mengmi Zhang, Keng Teck Ma, Joo Hwee Lim, Qi Zhao, and Jiashi Feng. Deep future gaze: Gaze anticipation egocentric videos using adversarial networks. In CVPR, 2017.
[20] Bolin Lai, Fiona Ryan, Wengi Jia, Miao Liu, and James M Rehg. Listen to look into the future: Audio-visual egocentric gaze anticipation. In ECCV, 2024.

---

[21]
Anjul Patney, Marco Salvi, Joohwan Kim, Anton Kaplanyan, Chris Wyman, Nir Benty, David Luebke, and Aaron Lefohn. Towards foveated rendering for gaze-tracked virtual reality. ACM TOG, 2016.
[22] Josef Spjut, Ben Boudaoud, Jonghyun Kim, Trey Greer, Rachel Albert, Michael Stengel, Kaan Akşit, and David Luebke. Toward standardized classification of foveated displays. IEEF TVCG, 2020.
[23] Zhaoyang Lv, Nicholas Charron, Pierre Moulon, Alexander Gamino, Cheng Peng, Chris Sweency, Edward Miller, Huixuan Tang, Jeff Meissner, Jing Dong, et al. Aria everyday activities dataset. arXiv:2402.33349 2024.
[24] Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitami, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In CVPR, 2024.
[25] Hyun Soo Park, Jyh-Jing Hwang. Yedong Niu, and Jianbo Shi. Egocentric future localization. In CVPR, 2016.
[26] Heeseung Yun, Ruohan Gao, Ishwarya Ananthabhotla, Anurag Kumar, Jacob Donley, Chao Li, Gunhee Kim, Vamsi Krishna Ithapu, and Calvin Murdock. Spherical world-locking for audio-visual localization in egocentric vidcos. In ECCV, 2024.
[27] Hao Jiang and Vamsi Krishna Ithapu. Egocentric pose estimation from human vision span. In ICCV, 2021.
[28] Xucong Zhang, Yusuke Sugano, Mario Fritz, and Andreas Bulling. Appearance-based gaze estimation in the wild. In CVPR, 2015.
[29] Kyle Krafka, Aditya Khosla, Petr Kellnhofer, Harini Kannan, Suchendra Bhandarkar, Wojeiech Matusik, and Antonio Toralla.. Eye tracking for everyone. In CVPR, 2016.
[30] Yihua Cheng, Haofei Wang, Yiwei Bao, and Feng Lu, Appearance-based gaze estimation with deep learning: A review and benchmark. JEEE TPAMI, 2024.
[31] Adria Recasens, Aditya Khosla, Carl Vondrick, and Antonio Torralba. Where are they looking? In NIPS, 2015,
[32] Adria Recasens, Carl Vondrick, Aditya Khosla, and Antonio Torralba. Following gaze in video. In ICCV, 2017.
[33] Danyang Tu, Xiongkuo Min, Huiyu Duan, Guodong Guo, Guangtao Zhai, and Wei Shen. End-to-end numan-gaze-targel detection with transformers. In CVPR, 2022.
[34] Benoît Massé, Silèye Ba, and Radu Horaud. Tracking gaze and visual facus of attention of people involved in social interaction. IEFE TPAMI, 2017.
[35] Yuchen Zhou, Linkai Liu, and Chao Gou. Learning from observer gaze: Zero-shot attention prediction oriented by buban-object interaction recognition. CVPR, 2024.
[36] Yanyu Xu, Yanbing Dong. Junru Wu, Zhengzhong Sun. Zhiru Shi, Jingyi Yu, and Shenghua Cao. Gaze prediction dynamic 360 immersive vidcos. CVPR, 2018.
[37] Julian Steil, Philipp Müller, Yusuke Sugano, and Andreas Bulling. Forecasting user attention during everyday mobile interactions using -itteeeinteeratte and wearable sensors. MobileHCI. 2018.
[38] Zhiming Hu, Sheng Li, Congyi Zhang, Kangrui Yi, Guoping Wang, and Dinesh Manocha. Dgaze: Can-based gaze prediction dynamic scenes. IEEE TVCG, 2020.
[39] Zhiming Hu, Andreas Bulling, Sheng Li, and Guoping Wang. Fixationnet: Forecasting eye fixations in task-oriented virtual environments. IEEE TVCG, 2021.
[40] Tim Rolff, H Matthias Harms, Frank Steinicke, and Simone Frintrop. Gazetransformer: Gaze forecasting for virtual reality using transformer networks. In DAGM German Conference on Pattern Recognition, 2022.
[41] Sigrid Klerke, Yoav Goldberg, and Anders Søgaard. Improving sentence compression by learning to predict gaze. In NAACL, 2016.
[42] Angela Lopez- Cardona, Carlos Segura, Alexandros Karatzoglou, Sergi Abadal, and loannis Arapakis. Seeing eye to ai: Human alignment via gaze-based response rewards for large language models. In ICLR, 2025.
12

---

[43] Yifei Huang, Minjie Cai. Zhenqiang Li, and Yoichi Sato. Predicting gaze in egocentric video by learning task-dependent attention transition. In ECCV, 2018.
[44] Yin Li, Miao Liu, and James M Rehg. In the eye of the beholder: Gaze and actions in first person video. IEEE TPAMI, 2021.
[45] Bolin Lai, Miao Liu,, Fiona Ryan, and James M Rehg. In the eye of transformer: Global-local correlation for egocentric gaze estimation. In BMVC, 2022.
[46] Sven Bambach, Stefan Lee, David Crandall. and Chen Yu. Lending a hand: Detecting hands and recognizing activities complex egocentric interactions. In ICCV, 2015.
[47] Guillermo Garcia Hernando, Shanxin Yuan, Seungryul Back, and Tae- Kyun Kim. First- pers-- hand action benchmark with rgb-d videos and 3d hand pose annotations. In CVPR, 2018.
[48] Fanqing Lin, Brian Price, and Tony Martinez. Ego2hands: and detection. arXiv:2011. 002552 2020.
A dataset for egocentric two-hand segmentation
[49] Minjie Cai, Feng Lu, and Yoichi Sato. Generalizing hand segmentation in egocentric videos with uncertainty-guided model adaptation. CVPR, 2020.
[50] Lingzhi Zhang, Shenghao Zhou, Simon Stent, and Jianbo Shi. Fine-grained egocentric hand-object segmentation: Dataset, model, and applications. In ECCV, 2022.
[51] Bugra Tekin, Federica Bogo, and Mare Pollefeys. H+o: Unified egocentric recognition of hand hand-object poses and interactions. CVPR, 2019.
[52] Tacin Kwon, Bugra Tekin, Jan Stuhmer, Federica Bogo, and Mare Pollefeys. H2o: Two hands manipulating objects for first person interaction recognition. arXiv:2104. 1/181, 2021.
[53] Takehiko Ohkawa, Kun He, Fadime Sener, Tomáš Hodañ, Luan Tran, and Cem Keskin. Assemblyhands: Towards egocentric activity understanding via 3d hand pose estimation. In CVPR, 2023.
[54] Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael Black, Ivan Laptev, and Cordelia Schmid, Learning joint reconstruction of hands and manipulated objects. In CVPR, 2019.
[55] Yu- Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar lqbal, Stan Birchfield, et al. Dexycb: A benchmark for capturing hand grasping of objects. In CVPR, 2021.
[56] Yunze Liu, Yun Liu, Che Jiang, Kangho Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. Hoi4d: A 4d egocentric dataset for category-level human-object interaction. In CVPR, 2022.
[57] Ahmad Darkhalil, Dandan Shan, Bin Zhu, Jian Ma, Amlan Kar, Richard Higgins, Sanja Fidler, David Fouhey, and Dima Damen. Epic-kitchens visor benchmark: Video segmentations and object relations. NeurlPS, 2022.
[58] Yiming Li, Ziang Cao, Andrew Liang, Benjamin Liang, Luoyao Chen, Hang Zhao, and Chen Feng. Egocentric prediction of action target in 3d. In CVPR, 2022.
[59] Wentao Bao, Lele Chen, Libing Zeng, Zhong Li. Xu, Junsong Yuan, and Kong. Uncertainty- aware state space transformer for egocentric 3d hand trajectory forecasting. In ICCV, 2023.
[60] Irving Fang, Yuzhong Chen, Yifan Wang, Jianghan Zhang, Qiushi Zhang, Jiali Xu, Xibo He, Weibo Gao, Hao Su, Yiming Li, et al. Egopat3dv2: Predicting 3d action target from 2d egocentric vision for human robot interaction. In ICRA, 2024.
[61] Kumar Ashutosh, Georgios Pavlakos, and Kristen Grauman. Fiction: 4d future interaction prediction from video. In CVPR, 2025.
[62] Hugh Durrant- Whyte and Tim Bailey. Simultaneous localization and mapping: part i. IEEE Robotics & Automation Magazine, 2006.
[63] Raul Artalr Jose Maria Martinez Montiel, and Juan D Tardos. Orb-slam: A versatile and accurate monocular slam system. IEEE Transactions on Robotics, 2015.
[64] Jakob Engel, Thomas Schöps, and Daniel Cremers. Lsd-slam: Large-scale direct monocular slam. In ECCV, 2014.
13

---

[65] Jakob Engel, Vladlen Koltun, and Daniel Cremers. Direct sparse odometry. JEEE TPAMI, 2017.
[66] Tong Qin, Peiliang Li, and Shaojie Shen. Vins-mono: A robust and versatile monocular visual-inertial state estimator. IEEE Transactions Robotics, 2018.
[67] Jakob Engel, Kiran Somasundaram, Michael Goesele, Albert Sun, Alexander Gamino, Andrew Turner, Arjang Talattof, Arnie Yuan, Bilal Souti, Brighid Meredith, et al. Project aria: A new tool for egocentric multi-modal ai research. arXiv:2308. 13561, 2023.
[68] Dima Damen, Teesid I Lelasaawssuk,, Osian Haines, Andrew Calway, and Walterio W Mayol-Cuevas. You-do, i-learm: Discovering task relevant objects and their modes of interaction from multi-user egocentric video. In BMVC, 2014.
[69]
Hyun Soo Park, Eakta Jain, and Yaser Sheikh. 3d social saliency from head-mounted cameras. NIPS, 2012.
[70] Hans Strasburger, Lewis O Harvey, and Ingo Rentschler. Contrast thresholds for identification of numeric characters direct and eccentric view. Perception & psychophysics, 1991.
[71] Pia Mäkelä, Risto Näsänen, Jyrki Rovamo, and Dean Melmoth. Identification of facial images in peripheral vision. Vision Research, 2001.
[72] Ryoichi Nakashima and Satoshi Shioiri. Why do we move head to look at object in our peripheral region? lateral viewing interferes with attentive search. PloS one, 2014.
[73] Yumiko Otsuka and Colin WG Clifford. Influence of head orientation perceived gaze direction and eye-region information. Journal of Vision, 2018.
[74] Andrew J Davison, Ian D Reid, Nicholas D Molton, and Olivier Stasse. Monoslam: Real-time single camera slam. IEEE TPAMI, 2007.
[75] Tobias Fischer, Hyung Jin Chang, and Yiannis Demiris. Rt-gene: Real-time eye gaze estimation in natural environments. ECCV, 2018.
[76] Meta, Introducing Aria Gen 2: Unlocking New Research in Machine Perception, Contextual AL, Robotics, and More. https://www.mett http:://www..taa..c//////////pg/ reality-lass-ai-rootics 2025.
[77] Özgün Çişek, Ahmed Abdulkadir, Socren S Lienkamp, Thomas Brox, and Olaf Ronneberger. 3d u-net: learning dense volumetric segmentation from sparse annotation. MICCAI, 2016.
[78] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NIPS, 2017.
[79] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.
[80] Jacob Gardner, Geoff Pleiss, Kilian Q Weinberger, David Bindel, and Andrew G Wilson. Gpytorch: Blackbox matrixx gaussian process inference with gpu acceleration. In NeurlPS, 2018.
[81] Yunpeng Zhang, Zheng Zhu, Dalong Du. Oceformer: Dual-path transformer for vision- basodd 3d semantic occupancy prediction. In ICCV, 2023.
[82] Yiming Li, Zhiding Yu, Christopher Choy, Chaowei Xiao, Jose M Alvarez, Sanja Fidler, Chen Feng, and Anima Anandkumar. Voxformer: Sparse voxel transformer for camera- based 3d semantic scene completion. In CVPR, 2023.
[83] Qiao Gu, Zhaoyang Lv, Duncan Frost, Simon Green, Julian Straub, and Chris Sweeney, Egolifter: Open-world 3d segmentation for egocentric perception. In ECCV, 2024.
14

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
We clearly state the scope and contributions of our work in the abstract and introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
15

---

Justification: We discuss the limitations of our work in the Appendix under 'Limitations and broader impacts' section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-ten system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: Our work does include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: In addition to the core information presented in Sections 3-5, we provide all necessary details for reproducing our experimental results and dataset curation process in the Appendix.
16

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide necessary details for reproducing experiments as well as source code supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ctiips.cc/puulic/guui.
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
17

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:
[Yes]
Justification: We provide details for data preparation and hyperparameters for training in Section 5 and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We briefly discuss the statistical significance of our results the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We report the computer resources we used for all experiments in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http://s..c/clllct/ttticciiieee.
Answer:
[Yes]
Justification:
We confirm that our work complies with the NeurlPS Code of Ethics in every
aspect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the potential societal impacts of our work in the Appendix under "Limitations and broader impacts' section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
19

---

Answer: [NA]
Justification: Our work does not involve assets that have a high risk for misuse, such as pretrained LLMs, image generators, scraped datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We enumerate assets we used for our work with their licenses in the Appendix.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Details regarding our data curation and model implementation are discussed in our main paper well the Appendix.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
20

---

14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsoureing / research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our work does involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Our work does involve LLMs as any important, original, or ntn-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy hitpes///earrppsccc////ofteeeeee for what should or should not be described.
21

---

FoVS-Aria & FoVS EgoExo
Figure 6: More examples from our curated FoVS-Aria and FoVS-EgoExo. Blue denotes previous observations and green denotes forecast targets.
A
Experimental Details
A.1 Dataset Curation Details
The detailed preprocessing procedures for FoVS-Aria and FoVS-EgoExo are as follows. We first temporally synchronize the raw streams present in each source dataset, namely gaze, RGB, audio, IMU-based trajectory, and SLAM observations. Since gaze data has the lowest sampling rate of 10Hz, we use it as the temporal anchor synchronize other sensory streams and group them into 7 1-second units. Using a I-second sliding window, we apply EgoSpanLift to obtain volumetric grids corresponding multi-level visual spans. Given the substantial volume of raw data (e.g., Ego-Exo4D exceeding 10TB), we store the processed data as bitpacked arrays to enable efficient storage and retrieval during training.
5
11 We perform manual validation for both datasets regarding gaze distribution and 3D point cloud 12 integrity. For each recording, we visualize gaze distribution across 2D frames throughout the entire 13 recording, excluding samples with extremely low variance as these indicated miscalibration. manually remove cases where SLAM capture quality is poor and visual spans with gaze fixations are
also
15 trivial such as instances of lying down while reading books using mobile phones. Additionally, we exclude samples where keypoints of interest fall below two standard deviations from the wide
22

---

Table 5: Comparison of forecasting accuracy the FoVa-Aria test split.
Posptepla-
F1
Pr
Be
101
F1
PV
Re
300
F1
Re
Piooe En
0L5581
05551
01:064
0.0885
0.1132
CSTS B
0.5955
033311
0.6313
0.8817
0.435
01006
0.5920
0.500
0.2342
0.3423
0.3889
0.5965
0.1388
1945
0.3099
0.2285
OcePereer
0.1395
002522
02340
0,2422
0.106
0.865
0.129
0.008
0,420
0.603
03216
00534
0AL5R
0.029
0,015
0.0226
VoaFemsrr
0.194
09453
0.144
0.405
06579
0.391
0.624
0.432
0.581
0.544
0,553
0.612
0.308
0.3111
0.290
0.997
0.311
0.3455
0,2335
Glonal Price
0.359
0.314
01875
0.79M
1048
01825
01374
0.661
08331
0,018
06319
03523
0.046
0.049
0.578
(etlgreeeeess spon)
Our (Cams)
0.5730
03154
08314
0.591
0.854
06017
0.750
05:54
0.3816
03875
0.585
03674
0.361%
0.291
Oar (Sirglo-ssk)
015813
01738
07711
0.700
0.771
06151
07735
0.501
0.311
0.689
0.330
04605
0.366
119
0.35%
0.111
Oan ouo alobal onbooklina)
0.5962
0.7042
0.760
0,577
0.664
06123
0.739
0516)
0.324
04476
0.3347
04755
0.2244
0.3487
0.3732
0,503
Our (ull))
00347
07002
0.485
0.683
06354
0.7022
0.5054
0.3366
0.4085
average, evaluating over the 2-second forecast windows. The resulting datasets comprise 23.2k 18 samples for FoVS-Aria and 341.4k samples for FoVS-EgoExo, providing rich visual span-basel forecasting samples across diverse scenarios, illustrated in Fig. 6
20
A.2
Implementation Details
21
For all reported experiments, we use the Adam optimizer [1] with learning rate of le-4, without applying any scheduler or weight decay. We train our models with a batch size of for 50 epochs and select the epoch that achieves the highest average IoU across all visual spans on the validation
22
24 split for final testing. Input volumes are augmented through axis permutation and flipping (excluding upside-down orientation), along with random translation applied up to 2 units.
26 The unidirectional transformer utilizes a feature dimension of C = 1024. To achieve this di- mensionality, each U-Net layer reduces the feature dimension by factor of 2, i.e., an encoding progression 1-282556-11-10 for 16 grids. Each U-Net layer consists of two applications of Con--BacchNorm-RRllU-DDoee
27
28
blocks, with dropout rate of 0.1.
30
While experimental results reported for single runs, conduct three random runs for our method and observe performance variations of less than 1,, which proved negligible compared to performance differences with other models. All experiments are conducted using NVIDIA RIX A6000 GPUs with 48GB memory and 16 CPU cores. Additional implementation details can be found in the source code.
31
32
33
34
35
A.3
Environment and Asset Usage
The raw egocentric data sources used for curating the testhed are Aria Everyday Activities 2 and Ego-Exo4D [3]. Aria Everyday Activitie is released under a custom license that permits academic research only, while Ego-Exo4D uses custom I allowing both academic and commercial usage. We confirm that our usage aligns with their intended purposes. Since both datasets collected using Aria glasse we utilize the Project Aria Tools (
37
38
39
40
for data 41 processing. Additionally, we conduct experiments using PyTorch 2.4. 1. employing Open3D 4 for data processing and the ByTorch-3DUNN (5) library for model construction, both of which are distributed under the MIT License.
42
43
B Additional Experiments
44
B.1
Additional Results on 3D Visual Span Forecasting
46 Table presents the complete experimental results for FoVS-Aria, including both Precision and Recall metrics. In some cases, the harmonic mean of Precision and Recall differs from the F1 score 48 because the harmonic mean was calculated at the sample level rather than globally, resulting in
httpps//eggo-exoddddaaa....r
23

---

66 To conduct sensitivity analysis examining how the performance of our framework changes when SLAM struggles, we explore several types of sensory corruption across mild to severe scenarios. First, multi-sensor devices may experience temporary frame drops or time drift due to hardware issues, 69 representing temporal corruption. Second, spatial degradation egocentric localization may occur, which we apply separately to translation and rotation components. Finally, we consider corruption Table 6: Comparison of overlap-basee metrics and saliency- based metries the FoVS-Aria test split.
Ours
Orientation
Peripheralge
Contrala-
Fowealg-
IoU
F1
CC
AUC
IoU
F1
CC
AUC
loU
F1
CC
AUC
loU
F1
CC
AUC
Single-tisk
0.583
0.724
0.725
0.861
0.472
0.615
0.630
0.791
0.335
0.449
0.468
0.674
0.249
0.319
0.340
0.524
wio global
0.560
0.704
0.711
0.847
0.465
0.613
0.631
0.827
0.324
0.448
0.477
0.765
0.262
0.349
0.376
0.613
Full
0.584
0.725
0.729
0.858
0.489
0.635
0.649
0,840
0.351
0.476
0.502
0.771
0.284
0.371
0.399
0.624
Table 7: SLAM sensitivity analysis on the Fori test split.
Orientation
Peripheralga
Centralge
Fovealg
loU
F1
loU
F1
loU
Fl
loU
Fl
EgoChoir
12
0.4959
0.6579
0.4302
0,5581
0.2612
0.3608
0.1987
0.2311
Ours (original)
0.5838
0.7247
0.4886
0.6350
0,3513
0.4762
0.2836
0.3709
w/ temporal (5%)
0.5592
0.6956
0.4666
0.6071
0.3343
0.4543
0.2697
0.3530
w/ temporal (10%)
0.5347
0.6674
0,4457
0,5810
0.3194
0.4344
0.2571
0.3371
w/ translation (5cm)
0.5814
0.7226
0.4871
0.6332
0.3399
0.4661
0.2611
0.3498
w/ translation (10cm)
0.5727
0.7158
0.4780
0.6258
0.3129
0.4407
0.2174
0.3022
w/ rotation (2.59)
0,5818
0.7227
0,4882
0,6338
0.3461
0.4717
0.2681
0.3563
w/ rolation (5")
0.5775
0.7193
0.4836
0.6298
0.3328
0.4575
0.2490
0.3357
w/ Gaussian (2.5cm)
0.5759
0.7190
0.4839
0,6313
0.3432
0.4697
0.2758
0,3661
w/ Gaussian (5cm)
0.5292
0.6831
0,4503
0.6037
0.3133
0.4436
0.2364
0.3336
w/ point drop (10%)
0.5823
0.7231
0.4890
0.6346
0.3500
0.4751
0.2823
0.3691
w/ point drop (20%)
0.5799
0.7212
0.4871
0.6330
0.3498
0.4741
0.2806
0.3676
different patterns of invalid values across metrics. For the 2D Gaze model, the numbers in gray (i.e. orientation) are mechanically assigned based on camera position regardless of gaze anticipation quality, thereby identical for all cases. Examining Precision and Recall performance, our full model demonstrates superior results across all Foveal span metrics. However, for broader spans, single-task models and those using BCE loss achieve slightly higher precision scores. Nevertheless, these models tend to make relatively sparse predictions, resulting in notably lower recall performance. Consequently, Dice loss is preferred in our multi-task model due its more balanced performance.
50
51
52
53
54
55
56
To provide complementary analysis on saliency metrics in our benchmarks, we report Correlation Coefficient (CC) and Area Under Curve (AUC) over the logit distribution of 3D grids using three variants of our framework in Table 6 We use AUC instead of KL.D due to zero-value sensitivity of KLD [3 in our evaluation. The results demonstrate patterns that are generally consistent with previously reported metrics.
58
60
61
B.2
Sensitivity Analysis on SLAM Keypoints
Since our approach utilizes semidense kcypoints derived from SLAM, their quality could affect performance. However, this applics to any framework that relies on SLAM keypoints as input. Still, it is feasible to utilize accurate pre-mapped information fallback option since our framework should typically be deployed in familiar everyday environments, e.g., home and office.
63
65
67
68
71 that can affect the set of keypoints by adding Gaussian noise to individual points or dropping certain points entirely.
73 Performance results measured on the FoVS-Aria test split for each corruption type are presented in
74 Table Under mild corruption, the impact on performance is negligible. However, applying higher- intensity corruption results in performance degradation of several percentage points. Among the different spans, those requiring wider coverage (Orientation and Peripheral) show smaller performance
75
76
77 drops. contrast, spans demanding higher precision (Central and Foveal) exhibit larger degradation.
78 Despite these challenges, our framework maintains superior performance compared to EgoChoir
79 even under severe corruption scenarios. This suggests that our framework can perform reliable forecasting despite some degree of SLAM-indueed imprecision. In practice, real-world scenarios
24

---

Fig. 7 provides additional qualitative analysis of examples from FoVS-Aria. Consistent with the findings in Fig. 5, these results demonstrate our model's 's ability to perform effective forecasting across diverse scenarios and show meaningfully closer approximations to ground truth compared to other competitive models. Fig. 8 presents additional qualitative analysis of examples from FoVS-EgoExo. 89 Our model performs well across various scenarios including bike repairing, cooking, and bouldering. For instance, it appropriately predicts the semantic significance of actions such as gathering items Previous Visual Span
CSTS
OuI'
Ground Truth
Figure 7: More qualitative examples FoVS-Aria test split.
involve multiple types of corruption occurring in complex and somewhat unpredictable combinations. 82 Therefore, training our framework to be robust against such corruption would represent an interesting extension for future work.
88
B.3 More Qualitative Examples
85
87
91 on a workbench or looking around while holding a knife in the kitchen, accurately anticipating the future visual focus these behaviors will lead to.
93 Fig. Sisows qualitative examples egocentric 2D gaze anticipation when our model's 3D inference results are projected to 2D. The results reveal that actual human gaze tends to be linked to specific 3D 95 positions or objects rather than simply following head rotation patterns, which our model captures effectively. The final example illustrates challenging case for our model in both 2D and 3D: when people interact with objects and create three-dimensional spaces that were not captured by semidense keypoints& at previous time steps, performing precise inference becomes relatively difficult.
94
97
C Limitations and Broader Impact
100
Our research interpret the primary source of perceptual intent in terms of gaze and peripheral span. However, human intent is comprehensively formed through the interplay of multiple sensory perceptions, including audio and proprioceptive inputs, beyond visual attention alone. Therefore, gaze alone may not fully capture the complexity of human intent. Future work could generalize toward forecasting human intent that can be identified through more broadly defined multisensory perception. Additionally, while we utilized semidense keypoints minimize latency for real-time applications while conveying an accurate sense of distance, we did not interpret the visual span continuous surface representation. From this perspective, approach that combines neural rendering [4] to represent visual span in a fine- fine-grained manner would be beneficial for modeling
102
104
106
108
109
intent with greater precision.
110
Moving forward, our research can be applied to facilitate service delivery in various egocentric sensncy-sensitive services by preemptively capturing the wearer's intent. For instance, can predict
25

---

Past
Fubure
Previous Visual Span
Our Forecast
Truth
Figure 8: More qualitative examples on FoVS-EgoExo split.
Previous Span
Our Forecast
Our 2D Forecat (+GT)
Figure 9: More qualitative examples on egocentric 2D gaze anticipation.
26

---

112 users' future focus or interest, enabling ambient computing to proactively adapt the surrounding environment in a more user-friendly manner. We anticipate that such technology will be useful for providing seamless access to desired objects or information for individuals with various impairments non-invasive manner. Furthermore, for general users, when providing augmented reality services, it will be possible to render or deliver information with higher fidelity in areas that align with user's intent.
113
114
115
116
117
On the other hand, using the wearer's perceptual input for model training or data utilization could potentially raise privacy issues. Our model is relatively unaffected by such concerns as it uses source data 22 3 that are in compliance with privacy requirements and does not explicitly exploit personally 120
118
119
121
identifiable information. However, careful consideration is needed when applying this technology in real-worls scenarios. Since our methodology adopted a direct and lightweight approach in both volumetric region representation as visual span and network design due to latency-sensitive aspect of the problem setup, suggesting that on-device processing could also be viable direction for mitigating privacy concerns.
122
123
124
125
References
126
127
[1] Diederik Kingma. Adam: A method for stochastic optimization.
2014.
128
[2] Zhaoyang Lv, Nicholas Charron, Pierre Moulon, Alexander Gamino, Cheng Peng, Chris Sweeney, Edward Miller, Huixuan Tang. Jeff Meissner, Jing Dong, et al. Aria everyday activities dataset. arXiv:2402.13349 2024.
129
130
131
[3] Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third- person perspectives. In CVPR, 2024.
132
134
[4] Qian- Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3D: A modern library for 3D data processing. arXiv: 110111001..000 098847, 2018.
136
[5] Adrian Wolny, Lorenzo Cerrone, Athul Vijayan. Rachele Tofanelli, Amaya Vilches Barro, Marion Lou- veaux, Christian Wenzl, Sören Strauss, David Wilson-Sánchez, Rena Lymbouridou, Susanne S Steigleder, Constantin Pape, Alberto Bailoni, Salva Duran-Nebreda, George W Bassel, Jan U Lohmann, Miltos Tsiantis, Fred Hamprecht, Kay Schneitz, Alexis Maizel, and Anna Kreshuk. Accurate and versatile 3d segmentation of plant tissues at cellular resolution. elife, 2020.
137
138
139
140
141
[6] Jacob Gardner, Geoff Pleiss, Kilian Q Weinberger, David Bindel, and Andrew G Wilson. Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu acceleration. In NeurlPS, 2018.
142
143
[7]
Bolin Lai, Miao Liu, Fiona Ryan, and James M Rehg. In the eye of transformer: Global-local correlation for egocentric gaze estimation. In BMVC, 2022.
144
145
[8] Bolin Lai, Fiona Ryan, Wenqi Jia, Miao Liu, and James M Rehg. Listen to look into the future: Audio-visual egocentric gaze anticipation. In ECCV, 2024.
146
147
99 Yunpeng Zhang, Zheng Zhu, and Dalong Du, Oceformer: Dual-path transformer for vision- based 3d semantic occupancy prediction. In ICCV, 2023.
148
149
[10] Viming Li, Zhiding Yu, Christopher Choy, Chaowei Xiao, Jose M Alvarez, Sanja Fidler, Chen Feng, and Anima Anandkumar. Voxformer: Sparse voxel transformer for camera-based 3d semantic scene completion. In CVPR, 2023.
150
152
[11] Yuhang Yang, Wei Zhai, Hongehen Luo, Yang Cao, Jiebo Luo, and Zheng-Jun Zha. Grounding 3d ohject affordance from 2d interactions in images. In ICCV, 2023.
153
154
[12] Yuhang Yang. Wei Zhai, Chengfeng Wang, Chengjun Yu, Yang Cao, and Zheng-Jun Jun Zha. Egochoir: Capturing 3d human-object interaction regions from egocentric views. In NewrIPS, 2024.
156
[13] Zoya Bylinskii, Tilke Judd, Aude Oliva, Antonio Torralba, and Frédo Durand. What do different evaluation metries tell us about saliency models? IEFE TPAMI, 2018.
157
158
[14] Qiao Gu, Zhaoyang Lv, Duncan Frost, Simon Green, Julian Straub, and Chris Sweeney. Egolifter: Open- world 3d segmentation for egocentric perception. In ECCV, 2024.
27


Reviews:
[{"review_id": "XUr05uk7aw", "reviewer": "Reviewer_k89X", "summary": "The authors propose an approach to forecast \"visual spans\" form egocentric recordings. The method predicts in 3D, by lifting from 2D. The method is evaluated both in the 3D scenario, as well as for 2D egocentric gaze anticipation.", "questions": "To re-consider my score, I would like to see a convincing clarification of the concept of \"visual span\" (and how it relates to and motivates the author's works), and a convincing argument why the problem can not simply be addressed by doing 3D gaze anticipation + post-processing.", "limitations": "Limitations should be discussed in the main paper.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths & Weaknesses:\n-----------------------\n\nThe problem is relevant and the paper quite readable. The technical description appears to be valid, and is supported by adequate figures. My main criticisms are concerning the conceptual motivation/background of the work and an inadequate representation of previous works. I am also questioning whether \"egocentric span prediction\" as formulated by the authors has relevance on top of the task of 3D gaze anticipation.\n\nConcept from visual attention research are used in an inconsistent and confusing way. Sometimes authors talk about forecasting gaze, sometimes about predicting future visual focus, most prominently (and in the title of the method), authors talk about forecasting future visual spans.\n\nConcerning the terminology of \"visual span\": Visual span is mainly used in the context of reading and denotes the number of letters to the left and right of the fixation point that can be recognised correctly. See Frey & Bosse (2018) for a clarification on the terminology. The visual span depends on different aspects of the stimulus, e.g. character orientation or crowding. The crucial point here is that the visual span is not a cone around the gaze point that is independent of the stimulus. In contrast, the authors in the present paper do not capture visual span, as they simply define a stimulus-independent stack of cones around the gaze point. They also do not evaluate visual span prediction. To evaluate visual span, they would need to check whether their method is able to predict which objects around the gaze point can be recognised by the user.\n\nI am not convinced that defining these regions around the gaze point gives us any new insights in the evaluation. Alternatively, the authors could simply evaluate the accuracy of 3D gaze prediction. My view is supported by Table 1 in the paper: the ordering of methods is highly consistent between Orientation metrics and Peripheral/Central/Foveal metrics. The same applies for Table 3. An alternative to the way the authors pose the problem is to simply anticipate 3D gaze, evaluate the quality of this gaze, and provide peripheral/central/foveal areas as a post-processing step.\n\nIn line 32, the authors state that \"However, attempts to forecast human visual perception itself remain less explored.\"\nI am not sure about this statement, as there are already quite a few works that try to do this (and are not referenced by the authors), e.g. see Steil et al. (2018), Hu et al. (2021), Rolff et al. (2022).\nA similar point concerns lines 40-42: \"While previous research has shown impressive results in predicting egocentric future gaze fixations on 2D image frames [16, 17], forecasting gaze for dynamic scenarios in 2D remains unclear.\n\nThe related work section is also misleading in these respects. For example, in the paragraph on \"Egocentric Gaze Prediction\", authors refer to works such as gaze following ([24, 25]) which is not an egocentric setting, and also not about anticipation. Relevant gaze anticipation works are not discussed (e.g. Steil et al., 2018; Hu et al., 2021; Rolff et al., 2022).\n\n\n\nReferences:\n-----------\n- Frey, A., & Bosse, M. L. (2018). Perceptual span, visual span, and visual attention span: Three potential ways to quantify limits on visual processing during reading. Visual Cognition, 26(6), 412-429.\n- Steil, J., Müller, P., Sugano, Y., & Bulling, A. (2018, September). Forecasting user attention during everyday mobile interactions using device-integrated and wearable sensors. In Proceedings of the 20th international conference on human-computer interaction with mobile devices and services (pp. 1-13).\n- Hu, Z., Bulling, A., Li, S., & Wang, G. (2021). Fixationnet: Forecasting eye fixations in task-oriented virtual environments. IEEE Transactions on Visualization and Computer Graphics, 27(5), 2681-2690.\n- Rolff, T., Harms, H. M., Steinicke, F., & Frintrop, S. (2022, September). Gazetransformer: Gaze forecasting for virtual reality using transformer networks. In DAGM German Conference on Pattern Recognition (pp. 577-593). Cham: Springer International Publishing.", "was_revised": true, "final_justification": "The author's promise to clarify the concept of visual span and the additional evaluations lead me to increase my score. I am still not totally convinced about where the difference to the postprocessing approach really lies. It would be great to convey a better intuition on this. I recommend not to use the name \"visual span\", as this terminology is too messy in the context of the author's work and will further confuse readers.", "rebuttal": "We thank reviewer k89X for the detailed comments.\nTo summarize all reviewer feedback, we are encouraged that the reviewers find our problem formulation interesting and advantageous (7cjL, LuMp, 23sp) while being methodologically solid and innovative (7cjL, LuMp) with clear writing (7cjL, k89X).\nAdditionally, they find that our comprehensive experimental results demonstrate significant performance improvements (7cjL, LuMp, 23sp) with the curation of two new large-scale benchmarks (LuMp, 23sp).\nBelow, we address the specific questions raised in reviewer k89X’s comments.\n\n## (k89X-Q1) Conceptual motivation of our work\n\nAs noted in L39-40, we acknowledge that the term 'visual span' originates largely from text and symbolic reading research in cognitive psychology. While we draw inspiration from this foundation, our work addresses daily and casual behaviors and interactions from an egocentric perspective, without aiming to provide precise computational modeling of the original terminology. \nWhile reading-based visual span depends on specific textual stimuli, forecasting visual perception in egocentric daily activities involve diverse, dynamic environments where stimulus-independent spatial regions provide a more practical and generalizable framework. We will clarify the terminology discrepancies in our final draft.\nPlease note that the other three reviewers have recognized the merit of this conceptual formulation, with reviewer 7cjL highlighting our 'practical problem formulation', reviewer LuMp noting that we are 'clearly motivated by cognitive theories and real-world applications', and reviewer 23sp stating that we 'frame the problem in a sensible fashion that provides many advantages,' respectively.\n\nOur cone-based multi-level interpretation serves as a plausible and practical representation for modeling eccentricity and periphery, which is reflected in our substantial performance gains over prior art. We note that gaze, visual focus, and visual span are known to be interconnected [VisCog’13, TPAMI’18, NatHumBehav’19], and similar concepts and wordings are commonly used in related computer vision work, including previous research [16, 17, 32, 33, 34, ETRA’10, ICCV’21].\nImportantly, our core methodological and empirical contributions, which have been appreciated by the other three reviewers, remain valuable regardless of naming choices. To address any remaining terminological concerns, we are open to adopting alternative terms.\n\n## (k89X-Q2) Rationale behind our multi-level representation\n\nWhile the monotonic results in Table 1 across different spans may appear redundant, this impression arises simply from the broad prediction target regions. In reality, multi-level interpretation carries significant implications due to uncertainties in self-motion forecasting and geometric correspondence between 2D and 3D spaces. To demonstrate this importance, we conducted an ablation study applying post-processing to 3D gaze forecasting using a model trained with the multi-level components removed from our framework:\n\n|                          | P-IoU  | P-F1   | C-IoU  | C-F1   | F-IoU  | F-F1   | 2D-F1 | 2D-Pr | 2D-Re |\n|--------------------------|--------|--------|--------|--------|--------|--------|-------|-------|-------|\n| 2D gaze + postprocessing | 0.4567 | 0.6076 | 0.2342 | 0.3423 | 0.1388 | 0.1948 | 0.515 | 0.497 | 0.535 |\n| 3D gaze + postprocessing | 0.4723 | 0.6214 | 0.2666 | 0.3791 | 0.2494 | 0.3193 | 0.505 | 0.432 | 0.608 |\n| Ours (Single-task)       | 0.4721 | 0.6154 | 0.3351 | 0.4485 | 0.2494 | 0.3193 | 0.505 | 0.432 | 0.608 |\n| Ours                     | 0.4886 | 0.6350 | 0.3513 | 0.4762 | 0.2836 | 0.3709 | 0.515 | 0.440 | 0.619 |\n\nPredicting only gaze and applying post-processing is clearly suboptimal. Compared to the foveal forecast performance when jointly predicting all multi-level spans, foveal forecast performance shows an F1 decrease of approximately 5.2, stressing the importance of our multi-level representation and multi-task objectives.\nFurthermore, the central forecast exhibits an F1 decrease of approximately 9.7, which is substantially larger than the 2.8 F1 performance decrease observed in our model trained to predict only the central span. This demonstrates that predictive performance across different spans involves complex geometric and learning dynamics that extend beyond simple monotonic relationships.\nWe will include this analysis in our final draft to clarify the benefit of our formulation.\n\n## (k89X-Q3) Suggestions on relevant papers\n\nThank you for your suggestion regarding the related work from mobile and VR research. We will cite these works [MobileHCI'18, TVCG'21, GCPR'22] and revise our discussion accordingly.\nPlease note that we reference works [24, 25] in our related work section before narrowing our scope to egocentricity, as these are some of the seminal works for gaze comprehension in computer vision to the best of our knowledge. \n\n## Reference\n\n[ETRA’10] Blignaut (2010). Visual Span and Other Parameters for the Generation of Heatmaps.\n\n[VisCog’13] Nuthmann (2013). On the Visual Span during Object Search in Real-world Scenes.\n\n[TPAMI’18] Masse et al. (2018). Tracking Gaze and Visual Focus of Attention of People Involved in Social Interaction.\n\n[NatHumBehav’19] van Ede et al. (2019). Human Gaze Tracks Attentional Focusing in Memorized Visual Space.\n\n[ICCV’21] Jiang and Ithapu (2021). Egocentric Pose Estimation from Human Vision Span."}, {"review_id": "IwRla7eTR9", "reviewer": "Reviewer_23sp", "summary": "This paper presents a new way to predict where a person will look next, shifting the problem from a 2D image to a 3D environment. The authors argue that this is more robust/fundamental because people naturally direct their attention to objects and locations in 3D space. Their solution has two main parts:\n- The first, EgoSpanLift, takes the video from a user's perspective and uses SLAM to build a 3D map of the surroundings in real-time. Then, using the user's gaze direction, it performs a simple geometric calculation to determine which 3D points fall within different levels of their visual span (ranging from a tiny foveal focus to a much wider peripheral one). The result is a set of 3D grids that represent where the user's attention is focused at any given moment.\n- The second is a forecasting network designed to predict what these 3D attention grids will look like in the near future. It takes a history of the grids created by EgoSpanLift and feeds them into a model that uses a 3D U-Net to understand the spatial patterns of attention and a transformer to understand how that attention moves over time. The final output is a predicted 3D volumetric region where the user's visual perception will focus next. To prove their method works, the authors created two new large-scale benchmarks from existing egocentric video datasets and showed that their approach outperforms a wide range of other methods.", "questions": "Please address my comments under Weaknesses.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n- IMO the paper frames the problem of visual attention/gaze estimation in a sensible fashion. By constructing a state of the 3D environment (via SLAM) and modeling the user's focus here, this formulation provides many advantages (e.g., disentanglement from the user's own motion, 3D is more fundamental than 2D).\n- The authors went through the process of curating two large-scale benchmarks, FoVS-Aria and FoVS-EgoExo, specifically for this new 3D forecasting problem.\n- They show that their framework significantly outperforms a wide range of other approaches, including models adapted from both 2D gaze prediction and other 3D localization tasks. The improvement is especially large when predicting the user's precise/foveal gaze.\n\n**Weaknesses**:\n- The proposed system relies on getting a good 3D map from SLAM. The authors harness \"3D semidense keypoints\" from SLAM, but I'm unclear how important it is / what happens when the SLAM algorithm struggles (e.g. during fast motion)  such that a good map is not produced. If the foundational map is inaccurate, the entire gaze forecasting pipeline will be thrown off, limiting its reliability in the real world.\n- It might be too computationally heavy for real-world use right now, and it would be nice if the authors can shed some light on this aspect. The forecasting network uses a 3D U-Net and a unidirectional transformer on a sequence of 3D volumetric grids. This kind of architecture is powerful but also known to be computationally hungry. For latency-sensitive applications like AR/VR running on a mobile headset, this could be too slow.", "was_revised": true, "final_justification": "I have read the rebuttal carefully and decided to update my review -- same rating (5/accept) but with a higher confidence (4).", "rebuttal": "We thank reviewer 23sp for the constructive feedback.\nTo summarize all reviewer feedback, we are encouraged that the reviewers find our problem formulation interesting and advantageous (7cjL, LuMp, 23sp) while being methodologically solid and innovative (7cjL, LuMp) with clear writing (7cjL, k89X).\nAdditionally, they find that our comprehensive experimental results demonstrate significant performance improvements (7cjL, LuMp, 23sp) with the curation of two new large-scale benchmarks (LuMp, 23sp).\nBelow, we address the specific questions raised in reviewer 23sp's comments.\n\n## (23sp-Q1) What happens when SLAM struggles?\nSince our approach utilizes semidense keypoints derived from SLAM, their quality could affect performance. However, this applies to any framework that relies on SLAM-derived keypoints as input. Still, it is feasible to utilize accurate pre-mapped information as a fallback option since our framework should typically be deployed in familiar everyday environments, e.g., home and office.\n\nTo conduct a sensitivity analysis examining how the performance of our framework changes when SLAM struggles, we explore several types of sensory corruption across mild to severe scenarios. First, multi-sensor devices may experience temporary frame drops or time drift due to hardware issues, representing temporal corruption. Second, spatial degradation in egocentric localization may occur, which we apply separately to translation and rotation components. Finally, we consider corruption that can affect the set of keypoints by adding Gaussian noise to individual points or dropping certain points entirely.\nPerformance results measured on the FoVS-Aria test split for each corruption type are presented in the following table:\n\n|                     | O-IoU  | O-F1   | P-IoU  | P-F1   | C-IoU  | C-F1   | F-IoU  | F-F1   |\n|---------------------|--------|--------|--------|--------|--------|--------|--------|--------|\n| Baseline (EgoChoir) | 0.4959 | 0.6579 | 0.4302 | 0.5581 | 0.2612 | 0.3608 | 0.1987 | 0.2311 |\n| Ours                | 0.5838 | 0.7247 | 0.4886 | 0.6350 | 0.3513 | 0.4762 | 0.2836 | 0.3709 |\n| Temporal (5%)       | 0.5592 | 0.6956 | 0.4666 | 0.6071 | 0.3343 | 0.4543 | 0.2697 | 0.3530 |\n| Temporal (10%)      | 0.5347 | 0.6674 | 0.4457 | 0.5810 | 0.3194 | 0.4344 | 0.2571 | 0.3371 |\n| Translation (5cm)   | 0.5814 | 0.7226 | 0.4871 | 0.6332 | 0.3399 | 0.4661 | 0.2611 | 0.3498 |\n| Translation (10cm)  | 0.5727 | 0.7158 | 0.4780 | 0.6258 | 0.3129 | 0.4407 | 0.2174 | 0.3022 |\n| Rotation (2.5°)     | 0.5818 | 0.7227 | 0.4882 | 0.6338 | 0.3461 | 0.4717 | 0.2681 | 0.3563 |\n| Rotation (5°)       | 0.5775 | 0.7193 | 0.4836 | 0.6298 | 0.3328 | 0.4575 | 0.2490 | 0.3357 |\n| Gaussian (2.5cm)    | 0.5759 | 0.7190 | 0.4839 | 0.6313 | 0.3432 | 0.4697 | 0.2758 | 0.3661 |\n| Gaussian (5cm)      | 0.5292 | 0.6831 | 0.4503 | 0.6037 | 0.3133 | 0.4436 | 0.2364 | 0.3336 |\n| Point drop (10%)    | 0.5823 | 0.7231 | 0.4890 | 0.6346 | 0.3500 | 0.4751 | 0.2823 | 0.3691 |\n| Point drop (20%)    | 0.5799 | 0.7212 | 0.4871 | 0.6330 | 0.3498 | 0.4741 | 0.2806 | 0.3676 |\n\nUnder mild corruption, the impact on performance is negligible. However, applying higher-intensity corruption results in performance degradation of several percentage points. Among the different spans, those requiring wider coverage (Orientation and Peripheral) show smaller performance drops. In contrast, spans demanding higher precision (Central and Foveal) exhibit larger degradation. Despite these challenges, our framework maintains superior performance compared to EgoChoir even under severe corruption scenarios.\nThis suggests that our framework can perform reliable forecasting despite some degree of SLAM-induced imprecision. In practice, real-world scenarios involve multiple types of corruption occurring in complex and somewhat unpredictable combinations. Therefore, training our framework to be robust against such corruption would represent an interesting extension for future work.\n\n## (23sp-Q2) Detailed analysis on the latency of our framework\n\nOur framework has two primary sources of latency: (i) extracting the relevant set of points from gaze and SLAM keypoints (performed every 100ms), and (ii) performing model inference every second on a set of points spanning two seconds.\n\nSince the point extraction can be pre-computed and stored at 10 fps for continuous use, we only need to consider the computation time for processing the final observation when calculating inference latency.\nWe measured this in a resource-constrained environment compared to our training setup, using 8 CPU cores and a GPU with 12GB VRAM (e.g., Titan X). In the table below, all operations except model inference are processed on the CPU, with the following execution time:\n\n| **Gaze and keypoint processing**   |                 |\n|------------------------------------|-----------------|\n| Point preprocessing                | 4.541±1.999ms   |\n| 3D visual span localization        | 1.811±0.824ms   |\n| **From keypoints to model output** |                 |\n| Voxelization                       | 45.406±26.223ms |\n| Model inference                    | 19.483±8.234ms  |\n|                                    |                 |\n| **Average inference latency**      | 71.241ms        |\n\nThe first stage, which handles outlier removal, axis-aligned bounding box cropping for keypoints, and selection of points within a certain degree of eccentricity from the gaze, can be processed within 10ms.\nIn the second stage, we identified that the primary bottleneck lies in voxelization rather than in the model itself. This occurs because the large number of keypoints from the previous stage should be voxelized, whereas the model operates efficiently once it receives the 3D voxelized representation.\n\nConsequently, the average inference latency is 71.241ms, yielding a real-time factor of 0.036 when processing 2-second input for multi-second forecasting. This confirms our claim that the framework supports fast processing. However, actual AR/VR environments typically operate with even fewer computational resources, and thus additional optimization techniques such as model quantization or more efficient voxelization could be considered for further performance improvements."}, {"review_id": "33Ioau11JB", "reviewer": "Reviewer_LuMp", "summary": "This paper tackles a novel problem of egocentric 3D visual span forecasting. It introduces EgoSpanLift, a cone-based lifting mechanism that transforms 2D gaze data into 3D multi-level volumetric representations, and proposes a temporal forecasting model to predict future human visual attention in 3D space. The authors also curate two datasets for benchmarking this task. The proposed method demonstrates superior performance compared to existing baselines and shows promising potential for downstream applications in egocentric vision.", "questions": "1. The paper mentions real-time or latency-sensitive applications. Is the current model suitable for deployment on mobile or AR/VR hardware platforms?\n2. Could the authors elaborate more on the practical applications of 3D visual span forecasting in the real world? What new possibilities does this 3D representation unlock that 2D gaze prediction cannot?\n3. Why did you choose IoU and F1 as the primary evaluation metrics? Could commonly used 2D saliency metrics like KLdiv or CC also be applicable or provide complementary insights?\n4. It might be interesting to consider subject-level information in future work, as different individuals can exhibit diverse gaze patterns. Incorporating such factors could potentially improve generalization across users.\n5. In the broader impact section, the authors mention that this technology may help individuals with cognitive or perceptual impairments gain non-invasive access to desired objects or information. I would be curious to learn more about this point. Could the authors clarify which specific groups of users they are referring to, and how such support might be practically implemented?\n\nPlease refer to the above weaknesses and questions. If addressed appropriately, I would be open to raising my score.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper introduces an interesting and timely new task that can significantly advance egocentric vision, especially in interaction-oriented settings.\n2. The proposed 3D gaze representation and forecasting framework are innovative and clearly motivated by cognitive theories and real-world applications.\n\nWeaknesses:\n1. The evaluation on FoVS-EgoExo (Table 3) is somewhat limited. Unlike FoVS-Aria (Table 1), it lacks comprehensive results for extreme baselines such as Global Prior, 2D methods lifted via EgoSpanLift, and 3D localization models like OccFormer or VoxFormer. A fuller set of comparisons would strengthen the conclusions.\n2. Several recent egocentric and interaction-guided attention prediction works are missing in the Related Work section. For example: CVPR'24 Learning from observer gaze: Zero-shot attention prediction oriented by human-object interaction recognition", "was_revised": true, "final_justification": "Thank you for your detailed response, which has addressed most of my concerns. The computation time may still be insufficient to meet the demands of real-world applications, but this exceeds the scope of contribution for a single paper. I suggest that the authors (1) include a discussion on the primary evaluation metrics in the final version, as this is crucial for presenting a new paradigm, and (2) supplement the paper with more related works on egocentric and interaction-guided attention prediction, as the current version is lacking in this regard. I am willing to maintain my positive initial score.", "rebuttal": "We thank reviewer LuMp for the constructive comments.\nTo summarize all reviewer feedback, we are encouraged that the reviewers find our problem formulation interesting and advantageous (7cjL, LuMp, 23sp) while being methodologically solid and innovative (7cjL, LuMp) with clear writing (7cjL, k89X).\nAdditionally, they find that our comprehensive experimental results demonstrate significant performance improvements (7cjL, LuMp, 23sp) with the curation of two new large-scale benchmarks (LuMp, 23sp).\nBelow, we address the specific questions raised in reviewer LuMp's comments.\n\n## (LuMp-Q1) Additional experiments on FoVS-EgoExo\nDue to space constraints, Table 3 previously reported only partial entries. We further report two additional ablation experiments, three 3D localization baselines, and one 2D gaze anticipation baseline:\n|           | O-IoU  | O-F1   | P-IoU  | P-F1   | C-IoU  | C-F1   | F-IoU  | F-F1   |\n|---------------------------|--------|--------|--------|--------|--------|--------|--------|--------|\n| CSTS + EgoSpanLift + [60] | -      | -      | 0.4978 | 0.6398 | 0.2867 | 0.4010 | 0.1556 | 0.2107 |\n| OccFormer                 | 0.1287 | 0.2280 | 0.0920 | 0.1685 | 0.0251 | 0.0490 | 0.0084 | 0.0167 |\n| VoxFormer                 | 0.1896 | 0.3188 | 0.1475 | 0.2571 | 0.0620 | 0.1168 | 0.0179 | 0.0350 |\n| EgoChoir                  | 0.3287 | 0.4948 | 0.2851 | 0.4437 | 0.1976 | 0.3300 | 0.1266 | 0.2247 |\n| Ours w/ BCEloss           | 0.5112 | 0.6621 | 0.4905 | 0.6338 | 0.3722 | 0.4867 | 0.2870 | 0.3578 |\n| Ours w/o global emb.      | 0.4998 | 0.6542 | 0.4892 | 0.6381 | 0.3954 | 0.5294 | 0.3475 | 0.4500 |\n| Ours                      | 0.5317 | 0.6827 | 0.5255 | 0.6694 | 0.4462 | 0.5730 | 0.3977 | 0.4932 |\n\nOverall, the results show a similar performance distribution to those reported on FoVS-Aria except for the 2D baseline often outperforming EgoChoir, presumably due to increased 3D scene complexity compared to FoVS-Aria. Due to time constraints, additional rows will be incorporated into the draft and the response once they become available.\n\n## (LuMp-Q2) Detailed analysis on the latency of our framework\nOur framework has two primary sources of latency: (i) extracting the relevant set of points from gaze and SLAM keypoints (performed every 100ms), and (ii) performing model inference every second on a set of points spanning two seconds.\n\nSince the point extraction can be pre-computed and stored at 10 fps for continuous use, we only need to consider the computation time for processing the final observation when calculating inference latency.\nWe measured this in a resource-constrained environment compared to our training setup, using 8 CPU cores and a GPU with 12GB VRAM (e.g., Titan X). In the table below, all operations except model inference are processed on the CPU, with the following execution time:\n| **Gaze and keypoint processing**   |                |\n|------------------------------------|-----------------|\n| Point preprocessing                | 4.541±1.999ms   |\n| 3D visual span localization        | 1.811±0.824ms   |\n| **From keypoints to model output** |                 |\n| Voxelization                       | 45.406±26.223ms |\n| Model inference                    | 19.483±8.234ms  |\n|                                    |                 |\n| **Average inference latency**      | 71.241ms        |\n\nThe first stage, which handles outlier removal, axis-aligned bounding box cropping for keypoints, and selection of points within a certain degree of eccentricity from the gaze, can be processed within 10ms.\nIn the second stage, we identified that the primary bottleneck lies in voxelization rather than in the model itself. This occurs because the large number of keypoints from the previous stage should be voxelized, whereas the model operates efficiently once it receives the 3D voxelized representation.\n\nConsequently, the average inference latency is 71.241ms, yielding a real-time factor of 0.036 when processing 2-second input for multi-second forecasting. This confirms our claim that the framework supports fast processing. However, actual AR/VR environments typically operate with even fewer computational resources, and thus additional optimization techniques such as model quantization or more efficient voxelization could be considered for further performance improvements.\n\n## (LuMp-Q3) Practical applications of 3D visual span forecasting\nAs our title \"Gaze Beyond the Frame\" implies, our 3D representation offers significant competitive advantages over 2D methods in the scenarios involving perception and interaction beyond the current visual frame. While 2D approaches are constrained to forecasting on existing visual frames, our method can predict attention patterns in previously unseen spatial regions.\nFoveated rendering in VR/MR systems exemplifies this advantage: when users turn their head to attend to different contexts, our 3D spatial awareness maintains consistent predictions across head movements and viewpoint changes, whereas 2D methods struggle with such dynamic scenarios.\n\nMoreover, our approach potentially enables various forms of proactive environmental assistance by integrating ambient intelligence. For instance, systems can adjust lighting or display configurations in spaces where users are predicted to look, facilitating faster information perception. Additionally, by considering users' established focusing patterns, ambient assistants can proactively remind users of information they might otherwise overlook, creating more intuitive human-environment interactions.\n\nTemporally persistent 3D representations offer another key advantage. While 2D-based forecasting becomes less reliable as self-motion increases, our framework utilizes 3D spatial representation that is relatively robust to self-motion. This opens up the potential for extending our framework for forecasting on a longer horizon, which could enable wearable devices to learn and anticipate user behaviors in their daily 3D environments rather than being limited to momentary 2D observations.\n\nLastly, our experimental results demonstrate that 3D visual span predictions, when back-projected to 2D image planes, achieve performance comparable to 2D-specific models without requiring 2D-specific training. This suggests that our 3D approach not only unlocks new capabilities but also maintains the effectiveness of traditional 2D methods while providing the additional spatial understanding necessary for future egocentric applications.\n\n## (LuMp-Q4) Rationale behind our primary evaluation metrics\nWe primarily build upon the recent 2D egocentric gaze estimation and anticipation literature [17,33,34], where F1, Precision, and Recall are well-established metric choices. We further complement these metrics by incorporating measures that reflect the nature of 3D forecasting: IoU for measuring overlaps between volumetric regions and metric distance (Table 2) to compensate for narrow regions from the foveal span.\n\nTo provide complementary analysis on saliency metrics in our benchmarks, we report CC and AUC over the logit distribution of 3D grids using three variants of our framework. Note that AUC is reported instead of KLD due to zero-value sensitivity of KLD [TPAMI’19] observed in our evaluation:\n|       | O-IoU  | O-F1   | O-CC   | O-AUC  | P-IoU  | P-F1   | P-CC   | P-AUC  |\n|-------------|--------|--------|--------|--------|--------|--------|--------|--------|\n| Single-task | 0.5831 | 0.7237 | 0.7253 | 0.8610 | 0.4721 | 0.6154 | 0.6295 | 0.7913 |\n| w/o Global  | 0.5602 | 0.7042 | 0.7113 | 0.8470 | 0.4647 | 0.6128 | 0.6308 | 0.8265 |\n| Ours        | 0.5838 | 0.7247 | 0.7293 | 0.8582 | 0.4886 | 0.6350 | 0.6492 | 0.8402 |\n|             | C-IoU  | C-F1   | C-CC   | C-AUC  | F-IoU  | F-F1   | F-CC   | F-AUC  |\n| Single-task | 0.3351 | 0.4485 | 0.4679 | 0.6736 | 0.2494 | 0.3193 | 0.3398 | 0.5240 |\n| w/o Global  | 0.3242 | 0.4476 | 0.4767 | 0.7651 | 0.2624 | 0.3487 | 0.3755 | 0.6132 |\n| Ours        | 0.3513 | 0.4762 | 0.5020 | 0.7714 | 0.2836 | 0.3709 | 0.3990 | 0.6242 |\n\nThe results demonstrate patterns that are generally consistent with previously reported metrics. We will include this additional analysis in the Appendix.\n\n## (LuMp-Q5) Elaboration on users with cognitive and perceptual impairments\nSeveral user groups with cognitive and perceptual impairments could potentially benefit from this technology. These include individuals with visual perceptual deficits who may miss objects in their peripheral vision, those with age-related visual degeneration who experience slower or less accurate visual targeting compared to typical users, and people with mild cognitive impairment who could benefit from contextual reminders about their behavior sequences and task progression.\n\nThe implementation of assistive support for these users would vary significantly depending on the underlying device or platform. For instance, in display-equipped systems, highlighted overlays could be used to draw attention to forecasted visual spans, helping users identify areas of upcoming interaction. In environments with voice assistants or ambient intelligence systems, direct or indirect cues (e.g., haptic or voice assistant notification) could be provided to users regarding perceptually overlooked contexts, thereby facilitating both perception and interaction. Such proactive assistance has the potential to reduce cognitive load while maintaining user autonomy by anticipating user needs based on their natural gaze patterns and environmental context.\n\n## (LuMp-Q6) Suggestions on future work and relevant papers\nThank you for your suggestion on the related work and future direction. Enhancing subject-aware personalization of 3D forecasting could be an interesting extension by incorporating methods like test-time adaptation. We will discuss them in our final draft.\n\n## Reference\n[TPAMI’19] Bylinskii et al. (2019). What Do Different Evaluation Metrics Tell Us about Saliency Models?"}, {"review_id": "o8l9CLwq0S", "reviewer": "Reviewer_7cjL", "summary": "This paper introduces EgoSpanLift, a method that (1) lifts egocentric 2D gaze history into 3D volumetric “visual spans” via SLAM-derived keypoints, and (2) forecasts future spans using a hybrid 3D U-Net + transformer framework. To evaluate, the authors curate large-scale benchmarks: FoVS-Aria and FoVS-EgoExo to demonstrate substantial gains over adapted 2D gaze and 3D localization baselines across multiple metrics", "questions": "1. While the author says the latency introduced by the method is \"trivial\", I wonder if there is any data related to this? I would think any AR/VR application would be very sensitive to latency.\n\n2. Some very minor formatting/grammar issues\n- Binary cross-entropy and BCE should be associated at some point (maybe when it first appears in line 280), so that in Table 1, $L_{BCE}$ can be less confusing.\n- In Fig. 1 and Fig. 3, Ground truth's truth is not capitalized, despite all other words being capitalized\n- In Sec 5.1 and Sec 5.2, it should be \"Forecasting\", not \"Forcasting\"", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "# Strength\n- Problem formulation is interesting and practical\n- Writing about methodology is clear\n- Methodology itself is solid, concise, with ample ablation study\n- Benchmarking results are comprehensive, and the performance gain is substantial\n\n# Weakness\n- No obvious weakness\n- However, I think the related work section can be enhanced.\n    - https://arxiv.org/abs/2203.13116, https://arxiv.org/abs/2403.05046 addresses egocentric 3D user intention prediction\n    - https://arxiv.org/abs/2307.08243 addresses 3D user hand prediction", "was_revised": true, "final_justification": "I believe this paper is a positive contribution to the community. I will maintain my original positive rating.", "rebuttal": "We thank reviewer 7cjL for the helpful feedback.\nTo summarize all reviewer feedback, we are encouraged that the reviewers find our problem formulation interesting and advantageous (7cjL, LuMp, 23sp) while being methodologically solid and innovative (7cjL, LuMp) with clear writing (7cjL, k89X).\nAdditionally, they find that our comprehensive experimental results demonstrate significant performance improvements (7cjL, LuMp, 23sp) with the curation of two new large-scale benchmarks (LuMp, 23sp).\nBelow, we address the specific questions raised in reviewer 7cjL’s comments.\n\n## (7cjL-Q1) Detailed analysis on the latency of our framework\n\nOur framework has two primary sources of latency: (i) extracting the relevant set of points from gaze and SLAM keypoints (performed every 100ms), and (ii) performing model inference every second on a set of points spanning two seconds.\n\nSince the point extraction can be pre-computed and stored at 10 fps for continuous use, we only need to consider the computation time for processing the final observation when calculating inference latency.\nWe measured this in a resource-constrained environment compared to our training setup, using 8 CPU cores and a GPU with 12GB VRAM (e.g., Titan X). In the table below, all operations except model inference are processed on the CPU, with the following execution time:\n\n| **Gaze and keypoint processing**   |                 |\n|------------------------------------|-----------------|\n| Point preprocessing                | 4.541±1.999ms   |\n| 3D visual span localization        | 1.811±0.824ms   |\n| **From keypoints to model output** |                 |\n| Voxelization                       | 45.406±26.223ms |\n| Model inference                    | 19.483±8.234ms  |\n|                                    |                 |\n| **Average inference latency**      | 71.241ms        |\n\nThe first stage, which handles outlier removal, axis-aligned bounding box cropping for keypoints, and selection of points within a certain degree of eccentricity from the gaze, can be processed within 10ms.\nIn the second stage, we identified that the primary bottleneck lies in voxelization rather than in the model itself. This occurs because the large number of keypoints from the previous stage should be voxelized, whereas the model operates efficiently once it receives the 3D voxelized representation.\n\nConsequently, the average inference latency is 71.241ms, yielding a real-time factor of 0.036 when processing 2-second input for multi-second forecasting. This confirms our claim that the framework supports fast processing. However, actual AR/VR environments typically operate with even fewer computational resources, and thus additional optimization techniques such as model quantization or more efficient voxelization could be considered for further performance improvements.\n\n## (7cjL-Q2) Suggestions on writing\n\nThank you for your suggestion on the related work. We will include these papers and fix formatting issues."}]

Author rebuttal:

Meta-review: This paper received 2 borderline accept scores and 2 accept scores as final ratings. Initially, reviewers liked the paper due to the curation of two datasets; strong motivations; interesting and timeliness of the proposed new task; and clear writing within the paper. They also had some criticisms towards the paper, namely that the concept of a visual span and how it could be solved in other ways; whether the model is too heavy for real-time applications; the underlying need for a strong SLAM map; and some missing related works. During the rebuttal stage, many of the reviewers' questions were answered with new results; additional descriptions; and analysis. Accordingly, all reviewers gave a positive final rating towards the paper. The AC agrees with the scores of the reviewers and sees no need to overturn the decision and recommends acceptance for the paper.

The AC reminds the authors that they should update the camera ready with the suggestions from the reviewers and promises that they made during the rebuttal stage.
Decision: Accept (spotlight)