Robust Ego-Exo Correspondence with Long-Term Memory
Authors: Yijun Hu
Bing Fan
Xin Gu
Haiqing Ren
Dongfang Liu
Heng Fan
Libo Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Object Segmentation
Primary area: applications
TL;DR: We propose LM-EEC, a SAM2-based dual-memory segmentation framework that improves egocentric-exocentric object correspondence by adaptive fusion and efficient memory compression.

Abstract: Establishing object-level correspondence between egocentric and exocentric views is essential for intelligent assistants to deliver precise and intuitive visual guidance. However, this task faces numerous challenges, including extreme viewpoint variations, occlusions, and the presence of small objects. Existing approaches usually borrow solutions from video object segmentation models, but still suffer from the aforementioned challenges. Recently, the Segment Anything Model 2 (SAM 2) has shown strong generalization capabilities and excellent performance in video object segmentation. Yet, when simply applied to the ego-exo correspondence (EEC) task, SAM 2 encounters severe difficulties due to ineffective ego-exo feature fusion and limited long-term memory capacity, especially for long videos. Addressing these problems, we propose a novel EEC framework based on SAM 2 with long-term memories by presenting a dual-memory architecture and an adaptive feature routing module inspired by Mixture-of-Experts (MoE). Compared to SAM 2, our approach features **(i)** a Memory-View MoE module which consists of a dual-branch routing mechanism to adaptively assign contribution weights to each expert feature along both channel and spatial dimensions, and **(ii)** a dual-memory bank system with a simple yet effective compression strategy to retain critical long-term information while eliminating redundancy. In the extensive experiments on the challenging EgoExo4D benchmark, our method, dubbed ***LM-EEC***, achieves new state-of-the-art results and significantly outperforms existing methods and the SAM 2 baseline, showcasing its strong generalization across diverse scenarios. Our code and model are available at https://github.com/juneyeeHu/LM-EEC.

Robust Ego-Exo Correspondence with Long-Term Memory
Yijun Hul*
Bing Fan 2*
Xin Gu
Haiqing Ren
Dongfang Liu"
Heng Fan
Libo Zhang
University of Chinese Academy of Sciences
Universitt of North Texas
Institute of Software Chinese Academy of Sciences
*Rochester Institute of Technology
Equal contribution and co-first authors
'Equal advising and corresponding authors
Abstract
Establishing object-level correspondence between egocentric and exocentric views is essential for intelligent assistants to deliver precise and intuitive visual guidance. However, this task faces numerous challenges, including extreme viewpoint varia- tions, occlusions, and the presence of small objects. Existing approaches usually borrow solutions from video object segmentation models, but still suffer from the aforementioned challenges. Recently, the Segment Anything Model 2 (SAM 2) has shown strong generalization capabilities and excellent performance in video object segmentation. Yet, when simply applied to the ego-exo correspondence (EEC) task, SAM encounters severe difficulties due to ineffective ego-exo feature fusion and limited long-term memory capacity, especially for long videos. Addressing these problems, we propose novel EEC framework based on SAM with long-term memories by presenting sual-memory architecture and an adaptive feature routing module inspired by Mixture-of-Fii of-Experts (MoE). Compared to SAM 2, our approach features (i) Memory View MoE module which consists of a dual-branch routing mechanism to adaptively assign contribution weights to each expert feature along both channel and spatial dimensions, and (ii) dtul-mmeoryy bank system with a simple yet effective compression strategy to retain critical long-term information while eliminating redundancy. In the extensive experiments the challenging EgoExo4D benchmark, our method, dubbed LM-EEC, achieves new state-of-the- art results and significantly outperforms existing methods and the SAM baseline, showcasing its strong generalization across diverse scenarios. Our code and model are available at https: / //gi thub. com/ wwt.coommeeeeeeeeeeeceeeeeeeeeee htt//jjnneeeuiiiee L
Introduction
Aligning observations of objects in egocentric and exocentric viewpoints can facilitate many applica- tions. In augmented reality (AR), person wearing smart glasses could quickly pick up new skills with virtual intelligent coach that provides real-time guidance. In robotics, robot watching people in its environment is able acquire new dexterous manipulation skills with less physical experience. The recent work EgoExo4D [1] makes significant contributions to establishing object-level corre- spondence between these ego-exo views by introducing benchmark featuring annotated, temporally synchronized egocentric and exocentric videos along with object segmentation masks.
The first-person (ego) perspective captures fine-grained details of hand-object interactions and the camera wearer's attention, while the third-person (exo) perspective provides broader view of full- body poses and the surrounding environment. Therefore, this task presents significant challenges, including extreme viewpoint variations, substantial object occlusions, and the presence of numerous small objects. Previous approaches, such XSegTx [1], which formulates the task matching
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

IoU
Location Score
Contour Accuracy
2.00
0.139
0.303
2:21-
0.525
2.50
0.282
2420
640 (L99)
Ziee 2P67 Rig--aa
50l (10))
New Iite 250
Gal (00)
Figure 1: Left: Comparison of segmentation results between "View-XMen and our model, using exocentric videos as an example. Right: Quantitative results on the EgoExo4D validation set.
problem, and -View-XMen [1], which adapts XMem [2] to track objects across different views, struggle to effectively address these challenges as depicted on the left of Figure I, highlighting the need for more robust solutions. Recently, unified model for both video and image segmentation, called Segment Anything Model (SAM 2) [3], has been introduced. Trained on the SA-V dataset [3], which contains 50.9K videos with 642.6K masklets, SAM 2 demonstrates strong generalization capabilities and adaptability across a wide range of scenarios and tasks [4, 5]. The right panel of Figure 1 shows the strong performance of SAM 2 on task. "Ego-Exo" refers to segmenting objects in the exocentric videos, given the corresponding egocentric videos and annotations, while "VOS" refers segmenting objects in the exocentric videos using only the first-frame annotations.
However, SAM still faces challenges establishing object-level correspondence between temporally synchronized egocentric and exocentric videos. A key issue arises when using annotations from one view as prompts to segment the other. Effective fusion between two views is crucial. yct SAM simply adds memory-awan features and prompt embeddings together, completely overlooking the gap in different features and distribution differences between two views. Another major challenge is the scale of the task, which involves processing large volumes of long videos. Yet SAM 2 retains only a few frames closest to the current frame in its memory bank. As a result, long-term information is not effectively preserved, leading to degradation in complicated scenarios.
To overcome these limitations, we first introduce an adaptive Hfittreeoof-xperre (MoE) strategy that dynamically integrates diverse knowledge- fused features based on their distinct characteristics, enabling them to complement each other effectively. Additionally, optimizing SAM 2's memory management is crucial. Current approaches [3, 5, 6, 7]. which directly store recent selected frames in the memory bank, introduce redundant features while failing to retain long-term information. Addressing these challenges is essential to fully hamness SAM 2's capabilities for robust object segmentation in ego-exo correspondence scenarios.
we propose LM-EEC, a SAM-based robust segmentation model with dual compressed long-term memory and an adaptive fusion module. Our method introduces two key advancements: (i) flexible Memory-View Mixture-of-Expper their characteristics, enabling more effective integration of complementary information, and (ii) dual-memory bank system that separates ego and exo memory banks and leverages a specially designed compression strategy to efficiently preserve critical long-term information. In our extensive evaluation on the challenging EgoExo4D benchmark, LM-EEC achieves new state-of-the-arr results and significantly outperforms other models including SAM 2.
module that dynamically reweights expert features based on
In summary, our main contributions are as follows: We enhance SAM 2's segmentation capabilities for the ego-exo correspondence task by introducing a specialized MoE mechanism that effectively integrates features from both egocentric and exocentric views; To efficiently manage long-term dependencies, we design a dual-memory bank system that incorporates a view-specific compression strategy, effectively reducing redundancy while retaining critical long-term information; + Exten- sive experiments on EgoExo4D demonstrate that our model achieves state-of-the-ar performance, showcasing its strong generalization ability across diverse real-world scenarios.
2
Related work
Video object segmentation. Video Object Segmentation (VOS) involves tracking an object through- out video given its mask in the first frame [8]. This task is classified as "semi-supervised "OS

---

since the initial object mask serves as a supervision signal available only in the first frame. VOS has attracted significant attention due its broad applications in areas such as video editing and robotics.
Early deep barning-based models often rely on online fine-tuning [9, 10, 11, 12, 13, 14, 15, 16], either the first frame or across all frames, to adapt the model to the target object. To improve inference speed, offline- models were introduced, leveraging conditioning on either the first frame alone [17, 18] incorporating information from previous frames [19, 20J. These conditioning strategies have since been extended to all frames using RNNs [21] and transformers [22, 23, 24, 25, 2, 26, 27].
Recently, the Segment Anything Model 2 (SAM 2) [3] has emerged as a unified foundational model for promptable object segmentation in both images and videos. Notably, SAM 2 [3] has established new state-of-the-ard henchmarks across various VOS tasks, significantly outperforming previous methods. For instance, SAM2-UNet [4] introduces simple yet effective U-shaped architecture for versatile segmentation across both natural and medical domains, while SAMURAI [5], an enhanced adaptation of SAM 2 tailored for visual object tracking, achieves substantial improvements both success rate and precision over existing tracking models. Building upon this foundation, we develop our model based on SAM 2 [3] further enhance segmentation performance in our task.
Long-term video models. Long-term video understanding focuses on capturing long-range depen- dencies in the videos. The primary challenge lies in balancing the retention of crucial information while maintaining computational efficiency. To address this, a common strategy involves utilizing pre- extracted features, eliminating the need for joint training of backbone architectures [28, 29, 30, 1,,31 32]. Alternatively, some studies have explored sparse video sampling techniques [33, 34], which reduce the number of input frames by selectively preserving salient content.
Another widely adopted approach is the use of memory banks to retain relevant features, particularly in video object segmentation. For instance, XMem [2] employs multiple independent yet highly interconnected feature memory stores: a rapidly updated sensory memory. a high-resolution working memory, and compact but persistent long- tegg-termmm memory. Similarly, RMem [35] and SAM 2 [3] constrain memory banks to a limited set of essential frames, striking a balance between relevance and freshness when updating feature representations.
3
The Proposed Method
3.1 Task definition
Given a pair of synchronized ego-exo videos and sequence of query masks for an object of interest in one of the videos, the objective is to identify the corresponding masks of the same object in each synchronized frame of the other view, if visible. This task consists of two scenarios: one where the ego view serves as the query to segment the object in the exo view, and the other where the exo view serves the query to segment the object in the ego view. These two settings are referred as Ego-to-Exo correspondence and Exo-to-Ego correspondence, denoted as Ego2Exo and Exo2Ego, respectively, the following for brevity.
Formally, taking Exo2Exx as an example, let an ego-view video with 7 frames and a corresponding exo-view video with T' frames be represented as {4, G#90 ] and {4[=0) yT= respectively. Here, G denotes the ground-truth object mask of frame 18500 Given these synchronized frames, the goal segment the object masks Mero in each frame of the exo-view video.
3.2 Preliminary on SAM 2
SAM 2 [3] begins with an image encoder that encodes each input frame into embeddings. In contrast to SAM [36], where frame embeddings directly fed into the mask decoder, SAM 2 designs memory module that conditions features of the current frame on both previous and prompted frames.
Specifically, for semi-supervised video object segmentation tasks, SAM 2 maintains memory bank each time step 2
M, (M, E R KxC
(I)
TEI
3

---

Dual memory compeeesion
Dual memory bank
Exo memory compression
Exo Hemory Bank
Exocentric Video
Multi-view Encoder
Ego memory
Ego Hemory
encode
image
module
attention
compression
Memory- View MoE module
Object Mask
Egocentric Vidco
shured
sbared
Prediction
mask decoder
incodar
image
module
attention
Friew
F
Lar
oncdder mommoy
Meko
Figure 2: Overview of our proposed model, which consists of three key components: multi-view encoding, dual memory compression, and object mask prediction. The multi-view encoder extracts features from egocentric and exocentric videos, using a Memory- View MoE module to adaptively combine memory-awan and view-specific representations. The object mask prediction module then generates segmentation masks for objects in the exocentric view. To capture long-term dependencies efficiently, we apply dual memory compression on the memory banks from both viewpoints.
where K is the number of memory tokens per frame, is the channel dimension, and I is the set of frame indices included in the memory. SAM 2, the memory bank stores up of the most recent frames, along with the initial mask, using First-In-Firt-OOu (FIFO) queue mechanism.
The encoded frame feature would be fused with the memory bank through the memory attention module, which is stacked by L transformer blocks, facilitating full interaction between the two. After this memory fusion, if mask prompts are present, the fused feature summed with the dense prompt embedding, which is then processed by the convolutions of the prompt encoder.
3.3 Overview
Figure 2 provides overview of proposed model LM-EEC. Taking Ego2Exo example, given the frames and target object masks from the ego view along with the corresponding frames from the exo view (left side of Figure 2), the workflow then unfolds follows:
First, the video frames from both views are encoded using the image encoder. The feature representa- tions of the current and ego frames then passed through the dual attention module, where the exo feature is integrated separately by the attention module with the stored features from both the ego and memory banks, well as the corresponding annotated ego feature which is encoded by the off-the-shelf memory encoder. Once the memory-aware and view-specific features are obtained, the Memory- View Mixture-of- -f-ppers module(sec 3.4) adaptively integrates the two expert features. The fused feature is subsequently processed by the mask decoder to generate the predicted object mask.
Finally, the predicted mask, along with the given ego- mask and their corresponding encoded features, are fed into the memory encoder to generate memory frames, which are stored in the ego and exo memory banks. As the dual memory banks have fixed capacity, the tailored memory compression mechanism(sec 3.5) triggered when the stored frames exceed this limit, ensuring efficient memory management.
3.4 Memory- View Mixture-of-F Experts module
SAM simply adds together the prompt embedding generated by the prompt encoder and the memory-awan feature. In our task, the memory aware feature and the view-specific feature, which

---

Pmem
Channel-wise Router
mmmm
Spatial-wise
Fimm
Router
MLPI
Convl
Weem
Wimm
Fear
MLP2
W
Conv2
W
siew
Felew
Fview
Friew
Channel-wit
Element wise Multiplication
Element
Concatenation
Addition
Figure 3: Overview of the proposed Memory- -Veew Mixture-of-FF of-ure--o--xpeertt Experts (MV-MoE) module. Channel- and spatial-wisg -wise routers generate dynamic weights to recalibrate memory-aware and view-specific features, enabling adaptive and context-aware fusion of complementary information.
prompt information, highlight different regions and have different feature distributions. This may cause the prompt information overwhelm the representation the other branch, leading to less discriminative and robust target representations.
To alleviate this issue, we propose the Memory-Viev Mixture-of-Experte (MV-MoE) module, which treats the memory-awan and view-specific features as two complementary experts. Rather than performing direct fusion, we design a lightweight dual-branch routing mechanism to adaptively assign contribution weights to each expert along both channel and spatial dimensions, allowing the network to determine the relative importance each expert in a data-dependent manner.
Specifically, as shown in Figure 3, given the memory aware feature Fmem IR Axxx and view- specific feature Foiew R wxe channel-wiss routing is performed firstly. We concatenate the two features along the channel dimension and apply global average pooling to obtain compact descriptor. This descriptor then passed through two separate MLPs each consisting of two linear layers, with ReLU and sigmoid activations to generate dynamic channel-wis importance weights w mems W modulation:
view E Rlxixe These weights are used to recalibrate each expert through residual
w "mem/view /view =MLP1,2 (Agg(CCcclll Fmem: Fview)))
Fmem/euee /eiew = w "mem/wiee /wiew @ F F /vee view + Fmmm views
(3)
"mem//
where Concat(.) and Aug(.) denote channel-wit concatenation and global average pooling, respec- tively, and @ represents element-wise multiplication. This operation adaptively emphasizes the more informative channels each expert while preserving the original feature content.
Subsequently, spatial-wise routing is applied to the channel-enhanced features Finm and Friews The two features are concatenated and fed into two parallel convolutional branches each composed of Conv-ReLU-Conv-Spp sequence-to generate spatial attention maps w Wmems w Witew E 1xx=x1 These spatial weights are then used to further refine each feature spatially, again using residual modulation:
w mem / /iiewe Cooou (ConeattCCo Foiew)),
F Feem /view w mmmm/ /view F Finem/vic /vie/ + Finem /vicus
(5)
Finally, the refined memory-aware and view-specific features are summed to produce the fused target feature which is then forwarded to the decoder for subsequent segmentation:
Ftar Fmem + Feieu-
(6)
These routing strategies enable the model to adaptively modulate the contributions memory- aware and view-specific features across both spatial and channel dimensions, aligning with the core intuition the Mixture-of-EEaee Experts (MoE) p selectively leveraging complementary sources of knowledge based on the input context. Unlike conventional MoE architectures that rely multiple sub-networks and sparse expert activation, our design performs dense, feature-level expert routing, which preserves the adaptive expert weighting behavior of MoE while avoiding the complexity of network-level sparsity. This fine-grained and lightweight formulation enables context-awan fusion and enhances the model's flexibility in dynamic video scenarios.
5

---

2's memory bank
f1
f2
f3
f*
f
f
pk+
ft-1
ft
memory bank by compression
Step I: Compute the Euclideun distance between corresponding feature points in adjacent memory frames.
Step 2: Select the feature pairs each spatial position with the smallest Euclidean distance.
Step Average selected features pairs.
kk
k
1.2
kk+
p++1
f
f2
0.6
2
2
k+22
fk
2
k++
f
pk+2
ft + +
fz
ck+2
fhw
1.6
pk+1
0.7
ck+2
flw
pk+1
pk+2
fhw
hw
Jhw
hw
Figure 4: An illustration of our memory bank compression strategy, which preserves fixed memory size in both ego-view and exo-view memory banks by aggregating temporally redundant information.
3.5 Dual Memory Compression
Previous work [1] constructs a single memory bank that stores both egocentric and exocentric features uniformly. However, egocentric and exocentric videos differ substantially in terms of viewpoints, motion patterns, and visual appearance. better leverage their complementary characteristies, we first design a dual memory bank based on SAM 2, in which ego and exo features are stored separately.
SAM maintains the most recent N frame features in its memory bank. Although effective in the short-term context, this strategy suffers from several limitations. As segmentation progresses, the memory bank gradually loses alignment with earlier frames, reducing temporal consistency. Meanwhile, since the target object typically occupies only a small region of each frame, the memory bank becomes saturated with redundant spatial information. More critically, this unified memory design fails to differentiate between egocentric and exocentric views, thereby undermining our dual-memory objective.
To address these challenges, we propose a view-specific memory compression strategy that sepa- rately compresses the ego and memory banks. This approach reduces redundancy, preserves discriminative long-term features, and improves both efficiency and robustness.
Figure 4 illustrates our memory compression process. The core idea is to temporally aggregate and compress video features by exploiting similarities between adjacent frames, while retaining informative long-term tengtermm content. This enables maintain compact and discriminative memory representations for both ego and exo views.
The compression algorithm is applied at each auto-regressive iteration whenever the length of the ego/exo memory bank exceeds a pre-defined threshold M. Formally, given an egolexo memory bank consisting of the following sequence:
JMI, f ERRP
where P h x w, representing the spatial dimensions of each memory frame feature. When new frame feature pM+1 arrives, the memory bank must be compressed by reducing its length by one.
For each spatial location i, we first compute the Euclidean distance between the corresponding feature points in the temporally adjacent memory frames:
df Eucli(fi, tE [L, M], ie [I,
(8)

---

Next, we identify the most temporally redundant features by selecting the feature with the minimum Euclidean distance, indicating the highest similarity across time:
k arg min(d)), E [1.
Finally, to reduce the memory bank length by one, we perform a simple feature averaging at each spatial location:
(10)
2
This three-in-one approach not only reduces redundancy but also preserves long-term information. Furthermore, adaptively updates the memory bank based on scene variations in each viewpoint, ensuring a compact and informative memory representation.
4 Experiments
4.1
Dataset
We conduct experiments the challenging EgoExo4D benchmark [1]. EgoExo4D is dataset specifically designed for the ego-exo correspondence task. It contains approximately 5.5K annotated objects across 1.3K temporally synchronized egocentric and exocentric video pairs. In total, around 4 million frames have been annotated, resulting in 742K ego-view and 1.IM exo-view paired segmentation masks. We adopt the official dataset split in our experiment, where 756 videos are used for training, 202 for validation, and 291 for testing.
4.2 Metrics
Following [1], employ four evaluation metries, including Intersection Over Union (IoU) between the predicted and ground-truth masks, Location Error (LE), which is defined as the normalized distance between the centroids of the predicted and ground-truth masks, Contour Accuracy (CA), which measures how well the predicted masks match the ground-truth masks on the boundary, and Existence Balance Accuracy (BA), which evaluates the method's 's ability to estimate object visibility in the target view, since, in practice, objects may often be occluded or fall outside the field of view.
4.3 Implementation Details
Our model is built upon the official SAM 2 base [3], with key modifications including the introduction of Memory-View MoE module and a refined memory bank construction strategy. Following the training protocol of SAM 2, we sample 8 consecutive frames for each object from ego-exo video pairs, and set the memory bank size to 6. To reduce computational overhead due to the large resolution of original video frames. we resize all frames to 480 480, following the practice in [1]. Given the large-scale nature of EgoExo4D, we train all modules jointly based on the pre-trained SAM 2 checkpoint, without freezing any components. The model is trained for 60 epochs on 8 NVIDIA A100 GPUs with batch size of 32 and evaluated on a single V100 GPU, achieving an inference speed of approximately 8.4 FPS. Memory compression is applied only during inference. Our code and model are available at https: //githuttttttttt com/ mamjjuna juneyeeHu/LLM LiteeLMMEERe EEC.
4.4 Comparison to State-of- Methods
Table presents the quantitative results in comparison with previous methods on the test set. We experiment with two settings: providing the ground- -ruth object track in the exo view (exo query mask) and predicting in the ego view Exo2Ego), and vice versa. Given the limited availability of existing
models tailored for this task, we selected several VOS models from 2021 onward [37, 38, 39, 40, 41] and adjusted their architectures from the other view with annotations before segmenting the current frame. All models are trained or fine-tuned on the EgoExo4D dataset to ensure a fair comparison. To provide a more comprehensive comparison, we additionally report the performance of our base model, which comprises only standard mual-memory bank and simple prompt addition, to intuitively highlight the effectiveness of our proposed method. As shown in Figure 5, we also evaluate and compare the segmentation
manner similar XView-XMem [1], namely, integrating features
7

---

Table Comparison for the ego-exo correspondence benchmark on test Best results are reported
in bold, whereas our results are highlighted in cyan and the results of the base model are underlined.
Query Mask
Method
loUt
LE
CAt
BAt
Ego
XSegTx [1]
18.99
0.070
0.386
66.31
Ego
XView-Xmem (w/ finetuning)
14.84
0.115
0.242
61.24
Ego
XVee--Xmem (+ XSegTxx) [1]
34,90
0.038
0,559
66.79
Ego
STCN [37]
27.39
0.109
0.378
61.97
Ego
QDMN [38]
27.03
0.108
0,346
63.16
Ego
GSFM [39]
3.98
0.146
0.057
52.39
Ego
VOS [40]
38.26
0.090
0,481
57.21
Ego
Cutie [41]
27.03
0.108
0.346
59.18
Ego
Base model
52.13
0.024
0.734
61,56
Ego
I.M-EEC(Ours)
54.98
0.017
0.778
64,22
Exo
XSegTx [1]
27.14
0.104
0.358
82.01
Exo
XView-Xmem (w/ finetuning)
21.37
0.139
0.269
61.72
Exo
XView-Xmem (+ XSegTx) [1]
25.00
0.117
0,327
59.71
Exo
STCN [37]
27.61
0.122
0,348
65.98
Exo
QDMN [38]
17.56
0.158
0,213
58.94
Exo
GSFM [39]
13.78
0.164
0.169
59.53
Exo
VOS [40]
40.67
0.099
0,481
66.62
Exo
Cutie [41]
47.52
0.070
0.579
70.71
Exo
Base model
57.27
0.047
0.677
57.11
Exo
LM- L.M-EEC(Ours)
65.77
0.031
0.774
58.14
Higo
Conttur Accurny
STON
STCM
Serviss
Cate
Cutee
DOMN NanvOn
Dgpd
0M0 Saat PONLNON
Okject VOLCETT
Figure 5: Performance evaluation across different object sizes in the target (exo) view, including IoU, shape accuracy, and location score.
performance of our method across varying object sizes under the Ego2Exo task. Specifically, divide the test set based on the proportion object pixels within each frame. Furthermore, we provide visual results, conduct qualitative comparisons with other baselines, and analyze the model's performance across diverse activity scenarios in the EgoExo4D dataset. For more details, please kindly refer the appendix.
Overall, our model consistently outperforms all baselines across different object sizes and delivers substantial improvements over the base model, particularly in the Exo2Ego task. XSegTx [1] achieves a high level of balanced accuracy because is a co-segmentation model and employs specialized data augmentation techniques during training.
4.5 Ablations
We conduct ablation studies on the validation set of EgoExo4D the Ego2Exo task. Evaluation metrics are reported, with our final configuration highlighted in cyan
MV-MoE module Table 2 presents the ablation study our MV-MoE module. We compare two different experimental settings: (1) excluding the prompt from other view of the current frame, just using the memory-aware feature for segmentation, (2) directly adding the prompt and the memory-aware feature, as done SAM Our results indicate that simple prompt addition fails to fully exploit the effectiivnnes of the prompts. In contrast, proposed module leverages routing

---

Table 2:
Ablation fusion mechanism.
Table 3: Ablation training frame number.
Setting
loUt
LE
CAT
Frame
IoUt
LE]
CA
w/o other view simply add (base model)
0.5691
0.0276
0.7660
0.5857
0.0204
0.7931
0,5673
0.0258
0.7643
0,5925
0.0198
0.8006
MV-MoE
0.5925
0.0198
0.8006
0.5927
0.0201
0.8003
Table 4: Ablation on our memory stores.
Table 5: Ablation memory size.
Setting
IoUt
L.E.L.
CA1
Memory
loUt
LEL
CA
All memory stores
0.5925
0.0198
0,8006
4
0.5921
0.0205
0.8004
No ego mem.
0.5748
0.0220
0.7837
6
0.5925
0.0198
0.8006
No
0.5420
0.0293
0.7495
8
0.5963
0.0196
0.8025
mechanism selectively emphasize and integrate key information from both sources, leading to more effective fusion. In appendix, we also demonstrate the modularity of MV-MoE by integrating it into other backbones.
Memory stores. Table 4 summarizes the performance of our model when each memory bank is ablated individually. The results clearly demonstrate the effectiivneess of our dual-memory design and underscore the importance of integrating both egocentric and exocentric views to achieve robust perception in complex scenarios.
Memory compression strategies. Table 6 compares dif- ferent memory compression strategies. The FIFO (first-in, first-out) approach stores the most recent N frames in the memory bank. The loU selection strategy [5] follows conditional FIFO approach based on IoU scores predicted by SAM 2. In contrast, the cluster selection method first applies average pooling to all frames, followed by cluster- ing identify and retain the most representative frames. Notably, these methods operate at the frame level, whereas our proposed method performs compression the feature point level. The results demonstrate that our strategy more effectively aids the model in segmenting objects.
Table 6: Ablation on our memory com- pression strategy.
Setting
IoUt
LE
CAt
FIFO
0,5823
0.0214
0.7874
Cluster
0.5867
0.0208
0.7942
IoU Sel.
0,5880
0.0204
0.7948
Ours
0.5925
0.0198
0.8006
Frame number and memory size. As show in Table 3, increasing the number of frames brings default of to balance speed and accuracy.
gains although there could be some variance and Meanwhile, increasing the (maximum) number of memories M, generally helps the performance, in Table 5. We use default value of 6 past frames strike balance between temporal context length and computational cost.
Due to space limitation, include more results, analysis, and discussion in the appendix. Please kindly refer to our appendix for details.
5 Conclusion
In this paper, we present LM-EEC, novel framework that builds upon the strong generalization capabilities of SAM to effectively tackle challenges such as severe object occlusions and the presence of numerous small objects. To better exploit multi-view information, LM-EEC introduces Memory- -Veww Mixture-of-Eppert module that dynamically integrates egocentric and exocentric fea- tures. In addition, enhance the memory management mechanism of SAM 2 by constructing dual long-term memory bank using a view-specific compression strategy, which preserves discriminative information while reducing redundancy. Extensive experiments the EgoExo4D benchmark show that LM-EEC consistently outperforms prior approaches, especially in complex and cluttered scenes.
Acknowledgment. Libo Zhang was supported by National Natural Science Foundation of China (No. 62476266). Heng Fan was not supported by any fund for this work.

---

References
[1] Kristen Grauman, Andrew Westhury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyl- los Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, Eugene Byrne, Zachary Chavis, Joya Chen, Feng Cheng, Fu-Jen Chu, Sean Crane, Avijit Dasgupta, Jing Dong, María Escobar, Cristhian Forigua, Abrham Gebreselasie, Sanjay Haresh, Jing Huang, Md Mohaiminul Islam, Suyog Dutt Jain, Rawal Khirodkar, Devansh Kukreja, Kevin J. Liang, Jia-Wei Liu, Sagnik Majumder, Yongsen Mao, Miguel Martin, Effrosyni Mavroudi, Tuuharr Nagarajan, Francesco Ragusa, Santhosh Kumar Ramakrishnan, Luigi Seminara, Arjun So- mayazulu. Yale Song. Shan Su, Zihui Xue, Edward Zhang. Jinxu Zhang, Angela Castillo, Changan Chen, Xinzhu Fu. Ryosuke Furuta, Cristina González, Prince Gupta, Jiabo Hu, Yifei Huang, Yiming Huang, Weslie Khoo, Anush Kumar, Robert Kuo, Sach Lakhavani, Miao Liu, Mi Luo, Zhengyi Luo, Brighid Meredith, Austin Miller, Oluwatumininu Oguntola, Xiaqing Pan, Penny Peng. Shraman Pramanick, Merey Ramazanova, Fiona Ryan, Wei Shan, Kiran K. Somasundaram, Chenan Song. Audrey Southerland, Masatoshi Tateno, Huiyu Wang. Yuchen Wang, Takuma Yagi. Mingfei Yan, Xitong Yang. Zecheng Yu, Shengxin Cindy Zha, Chen Zhao, Ziwei Zhao, Zhifan Zhu, Jeff Zhuo, Pablo Arbeláez, Gedas Bertasius, Dima Damen, Jakob J. Engel, Giovanni Maria Farinella, Antonino Furnari, Bernard Ghanem, Judy Hoffman, C. Jawahar, Richard A. Newcombe, Hyun Soo Park, James M. Rehg, Yoichi Sato. Manolis Savva, Jianbo Shi, Mike Zheng Shout, and Michael Wray. Ego-exo4d: Understanding skilled human activity from first- and third-persoe person perspectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19430, 2024.
[2] Ho Kei Cheng and Alexander G Schwing. Xmem: Long-term video object segmentation with an aukinson-shiffrin memory model. In European conference on computer vision, pages 640-658. Springer, 2022.
[3] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything images and videos. arXiv preprint arXiv:2408.0007 2024.
[4] Xinyu Xiong, Zihuang Wu, Shuangyi Tan, Wenxue Li, Feilong Tang, Ying Chen, Siying Li, Jie Ma, and Guanbin Li. Sam2-unet: Segment anything 2 makes strong encoder for natural and medical image seg- mentation. arXiv preprint arXiv:2408.0888 2024.
[5] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alexander Sorkine-Hornung and Lue Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:2411. / 1922, 2017.
[6] Shuangrui Ding, Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Yuwei Guo, Dahua Lin, and Jiaqi Wang. Sam2long: Enhancing sam 2 for long video segmentation with a training-free free memory tree. arXiv preprint arXiv:2410.16268, 2024.
[7] Jiayuan Zhu, Abdullah Hamdi, Yunli Qi, Yueming Jin, and Junde Wu. Medical sam 2: Segment medical images as video via segment anything model 2. arXiv preprint arXiv:2408.0053 00874v2, 2024.
[8] Cheng- Yen Yang, Hsiang-Wai Huang, Wenhao Chai, Zhongyu Jiang, and Jenq-Neng Hwang. Samurai: Adapting segment anything model for zero-shot visual tracking with motion-aware memory. arXiv preprint arXiv: / 00675,, 2024.
[9] Sergi Caelles, Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Laura Leal-Taixé, Daniel Cremers, and Luc Van Gool. One-shot video object segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 221-230, 2017.
[10] Federico Perazzi, Anna Khoreva, Rodrigo Benenson, Bernt Schiele, and Alexander Sorkine- Hornung. Learning video object segmentation from static images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2663-2672, 2017.
[11] Jae Shin Yoon, Francois Rameau, Junsik Kim, Seokju Lee, Seunghak Shin, and In So Kweon. Pixel-level matching for video object segmentation using convolutional neural networks. In Proceedings of the IEEE international conference computer vision, pages 2167-2176, 2017.
10

---

[12] K-K Maninis, Sergi Caelles, Yuhua Chen, Jordi Laura Leal-Taixé, Daniel Cremers, and Luc Van Gool. Video object segmentation without temporal information. IEEE transactions on puttern analysis and machine intelligence, 41(6):1515-1530 2018.
[13] Yuan-Ting Hu, Jia-Bin Huang, and Alexander Schwing, Maskmmn: Instance level video object segmentation. Advances in neural information processing systems, 30, 2017.
[14] Goutam Bhat, Felix Järemo Lawin, Martin Danelljan, Andreas Robinson, Michael Felsberg. Luc Van Gool, and Radu Timofte. Learning what to learn for video object segmentation. In European Conference on Computer Vision, pages 777-794. Springer, 2020.
[15] Andreas Robinson, Felix Jaremo Lawin, Martin Danelljan, Fahad Shahbaz Khan, and Michael Felsberg. Learning fast and robust target models for video object segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7406-7415, 2020.
[16] Paul Voigtlaender and Bastian Leibe. Online adaptation of convolutional neural networks for video object segmentation. arXiv preprint arXiv:17066 09364, 2017.
[17] Yuan- Ting Hu, Jia-Bin Huang, and Alexander G Schwing. Videomatch: Matching based video object segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 54-70, 2018.
[18] Yuhua Chen, Jordi Pont-Tusel Alberto Montes, and Luc Van Gool. Blazingly fast video object segmentation with pixel-wise metric learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1189-1198, 2018.
[19] Seoung Wug Oh, Joon- Joon-Young Young Lee, Kalyan Sunkavalli, and Seon Joo Kim. Fast video object segmentation by reference-guiee -uided mask propagation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 7376-7385, 2018.
[20] Zongxin Yang, Yunchao Wei, and Yi Yang. Collaborative video object segmentation by foreground-background integration. In European Conference on Computer Vision, pages 332- 348. Springer, 2020.
[21] Ning Xu, Linjie Yang, Yuchen Fan, Jianchao Yang, Dingcheng Yue, Yuchen Liang, Brian Price, Scott Cohen, and Thomas Huang. Youtube-vos: vos: Sequence-to-sequenee video object segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 585-601, 2018.
[22] Scoung Wug Oh, Joon- Loon-YYung Lee. Ning Xu, and Scon Joo Kim. Video object segmentation using space-time memory networks. In Proceedings of the IEEE/CVF international conference on computer vision, pages
2019.
[23] Ho Kei Cheng, Yu-Wing Tai, and Chi- Chi-Keung Tang. Rethinking space-time networks with improved memory coverage for efficient video object segmentation. Advances in neural information processing systems, 34:11781-11794 2021.
[24] Mingxing Li, Li Hu, Zhiwei Xiong, Bang Zhang, Pan Pan, and Dong Liu. Recurrent dynamic embedding for video object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1332-1341, 2022.
[25] Zongxin Yang, Yunchao Wei, and Yang. Associating objects with transformers for video object segmentation. Advances in Neural Information Processing Systems, 34:2491-2502, 2021.
[26] Zongxin Yang and Yi Yang. Decoupling features in hierarchical propagation for video object segmentation. Advances in Neural Information Processing Systems, 35:36324-36336, 2022.
[27] Qiangqiang Wu, Tianyu Yang, Wei Wu, and Antoni B Chan. Scalable video object segmentation with simplified framework. In Proceedings of the IEEF/CVE international conference on computer vision, pages 13879-13889, 2023.

---

[28] Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan. Kate Saenko, and Trevor Darrell. Long-term recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2634, 2015.
[29] Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijyanarasiimann Oriol Vinyals, Rajat Monga, and George Toderici. Beyond short snippets: Deep networks for video classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4694-4702, 2015.
[30] Rohit Girdhar, Deva Ramanan, Abhinav Gupta, Josef Sivie, and Bryan Russell, Actionvlad: Learning spatio-temporal aggregation for action classification. In Proceedings of the IEEE conference on computer vision and patlern recognition, pages 971-980) 2017.
[31] Noureldien Hussein, Efstratios Gavves, and Arnold WM Smeulders. Timeception for complex action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 254-263, 2019.
[32] Chao-Yuar Wu and Philipp Krahenbuhl. Towards long-form video understanding. Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1884,1894, 2021.
[33] Bruno Korbar, Du Tran, and Lorenzo Torresani. Scsampler: Sampling salient clips from video for efficient action recognition. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 6232-6242, 2019.
[34] Zuxuan Wu, Caiming Xiong, Chih- Yao Ma, Richard Socher, and Larry S Davis. Adaframe: Adaptive frame selection for fast video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1278-122 1188,, 2019.
[35] Junbao Zhou, Ziqi Pang, and Yu-Xiong Wang. Rmem: Restricted memory banks improve video object segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18602-18611 2024.
[36] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander Berg, Wan Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015 2023.
[37] Ho Kei Cheng, Yu-Wing Tai, and Chi-Keung Tang. Rethinking space-time networks with improved memory coverage for efficient video object segmentation. Advances in neural information processing systems, 1111111111994, 2021.
[38] Yong Liu, Ran Yu, Fei Yin, Xinyuan Zhao, Wei Zhao, Weihao Xia, and Yujiu Yang. Learning quality-aware dynamic memory for video object segmentation. In European Conference on Computer Vision, pages 468 486. Springer, 2022.
[39] Yong Liu, Ran Yu, Jiahao Wang. Xinyuan Zhao, Yitong Wang, Yansong Tang, and Yujiu Yang. Global spectral filter memory network for video object segmentation. In European Conference on Computer Vision, pages 468-665.66 Springer, 2022.
[40] Qiangqiang Wu, Tianyu Yang, Wei Wu, and Antoni B Chan. Scalable video object segmentation with simplified framework. In Proceedings of the IEEE/CVE international conference on computer vision, pages 13879-13889, 2023.
Joon-Young [41] Ho Kei Cheng. Scoung Wug Oh, Brian Price, Joon- Young Lee, and Alexander Schwing. Putting the object back into video object segmentation. In CVPR, 2024.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims in the abstract and introduction is that we propose the LM- EEC framework for ego-exo correspondence task, and the paper is all about introducing its design and effectiveness.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations of the work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiaatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
13

---

Answer: [NA]
Justification: The paper does not propose new theorem. We conduct experiments and show illustrations evaluate our model design.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer to Section 4 for experiment settings and implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. .., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
14

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Our code and model are available httpss///gtthubb..o///jjmymyeee LM-EEC.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //ippsccc/ppppp wttpcc/pppplcccc wwwpc//gggeeeeeeeeeee www.odSuumssssiooooee....
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer to Section 4 for experiment settings and implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We do not report error bars, as they are not commonly included in prior literature, and running multiple trials on the scene-level dataset is computationally expensive. Nevertheless, conduct extensive experiments consistently observe performance improvements over existing methods. These consistent gains across diverse settings provide strong evidence supporting the statistical significance our results.
the large-scale EgoExo4D dataset and
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Section 4 for experiments compute resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: //tpurip////ur........ http://twrrp...cppiiiec/ee/e//ea haabbliieEthisstuddeleeeeeeeee
Answer:
[Yes]
Justification:
We have carefully read the Ethics Guidelines and strictly followed them.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the societal impact of the work in appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
16

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited the original paper of the pre-trained models and data we used.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode. thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
17

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
18

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https:/ /ttps/////euu...eeeeee www.arips.c.oooefeeeeeeeeeeeeeee /00/////eeneeee/225////// 20LL/LLM for what should or should be described.
19

---

Appendix
The appendix is structured as follows:
Section A: We first analyze model performance across diverse activity scenarios to evaluate its generalization capability.
Section B: We then provide qualitative results, comparing the proposed LM-EEC to base- lines.
Section C: We discuss the limitation of model and reflect its broader impact.
Section D: We also visualize attention maps demonstrate how our model establishes precise object-level correspondence between egocentric and exocentric views, enabling cross-view alignment.
We validate long-term ego-exo correspondence using association accuracy over videos of varying lengths.
Section E:
We demonstrate the modularity of MV-MoE by integrating it into alternative backbones, showing the architectural transferability.
Section F:
Section G: We also provide basic training configurations for all compared methods.
Cooking
STCN
Cooldnng
STCN
0DMN
Bike Repeir
Cutie
Bike Repóir
Outle
Basketbol
Basketball
Health
Health
Location Score
Ours
Bal. Acc. 1
Ours
STCN
Cooldng
ODMN
QDMN
SimVDS
SimVOS
Repair
Repoir
Musi
Music
Basketbell
BesketDell Dell
Figure 6: Comparison across different activity scenarios for each model.
20

---

QPMN
SmmVO
Ours
Figure 7: Ego to Exo results of different approaches.
A Analysis on Different Scenarios
Beyond object size, we further analyze model performance across different activity scenarios in EgoExo4D, which consists of Cooking, Basketball, Bike Repair, Health, and Music. We compare the results of various models across these scenarios to assess their robustness and adaptability.
As shown in Figure 6, some activities, such as Basketball, are generally easier to model due to limited variation in object shape and appearance. In contrast, activities like Cooking and Bike Repair pose greater challenges, objects exhibit more diverse appearances and shapes across different views.
Notably, in these more challenging scenarios, our model demonstrates clear advantage. The perfor- mance gap between our method and other baselines is more pronounced in these cases, highlighting the effectiveness of our approach in handling complex and highly variable environments.
Qualitative Results Comparison
We further compare the qualitative results of our model with other baselines. As shown in Figure 7 and 8, while some models struggle to fully segment the object (e.g., the stainless steel spatula) or to distinguish it from similar objects in the background, effectively handles occlusion and variations in its appearance.
model accurately segments the object and
C Limitation and Broader Impact
Discussion Limitation. As observed, our model does not achieve the highest performance in terms of balanced accuracy. This indicates that when the target object disappears from the scene, the
21

---

Ground Truth
STCN
QDMN
SimVOS
Cutie
Ours
Figure 8: Exo to Ego results of different approaches.
model may mistakenly segment visually similar background objects. Representative cases of such failures are illustrated in Figure 9.
This limitation primarily stems from the fact that existing baselines do not explicitly consider object disappearance during training. As result, the model tends associate and segment visually similar objects in the current frame, even when the target object no longer present. This observation reveals an inherent trade-off among IoU, contour accuracy, location score, and balanced accuracy.
In this work, we build upon SAM as the baseline architecture to develop our model. While SAM demonstrates strong matching capability and performs effectively in video segmentation, it lacks explicit mechanisms to handle object disappearance. Consequently, our model inevitably inherits this limitation to some extent. We believe that incorporating targeted data augmentation during training
22

---

Ground Truth
Prediction
Ground
Prediction
Figure 9; Failure cases.
Frame to be segmented
Corresponding ego frame with annotations
Heatmap of attention map
Figure 10: Attention map visualization.
introducing additional branch to explicitly predict object presence absence could be promising directions to address this issue in future work.
Discussion of Broader Impact. The proposed LM-EEC which focuses establishing object- level correspondence between egocentric and exocentric views has the potential to benefit a wide range of real-world applications, including intelligent robotics, augmented reality (AR), and assistive technologies. By enhancing the ability of Al systems to reason about objects from multiple viewpoints, LM-EEC could contribute more accurate and intuitive human-Al interaction, such guiding users through complex tasks or supporting remote collaboration.
It's worth noting that work carries potential risks, particularly concerning privacy and surveillance, as the ability to associate first-person and third-person views could be misused for unauthorized tracking monitoring of individuals beyond intended applications. Therefore, emphasize that our work must strictly applied within ethical and privacy complict contexts.
D Attention Map Visualization
In addition, we visualize the attention map of both the current frame and the corresponding annotated frame. Specifically, display the attention map related to the current target be segmented from
23

---

Table Association accuracy of our method and the baseline under different video lengths.
Video Length
200 frames
200-600 frames
>600 frames
Baseline Model
0.5580
0.6260
0.4724
Our Model
0.6054
0.6732
0.6150
an alternate view. As shown in Figure 10, model accurately matches objects from both ego and cxo views (c.g., the guitar).
E Long-term Correspondence Evidence
To verify the effectiivneess of method in modeling long-term ego-exo correspondence, conduct an experiment to evaluate the association accuracy across videos of different lengths. Specifically, compare our model with the baseline (i.e., SAM 2) that excludes the core MV-MoE and memory compression modules.
In this experiment, the association accuracy is defined as the ratio of frames where the Intersection over Union (IoU) between the predicted and ground truth masks exceeds threshold of 0.5. The comparison results are summarized in Table 7.
As shown in Table 7, our model consistently outperforms the baseline across all video lengths. Notably, for longer videos with more than 600 frames, our method achieves a significant improvement of approximately 14% in association accuracy compared with the baseline model. This result effectively validates the capability of our approach to maintain robust long-term ego-exo correspondence over extended temporal durations.
F
Modularity of MV-MoE
While the proposed MV-MoE. is initially designed on top of the SAM architecture, its core component, i.e., the dual-branch routing mechanism that adaptively assigns contribution weights to each expert feature along both channel and spatial dimensions, is conceptually modular and can be readily integrated into other architectures that involve multi-source feature fusion. To demonstrate the generality of MV-MoE, we conduct additional experiments by integrating it into two alternative frameworks with different segmentation bones, including STČN and QDMN, the Ego-Exo Correspondence task. The results on the Ego2Exo test split of the EgoExo4D dataset are shown in Table 8.
From Tab. B, we can clearly observe that, incorporating MV-MoE consistently enhances performance across metrics, indicating that the module not only effective but also transferable to different backbone architectures. This suggests that MV-MoE can be easily adapted to other segmentation wechttetures/backkones and future versions of SAM 2-like frameworks.
Table 8: Application of MV-MoE on other segmentation backbones.
Model
IoU 1
LE 4
CA f
BA f
STCN
27.39
0.109
0.378
61.97
STCN MV-MoR
30.51
0.083
0.436
62.81
QDMN
27.03
0.108
0.346
63.16
QDMN + MV-MoE
28.75
0.100
0.376
64.02
G Basic Training Configurations
All baselines were implemented from official code with minimal changes for EgoExo44, following original training settings. Table summarizes the training configurations of our method and other approaches. Notably, all the compared method are trained for more epochs/iterations than our

---

proposed model to ensure sufficient optimization. Our goal is to ensure fair and optimal performance for all methods on EgoExo4D.
Table 9: Training configurations of compared methods.
Model
Epochs (Iterations)
Optimizer
XSegTx
250 (250K iter.)
Adam
X XView-Xmen (w/ finetuning)
358 (100K iter.)
Adam W
XView-Xmem (+ Xx)
358 (100K iter.)
Adam W
STCN
258 (50K iter.)
Adam
QDMN
207 (80K iter.)
Adam
GSFM
213 (55K iter.)
Adam
SOs
193 (300K iter.)
Adam
Cutie
167 (125K iter.)
Adam W
LM-EEC(OUrs)
60 (12K iter.)
Adam
25


Reviews:
[{"review_id": "NA8D6bjAS2", "reviewer": "Reviewer_HNsS", "summary": "The paper tackles ego–exo object correspondence by extending Segment Anything Model 2 with a specialised framework called LM-EEC. The method introduces (i) a Memory-View Mixture-of-Experts module that dynamically fuses egocentric and exocentric features across channel and spatial dimensions, and (ii) a dual-memory bank with compression to preserve long-term context while pruning redundancy. Evaluated on EgoExo4D, LM-EEC sets new state-of-the-art performance and surpasses both SAM 2 and recent video-segmentation baselines", "questions": "Please check above weaknesses.", "limitations": "Limitations have been provided in appendix.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**\n\n- The manuscript is clear and engaging; the motivation naturally leads to the proposed methodology.\n\n- Figures are well designed and effectively convey the pipeline, making the approach easy to follow.\n\n- The Memory-View Mixture-of-Experts module is a thoughtful architectural addition that appears carefully engineered.\n\n- The experimental section is thorough, combining strong baseline comparisons with detailed ablations that convincingly demonstrate the method’s effectiveness.\n\n**Weaknesses**\n\n- *Long-term correspondence evidence.* The core claim is improved long-term ego–exo correspondence, yet no explicit metric or qualitative example is provided. Please include quantitative results (e.g., association accuracy against clip length) or visualisations that illustrate successful long-range tracking.\n\n- *Attribution of BA limitations.* The manuscript states that BA shortcomings “originate from existing baselines,” but the connection is not explained. Clarify precisely which baseline constraints carry over to LM-EEC and why they cannot be alleviated within your framework.\n\n- *Inconsistent ablation scores.* Tables 2–6 are reported on the same validation set, yet the final scores differ. Explain whether these discrepancies arise from stochastic variation, different checkpoints, or another factor, and unify the reporting format to make cross-table comparisons straightforward.", "was_revised": true, "final_justification": "Most of the concerns I raised have been satisfactorily addressed during the rebuttal. After reviewing the feedback from other reviewers, I maintain a positive assessment of this paper.", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. We greatly appreciate the reviewer's recognition of the clarity of our manuscript, the soundness of our proposed framework, and the comprehensiveness of our experimental evaluations. Below, we provide detailed responses to each of the concerns the reviewer raised.\n\n> **Q1**: *Long-term correspondence evidence*. The core claim is improved long-term ego-exo correspondence, yet no explicit metric or qualitative example is provided. Please include quantitative results (e.g., association accuracy against clip length) or visualisations that illustrate successful long-range tracking.\n\nTab. A: Association accuracy for our method and the baseline against different clip length.\n\n| Video Length   | < 200 frames | 200 - 600 frames | > 600 frames |\n|----------------|--------------|------------------|--------------|\n| Baseline Model | 0.5580       | 0.6260           | 0.4724       |\n| Our Model      | 0.6054       | 0.6732           | 0.6150       |\n\n**A1**:\nThanks for this helpful comment. As suggested, to demonstrate the effectiveness of our long-term ego-exo correspondence, we conduct an experiment under the ego-exo correspondence setting in Tab. A to analyze the **association accuracy** of our method against different video clip length and compare it with the baseline (i.e., SAM 2) that excludes the core MV-MoE and the memory compression modules. In the experiment, the **association accuracy** is defined as the ratio of frames where the Intersection over Union (IoU) between the predicted and ground truth masks exceeds a threshold (set to 0.5). The comparison results are shown in Tab. A.\n\nAs shown in Tab. A, our model consistently outperforms the baseline model across all video lengths. Notably, on the relatively longer videos with more than 600 frames, our model achieves a significant improvement of approximately 14% in association accuracy compared to the baseline model, which effectively validates the long-term ego-exo correspondence of our method.\n\nWe appreciate the reviewer for this careful comment and will include the above analysis and results in the revision. Thanks again!\n\n> **Q2**: *Attribution of BA limitations*. The manuscript states that BA shortcomings \"originate from existing baselines,\" but the connection is not explained. **(a)** Clarify precisely which baseline constraints carry over to LM-EEC and **(b)** why they cannot be alleviated within your framework.\n\n**A2**:\nThanks for this insightful comment. \n\n**To sub-question (a):** The balanced accuracy (BA) primarily measures the ability of a model in determining if the object appears or not in the frame. In this paper, we adopt the architecture of SAM 2 as the baseline to develop our model. While SAM 2 (our baseline) is powerful in video segmentation, it **lacks** specific mechanisms to handle cases where the target object disappears from the scene. For example, it does not incorporate strategies to verify object presence or absence. As a consequence, our model inevitably inherits the limitation of SAM 2 to some extent. It is worth noting that, despite this, our method still achieves noticeable gains in BA compared to the baseline model, as shown in Tab. 1 of the paper.\n\n**To sub-question (b):** Thanks for this careful comment. In fact, this limitation can be alleviated within our framework (and also the baseline SAM 2 framework). A possible solution is to introduce a simple branch within our framework to predict the presence or absence of the object. However, since the primary focus of our work is on improving object-level correspondence and retaining informative long-term content. To this end, we introduce the memory-view mixture-of-experts (MV-MoE) module, which effectively fuses the memory-fused and view-guided information, as well as a dual-memory bank system with a dedicated compression strategy that preserves long-term information while removing redundancy. As demonstrated in our experiments, these components significantly improve the correspondence performance.\n\nWe thank the reviewer and agree that handling object disappearance remains an important and challenging problem in ego-exo correspondence. We leave this to our future by exploring specific solution for BA. We will include the above analysis in the revision. Again, thanks!\n\n> **Q3**: *Inconsistent ablation scores*. Tables 2-6 are reported on the same validation set, yet the final scores differ. Explain whether these discrepancies arise from stochastic variation, different checkpoints, or another factor, and unify the reporting format to make cross-table comparisons straightforward.\n\n**A3**:\nSorry for the confusion. The discrepancies in the final scores across Tab. 2-6 result from evaluations performed under *distinct controlled settings*. These settings are designed to isolate the effect of each individual module and eliminate potential interference from others, thereby allowing for a more intuitive validation of the motivation and effectiveness of each proposed component relative to the SAM 2 baseline. Below, we provide detailed clarifications for each table:\n\n**(i). Tab. 2**: To isolate the impact of the fusion mechanism (MV-MoE), we compare it using a basic FIFO strategy for memory storage upon the SAM 2 baseline, thereby excluding the influence of memory compression. The final score (highlighted in cyan) thus reflects the performance of the model with only the MV-MoE module.\n\n**(ii). Tab. 3 and 5**: After validating the individual and joint effectiveness of our proposed modules, we conduct further ablations on key hyperparameters, including the number of frames and memory size. The final scores (highlighted in cyan) in these tables correspond to our full model, integrating all proposed modules.\n\n**(iii). Tab. 4**: Here, we aim to analyze the memory storage strategy in isolation. To do so, we perform ablation on the SAM 2 baseline model, excluding both the MV-MoE and memory compression modules. This allows us to observe the individual contribution of memory storage without interference from other components.\n\n**(iv). Tab. 6**: To isolate the effect of the memory compression strategy, we adopt a simple addition fusion mechanism upon the SAM 2 baseline, removing the influence of MV-MoE. The final score (highlighted in cyan) represents the performance of the model with only the memory compression module.\n\n\nWe thank the reviewer for this careful comment. As suggested, we will unify the reporting format for these ablation studies and clearly annotate which version of the model each score corresponds to, ensuring that cross-table comparisons are intuitive and consistent.\n\nOnce again, we thank the reviewer for all the constructive feedback, which has significantly helped improve the clarity and rigor of our paper."}, {"review_id": "vxm4BiF65t", "reviewer": "Reviewer_r7ib", "summary": "This paper addresses the challenging problem of object-level correspondence across egocentric and exocentric views, which an important problem with applications in AR guidance, robotic perception, etc. The core difficulty lies in large viewpoint changes, object occlusions, and scale variance.\n\nTo tackle this, the authors propose **LM-EEC**, a SAM2-based video object segmentation framework that introduces:\n- **Memory-View Mixture-of-Experts (MV-MoE)**: A dual-branch dynamic fusion module that adaptively weighs memory-aware and view-specific features across channel and spatial dimensions using learned routing mechanisms.\n- **Dual Memory Banks with Compression**: Separate memory banks for ego and exo views, with a lightweight temporal feature compression strategy to reduce redundancy and preserve long-term dependencies efficiently.\n\nThe model is benchmarked on the EgoExo4D dataset and achieves strong improvements over both prior SOTA methods (e.g. XSegTx, XView-XMem, SimVOS) and the SAM2 baseline across multiple metrics (IoU, location error, contour accuracy, etc.).", "questions": "Please see the weaknesses listed above, which also contains a few questions for the authors.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n1. **Well-motivated technical design**: The use of dual memory banks and expert-weighted fusion directly addresses the issues of feature misalignment and memory inefficiency that plague SAM2 when applied to the Ego-Exo task.\n2. **Novel implementation**: The MV-MoE module integrates both spatial and channel attention in a residual fashion, providing fine-grained expert weighting and avoiding simple addition-based prompt fusion.\n3. **Efficient memory compression**: The memory pruning strategy is based on pointwise Euclidean redundancy, which is simple yet effective, and goes beyond frame-level heuristics like FIFO or clustering.\n4. **Experiments**: \n    - The paper clearly isolates the contribution of each component - MV-MoE, memory compression, and training frame length - showing consistent improvement in isolation and combination.\n    - The method outperforms previous methods in IoU, contour accuracy, and location error across both Ego2Exo and Exo2Ego settings, and across small/large object subsets.\n    - Qualitative analysis and attention visualizations further support the method’s robustness to appearance variations and viewpoint shifts.\n\n# Weaknesses\n1. **Limited generalization claims. More datasets?**: Despite strong performance on EgoExo4D, the model is evaluated on a single dataset. Claims about generalization could be strengthened with tests on other multi-view video datasets. For example, how about applying this method to **Assembly101** dataset (CVPR 2022) which has multi-view videos for various activities?\n    - There's also the dataset from \"Replay: Multi-modal Multi-view Acted Videos for Casual Holography\", which can be used for evaluation.\n\n2. **Worse performance in Balanced accuracy**: The method underperforms in balanced accuracy compared to XSegTx, indicating poor modeling of object disappearance (i.e., false positives when objects are out of view). *This is also noted by the authors*.\n\n3. **Compression heuristic could be stronger?**: The proposed memory compression merges features purely based on Euclidean similarity. While efficient, it lacks semantics (e.g., object-aware grouping or scene dynamics).\n\n4. **Modularity of MV-MoE**: The fusion module is tightly coupled with SAM2’s architecture. It's not entirely clear how portable or reusable the module is with other segmentation backbones. This is not a big weakness, although it would be nicer is the module was such that it could be used with future versions of SAM2-like architectures.", "was_revised": true, "final_justification": "This is a strong paper, with interesting algorithmic components and satisfactory experiments. The authors' rebuttal has addressed my concerns, especially about the dataset and application of the MV-MoE module on other segmentation backbones. The newly added experiments confirm the author's hypothesis that the MV-MoE is indeed useful for the Ego-Exo task.\n\nI maintain my rating at \"Accept\".", "rebuttal": "We sincerely thank the reviewer for all the detailed and insightful comments. We truly appreciate the recognition of our technical contributions-including the MV-MoE design, the dual-memory structure, and the efficient compression strategy-as well as the positive assessment of the rigor of our experimental evaluations. Below, we provide point-by-point responses to all the concerns.\n\n> **Q1**: **Question about limited generalization claims. More datasets?**: Despite strong performance on EgoExo4D, the model is evaluated on a single dataset. Claims about generalization could be strengthened with tests on other multi-view video datasets. For example, how about applying this method to **Assembly101** dataset (CVPR 2022) which has multi-view videos for various activities?\n> - There's also the dataset from \"Replay: Multi-modal Multi-view Acted Videos for Casual Holography\", which can be used for evaluation.\n\n**A1**: Thanks for this comment. The reason why we only conduct experiments on EgoExo4D is because it is the only one that provides unique fine-grained ego-exo correspondence annotations (i.e., object masks in both ego and exo views). In addition, EgoExo4D itself is a large and diverse dataset that contains numerous scenarios. Please note that, the ego-exo correspondence task is firstly proposed in the EgoExo4D dataset. \n\nWe understand the reviewer's concern and fully agree with the reviewer that broader evaluation is essential to support claims of generalization. We have carefully reviewed the datasets the reviewer suggested. We found that, although the Assembly101 dataset provides multi-view videos, it primarily targets **action recognition and understanding** tasks (e.g., action recognition, anticipating actions, temporal action segmentation, hand-object interaction, etc.), with annotations limited to action labels and 3D hand poses, but it **lacks the dense object-level mask annotations and correspondence** which are necessary for our ego-exo correspondence task. Similarly, Replay focuses on multi-modal, multi-view synthesis of dynamic content, rather than dense segmentation in synchronized ego-exo views. As such, these datasets are ***not*** directly suitable for evaluating ego-exo correspondence in our method.\n\nThat said, we appreciate the reviewer's suggestion and agree that extending evaluation to broader contexts is a valuable direction. As part of our future work, we plan to develop a ***new*** dataset, as a ***supplement*** to the existing EgoExo4D, by featuring more complex outdoor environments and broader activity domains beyond those captured in EgoExo4D, in order to further assess the generalization capabilities of our approach and future research.\n\nThanks again for this helpful comment. We will include the above clarification in the revision.\n\n> **Q2**: **Question about performance in Balanced accuracy**: The method underperforms in balanced accuracy compared to XSegTx, indicating poor modeling of object disappearance (i.e., false positives when objects are out of view). This is also noted by the authors.\n\n**A2**:\nThanks for this careful comment. As discussed in the manuscript (Line 435-439 in paper), there exists an inherent trade-off among IoU, contour accuracy, location score, and balanced accuracy. Specifically, our model tends to segment the corresponding objects in the current frame, even when the target objects are not present. This tendency can lead to a higher false positive rate in scenarios involving object disappearance, negatively affecting balanced accuracy. This issue is particularly pronounced in the EgoExo4D dataset due to extreme viewpoint variations and frequent occlusions, resulting in numerous frames where the target object is absent. Interestingly, we observe that models with relatively weaker segmentation capability sometimes achieve higher balanced accuracy in such challenging scenarios, as they tend to refrain from predicting any segmentation in uncertain cases—an effect illustrated in Fig. 7 in the appendix, where SimVOS, QDMN, and STCN show such behavior.\n\nTo improve our balanced accuracy, a possible and simple solution is to include a branch in our model to directly predict object presence or absence. However, since our current primary goal is to enhance object-level correspondence and effectively retain informative long-term content. We leave developing specific mechanisms for improving balance accuracy to our future work.\n\nAgain, thanks for this helpful comment. We will include the above clarification in the revision.\n\n> **Q3**: **Compression heuristic could be stronger?**: The proposed memory compression merges features purely based on Euclidean similarity. While efficient, it lacks semantics (e.g., object-aware grouping or scene dynamics).\n\n**A3**:\nThanks for this insightful comment. Our goal is to develop a simple, efficient, and effective memory compression strategy to main long-term information for ego-exo correspondence. For this purpose, we design the proposed memory compression method using Euclidean similarity. Despite its simplicity, it allows our model to effectively maintain useful long-term information for segmentation and execute efficiently in training and inference. \n\nHowever, we fully agree with the reviewer that incorporating higher-level cues, such as object-aware grouping, semantic relationships, or motion segmentation priors, could lead to a more informed and selective memory compression process. These enhancements have the potential to further improvement. We believe this is a highly promising direction for the ego-exo correspondence task, and leave exploring this semantic-aware memory compression design to our future work.\n\n Again, thanks for this comment! We will add the above clarification in the revision.\n\n> **Q4**: **Modularity of MV-MoE**: The fusion module is tightly coupled with SAM2's architecture. It's not entirely clear how portable or reusable the module is with other segmentation backbones. This is not a big weakness, although it would be nicer is the module was such that it could be used with future versions of SAM2-like architectures.\n\nTab. B: Application of MV-MoE on other segmentation backbones.\n\n| Model         | IoU ↑ | LE ↓  | CA ↑  | BA ↑  |\n|---------------|-------|-------|-------|-------|\n| STCN          | 27.39 | 0.109 | 0.378 | 61.97 |\n| STCN + MV-MoE | 30.51 | 0.083 | 0.436 | 62.81 |\n| QDMN          | 27.03 | 0.108 | 0.346 | 63.16 |\n| QDMN + MV-MoE | 28.75 | 0.100 | 0.376 | 64.02 |\n\n**A4**:\nThanks for the insightful comment. While the proposed MV-MoE is initially designed on top of the SAM 2 architecture, its core component, i.e., the dual-branch routing mechanism that adaptively assigns contribution weights to each expert feature along both channel and spatial dimensions, is ***conceptually modular*** and can be ***readily integrated*** into other architectures that involve multi-source feature fusion. To demonstrate the generality of MV-MoE, we conduct additional experiments by integrating it into two alternative framework with different segmentation bones, including STCN and QDMN, for the Ego-Exo Correspondence task. The results on the Ego2Exo test split of the EgoExo4D dataset are shown in Tab. B.\n\nFrom Tab. B, we can clearly observe that, incorporating MV-MoE consistently enhances performance across metrics, indicating that the module is not only effective but also transferable to different backbone architectures. This suggests that MV-MoE can be easily adapted to other segmentation architectures/backbones and future versions of SAM 2-like frameworks.\n\nOnce again, we sincerely appreciate the reviewer's valuable suggestion, which has helped improve the clarity and rigor of our work. We hope the additional experiments and clarifications satisfactorily address the reviewer's concerns."}, {"review_id": "n9jYMrDIyB", "reviewer": "Reviewer_dkHk", "summary": "This paper proposes a new framework for the ego-exo object correspondence task. Given a pair of synchronized ego-exo videos and a sequence of query masks for an object of interest in one of the videos, the objective of the task is to identify the corresponding masks of the same object in each synchronized frame of the other view. The proposed framework adapts the Segment Anything Model 2 (SAM2) to this cross-view segmentation task, and develops two new modules, a memory-view mixture-of-experts module and a dual-memory bank system to enhance cross-view feature fusion and long-term memory management. Experiments on the EgoExo4D benchmark show that the proposed method achieves state-of-the-art results.", "questions": "1. The use of term “mixture-of-experts” may cause confusion. Often the term of expert refers to some independent models, but in this paper it is used for the features from certain dimension (channel or spatial).  \n2. Brief introduction (e.g., data amount, data split) of the object correspondence benchmark from Ego-Exo4d dataset would help the reader comprehends the task but is not given either.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**  \n1) *Clarity*: The paper is well-written and easy to follow. The motivation and idea are clearly stated and the details of method design is well explained.  \n2) *Significance*: The paper adapts the well-known SAM2 to the ego-exo object correspondence task with two newly developed modules, and significantly outperforms existing baselines of video object segmentation, largely advancing the state-of-the-art performance.\n3) *Originality*: The proposed Memory-View Mixture-of-Experts (MV-MoE) module and the dual-memory compression strategy are reasonable and validated by the experiments.\n\n**Weaknesses**\n1. The proposed method is largely based on the segmentation framework of SAM2, and the main technical novelties come from the two proposed module of MV-MoE and dual-memory compression. However, MV-MoE is essentially the combination of channel attention and spatial attention which are common (or even “old-fashioned”) design of attention modeling. Dual-memory compression is also simply designed and the improvement over SAM2’s memory design is marginal (less than 1% of IoU in Table 6). Thus the overall technical novelty seems to be of limited significance.  \n2. The paper introduces SAM2 to the object correspondence task and largely advance the SOTA performance, which is worth of praise. However, as far as my understanding, the performance improvement largely owes to the strong base model of SAM2. The base model already outperforms related methods for most metrics (IoU, LS, CA).\n3. Some important implementation details are missing. It is not clarified which parts of the framework are updated during training and which are frozen. The data amount, training time cost, and inference efficiency are not given either. Since the memory compression strategy relies on the computation of feature similarity over all temporal and spatial location in the memory bank, it may lowers the inference speed.", "was_revised": true, "final_justification": "Most of my concerns have been addressed by the rebuttal, and I have no major concern. Thus I recommend acceptance.", "rebuttal": "Thanks for the thoughtful and detailed feedback. We sincerely appreciate the reviewer's recognition of the clarity, significance, and originality of our work in adapting SAM 2 to the ego-exo object correspondence task through the introduction of two novel modules. Below, we provide point-by-point responses to the concerns raised.\n\n> **Q1**: The proposed method is largely based on the segmentation framework of SAM2, and the main technical novelties come from the two proposed module of MV-MoE and dual-memory compression. However, MV-MoE is essentially the combination of channel attention and spatial attention which are common (or even \"old-fashioned\") design of attention modeling. Dual-memory compression is also simply designed and the improvement over SAM2's memory design is marginal (less than 1% of IoU in Table 6). Thus the overall technical novelty seems to be of limited significance.\n\n**A1**: We appreciate the reviewer's perspective and would like to further clarify the novelty and significance of our contributions:\n\n**(i) About the MV-MoE module**: We understand the reviewer's concern regarding the \"old-fashioned\" nature of channel and spatial attention. While the attention is common, they are ***under-explored*** in the ego-exo correspondence task for representation learning. Our ***contribution*** lies not in simply reusing these mechanisms, but in how they are jointly leveraged within a modular mixture-of-experts framework specifically designed for multi-feature fusion to improve ego-exo correspondence. The experiments reported in our paper effectively validate that the proposed MV-MoE is crucial for performance improvement.\n\n**(2) About the dual-memory compression**: Thanks for the comment. Our goal is to develop a simple, efficient, and effective memory compression strategy to main long-term information for ego-exo correspondence. For this purpose, we design the proposed memory compression method using Euclidean similarity. Despite its simplicity, it allows our model to effectively maintain useful long-term information for segmentation and execute efficiently in training and inference. Using our dual-memory compression, the improvement in IoU is around 1%, which is non-trivial when using the rigorous IoU metric on the whole video. In addition, we show that it clearly improves the contour accuracy (CA) with around 1.6% and can decrease the location error (LE) from 0.0276 to 0.0258 compared to the FIFO strategy, showing its efficacy in consistently improving ego-exo correspondence performance.\n\n\nWe will incorporate the above clarifications in the revision to make our contributions more clear. Again, thanks!\n\n\n> **Q2**: The paper introduces SAM2 to the object correspondence task and largely advance the SOTA performance, which is worth of praise. However, as far as my understanding, the performance improvement largely owes to the strong base model of SAM2. The base model already outperforms related methods for most metrics (IoU, LS, CA).\n\n**A2**: Thanks for this careful comment. SAM 2 is chosen as our baseline because of its strong generalization ability. Yes, as noted by the reviewer, SAM 2 itself already shows impressive results by outperforming related methods, but directly applying SAM 2 to the exo-ego correspondence faces several challenges, such as suboptimal feature fusion across ego and exo views and ineffective long-term memory maintenance. Our proposed method addresses these challenges by introducing two key contributions, including the MV-MoE module for improved feature integration, and a dual-memory compression mechanism to manage memory more effectively. On the already-powerful SAM 2 baseline, our method still improves its performance by a large margin with 8.5% and 2.9% IoU gains in Exo2Ego and Ego2Exo settings (please see Tab. 1 in our paper). These substantial performance gains, achieved over an already powerful baseline, clearly demonstrate the efficacy and value of our proposed method in improving ego-exo correspondence.\n\nWe will include the above clarification in the revision. Again, thanks!\n\n> **Q3**: Some important implementation details are missing. It is not clarified which parts of the framework are updated during training and which are frozen. The data amount, training time cost, and inference efficiency are not given either. Since the memory compression strategy relies on the computation of feature similarity over all temporal and spatial location in the memory bank, it may lower the inference speed.\n\n**A3**: Sorry for the confusion. Here we provide clarifications for the implementation details mentioned by the reviewer. \n\n***(i) Trainable Components***: Given the large-scale nature of EgoExo4D, we train all modules jointly based on the pre-trained SAM2 checkpoint, without freezing any components. \n\n***(ii) Data Volume***: The EgoExo4D dataset contains 5,500 annotated objects across 1,249 ego-exo video pairs, with around 4 million annotated frames. This results in 742K egocentric and 1.1M exocentric segmentation masks. We adopt the official dataset split: 756 takes for training, 202 for validation, and 291 for testing. \n\n***(iii) Training Cost***: The training process takes approximately 13 hours on 8xA100 GPUs with a batch size of 4 for 60 epochs. \n\n***(iv) Inference Efficiency***: Inference is conducted on a single V100 GPU at ~8.4 FPS, compared to ~9.0 FPS for the baseline SAM 2. The proposed memory compression strategy slightly reduces inference speed but yields improved performance, which we believe is a worthwhile trade-off.\n\nWe will include the above details explicitly in the revision to improve clarity of our method. Our full code and model will be released for reproducibility. Again, thanks for this helpful comment!\n\n> **Q4**: The use of term \"mixture-of-experts\" may cause confusion. Often the term of expert refers to some independent models, but in this paper it is used for the features from certain dimension (channel or spatial).\n\n**A4**:\nThanks for this careful comment. We agree and understand the reviewer's concern. As suggested, to avoid potential confusion, we will modify the term \"mixture-of-experts\" in our method in the revision. Again, thanks!\n\n\n> **Q5**: Brief introduction (e.g., data amount, data split) of the object correspondence benchmark from Ego-Exo4d dataset would help the reader comprehends the task but is not given either.\n\n**A5**: Thanks for this comment. The EgoExo4D dataset contains 5,500 annotated objects across 1,249 ego-exo video pairs, with around 4 million annotated frames. This results in 742K egocentric and 1.1M exocentric segmentation masks. We adopt the official dataset split in our experiment, where 756 videos are used for training, 202 for validation, and 291 for testing. We will include this introduction of the EgoExo4D dataset in the revision. Thanks again!\n\n\nOnce again, we deeply appreciate the reviewer's constructive feedback. We have taken all suggestions seriously and would revise the paper to improve clarity, rigor, and transparency."}, {"review_id": "Naf52329a8", "reviewer": "Reviewer_2TFe", "summary": "This paper introduces LM-EEC, a novel framework for establishing object-level correspondence between egocentric (first-person) and exocentric (third-person) videos by enhancing SAM 2 with two key innovations: (1) a Memory-View Mixture-of-Experts module that adaptively fuses cross-view features using channel and spatial attention, and (2) a dual-memory bank system with compression to preserve long-term object context. Evaluated on the EgoExo4D benchmark, LM-EEC achieves state-of-the-art results, outperforming methods like XSegTx and XView-XMem, particularly in handling small objects and viewpoint variations.", "questions": "See in Weaknesses.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Paper strengthens: \n\n(1) This paper introduces LM-EEC, a novel framework enhancing SAM 2 for ego-exo correspondence. It proposes Memory-View Mixture-of-Experts (MV-MoE) for adaptive feature fusion across views, addressing feature distribution gaps, and designs a dual-memory bank system with view-specific compression to retain long-term information efficiently.\n\n(2) In comprehensive ablation studies, validate each component (MV-MoE, dual memory, compression). This work outperforms 10+ baselines (e.g., XSegTx, XView-XMem, STCN) by significant margins (e.g., +18% IoU in Exo2Ego tasks).\n\n(3) It solves critical challenges: extreme viewpoint variations, occlusions, small objects, and long-video redundancy. Meanwhile, it achieves SOTA results on the challenging EgoExo4D benchmark, demonstrating real-world applicability (e.g., AR, robotics).\n\n(4) Well-structured with clear task definitions (Ego2Exo/Exo2Ego), SAM 2 adaptations, and visualizations (Figures 1-5).\n\nPaper weakness:\n\n(1) In Table 3 (Section 4.4), the paper reports inconsistent performance when varying training frame numbers, showing degraded IoU (0.5873) at 10 frames compared to 8 frames (0.5882). The authors fail to provide ablation studies on regularization strategies or frame sampling methods that could explain this anomaly, undermining their claims about optimal memory configuration. \n\n(2) The description of the memory encoder in Section 3.3 contains critical gaps in technical specification. While the paper repeatedly references an \"off-the-shelf memory encoder\" for processing features before storage in the dual-memory system, it completely omits essential architectural details, including layer types, parameter counts, and input/output dimensions. \n\n(3) The compression approach described in Section 3.5 (Equations 8-10) demonstrates concerning conceptual simplicity. The method relies entirely on Euclidean distance thresholds between adjacent frames followed by naive feature averaging, without any consideration of more sophisticated techniques like attention-based pooling or entropy analysis to preserve critical information. This is particularly problematic for handling non-adjacent critical events like object reappearance after occlusion, as shown in the paper's own Figure 4. The authors provide no quantitative evidence that their compression method actually preserves the most discriminative features, despite making strong claims about long-term information retention.\n\n(4) The comparison with baseline methods in Table 1 (Section 4.3) suffers from serious methodological shortcomings. While the paper states that all models were \"trained or fine-tuned on the EgoExo4D dataset,\" it provides no details about the fine-tuning protocols used-including number of epochs, optimizer settings, or data splits. This lack of transparency makes it impossible to verify whether the baseline implementations achieved their full potential performance, maybe it’s better show more details of the implementation of various comparison methods.\n\n(5) Despite bold assertions in the Abstract about \"strong generalization across diverse scenarios,\" the paper provides no evidence beyond single-dataset evaluation on EgoExo4D. The complete absence of cross-dataset validation (e.g., on EPIC-KITCHENS or other standard benchmarks) renders these generalization claims entirely speculative. This is particularly concerning for a method that purports to handle challenging real-world conditions like viewpoint variation and occlusion conditions that can vary significantly across different datasets and environments.\n\n(6) The paper's treatment of occlusion challenges reveals significant limitations. While Section 1 emphasizes handling \"severe object occlusions\" as a key contribution, the actual performance gains in Balanced Accuracy (BA) metrics are minimal-just +2.66% for Ego2Exo and +1.03% for Exo2Ego compared to the base model (Table 1). Moreover, the qualitative results in Figure 1 conspicuously avoid showing challenging occlusion cases, focusing instead on relatively straightforward examples where objects remain largely visible. This selective presentation undermines the paper's claims about robust occlusion handling.", "was_revised": true, "final_justification": "Thanks for the response. I have no further questions. I will keep the score.", "rebuttal": "We sincerely thank the reviewer for the thoughtful and detailed review. Below, we address all concerns point-by-point.\n\n> **Q1**: Table 3 (Section 4.4) shows inconsistent performance—IoU drops at 10 frames (0.5873) compared to 8 frames (0.5882)—without ablation on regularization or frame sampling.\n\n**A1**:\nThanks for this careful comment. Upon further analysis, we believe the performance degradation when increasing the number of training frames from 8 to 10 is likely due to the introduction of redundant or noisy information. Specifically, the additional frames may not always provide complementary cues, but instead introduce irrelevant context or conflicting temporal signals that interfere with effective learning. This phenomenon is not uncommon in video object segmentation tasks, as similarly observed in prior works such as “End-to-End Video Instance Segmentation with Transformers” and “Segment Anything in Images and Videos (SAM 2)”.\n\nTab. C: Results of the random sampling strategy.\n| Frame | IoU ↑  | LE ↓   | CA ↑   |\n|-------|--------|--------|--------|\n| 8     | 0.5894 | 0.0199 | 0.7967 |\n| 10    | 0.5905 | 0.0202 | 0.7997 |\n\nAs suggested, to further explain this anomaly, we conduct an ablation study on the frame sampling strategy. Different from the adjacent frame sampling method in the paper, we utilize a random frame sampling strategy to increase temporal diversity. The experimental results are shown in Tab. C. From Tab. C, we can see, similar to our used adjacent frame sampling method (see Tab. 3), when increasing frames from 8 to 10, the performance slightly decreases, which is consistent with our above analysis.\n\nThanks again, we will include the above analysis and results in the revision. \n\n> **Q2**: Section 3.3 lacks key technical details about the “off-the-shelf memory encoder,” such as its architecture, layer types, and input/output dimensions.\n\n**A2**:\nThanks for this helpful comment. Here we provide the architectural details mentioned by the reviewer.  ***(i) Layer Types***: The memory encoder consists of two main components: (1) MaskDownSampler, which processes the binary object mask before feature fusion; and (2) Fuser, a ConvNeXt-style fusion block that integrates pixel-level visual features with the downsampled mask signal. ***(ii) Parameter Count***: The total number of trainable parameters in the memory encoder is 5.3 M. ***(iii). Input/Output Dimensions***: The memory encoder takes a visual feature map of shape [B, 256, 30, 30] and a binary mask of shape [B, 1, 480, 480] as input, and outputs a fused embedding of shape [B, 64, 30, 30], where B=4 during training and B=1 during testing.\n\nWe will incorporate the above details in the revision. Our code will also be released. Again, thanks!\n\n> **Q3**: **(a)** The compression method in Section 3.5 is overly simplistic, using only Euclidean distance and feature averaging, with no advanced techniques to retain more critical information—especially for non-adjacent events like reappearance after occlusion. **(b)** The authors provide no quantitative evidence that their compression method actually preserves the most discriminative features, despite making strong claims about long-term information retention.\n\n**A3**:\nThanks for the insightful comment. \n\n**To sub-question (a):** Our current goal is to develop a simple, efficient, and effective memory compression strategy to maintain long-term information for ego-exo correspondence. For this purpose, we design the proposed memory compression method using Euclidean similarity. Despite its simplicity, it enables efficient training and inference while preserving useful information for segmentation. Compared to the FIFO strategy, our method consistently improves IoU by ~1%, contour accuracy by ~1.6%, and reduces location error from 0.0276 to 0.0258 (see Tab. 6), demonstrating its effectiveness.\n\nHowever, we agree that more advanced techniques could better preserve critical features, especially for non-adjacent key events like reappearance after occlusion. Such improvements may further enhance segmentation in complex scenarios, and we leave this exploration to future work.\n\n**To sub-question (b):** To further show quantitative evidence that our comprehension method helps retain discriminative long-term information, we conduct a quantitative comparison between the baseline model (i.e., SAM 2) and a variant using our compression module. Specifically, we evaluate ***association accuracy***—the proportion of frames with IoU > 0.5—under different video length as a proxy for the model’s ability to maintain consistent object tracking and segmentation over time. Results are shown in Tab. D.\n\nTab. D: Quantitative analysis of our memory compression method via association accuracy on Ego2Exo.\n| Video Length            | <200 frames | 200-600 frames | >600 frames |\n|-------------------------|-------------|----------------|-------------|\n| Baseline                | 0.5580      | 0.6260         | 0.4724      |\n| Baseline w/ compression | 0.5646      | 0.6381         | 0.5385      |\n\nTab. D shows our compression method consistently improves association accuracy, especially on longer videos (>600 frames) with over 6% gain, demonstrating effective retention of long-term information.\n\nThanks again! We will include the above analysis in revision.\n\n> **Q4**: The baseline comparison in Table 1 lacks transparency—key fine-tuning details (e.g., epochs, optimizer, data splits) are omitted, making it unclear whether baselines reached their full potential.\n\n**A4**: Thanks for this comment. Here we clarify the implementation details of other compared methods.\n\n***(i) Training Configurations***: All baselines were implemented from official code with minimal changes for EgoExo4D, following original training settings. Tab. E summarizes the training configurations of our method and other approaches. Notably, all the compared method are trained for ***more*** epochs/iterations than our proposed model to ensure sufficient optimization. Our goal is to ensure fair and optimal performance for all methods on EgoExo4D.\n\nTab. E: Summary of training configurations of different methods.\n| Model                      | Epochs (Iterations)     | Optimizer |\n|----------------------------|-------------------------|-----------|\n| XSegTx                     | 250 (250K iter.)        | Adam      |\n| XView-Xmem (w/ finetuning) | 358 (100K iter.)        | AdamW     |\n| XView-Xmem (+ XSegTx)      | 358 (100K iter.)        | AdamW     |\n| STCN                       | 258 (50K iter.)         | Adam      |\n| QDMN                       | 207 (80K iter.)         | Adam      |\n| GSFM                       | 213 (55K iter.)         | Adam      |\n| SimVOS                     | 193 (300K iter.)        | Adam      |\n| Cutie                      | 167 (125K iter.)        | AdamW     |\n| Ours                       | 60 (12K iter.)          | AdamW     |\n\n***(ii) Data Splits***: We follow the official data split of EgoExo4D, which contains 756 takes for training, 202 takes for validation, and 291 takes for testing. These splits were applied consistently across all methods for fair comparison.\n\nWe will include the above clarifications in revision. Besides, we will release the implementations of all methods. Again, thanks!\n\n> **Q5**: The paper claims strong generalization but only reports results on EgoExo4D, without cross-dataset validation (e.g., EPIC-KITCHENS), making the generalization unsubstantiated.\n\n**A5**: Thank you for the insightful comment. We use EgoExo4D because it is the only dataset that provides ego-exo correspondence annotations (i.e., object masks in both views), and it is also large and diverse. Notably, the ego-exo correspondence task was first proposed in EgoExo4D.\n\nWe agree that broader evaluation is important. And we have carefully reviewed the EPIC-KITCHENS dataset. We found that, the EPIC-KITCHENS dataset mainly focuses on the egocentric perspective and do not offer dense segmentation mask annotations. Therefore, it is not directly applicable to evaluate our method. We also surveyed other datasets but found none supporting ego-exo correspondence.\n\nThat said, we appreciate the suggestion and see evaluation on more datasets as a valuable direction. In our future work, we plan to develop a ***new*** dataset, as a ***supplement*** to EgoExo4D, by featuring more complex outdoor environments and broader activity domains, to better assess generalization.\n\nThanks again! We will include this clarification in revision.\n\n> **Q6**: **(a)** The paper emphasizes occlusion handling but shows only marginal BA gains (+2.66%, +1.03%) **(b)** and avoids hard occlusion cases in Figure 1, weakening its claims of robustness.\n\n**A6**: Thanks for this insightful comment. \n\n**To sub-question (a):** Balanced Accuracy (BA) evaluates the model’s ability to detect the presence or absence of the target in a frame. In contrast, occlusion refers to cases where the target is present but partially or heavily blocked. Thus, occlusion handling is better reflected in metrics like IoU, location error, and contour accuracy, where our method shows clear improvements over baselines.\n\n**To sub-question (b):** Sorry for the confusion. Fig. 1 illustrates overall performance comparisons, not specifically occlusion. Occlusion cases are shown in Fig. 7 and Fig. 10 (e.g., a shovel occluded by eggs, a tire blocked by a hand). To further substantiate our claims, we will add more visualizations focusing on occlusion in the revision. Again, thanks!\n\n> **Q7**: Ethical Concerns: Data privacy, copyright, and consent\n\n**A7**:\nThank you for raising this important concern. We use the EgoExo4D dataset under an official research license from the University of Bristol, with proper attribution. We confirm that all usage complies with the license and no additional personally identifiable information is involved.\n\nOnce again, thanks for the helpful feedback. We will revise the paper to enhance clarity."}]

Author rebuttal:

Meta-review: The paper introduces LM-EEC for establishing object correspondence between egocentric and exocentric views in videos. It developed on top of SAM2 with two new technical designs: a Memory-View Mixture-of-Experts module and a dual-memory compression module. The designs are clearly presented, supported by strong empirical results on EgoExo4D, and validated through ablations. Reviewers consistently praised the clarity of writing, well-structured figures, and convincing baseline comparisons. All reviewers voted to accept the work. Congrats!

Concerns remain around generalization (evaluation is limited to a single dataset), efficiency of the memory compression strategy, and weaker performance in balanced accuracy for occluded/disappearing objects. Some of these are addressed through experiments during rebuttal. The main ones remained are the generalization of the method and improving BA. AC believes both are valid shortcomings of the work, and having both can potentially boost the impact of the work. However, the paper in its post-rebuttal format meets the bar for acceptance.
Decision: Accept (poster)