Ego3DT: Tracking Every 3D Object in Ego-centric Videos

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-01 06:47:20

Abstract: The growing interest in embodied intelligence has brought ego-centric perspectives to contemporary research. One significant challenge within this realm is the accurate localization and tracking of objects in ego-centric videos, primarily due to the substantial variability in viewing angles. Addressing this issue, this paper introduces a novel zero-shot approach for the 3D reconstruction and tracking of all objects from the ego-centric video. We present Ego3DT, a novel framework that initially identifies and extracts detection and segmentation information of objects within the ego environment. Utilizing information from adjacent video frames, Ego3DT dynamically constructs a 3D scene of the ego view using a pre-trained 3D scene reconstruction model. Additionally, we have innovated a dynamic hierarchical association mechanism for creating stable 3D tracking trajectories of objects in ego-centric videos. Moreover, the efficacy of our approach is corroborated by extensive experiments on two newly compiled datasets, with 1.04× - 2.90× in HOTA, showcasing the robustness and accuracy of our method in diverse ego-centric scenarios.

Main paper: 
 related work 2.1 open vocabulary detection: 
 Open vocabulary (OV) detection [79] has emerged as a novel approach to modern object detection, which aims to identify objects beyond the predefined categories. Early studies [18] followed the standard OV Detection setting [79] by training detectors on the base classes and evaluating the novel or unknown classes. However, this open-vocabulary setting, while capable of evaluating the detectors' ability to detect and recognize novel objects, is still limited to open scenarios and lacks generalization ability to other domains due to training on a limited dataset and vocabulary. Inspired by vision-language pre-training [23, 52] , recent works [9, 26, 73, 89, 91] formulate open-vocabulary object detection as image-text matching and exploit large-scale image-text data to increase the vocabulary at scale. GLIP [32] presents a pre-training framework for openvocabulary detection based on phrase grounding and evaluates in a zero-shot setting. Grounding DINO [39] incorporates the grounded pre-training into detection transformers [80] with cross-modality fusions. Several methods [36, 77, 78, 81] unify detection datasets and image-text datasets through region-text matching and pre-train detectors with large-scale image-text pairs, achieving promising performance and generalization. However, these methods often use heavy detectors, leading to high computational demands and deployment challenges. Utilizing the YOLO framework with an effective pretraining strategy, some works [4, 75] enhance openvocabulary performance and generalization. GLEE [72] excels in recognizing and tracking objects across both images and videos.Over the last few decades, the introduction of numerous ego-centric video datasets [6, 14, 16, 28, 50, 60] , has presented a wide range of fascinating challenges. Although many methodologies utilize tracking to address these challenges [7, 16, 29, 37, 41] , it's notable that only a few studies have focused solely on the crucial issue of tracking. The works by Dunnhofer et al. [11, 12] address the specific challenges associated with ego-centric object tracking and represent the research most closely related to our own. However, a significant distinction exists in the scale of the dataset they utilized, which comprises 150 tracks designed purely for assessment purposes. In the realm of ego-centric video comprehension, Ego4D [16] , EPIC-KITCHENS VISOR [7] and EgoTracks [61] are critical to our work. Ego4D stands out for its extensive compilation of ego-centric videos captured in natural settings and introduces numerous innovative tasks, including Episodic Memory, where tracking plays a pivotal role. Concurrently introduced, VISOR focuses on annotating brief videos (averaging 12 seconds in length) from EPIC-KITCHENS [6] with instance segmentation masks, illustrating the dynamic and detailed nature of research in this field.The study of 3D object detection has made considerable advancements through the utilization of images [1, 20, 43, 54] , point clouds [15, 30, 51, 58] , and videos [2, 22] . To convert 2D images into 3D scenes, researchers have extensively employed Structure from Motion (SfM) techniques [47] . These techniques are divided into geometric-based methods [27, 45, 56] , which rely on multiview geometry; learningbased methods [24, 67, 90] , which utilize deep neural networks; and hybrid SfM approaches [62, 63] , which integrate both strategies. SfM has been adapted for extensive videos in dynamic settings [85] and casual videos capturing everyday life [38, 84] . Yet, the distinct nature of ego-centric videos, characterized by their dynamic content, motion blur, and unconventional viewpoints, poses substantial hurdles to 3D scene understanding. While numerous studies have explored the reconstruction of 3D human poses from ego-centric footage [5, 19, 31, 53, 64, 68, 82] , the field of 3D perception from an ego-centric perspective has seen exciting developments recently. HuCenLife [76] dataset is for large-scale human-centric scenarios, providing benchmarks for segmentation and action recognition tasks. Some work [48] tackled the challenges of ego-centric 3D human pose estimation with their domain-guided spatiotemporal transformer model, Ego-STAN, achieving significant improvements.EgoFish3D framework [40] employed self-supervised learning for accurate egocentric 3D pose estimation. Noteworthy efforts include investigation into ego-centric indoor localization using the Manhattan world assumption for room layouts [3] , the development of EGO-SLAM for outdoor ego-centric videos through SfM over time [49] , and the creation of NeuralDiff [66] and N3F [65] , which innovate in dynamic NeRF technology for identifying and segmenting moving objects in ego-centric videos. Additionally, some work [46] proposed a method that correlates camera positions with video data to anticipate human-centric scene contexts. Our approach focuses on 3D tracking via dynamic matching in the 3D field. It is a zeroshot, RGB-only approach for open-vocabulary object tracking by constructing a 3D scene from an ego-centric video. 
method 3.1 overview: 
 As shown in Figure 2 EQUATIONwhere 𝑌 = {𝑂 𝑖 } 𝑖 ≤𝑁 is the 3D object tracking output of the video with 𝑁 frames, 𝑂 𝑖 = [(𝑥 𝑗 , 𝑦 𝑗 , 𝑧 𝑗 , ID 𝑗 )] 𝑗 ≤𝐾 is a matrix containing 3D coordinates of tracked objects in each frame with identification ID, and 𝐾 is the total number of tracked objects. First, we conduct object detection Det on videos 𝑋 , and semantic segmentation Seg based on detection output 𝑂 𝐷𝑒𝑡 2𝐷 as prompts:EQUATIONwhere 𝑂 𝑆𝑒𝑔 2𝐷 and 𝑂 𝐷𝑒𝑡 2𝐷 are the semantic segmentation and detection output respectively.Then, we utilize a 3D estimation model G to map segmentation coordinates from 2D spaceEQUATIONwhere 𝑂 3𝐷 forms a one-to-one mapping between image pixels and 3D scene points, i.e., 𝑂 2𝐷 ↔ 𝑂 3𝐷 , for all object coordinates (𝑥, 𝑦) ∈ {1 . . . 𝐾 } × {1 . . . 𝑁 }.Finally, Ego3DT involves matching the 3D positions of objects using a hierarchical method, avoiding the instability issues that can arise from relying solely on 2D image tracking:EQUATIONwhere the matching module M compares all the 3D points from frame to frame for precise object tracking 𝑌 with identification ID, and A is a 3D scene registration method aligning adjacent points. We use the additional Hungarian process to initialize matching ID.The foundational step in our method involves the precise identification and segmentation of objects within each frame of an egocentric video. As shown in Equation ( 2), this process is bifurcated into two pivotal operations: 2D Open Vocabulary (OV) Detection Det and 2D Segmentation Seg, applied sequentially to the raw video frames to ensure a comprehensive understanding of the scene.To achieve accurate object detection within our framework, we leverage the capabilities of the pretrained GLEE [72] in the experiment. Our efficient object detection model can identify a wide range of objects in 2D space across video frames, even those not explicitly labeled in the training data. We obtain precise 2D bounding boxes for all detectable objects by processing each frame through the model, setting the stage for subsequent segmentation.Following the detection phase, the identified objects are further processed through SAM [25] , a segmentation foundation model designed to delineate the precise boundaries of objects within an image. The bounding boxes obtained from GLEE [72] serve as prompts for SAM [25] , enabling it to focus on specific regions of interest within the frame. This approach generates detailed segmentation maps for each object, including shape and location. We maintain window-level 3D fields with a 3D estimation model called G, a pretrained DUSt3R [69] . This dual-branch system consists of image encoders, decoders, and regression heads. The image encoders are designed to extract detailed feature maps from segmented 2D object points, which are inputs derived from the preceding object detection and segmentation phases. The decoders then process these feature maps, which focus on extracting spatial relationships and depth cues from the encoded data. As shown in Equation ( 3), the 3D estimation model G processes segmented 2D object points, transforming them into their 3D counterparts through the pretrained DUSt3R [69] . This process is predicated on accurately detecting and segmenting objects within the 2D space, followed by their elevation into the 3D domain.From 2D Segmentation to 3D Localization. As shown in Figure 2 , the image encoders are designed to extract detailed feature maps from segmented 2D object points, inputs derived from the preceding object detection and segmentation phases. The decoders then process these feature maps, which focus on extracting spatial relationships and depth cues from the encoded data.Integration and Alignment of 3D Data. The output from G consists of accurate 3D coordinates inherently aligned with the original RGB video frames. This alignment is critical as it ensures that each 3D point precisely represents its corresponding 2D point and is correctly positioned within the global context of the video sequence. This meticulous alignment facilitates the seamless integration of 2D and 3D data, enhancing the robustness and accuracy of the subsequent object-tracking processes.Keeping window-level 3D fields in our framework advances the field of 3D object tracking and sets a new standard for the accuracy and efficiency of converting 2D video data into actionable 3D information. The rigorous processing and alignment of data ensure that our model is highly effective in the challenging environment of ego-centric videos, paving the way for innovative applications in multiple domains.The Matching Module M is a crucial component of the Ego3DT framework for tracking 3D objects across the video sequences. As shown in Equation ( 4), the Matching Module M consists of point-matching algorithms and a sliding window mechanism to ensure accurate and robust object tracking, even in occlusion or rapid movements. To minimize errors in point matching, we retain mutual correspondences between two images. This is achieved by performing KDTree search [69] in the 3D pointmap space.Sliding Window Mechanism. We adapt the sliding window mechanism in the matching module, defined by the window size 𝑊 , ensuring an overlap size 𝑇 to maintain temporal continuity between frames. This design choice allows for the efficient processing of video frames by dividing the extensive task of 3D object tracking into manageable segments, each containing 𝑊 frames. The step distance 𝑆 = 𝑊 -𝑇 dictates the window's movement across the video sequence, ensuring that every frame is analyzed while optimizing computational resources. Initial Object Tracking. The process begins by establishing a baseline of object tracking within the first window. For each frame 𝑖, up to the window size 𝑊 , the 3D coordinates of detected objects 𝑂 𝑖 3𝐷 = {(𝑥 𝑗 , 𝑦 𝑗 , 𝑧 𝑗 )} 𝑗 ≤𝐾 are determined, where 𝐾 represents the number of objects detected within a frame. Utilizing KDTree distance calculations between every two consecutive frames, we employ the Hungarian algorithm to match objects based on their spatial proximity, thus assigning a unique Identification Number ID to each object. The result, 𝑌 0 , comprising tracked objects with their respective IDs within the first window, is stored in a buffer B for subsequent processing.Dynamic matching across windows. As shown in the Algorithm 1, the module employs a hierarchical object-tracking approach. As the window slides by step 𝑆, each new set of frames is processed based on the previous window's data. Specifically, we employ a 3D scene registration method A, an optimized homography process to align the 3D points of objects between the current and previous windows, thus 𝑂 𝑡 3𝐷 = A (𝑂 𝑡 -1 3𝐷 , 𝑂 𝑡 3𝐷 ) to keep the current windows 𝑂 𝑡 3𝐷 into the same space of the previous 𝑂 𝑡 -1 3𝐷 . The homography process is shown as follows:EQUATIONwhere 𝐻 𝑡 is the homography matrix between the current points 𝑂 𝑡 3𝐷 and the previous points 𝑂 𝑡 -1 3𝐷 of overlapped frames, the ground points of all current frames are unified into the previous space. To further refine the alignment process, M employs an optimization strategy that minimizes the Euclidean distance between matched points across the homography transformations:EQUATION)where 𝐴 is the total number of matching points, 𝐻 𝑡 is a 4 × 4 matrix with rotation matrix K ∈ R 3×3 and translation matrix T ∈ R 3×1 . All parameters are random numbers in the (0, 1) range during initialization.By recalculating KDTree distances for the newly aligned 3D points and based on the applied Hungarian algorithm, PointMatch matches pixels of objects from frame to frame. Each object in the current window is then assigned the ID of its closest match from the previous window, thus extending the tracking sequence. This process is repeated for each window throughout the video, culminating in comprehensively tracking all objects across the sequence.The Matching Module M of Ego3DT achieves high precision and robustness in 3D object tracking through these sophisticated algorithms and mechanisms. It provides a global 3D field as shown in Figure 3 . This innovative approach ensures that Ego3DT can effectively handle the complexities of ego-centric video analysis, paving the way for advancements in interactive and immersive technologies. We summarize the matching process in Algorithm 1, which outlines the step-by-step procedures for achieving accurate and reliable tracking results.  
experiment: 
 This section evaluates the Ego3DT framework for 3D object tracking in ego-centric videos using the Ego3DT Benchmark. We form two datasets: Ego3DT-daily and Ego3DT-indoor, and advance metrics to evaluate tracking accuracy. We test the state-of-the-art detectors and compare their performance to baseline models, demonstrating the efficacy and robustness of the Ego3DT in handling the unique challenges of ego-centric video analysis. Through rigorous testing and validation, this section illustrates the robustness, precision, and scalability of the Ego3DT framework.Since there is no existing 3D object tracking benchmark based on ego-centric videos, we build a new benchmark called Ego3DT Benchmark to evaluate the performance of our model.We collected and re-annotated two datasets, Ego3DT-daily and Ego3DT-indoor, from Ego4D [16] and EmbodiedScan [70] . These datasets include 2D detection boxes and daily object trajectories in indoor and outdoor scenes.Ego3DT-daily. contains six indoor and outdoor scenes, from which we collected videos from EGO4D. Each video, sampled at 10 FPS, consists of 500 consecutive frames. There are two outdoor scenes and four indoor scenes. The video collection locations include supermarkets, gardens, corridors, and kitchens. These ego-centric videos feature noticeable shaking and diverse object changes.Ego3DT-indoor. includes data from five indoor scenes. Based on the Embodied Scan dataset, we collected ego-centric videos following predefined camera trajectories. We collected about 100 frames per video at 3 FPS from five scenes.Our annotation pipeline is semiautomatic. We annotated the same objects with detection boxes and a global ID in a single video. For the Ego3DT-daily dataset, we first used the existing open vocabulary detector GLEE to extract object detection boxes to save annotation time. We then calibrated and aligned each object's detection boxes and IDs frame by frame. For objects that disappeared and then reappeared, we assigned them a consistent global ID. For the Ego3DT-indoor dataset, since Embodied Scan provides 3D detection boxes for each object, we projected the 3D detection boxes onto the current frame based on the camera's pose in each frame, thus determining the object's 2D detection boxes and global ID.We evaluate the performance of our method using HOTA [42] and the MOT Challenge [8] evaluation metrics, including MOTA, IDF1, MT, ML, Frag etc. MOTA is computed based on false positives, false negatives, and ID switches and primarily focuses on detection performance. IDF1 assesses the consistency of IDs and places more emphasis on association performance. HOTA explicitly balances the accuracy of detection, association, and localization. We conduct experiments on Ego3DT using different detectors for open vocabulary detection, namely GLEE [72] via GLEE-Plus backbone Swin-L and YOLO-World [4] via YOLO-Worldv2-X. We also use SAM [25] with ViT-H backbone for open vocabulary segmentation. Then, we utilize the 3D estimation model via DUSt3R [69] with DPT Head, ViT-L Encoder, and ViT-B Decoder. Note that our experiments are conducted using only a single RTX3090-24G.We critically evaluate the Ego3DT framework against established baselines: ByteTrack [83] , DeepSort [71] , OVTrack [34] , and TET [33] , each offering unique strengths in multi-object tracking (MOT) and providing a comprehensive context for benchmarking our model's performance.ByteTrack [83] is a powerful multi-object tracking (MOT) system designed to associate almost every detection box, regardless of the score, to improve tracking consistency, especially in cases with occluded objects. It stands out due to its simplicity, efficiency, and robustness against occlusions and low-confidence detections. The system has been successfully applied to different tracking benchmarks, confirming its versatility and strength as a baseline model for MOT tasks.DeepSort [71] is an effective MOT method in videos, enabling accurate identity retention over time, particularly in scenarios where objects are frequently occluded. This system is a go-to choice for practitioners seeking a balance between performance and computational efficiency. The system proved versatile and robust, excelling as a baseline model in various MOT tasks.OVTrack [34] is an open-vocabulary MOT method, utilizing vision-language models for classification and association, applying knowledge distillation and data hallucination techniques for feature learning. The approach aims to be highly data-efficient and is tailored for large-scale tracking, focusing on using static images for training.TET [33] is a large-scale MOT method. It critically examines the limitations of current MOT metrics and methods, which often assume near-perfect classification performance, a presumption rarely met in practice. TET performs associations using Class Exemplar Matching, showing notable improvements in challenging tracking.As shown in Table 1 , we have evaluated the open-vocabulary multiobject tracking performance using a comprehensive range of metrics from the MOT Challenge and HOTA. Ego3DT greatly outperforms well-established baselines with a unique approach to 3D point association. It has been assessed on additional performance indicators, thus enhancing the breadth of our evaluation.Notably, the Ego3DT framework with the GLEE detector [72] achieves the highest HOTA score of 30.83 among all evaluated trackers, indicative of a well-balanced detection and association accuracy. It excels in DetA (Detection Accuracy) with a leading score of 47.91, demonstrating our framework's exceptional capability in precise object detection. Note that DetA is not the same across different methods, even if the same detector is used. This is because different methods adopt different association and post-processing strategies that may affect the detection results. Furthermore, Ego3DT maintains a competitive edge with a high number of Mostly Tracked (MT) targets and the fewest Mostly Lost (ML) targets among the automatic tracking methods, with respective scores of 24 and 49, highlighting the framework's robustness in persistent object tracking over time. The TET detector [33] produces the highest number of Fragmentations (Frag), indicating that our tracking is accurate and the object identity is stable compared to the real object trajectories in the ground truth data.These expanded metrics provide a holistic view of our framework's performance, affirming its strengths in maintaining object identities (as evidenced by its IDF1 score of 29.71) and effectively tracking objects throughout the video sequence. Despite the high Frag count, the Ego3DT framework's overall leading performance in key metrics solidifies its status as a robust solution for the MOT challenge, particularly within the demanding context of ego-centric videos.To refine the Ego3DT framework, we conduct a comprehensive ablation study to discern the individual contributions of detector quality and memory mechanisms to the framework's overall performance. The experiments are carefully designed to isolate the impact of these components, providing insights into their respective significance and interplay. As shown in Table 2 , a high-quality detector Accurate Detector is Pivotal. The choice of detector plays a pivotal role. The GLEE detector [72] , being a high-quality pretrained detector, synergizes exceptionally well with our 3D association approach, yielding a HOTA score of 30.83, which robustly indicates superior detection and ID association. This confirms that a proficient detector, coupled with our sophisticated 3D tracking methodology, can substantially boost the overall tracking quality, ensuring that high-quality detections translate into high-quality ID annotations.Appropriate Memory Mechanism is Critical. It reflects on the balance between memory usage and tracking performance. Notably, using a 30-frame memory mechanism offers the best performance across all metrics. This optimized setting achieves a HOTA of 30.83 and an IDF1 of 29.71, underscoring the effectiveness of a limited temporal memory that captures the immediate past to maintain context without being burdened by the noise of distant frames. On the other hand, the absence of a memory mechanism and the use of full frame memory result in reduced performance, demonstrating the importance of a focused temporal window for accurate tracking. This suggests that an excessive memory span can dilute the relevancy of information, leading to higher fragmentation and decreased detection accuracy. The results highlight the delicate trade-off between the memory's depth and the tracking accuracy, suggesting that moderate memory size is instrumental in improving the consistency and precision of object tracking in ego-centric videos.Our Ego3DT framework exhibits significant advancements in 3D reconstruction and 2D tracking, showcasing robust performance even under challenging first-person motion scenarios. We provide a qualitative analysis of these two core aspects to highlight the efficacy and improvements over existing methodologies. 4.6.1 Qualitive Results on 3D Reconstruction. A closer examination of Ego3DT's performance on our meticulously collected datasets reveals the nuanced capability of our framework in handling complex 3D environments. As shown in Figure 3 , we present two distinct scenarios that showcase the efficacy of Ego3DT in real-world applications.Outdoor Tracking in Ego3DT-daily. The Ego3DT-daily dataset, representing an array of outdoor settings, challenges the framework with dynamic lighting, diverse object shapes, and sizes. Our model demonstrates robustness in these conditions, accurately tracking and maintaining consistent IDs across different object types, from smaller items like a wok (ID 3) to larger potted plants (IDs 5 and 6). The 3D tracking field captures the spatial relations and movement paths, illustrating the model's adaptability to outdoor environments.Indoor Persistence in Ego3DT-indoor. Transitioning to the indoor domain, the Ego3DT-indoor dataset offers a contrasting setting with more controlled lighting but equally complex object interactions. The model successfully delineates and tracks objects such as furniture (IDs 1 to 4) in a typical room scenario, highlighting its precision in cluttered, confined spaces. The tracking continuity is evident, with the framework skillfully handling occlusions and varying distances from the camera. occasional ID inconsistencies and missed detections. c) Our Ego3DT approach, which maintains stable object identification, accurately captures dynamic objects and shows superior consistency in ID assignment, especially evident in motion-rich ego-centric perspectives. From left to right, the frames progress temporally, illustrating each method's tracking continuity and precision.4.6.2 Qualitive Results on 2D Tracking. As shown in Figure 4 , we compare our method to ByteTrack [83] and demonstrate how our approach offers improved detection stability. Our framework, Ego3DT, is particularly effective in dynamic scenes that involve uniform motion from a first-person perspective. It consistently identifies and tracks a higher number of objects with greater reliability, making it an ideal solution for applications that require real-time responsiveness and accuracy, such as augmented reality and autonomous navigation systems.As seen in the side-by-side comparison of 2D tracking techniques, our Ego3DT framework excels at preserving object identity across frames. The Ground Truth (a) provides a benchmark with flawless tracking and ID fidelity. ByteTrack coupled with GLEE detection (b) provides a strong baseline but occasionally falters with ID switches and detection lapses, especially under the erratic motion intrinsic to ego-centric videos. In contrast, our approach (c) consistently demonstrates a remarkable grasp on object trajectories, maintaining accurate IDs even in the presence of motion blur and rapid scene changes. Moving from left to right through the temporal sequence, this comparison underscores the Ego3DT framework's advanced capability to deliver reliable and coherent tracking performance in dynamic and challenging first-person video scenarios. These qualitative results underscore the versatile 3D tracking capabilities of Ego3DT, cementing its potential for comprehensive scene understanding and robust object tracking in diverse environments. 
limitation and future work: 
 Although our proposed Ego3DT can successfully detect and track almost every 3D object in the scene, it might still fail in tracking some rapidly moving objects like cats, dogs, or humans. We leave this in future work, including tracking the moving objects and detecting the interaction with the scene and other objects. 
conclusion: 
 We have introduced the Ego3DT framework for accurately tracking 3D objects in ego-centric videos. The framework uses a sophisticated 3D estimation model and state-of-the-art detection and segmentation technologies. Our experimental results demonstrate that the Ego3DT framework outperforms established baselines and can ensure accurate detection, consistent ID tracking, and precise localization. The Ego3DT framework can facilitate practical applications in augmented reality, robotics, and advanced surveillance systems.  
: 
 datasets, essential for training and evaluating tracking algorithms [13] . Although the research community has introduced several popular tracking datasets such as OTB [74] , TrackingNet [44] , GOT-10k [21] , and LaSOT [13] , the high performance achieved by state-of-the-art trackers on these benchmarks does not effectively translate to egocentric videos. This discrepancy underscores the urgent need for a dedicated ego-centric tracking dataset, particularly one that can support the unique requirements of ego-centric applications.The distinct characteristics of ego-centric videos, as opposed to conventional third-person videos, pose unique challenges. These videos often capture a wide range of activities, objects, and locations without specific focus, reflecting the wearer's shifts in attention. Large head movements from the camera wearer frequently cause objects to exit and re-enter the field of view, and objects manipulated by hands may undergo frequent occlusions, along with rapid changes in scale, pose, and even state or appearance [57] . These unique aspects make object tracking significantly more demanding than in scenarios typically presented in existing datasets, highlighting a critical gap in current evaluation methodologies. Traditional MOT tasks [83] , when applied to ego-centric videos [61] , often result in poor tracking accuracy and robustness.To better suit the variable conditions of ego-centric videos, our approach utilizes a 3D field representation, which offers a more adaptable and comprehensive framework for tracking. As shown in Figure 1 , the 3D field captures the spatial layout and the temporal dynamics of objects within the scene, making it exceptionally suitable for the complexities of ego-centric views. This concept involves maintaining a dynamic 3D scene to enhance perceptual tasks [69, 70] . 3D perception can improve task robustness by ensuring stable object properties and relationships throughout the scene. By maintaining a dynamic 3D field, our approach preserves stable relationships and properties of 3D objects, significantly enhancing performance. Moreover, our method employs training-free, plugand-play modules that deliver few-shot capabilities, distinguishing it from conventional approaches.We summarize our contributions as follows:• We propose a method for constructing a 3D scene from an ego-centric video and achieving open vocabulary object tracking, which requires only RGB videos as input and is a zero-shot approach. • We implement object 3D position matching through a dynamic cross-window matching method, thereby alleviating the instability caused by relying solely on 2D image tracking. • We achieve state-of-the-art performance on the open vocabulary multi-object tracking in ego-centric videos of daily activities, with 1.04× -2.90× in HOTA. 
 
 Appendix:  


Decision: This paper proposes a novel  method for accurately localizing and tracking objects from egocentric videos, which is specially robust against rapid camera movement. The paper is clearly written, technically sound and present a solid validation. 
All reviewers lean towards acceptance after rebuttal. I recommend to accept the paper.

Reviews: review: paper_summary: paper_summary: - This paper proposes a method for accurately localizing and tracking objects from ego-centric videos, robust against rapid camera movement. The method combines existing modules, specifically open vocabulary object detection GLEE, 2D object segmentation SAM, and 3D pointmap estimation DUSt3R, with the cross-window matching and projection component. A dynamic hierarchical association mechanism is introduced for robust object tracking in the presence of occlusions and camera’s rapid movement.
- Experiments showed that the proposed method's tracking accuracy exceeds existing trackers such as Bytetrack, OVTrack, and TET.
strength_weakness: strength_weakness: - The proposed tracking method combines existing modules (GLEE, SAM, DUSt3R) and dynamic hierarchical matching, which is somewhat novel.
- Experimental results show that the proposed method outperforms existing tracking methods.
limitations: limitations: - The paper is difficult to follow for me.
    - I wonder why Algorithm 1 is titled "cross-window" matching process. I do not see which steps correspond to "cross-window" matching. Is the for loop in Algorithm1, which repeats from 1 to T(overlap size), correct? How is Algorithm 1 related to Algorithm A1 in the supplemental material?
    - Is eq. 4 in l. 425 correct? I do not see why it has $\prod$ notation? Does the index $t$ represent time or frame in a sliding window?
    - What does the memory mechanism mean?
- The Impact of each component on tracking accuracy needs further discussion.
    - The impact of the object detection accuracy and DUSt3R's 3d estimation accuracy on the final tracking accuracy has not been discussed.
    - Why was the potted plant 251 detected in the fourth frame of Fig. 4(b) (Bytetrack+GLEE), while the same potted plant was not detected in Fig. 4 (c) (ours)? What caused the missed detection of the potted plant in the second frame in Fig. 4(c)? Why did the tracking of potted plant 47 in Fig. 4(c) fail in the fourth frame?
paper_summary: paper_summary: The paper presents Ego3DT, a novel framework for 3D object tracking in egocentric videos. Utilizing an Open Vocabulary detector and semantic segmentation, it maps detected 2D objects into a 3D space, ensuring consistent tracking across frames with a dynamic cross-window matching technique. Tested on two proposed datasets, Ego3DT-daily and Ego3DT-indoor, Ego3DT outperformed existing methods, demonstrating robustness and precision in handling rapid movements and occlusions typical of egocentric videos.
strength_weakness: strength_weakness: - Robust Testing: The proposed method is validated through extensive testing on two newly compiled datasets. The results demonstrate improvements in tracking accuracy and robustness, achieving significant gains compared to existing methods.

- Technical Contributions: The paper presents several technical innovations, including a cross-window matching approach. The proposed dataset also contributes to the overall novelty of this paper.

- The paper writing is clear and easy to follow
limitations: limitations: - The Ego3DT model is computationally intensive, requiring sophisticated components like SAM, GLEE, and DUSt3R for 3D object tracking. This high computational load can limit its use in real-time or resource-constrained environments.

- How effectively does the Ego3DT handle occlusions and lighting variations within ego-centric videos?
paper_summary: paper_summary: The paper introduces Ego3DT, a novel zero-shot framework for 3D reconstruction and tracking of all objects in ego-centric videos, addressing challenges of varying viewing angles. The proposed method identifies and extracts object detection and segmentation information, dynamically constructing a 3D scene using a pre-trained model and employing a dynamic hierarchical association mechanism for stable tracking. Extensive experiments on two new datasets demonstrate the robustness and accuracy of the approach, achieving improvements of 1.04× to 2.90× in HOTA metrics.
strength_weakness: strength_weakness: The use of 3D scene reconstruction as an intermediate step in object tracking from an egocentric perspective unlocks new opportunities, making this approach particularly relevant for applications such as scene navigation, virtual reality, and scene understanding. Additionally, the paper's application of Open Vocabulary Tracking for ego-centric videos is noteworthy, as it can enhance and expand current egocentric tracking datasets, facilitating further experimental research.
limitations: limitations: 1. **Major Contributions**

    The paper claims ego-tracking as its major contribution, yet it emphasizes 3D scene construction from ego-centric video as a significant aspect. This dual focus is somewhat confusing. If 3D scene construction is indeed a major contribution, the paper lacks significant performance comparisons with existing methods. To substantiate this claim, the authors should highlight existing 3D scene generation methods and provide relevant metrics for comparison.

2. **Explanation of Ego-Centric Nature**

    The paper's definition of ego-centric view lacks specific context, such as the field of view (FOV) angle being used. Larger FOVs can cause distortions, which may impact performance. Although detailed comparisons are not necessary, the authors should explain how the ego-centric nature and potential distortions are handled in their approach.

3. **Necessity of 3D Scene Generation**

    There are several works on object detection in 360-degree videos, which are inherently more complex. Most do not utilize 3D scene generation, indicating that object detection is feasible without it. The paper does not clearly motivate the need for 3D scene generation. A comparison with scenarios that do not use 3D scene generation would clarify whether this added complexity is justified.

4. **Writing Etiquette**

    - L96-L97: "*these benchmarks does not effectively translate..*" - What does "effectively" mean in this context?
    - L97-L99: "*This discrepancy underscores..*" - It is unclear what discrepancy the authors are referring to. Clarification is needed.

5. **Table 2: Ablation Study**

    The ablation study shows that Memory mechanisms with 30-Frames and Detector with GLEE have similar results, except for a significant difference in the Frag-score. This interesting finding needs explanation. Is it coincidental, or is there an underlying reason for this behavior?

6. **Motion Blur**

    Are there qualitative results showcasing the model's performance on objects affected by motion blur during sudden perspective changes? Since the authors mention this as a challenge, it would be valuable to see how the model addresses such issues.

7. **Throughput**

    What is the model's performance in terms of throughput? Additionally, what compute infrastructure is being used? This information is crucial for readers to understand the limitations and capabilities of the proposed methods in real-world applications.