TAO-Amodal: A Benchmark for Tracking Any Object Amodally

## Abstract

Abstract Amodal perception, the ability to comprehend complete object structures from partial visibility, is a fundamental skill, even for infants. Its significance extends to applications like autonomous driving, where a clear understanding of heavily occluded objects is essential. However, modern detection and tracking algorithms often overlook this critical capability, perhaps due to the prevalence of modal annotations in most benchmarks.
To address the scarcity of amodal benchmarks, we introduce TAO-Amodal, featuring 833 diverse categories in thousands of video sequences.
Our dataset includes amodal and modal bounding boxes for visible and partially or fully occluded objects, including those that are partially out of the camera frame.
We investigate the current lay of the land in both amodal tracking and detection by benchmarking state-of-the-art modal trackers and amodal segmentation methods. We find that existing methods, even when adapted for amodal tracking, struggle to detect and track objects under heavy occlusion. To mitigate this, we explore simple finetuning schemes that can increase the amodal tracking and detection metrics of occluded objects by 2.1% and 3.3%.

###### Keywords:

## 1 Introduction

Machine perception, particularly in object detection and tracking, has focused primarily on reasoning about visible or modal objects. This modal perception ignores parts of the three-dimensional world that are occluded to the camera. However, amodal completion of objects in the real-world (e.g., seeing a setting sun but understanding it is whole) and their persistence over time (e.g., person walking behind a car in Fig. 2) are fundamental capabilities that develop in humans in their early years [, , ]. In autonomous systems, this online amodal reasoning finds a direct application in downstream motion planning and navigation. Despite this, object detection and tracking stacks give little importance to partially or completely occluded objects; this becomes apparent in datasets that are only annotated modally [, , , , , , , , , ] but are still widely used and built upon by algorithms. These algorithms [, , , , , , , , ] in turn learn to perceive only modal objects.

To address this gap, we introduce a benchmark for large-scale amodal tracking, which requires estimating the full extent of objects through heavy and even complete occlusions.
Our benchmark annotates 17,000 objects with amodal bounding boxes, along with human confidence estimates, from 833 classes in 2,907 videos.
While prior datasets focus on images or are limited to a small vocabulary of classes (Tab. 1), our benchmark evaluates amodal tracking for hundreds of object classes.
Since objects can get occluded because of other objects in the scene, and because of the limited field-of-view of cameras in casual captures, we define and address two kinds of occlusions – in-frame, and out-of-frame.
As annotating amodal bounding boxes can be ambiguous and challenging, we design a new annotation protocol with detailed guidelines to improve human annotation.
Importantly, we base our benchmark on a large-vocabulary multi-object tracking dataset, TAO [].
This choice allows us to pair our amodal box annotations with class labels, modal boxes, and precise modal mask annotations [] collected in prior work.

Equipped with this data, we set out to evaluate the difficulty of amodal tracking.
We evaluate using standard metrics, including detection and tracking AP, and variants [] that evaluate tracking specifically under partial and complete occlusions.
As expected, we find that standard trackers trained with modal annotations do not suffice for amodal tracking.

To adapt existing modal trackers into amodal ones, we finetune them on TAO-Amodal. The closest line of work to amodal tracking is amodal segmentation [, , ]. We benchmark recent amodal segmentation algorithms by running a Kalman-Filter based association during post-processing on their predictions. While this addresses the gap between modal and amodal tracking to some extent, the performance is far from good due to the challenging occlusion scenarios in TAO-Amodal. To mitigate this, we explore different but simple finetuning and data-augmentation strategies inspired by prior work [, ]. This lets us set a new baseline on the tasks of amodal detection and tracking.

In summary, our contributions are as follows: (1) we annotate a large-scale dataset of amodal tracks for diverse objects, consisting of 17k objects spanning 833 categories, (2) we adapt evaluation metrics to handle amodal settings, and evaluate state-of-the-art trackers for our new task, and finally, (3) we investigate multiple finetuning and data-augmentation schemes as simple extensions to improve the existing modal tracking algorithms.

**Table 1: Statistics of amodal datasets. TAO-Amodal is proposed as an evaluation benchmark for amodal tracking. We compare our dataset to prior image (first block), synthetic video (second block), and real video (last block) datasets. TAO-Amodal is notable for being real-world videos that span far more categories and far more annotated frames for evaluation. Track length is averaged over the dataset in seconds, while total length is the length of eval sequences in seconds. We define heavy occlusion as objects with visibility below 10%, and partial as between 10%-80%. Occluded tracks are those that have heavy or partial occlusions for more than 5 seconds. Out-of-frame (OoF) objects are ones that extend partially beyond the image boundary.**
|  | # Sequences |  | Track | Total | # Occluded Boxes | # Occluded | Ann |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Total | Test | Val | Train | Classes | length | length | Partial | Heavy | OoF | tracks | fps |
| COCO-Amodal [] | 5000 | 1250 | 1250 | 2500 | - | - | - | 8.8k | 0.2k | 0 | - | - |
| Sail-VOS [] | 201 | 0 | 41 | 160 | 162 | 14.14 | 3,359 | 559.5k | 704.8k | 0 | 7.9k | 8 |
| Sail-VOS-3D [] | 202 | 0 | 41 | 161 | 24 | 13.10 | 2,808 | 295.0k | 387.5k | 0 | 5.0k | 8 |
| NuScenes [] | 1000 | 150 | 150 | 700 | 23 | 9.06 | 6,000 | 571.1k | 139.5k | 219k | 24.5k | 20 |
| MOT17 [] | 14 | 7 | 0 | 7 | 1 | 6.98 | 248 | 51.2k | 16.4k | 16k | 0.1k | 30 |
| MOT20 [] | 8 | 4 | 0 | 4 | 1 | 20.55 | 178 | 729.4k | 88.1k | 88k | 1.6k | 25 |
| TAO-Amodal | 2907 | 1419 | 988 | 500 | 833 | 22.24 | 88,605 | 158.2k | 35.1k | 139k | 9.6k | 1 |

Figure: Figure 3: Class distribution. We present counts of instances from top 8 most frequent categories and other categories, using a logarithmic scale.
Refer to caption: extracted/2312.12433v3/fig/class_distribution.png

## 2 Related work

Amodal perception has been studied in the past by benchmarks and algorithms, in both the single-frame (detection) and multi-frame (detection and tracking) settings. Since amodal object annotations are hard to obtain due to the uncertainty in human annotations (c.f. prior work [] on a human vision experiment), the community has depended heavily on synthetic datasets, or real-world datasets with few classes and limited diversity. We provide an overview of this prior wrok in the rest of this section.

### 2.1 Benchmarks

#### 2.1.1 Real-world datasets.

Amodal object annotations for real-world scenes are largely limited to the surveillance and self-driving domains.MOT 15-20 [, , ] evaluate multi-object tracking on amodal person detections obtained from detectors trained on MOT annotations. However, these amodal annotations are automatically propagated via linear interpolation of annotations in frames where objects are visible. Additionally, the metrics used by MOT weigh all modal and amodal annotations equally. This largely ignores tracking performance on amodal objects, which form only a small fraction of all annotations.

A number of multimodal (images and 3D LiDAR) datasets for autonomous driving have recently become popular. These include ArgoVerse (1.0 and 2.0) [, ], Waymo [], nuScenes [] and KITTI []. These datasets aim to focus on 3D tasks, and therefore use human annotators to label all objects in 3D to their full extent. In this setting, amodal annotations arise naturally due to the 3D nature of the data. These 3D boxes, when projected onto 2D images, would be useful for amodal perception; unfortunately, these annotations cover only a small number of object classes. Another way to obtain amodal object annotations is in a multi-view setting. Datasets like CarFusion [] and MMPTrack [] follow this data curation scheme, but, due to the cumbersome data collection process, they are limited to only a single or few categories.

In the single-frame setting, COCO-Amodal, Amodal KINS and NuImages [, , ] contain amodal annotations, but only cover the cases of partial occlusion: complete occlusions can only be recovered with temporal information, which is missing in image datasets. Moreover, COCO-Amodal [] and KINS [] do not provide class labels, which makes it difficult to learn object priors for amodal completion, and to evaluate the accuracy of amodal tracking in the long tail.

Synthetic datasets.
An alternative approach to the above is use synthetic data generation pipelines to get amodal annotations. SAIL-VOS and SAILVOS-3D [, ] are such datasets that exploit synthetic dataset curation and come with a number of different types of annotations (bounding boxes, object masks, object categories, their long-range tracks, and 3D meshes). Some of these even suit our case of detecting ‘out-of-frame’ occlusions, where one could project 3D meshes onto the image plane. While the number of categories are slightly larger for these datasets (including others like ParallelDomain [] and DYCE []), the sim-to-real transfer remains a challenge even for modal perception [, ].

### 2.2 Algorithms

Based off of some amodal datasets, there has been a growing interesting in developing algorithms suitable for amodal perception.
Some methods aim to track objects with object permanence [, , , ]. Previous work also segment objects amodally [, , , , ]. Some approaches utilize prior-frame information [, , , , , , , ]. For instance, GTR [] employs a transformer-based architecture and uses trajectory queries to group bounding boxes into trajectories. We lean on similar approaches in this work, and devise a mechanism to generate occlusion cases in the flavor of the data augmentation used by GTR [], and show that this is essential to the goal of enabling amodal perception.

Figure: Figure 5: ROI Head [] with Amodal Expander. Amodal Expander serves as a plug-in fine-tuning scheme to “amodalize" existing detectors or trackers with limited (amodal) training data. It operates by taking as input region proposal features and modal box predictions (often represented as a residual delta with respect the region proposal) and generates amodal box outputs (again represented as residual deltas). We freeze all modules except the amodal expander during fine-tuning.

Figure: Figure 6: Amodal Expander Architecture. Given $N$ flattened proposal features and modal box (delta) predictions represented with 256-dim positional encodings [], we predict amodal box (deltas) with a two-layer MLP (unless otherwise specified). Further architecture details are in Sec. 5.2.

**Table 2: Amodal trackers on TAO-Amodal validation set. We define metrics in Sec. 5.1. The visibility range is indicated by the superscript to denote various levels of occlusion. We fine-tuned modal trackers on TAO-Amodal-train for 20k iterations. Detector [] and amodal segmentation methods [, , ] were evaluated using Kalman filter based association []. We evaluated models that predict COCO vocabulary [] using objects within COCO category. GTR is used as the basis for subsequent experiments, considering its performance in detection and tracking metrics. We run all trackers at 1 fps and average AP across categories with an IoU threshold of 0.5.**
|  |  | Detection Metrics | Tracking Metrics |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | FT | AP^[0,0.1] | AP^[0.1,0.8] | AP^[0.8,1] | AP${}^{\textrm{OoF}}$ | AP | AP | AP${}^{\textrm{[0,0.8]}}$ |
| PCNet [] |  | 0.48 | 7.15 | 21.43 | 8.69 | 15.59 | 5.80 | 3.91 |
| QDTrack [] |  | 0.35 | 8.03 | 21.82 | 8.05 | 15.62 | 7.84 | 4.03 |
| TET [] |  | 0.24 | 5.77 | 14.98 | 4.87 | 10.86 | 4.84 | 3.44 |
| ViTDet-B [] |  | 0.77 | 12.57 | 34.33 | 14.18 | 25.94 | 7.66 | 4.38 |
| ViTDet-L [] |  | 1.25 | 15.06 | 38.16 | 15.84 | 29.04 | 9.70 | 5.90 |
| ViTDet-H [] |  | 1.13 | 15.80 | 40.09 | 16.97 | 30.20 | 9.72 | 5.63 |
| GTR [] |  | 0.77 | 14.62 | 38.17 | 15.31 | 29.24 | 16.07 | 9.28 |
| COCO category eval |  |  |  |  |  |  |  |  |
| ORCNN [] |  | 0.33 | 11.78 | 37.88 | 16.68 | 26.09 | 5.72 | 3.43 |
| AmodalMRCNN [] |  | 0.46 | 14.74 | 42.65 | 18.35 | 29.58 | 7.57 | 4.47 |
| AISFormer [] |  | 0.36 | 14.23 | 39.76 | 18.61 | 27.70 | 7.88 | 5.90 |
| PCNet [] |  | 1.30 | 20.61 | 53.13 | 24.21 | 37.04 | 11.19 | 8.78 |

**Table 3: Exploring fine-tuning strategies on TAO-Amodal validation set. We ablate different strategies for repurposing GTR for amodal tracking. Fine-tuning plug-in expander modestly outperforms fine-tuning all or part of the model. Combined with data augmentation, PasteNOcclude (PnO), for generating synthetic occlusions (as detailed in Sec 4.2), expander produces noticeable gains for partially occluded and out-of-frame objects. All models (other than the baseline) were trained on TAO-Amodal training set for 20k iterations, while ^† denotes 45k iterations of training.**
|  | Detection Metrics | Tracking Metrics |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Method | AP^[0,0.1] | AP^[0.1,0.8] | AP^[0.8,1] | AP${}^{\textrm{OoF}}$ | AP | AP | AP^[0,0.8] |
| Baseline (GTR []) | 0.78 | 13.24 | 37.54 | 14.18 | 28.19 | 16.02 | 8.86 |
| Fine-tune entire model | 0.52 | 10.36 | 24.08 | 10.34 | 17.93 | 7.70 | 3.93 |
| FT entire model + PnO | 0.79 | 9.68 | 26.56 | 10.10 | 20.16 | 9.05 | 4.30 |
| Fine-tune regression head<br>& proposal network | 0.79 | 10.57 | 27.91 | 11.37 | 21.42 | 9.04 | 4.53 |
| Fine-tune regression head | 0.77 | 14.62 | 38.17 | 15.31 | 29.24 | 16.07 | 9.28 |
| FT regression + PnO | 0.87 | 14.36 | 38.18 | 15.47 | 29.04 | 15.95 | 9.23 |
| Amodal Expander | 0.67 | 16.29 | 37.11 | 17.39 | 29.50 | 16.10 | 10.44 (+1.58) |
| Amodal Expander + PnO | 0.80 (+0.02) | 16.41 | 37.74 | 17.64 | 29.87 | 16.35 (+0.33) | 10.13 |
| Amodal Expander + PnO^† | 0.77 | 16.53 (+3.29) | 37.80 (+0.26) | 17.65 (+3.47) | 29.96 (+1.77) | 16.35 (+0.33) | 10.28 (+1.42) |

## 3 Dataset Annotation

#### 3.0.1 Base dataset.

Existing datasets for modal perception are limited either in terms of their diversity, or the vocabulary of classes labelled. To this end, we build upon the modally annotated TAO dataset. It contains bounding box track annotations of 833 object categories at 1FPS spanning a total of 2,921 videos from 7 different data sources (AVA [], Argoverse [], Charades [], HACS [], LaSOT [], BDD100K [], YFCC100M []). Bootstrapping from this dataset allows us to add amodal box annotations to an already existing set of multimodal annotations in TAO – i.e., object classes, modal bounding boxes and modal segmentation masks. TAO follows the single-frame detection datasets, such as LVIS and OpenImages [, ], in adopting a federated annotation protocol for object tracking: i.e., not every object class is exhaustively annotated in every video. We refer the reader to [, ] for details on federated annotation and evaluation setup, and focus here on our amodal annotation of objects in TAO.

Scope. Since annotators can exhibit a large variation in annotating the precise shape of objects while they undergo partial or even complete occlusion, we annotate using bounding boxes instead of segmentation masks to mark the full extent of objects in the visible scene.
We define ‘in-frame’ occlusions as those occurring from the presence of occluders (which may be other dynamic objects, or static scene elements), and ‘out-of-frame’ occlusions as those resulting from objects leaving the camera field-of-view.
We do not label the extent of occlusion in cases where an object may be partially present behind the camera (e.g., a person holding the camera who has their hands visible in the image). For labelling ‘out-of-frame’ occlusions, we need to fix bounds for annotation on the image plane. We ask annotators to work within an annotation workspace that extends to twice the image dimensions in consideration, with the image itself horizontally and vertically center-aligned in this workspace.
Annotation Protocol. Since object tracks in TAO are modal in nature, extending boxes to account for in-frame and out-of-frame occlusions requires (1) (in the case of partial occlusion) complementing TAO bounding boxes with amodal boxes, and (2) (in the case of complete occlusion) adding new boxes to object tracks for occluded frames.
Out of a total of 358,862 boxes in TAO, our annotators modify 266,902 (74.4%) to account for partial occlusions.
Further, TAO-Amodal introduces an additional 23,449 bounding boxes for frames where objects were invisible and unlabeled in TAO.
These annotations follow the guidelines detailed in the appendix, covering a wide range of both in-frame and out-of-frame occlusion scenarios.
Importantly, we only consider occlusion cases where an object has appeared in the scene before.
We exclude occlusions where an object might be partially behind the camera or outside the annotation workspace defined above.
Within the strict purview of the guidelines, when an object’s location cannot be discerned confidently by the annotators, annotators are instructed to mark an is_uncertain flag.
From the 23,449 boxes for invisible objects, 20,218 (85.8%) boxes are annotated confidently (i.e., without the uncertain flag), indicating that there is inherent uncertainty in localizing objects when they undergo heavy occlusions (matching observations from prior work [] which indicate human uncertainty of object location under occlusion).

Finally, equipped with both modal and amodal annotations for all objects, we add a visibility field to the TAO-Amodal annotations, using the overlap (intersection-over-union) between the modal and amodal boxes as a proxy.

Quality Control. We conduct two rounds of professional quality checks on TAO-Amodal annotations: all bounding box annotations are refined twice by annotators. Finally, the authors of this work conducted a manual quality check reviewing 349 tracks from 7 randomly sampled videos, and found only 2 (<1%) tracks without an uncertainty flag to be erroneous. Both tracks were for objects with complete occlusions (visibility 0.0%) in the video. Our analysis show that nearly all inspected tracks ($>99\%$) are accurate, indicating the high-quality of amodal tracking annotations in TAO-amodal.

### 3.1 Dataset statistics

We compare the statistics of TAO-Amodal to other amodal benchmarks in Table 1. For NuScenes, which only categorizes object visibilities into four buckets, we use interpolation to estimate the number of boxes below visibility 0.1 and 0.8. A few amodal datasets are omitted from the table, either because they have been incorporated into TAO-Amodal [, ] or because these datasets lack quantified visibilities for categorizing different occlusion scenarios [, ]. TAO-Amodal covers annotations across an extensive 833 categories, which can be used to learn and evaluate object priors in a large-vocabulary setting. Furthermore, TAO-Amodal features a 10$\times$ longer evaluation duration, ensuring a comprehensive evaluation.

### 3.2 Dataset Splits Design

Following TAO [], we propose TAO-Amodal primarily as an evaluation benchmark. We choose to make the ‘validation’ and ‘test’ set larger to reliably benchmark trackers.
We are not the first to do this: datasets in the single-object tracking [] community have similarly focused on evaluation. With the success of foundation models trained on internet data, high quality evaluation benchmarks are more important than ever, as evidenced in the NLP community (*e.g*., MMLU []).
Our training set is constructed in the spirit of instruction-tuning datasets: a small amount of data to align models to perform amodal tracking.
We also find that increasing the size of the train set (by including test videos as train) does not significantly improve accuracy: adding $4\times$ more training data only increases AP^[0.1,0.8] from +3.3% to +3.7%, further validating our choice to dedicate limited annotated data for evaluation. See appendix for details.

**Table 4: Multi-frame-aware amodal baselines on TAO-Amodal validation set. We explore extensions to include multi-frame signals for fine-tuned expander. Following [], we use a Kalman filter to predict the positions of occluded objects, augmented by a monocular depth estimator to filter out spurious predictions. This leads to an increase in AP^[0,0.1]. Further, we integrate multi-frame cross-attended Re-ID features, feeding them into the expander with concatenation. This boosts tracking and out-of-frame metrics.**
|  | Detection Metrics | Tracking Metrics |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Method | AP^[0,0.1] | AP^[0.1,0.8] | AP^[0.8,1] | AP${}^{\textrm{OoF}}$ | AP | AP | AP^[0,0.8] |
| Baseline (GTR []) | 0.8 | 13.2 | 37.5 | 14.2 | 28.2 | 16.0 | 8.9 |
| Amodal Expander | 0.8 | 16.4 (+3.2) | 37.7 (+0.2) | 17.6 | 29.9 (+1.7) | 16.4 | 10.1 |
| + Kalman filter | 1.8 | 15.8 | 36.3 | 16.4 | 29.0 | 16.0 | 10.1 |
| + Depth [] | 2.0 (+1.2) | 16.1 | 36.8 | 16.8 | 29.4 | 15.9 | 10.0 |
| Amodal Expander<br>+ Temporal Re-ID | 0.7 | 16.2 | 37.7 | 17.8 (+3.6) | 29.8 | 17.1 (+1.1) | 11.0 (+2.1) |

## 4 Amodal tracking

#### 4.0.1 Traditional and amodal tracking.

Given a sequence of images $I^{1},I^{2},...,I^{t}$, tracking approaches aim to output modal bounding boxes $b$, trajectory identifiers $\tau$, and class labels $s$ for objects across all frames. If an object is partially occluded, the box marks only the visible extent of the object, as illustrated in Fig. 2. We focus here on amodal trackers, which similarly take as input a sequence of images, but, in addition to the modal tracker outputs, they generate amodal boxes $b_{a}$, which cover the full extent of partially / fully occluded objects.

In practice, training an amodal tracker end-to-end is infeasible due to the limited amount of amodal training data.
We focus instead on transforming a conventional tracker into an amodal one by leveraging its understanding of modal objects. To do this, we introduce a light-weight class-agnostic module $E$.

**Table 5: Evaluating the ‘people’ category. We follow the conventions of Table 3 but evaluate performance only on the people category. Fine-tuned expander shows improvements over modal baseline, which can be observed in Fig. 7. We posit that this dramatic performance increase comes from the fact that people is the most common category. PasteNOcclude (PnO) leads to a slight drop for this category, which suggests that adding synthetic (occluded) examples is more helpful for less common categories.**
|  | Detection Metrics | Tracking Metrics |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Method | AP^[0,0.1] | AP^[0.1,0.8] | AP^[0.8,1] | AP${}^{\textrm{OoF}}$ | Overall | Overall | AP^[0,0.8] |
| GTR [] | 0.29 | 37.15 | 71.49 | 42.07 | 53.81 | 17.47 | 14.39 |
| FT regression head | 0.41 | 49.32 | 78.93 | 53.26 | 61.36 | 20.44 | 18.74 |
| Amodal Expander | 2.26 | 71.64 | 84.07 | 73.74 | 74.22 | 26.77 (+9.30) | 28.94 |
| Amodal Expander^† | 2.46 (+2.17) | 71.86 (+34.71) | 84.21 (+12.72) | 73.96 (+31.89) | 74.34 (+20.53) | 26.72 | 28.95 (+14.56) |
| Amodal Expander + PnO | 1.94 | 69.87 | 83.86 | 72.58 | 73.20 | 26.68 | 28.76 |
| Amodal Expander + PnO^† | 1.99 | 70.23 | 84.00 | 72.85 | 73.38 | 26.61 | 28.64 |

### 4.1 Amodal expander

We design an amodal expander $E$, which serves as a plug-in module to conventional trackers. For each object, the amodal expander takes as input the modal box $b$ and an embedding $f$ (which can be extracted from the tracker), and generates amodal bounding boxes $b_{a}$.

#### 4.1.1 Predicting amodal boxes in a residual manner.

The amodal expander operates as a refinement step, similar to the second stage of two-stage detectors [, ] and trackers [].
We introduce the amodal expander in the context of GTR [], although it can be applied to most standard modal trackers.
As illustrated in Fig. 5, GTR produces modal boxes $b$ with corresponding object features $f$, and subsequently refines $b$ through a regression head $R$ by predicting a modal box delta $\Delta b$.
Our amodal expander takes as input the modal box delta $\Delta b$ and object feature $f$ as input, generating an amodal box delta.
This delta is then applied to the modal proposal $b$ to generate amodal boxes $b_{a}$, denoted as $E(\Delta b,f)+b\approx b_{a}$.
The training of the amodal expander follows the training of regression head [] by matching box proposals with a ground truth and applying regression loss.
We first match modal box predictions $b$ to a modal ground truth $b^{*}$. Then, we apply the regression loss, selected as smooth L1 [], with the corresponding amodal ground truth $b^{*}_{a}$:

$$ $\displaystyle L(b,\Delta b,f)=L_{reg}(E(\Delta b,f)+b,b^{*}_{a})$ (1) $$

As shown in Tab. 6, the matching strategy is crucial for training the expander.
Fig. 6 illustrates the amodal expander, and we provide implementation details in Sec. 5.2. We demonstrate the effectiveness of amodal expander in Tabs. 3 and 5.

**Table 6: Ablation: Region proposal matching strategy. Given that modal trackers generate modal proposals, an improved strategy involves matching region proposals with modal ground truth (GT) while applying regression loss to amodal predictions against the amodal GT. Both expander models are trained with Paste-and-Occlude (PnO) on TAO-Amodal training set for 20k iterations.**
|  | Detection AP | Tracking AP |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Matching | AP^[0.1,0.8] | AP${}^{\textrm{OoF}}$ | AP | AP | AP^[0.1,0.8] |
| Modal GT | 13.96 | 14.92 | 28.64 | 16.45 | 8.96 |
| Amodal GT | 16.41 | 17.64 | 29.87 | 16.35 | 10.13 |

### 4.2 Synthesizing occlusion with Paste-and-Occlude (PnO)

To simulate occlusion scenarios during training, we use a data augmentation technique inspired by [, ], which we refer to as Paste-and-Occlude (PnO).
Paste-and-Occlude functions by pasting object segments into the original images to act as occludees.
The segment collection comprises 505k objects extracted from LVIS [] and COCO [] images using segmentation masks.
For each input image, we randomly select 1 to 7 segments from the collection and paste them at arbitrary locations, allowing for partial extension beyond the image boundary to replicate out-of-frame situations.
Subsequently, we incorporate the ground truth boxes of the pasted segments into the original ground truth masks. We find that PnO leads to improvements in detection across all occlusion scenarios, shown in Table 3.
We posit that this synthetic strategy is particularly important for the long-tailed nature of TAO-amodal, unlike COCO-amodal, where a similar synthetic occlusion strategy leads to limited improvement [].
We provide visual examples of synthetic occlusions in the appendix.

Figure: Figure 7: Qualitative results of Amodal Expander on TAO-Amodal val. Trackers fine-tuned with expander produce both modal and amodal predictions. The expander amodally complete objects that are occluded by objects in the scene (bottom-left) or objects that lie partially out of frame. We further verify that fine-tuned expander can amodally complete objects that were occluded in the past as well as objects that become occluded later.

## 5 Empirical analysis

In  Sec. 5.1, we assess the challenges of amodal detection and tracking by evaluating a number of amodal trackers and segmentors.
Next, we investigate fine-tuning strategies and extensions for amodal baselines in Sec. 5.2.
We present implementation details and further ablations in the appendix.

#### 5.0.1 Evaluation Metrics.

Using the estimated visibility attributes, we assess the tracking and detection capabilities of the model through variations of detection AP [] and Track-AP [], representing the average precision across all categories at an IoU threshold of 0.5.
We label objects with visibility less than 0.1 as heavily occluded, evaluated as AP^[0.0, 0.1], where the superscript indicates the range of object visibility. If the visibility falls between 0.1 and 0.8, we categorize them as partially occluded, while those with visibility greater than 0.8 are considered non-occluded.
Objects that extend beyond the image boundary are referred to as out-of-frame (OoF) and evaluated with AP^OoF.
Additionally, we assess the model’s performance on modal annotations with Modal AP.
In tracking, we evaluate highly or partially occluded tracks (Track-AP^[0, 0.8]), which are track with visibility at or below 0.8 for more than 5 frames (seconds).
We also evaluate performance on modal annotatiorns (Modal Track AP).
We provide a table of metric definitions in the appendix for quick reference.

### 5.1 Benchmarking state-of-the-art trackers

Evaluation of modal detectors and trackers. We use three recent modal trackers, QDTrack [], TET [] and GTR [] and a detector, ViTDet [] for benchmarking. Every modal tracker is pre-trained on either TAO [] or LVIS [], ensuring alignment of category vocabulary with our dataset. GTR [] is trained on the combination of LVIS and COCO [] by generating synthetic videos using the training strategy in []. QDTrack [] and TET [] follow similar training procedures, pretraining detectors on LVIS and instance similarity heads on TAO for association. ViTDet [] is trained on LVIS and combined with online SORT [] tracker. We further adapt all models for amodal tracking by fine-tuning the regression head on TAO-Amodal training set for 20k iterations.

Evaluation of off-the-shelf amodal segmentors.
While we finetune modal algorithms for the amodal task, we note that these may not be architecturally optimized for amodal perception. To this end, we also evaluate state-of-the-art methods from the amodal segmentation line of work (note that mask prediction is more prevalent for amodal perception than box prediction). We benchmark ORCNN [], Amodal Mask-RCNN [], AISFormer [] and PCNet []. ORCNN proposes a loss which brings occluders and ocludees spatially close. Amodal Mask-RCNN trains an additional amodal mask head on top of Mask-RCNN. AISFormer, also based on Mask-RCNN, uses transformer blocks in its amodal mask head to learn the spatial relations between visible and occluded objects. These methods only need an image as input and are trained on COCOA-cls []. PCNet takes in modal masks of all objects in the scene as input, and recovers their relative ordering in the scene, before expanding modal masks into amodal ones. We use Detic [] to get these modal masks. Finally, we run SORT [] on top of all boxes obtained from aforementioned methods and evaluate with proposed metrics only on COCO classes. PCNet shines likely because it only needs to expand modal masks, which is a smaller lift than other baselines.

How well do SOTA methods handle amodal perception?
In Tab. 2, we see that both amodal segmentation baselines and fine-tuned modal trackers struggle in handling heavy occlusion and out-of-frame cases. To bridge the gap, we further explore different fine-tuning schemes and effects of data augmentation in Tab. 3, introduced in the next section. We report the performance of modal trackers on TAO-Amodal validation set as an ablation in the appendix.

Figure: Figure 8: Qualitative results of Amodal Expander across diverse categories on TAO-Amodal val. Though we achieve the most impressive results for people, our Amodal Expander is effective across a diverse set of categories.

### 5.2 Building amodal baselines with amodal expander

We illustrate amodal expander architecture in Fig. 6. We build the expander on top of GTR [] as this method shows reasonable performance in both detection and tracking aspects in Table 2, likely due to its transformer-based association architecture that links identities over longer time periods with a sliding window of size 16. The hidden dimension of MLP is 256. We apply ReLU [] and dropout [] with a probability of 0.2 to each layer except the last one. We train the amodal expander on the TAO-Amodal training set, along with PasteNOcclude (PnO) and augmentation used in GTR []. All the modules except the amodal expander are frozen during training. More ablation studies, hyperparameter details for training and PnO can be found in the appendix.

Explore fine-tuning strategies for amodal perception.
We explored several fine-tuning strategies including amodal expander on TAO-Amodal validation set as shown in Tab. 3. Amodal expander trained with PnO for 45k iterations achieves 3.29% and 3.47% performance win under partially occluded (AP^[0.1,0.8]) and out-of-frame (AP^OoF) scenario. Fine-tuning entire model or solely the regression head and proposal network results in performance degradation. We posit that, with only 500 amodal training sequences, the models struggle to completely discard modal knowledge. Fine-tuning box regression head is suboptimal when compared to amodal expander. Amodal expander further provides flexibility to adjust the architecture and select different input information, which are both important as shown in the ablation provided in appendix.

Integrating multi-frame signals into amodal baselines.
In Tab. 4, we present two strategies for using temporal information within the amodal expander: 1) using a Kalman filter to forecast occluded object locations, with a monocular depth estimator to filter erroneous predictions, following [], and 2) incorporating temporal Re-ID features. Note that (1) can associate single-frame detections, while also predicting new boxes when an object is completely occluded. This significantly improves AP^[0,0.1].
For (2), we take multi-frame Re-ID features and feed them into the amodal expander with channel concatenation. This helps handle out-of-frame occlusion while improving tracking metrics.

Detecting people with amodal expander.
In Table 5, we study how well the expander baseline detects and tracks people, which serves as a crucial category in many autonomous driving and tracking benchmarks. Amodal expander obtains a significant improvement compared to the modal baseline, particularly on AP^[0.1, 0.8] and AP^OOF. Tracking on highly or partially occluded people (Track-AP^[0.0,0.8]) also increases by 14.6%. This shows that one can obtain an effective amodal people tracker that could also track objects of diverse category vocabulary with our dataset using a simple fine-tuning scheme.

Importance of proposal matching strategies.
To apply regression loss, training a box prediction head requires matching each region proposal to a ground truth box. A naive strategy is to directly match the region proposals to the amodal ground truth box. However, direct matching with amodal boxes leads to suboptimal results as shown in Tab. 6. As standard trackers generate modal region proposals, the model faced challenges in aligning proposals with the accurate ground truth due to a low Intersection over Union (IoU) between modal proposals and amodal ground truth. Matching proposals with modal boxes and applying regression loss using amodal ground truth yield better results.

## 6 Discussion

In this work, we focus on amodal perception of real-world objects. We draw inspiration from cognitive functions of amodal completion and object permanence in humans, that develop at an early age. Despite this, advancements in perception stacks like object detection and tracking, do not make amodal understanding central. We bring focus to three aspects/stages of building amodal perception stacks. First, we contribute a benchmark that annotates 833 categories of objects amodally in unconstrained indoor and outdoor settings, under partial and complete occlusion. Second, we contribute a benchmarking protocol in the form of metrics that evaluate detection and tracking specifically for the cases of partial or complete occlusions. Our key finding here is that existing algorithms struggle on these metrics. To address the observed limitations in amodal detection and tracking performance, and as a third contribution, we investigate data augmentation and fine-tuning schemes to boost existing tracking algorithms.
We hope our empirical evaluation provides a foundation for improving amodal perception.

## 7 Implementation details

#### 7.0.1 Training Amodal Expander.

We trained amodal expander on TAO-Amodal training set for 20k iterations for all experiments unless specified. We used a 2-layer MLP as the architecture. The hidden dimension of MLP is 256. We apply ReLU [] and dropout [] with a probability of 0.2 to each layer except the last one. We implemented the expander in conjunction with GTR []. Architecture details of GTR align with the selection in the prior work []. We used 0.01 as the base learning rate and applied WarmupCosineLR [] as the scheduler. The optimizer is AdamW []. The batch size for training is 4. We adopted the training methodology outlined in [], treating each image as an independent sequence. We applied data augmentation [], including random cropping and resizing, to each image to produce synthetic videos with a length of 8 frames. Beyond this, we further applied PasteNOcclude, introduced in Sec. 4.2 in the main paper, on top of the synthetic videos to automatically generate more occlusion scenarios. We provide the hyperparameter details of PasteNOcclude in the next section.

PasteNOcclude (PnO).
We illustrated visual examples of PnO in Fig. 9. We mask the background area with the segmentation mask and collect the cropped object from LVIS [] and COCO [] to serve as occluders. We filter out segments where the mask area is less than 70% of the bounding box area to ensure that the occluder is not occluded. In the training process, we view each image as a sequence and create an 8-frame sequence employing the data augmentation strategy in GTR [] based on each image. Subsequently, we randomly select 1 to 7 segments from the collection and place them at random locations. Further, we randomly adjust the height and width of the inserted segments within the range of $[12,192]$. We randomly determine the object’s location and size only in the first and last frames to ensure smooth transitions between consecutive frames. The size and location in intermediate frames are obtained through interpolation.

## 8 More empirical analysis

We used the evaluation metrics defined in Sec. 5.1 in the main draft. We summarize all the definitions in Table 13. We presented additional experiments involving state-of-the-art trackers in Section 13 and the amodal expander in Section 8.2.

### 8.1 Benchmarking off-the-shelf-trackers

#### 8.1.1 Evaluation on TAO-Amodal validation set.

We report detection and tracking average precision (AP) numbers of SOTA off-the-shelf trackers on TAO-Amodal validation set running at 1fps with an IoU threshold 0.5 in Tab. 7. We also observed similar performance trends when running at 5fps with higher IoU thresholds, shown in Tabs. 9 and 8. Every off-the-shelf tracker was trained on either TAO [] or LVIS [], ensuring alignment of category vocabulary with our dataset as detailed in Sec. 5.1. We reproduced AOA [] using their released implementation, with object detector trained on LVIS and tracking ReID head trained on TAO.

#### 8.1.2 Using off-the-shelf modal trackers for amodal perception.

Table 7 reveals notable differences in detection AP between modal (Modal AP) and amodal annotations (AP), amounting to an 8.49% difference. Additionally, the amodal tracking AP experiences a substantial decline compared to modal tracking AP.
These results highlight the difference between amodal and modal perception.

#### 8.1.3 How well do standard trackers handle occlusion?

Existing off-the-shelf trackers exhibit reasonable performance in detecting non-occluded objects, with ViTDet achieving 39.71% AP^[0.8,1] as revealed in Table 7. However, all trackers face challenges in handling heavily occluded, partially occluded (AP^[0.1,0.8]) and out-of-frame (OoF) scenarios. We noticed that ViTDet operating at 5 fps benefits from the property of SORT to estimate the location in the current frame using past information in Tab. 9. Nevertheless, this improvement comes at the cost of processing ViT-Det on 5x more frames than models running at 1 fps. In contrast, amodal completion could be a promising way for efficiently handling occlusion.

**Table 13: Evaluation metrics with IoU threshold 0.5. We define variations of AP [] and Track-AP [] based on levels of occlusion.**
| Metric | Definition | Type |
| --- | --- | --- |
| AP | Average Precision (AP) averaged across all categories at an | Detection Metrics |
| IoU threshold 0.5. |  |  |
| AP^[0, 0.1] | AP for heavily occluded objects, with visibility smaller than 0.1. |  |
| AP^[0.1, 0.8] | AP for partially occluded objects, with visibility in [0.1, 0.8]. |  |
| AP^[0.8, 1.0] | AP for non-occluded objects, with visibility larger than 0.8. |  |
| AP^OoF | AP for partially out-of-frame (OoF) objects. |  |
| Modal AP | AP on modal annotations. |  |
| Track-AP [] | Average Precision of a track averaged across all categories at an | Tracking Metrics |
| 3D IoU threshold 0.5. |  |  |
| Track-AP^[0, 0.8] | Track-AP for any track that is occluded, with visibility |  |
| at or below 0.8, for more than 5 frames (seconds). |  |  |
| Modal Track-AP | Track-AP on modal annotations |  |

Evaluation with higher IoU thresholds.
In Table 8, we evaluate the trackers with average precision (AP) averaged over 10 IoU thresholds from 0.5 to 0.95 at a step 0.05. The performance trend basically aligns with what we observed in Table 7 in the main paper. GTR [] obtained strong performance in both detection and tracking. When evaluated with higher IoU thresholds, ViTDet [] and SORT [] demonstrate inferior detection performance compared to GTR, indicating a contrasting outcome compared to the results obtained at a 0.5 threshold. This shows the limitations of SORT [] in accurately estimating bounding boxes.

Running trackers at higher fps.
We reported the performance of state-of-the-art trackers running at 5 fps in Tab. 9. We noticed that ViTDet [] along with SORT [] achieved the best performance among all the trackers. This aligns with our intuition as SORT estimates the location in the current frame based on prior-frame locations. This property benefits from running at higher fps, but it requires processing ViTDet on 5×more frames than models operating at 1 fps, heavily increasing computational demands.

### 8.2 Amodal expander experiments

#### 8.2.1 Scaling up training data.

In Tab. 10, we scale up the training data to 4x by including test videos as train set and evaluate the amodal expander on the validation set. We note that simply increasing the size of the training data does not significantly improve the metrics compared to results shown in Tab. 2. This validates our design to propose TAO-Amodal as an evaluation benchmark.

#### 8.2.2 Investigating key information for amodal box inference.

Table 11 reports different input choices to the amodal expander. Modal box (deltas) $\Delta b$, output by the regression head as shown in Fig. 5 in the main paper, are used to yield final modal box predictions when applied to region proposals and thus contain information about the exact location of modal box predictions. Proposal features includes visual appearance information of the detected region proposals. Absence of visual cues significantly diminishes the performance of both detection and tracking under occlusion. Interestingly, the amodal expander, incorporating both modal delta and proposal features, yielded the most favorable outcomes. This indicates that estimating modal box locations also contributes to effective amodal reasoning.

#### 8.2.3 Number of MLP layers.

We tested with the depth of amodal expander architecture in Table 12. We observe a reverse-U pattern concerning the number of MLP layers, with two-layer MLPs demonstrating superior performance compared to other models. A one-layer MLP proves suboptimal in both detection and tracking. Notably, using a 1-layer MLP results in slightly inferior outcomes compared to fine-tuning the regression head, as indicated in Table 3 in the main paper. We argue that the regression head may derive benefits from pre-training on modal benchmarks.

**Table 14: Annotation guidelines. TAO-Amodal is annotated with the guidelines above, which taxonomizes occlusions across severity (partial versus complete) and type (in/out-of-frame). As mentioned in Sec. 3 in the main paper, we scope out the case where an object may be present behind the camera. For out-of-frame occlusions, we limit the annotation workspace to be twice the image size.**
| Occlusion type | Extent | Cases | Instructions |
| --- | --- | --- | --- |
| In-frame | Partial | Partially occluded before being fully visible | Annotate with best estimate using category label |
|  | Partially occluded after being fully visible | Annotate with best estimate |  |
| Complete | Invisible before being (partially) visible | Only annotate if the object has been visible before |  |
|  | Invisible after being (partially) visible | If confident, annotate with best estimate |  |
|  |  | If not, only annotate till the last visible frame |  |
|  | Invisible for a while | If confident, annotate with best estimate |  |
|  |  | If not, still annotate but add an uncertainty flag |  |
| Out-of-frame | Partial | Object goes beyond image border | Only annotate inside the annotation workspace |
|  | Object goes beyond the padded image | Clip at the border of the padded image |  |
| Complete | - | - |  |
| Behind-the-frame | Partial | Object is in front of and behind the camera | Only label the part of object in front of camera |
| Complete | - | - |  |

## 9 Annotation guidelines

We ensure high-quality annotations by requiring annotators to follow the guidelines detailed in Table 14. Our coverage spans various occlusion scenarios, encompassing in-frame, out-of-frame, or behind-the-scene situations, where an object may be partially obscured behind the camera.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: The contributions of this work seem small and the motivation is unclear to me.
soundness: 2
strengths: 1. A comprehensive annotation for amodal tracking based on TAO dataset is performed.

2. A plugin module for amodal tracking and some data augmentation methods are designed.

3. Extensive experiments and analysis are conducted.
summary: This work presents a large-scale video benchmarks for amodal tracking, and the evaluation metrics and baseline trackers are also designed.
The challenges of amodal detection and tracking by evaluating a number of amodal trackers and segmentors are validtated on the created benchmark.
weaknesses: 1. The contributions of this work seem small. TAO dataset is existing, and the contribution to the benchmark is large-scale annotations. The designed expander is also a simple regressor and data augmentation schemes are also based on existing ones.

2. The motivation of this task is unclear to me. When an object is totally occluded, its state, including position, size and motion, is very difficult to predict. Although authors consume much time to annotate such objects, but the quality can not be guaranteed because we do not know their real states. What are the potential downstream applications or benefits of amodal tracking that motivate this work? How might uncertainty in amodal predictions be handled or utilized in subsequent tasks?

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: Please see the weakness. If my concerns are solved, I will consider raising my score.
soundness: 3
strengths: The authors present TAO-Amodal, a comprehensive and high-quality dataset designed to advance amodal perception by providing diverse occlusion annotations across 833 categories and introducing the Paste-and-Occlude (PNO) augmentation method, which simulates various occlusion scenarios to improve model robustness. This dataset enables rigorous benchmarking and reveals the limitations of current tracking algorithms under challenging conditions, offering a reliable foundation for future developments in amodal tracking research.
summary: The TAO-Amodal dataset method was introduced to address the amodal perception problem. It provides video sequences with 833 categories, including amodal and modal bounding boxes for visible and partially or fully occluded objects. This approach aims to address the scarcity of amodal benchmarks. Benchmarking current modal trackers and amodal segmentation methods reveals that existing methods struggle with detecting and tracking heavily occluded objects. Simple fine-tuning schemes were explored to mitigate this, improving the amodal tracking and detection metrics for occluded objects by 2.1% and 3.3%, respectively.
weaknesses: 1、Limited Training Data: While the dataset is extensive for evaluation, the relatively small training set may restrict model generalization, especially for categories with fewer instances. This also means that subsequent research can only use this dataset for validation, making it ineffective for robust training.
2、During the data annotation phase, I noticed that the authors conducted detailed annotations, but I have a question: how did you determine the size of the occluded objects for annotation clearly?
3、The comparison is insufficient. The TAO dataset itself is an open-world dataset, but your evaluation method does not incorporate any open-world detection or open-world tracking approaches.
4、The examples of PasteNOcclude (PnO) shown in Figure 5 are overly simplistic. I want to know whether all data augmentation is done in this way.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: Please see the above.
soundness: 2
strengths: The paper annotated a large-scale dataset that extends the capabilities of existing modal object tracking systems to amodal scenarios.

The authors benchmark several state-of-the-art tracking methods to evaluate the dataset.

To address occlusion challenges, the paper presents a series of new evaluation metrics suitable for tracking and detecting heavily and partially occluded objects. These metrics enhance the granularity and accuracy of existing evaluation methods.

The writing of the paper is good and easy to follow.
summary: The paper introduces a new benchmark dataset, TAO-Amodal, which focuses on amodal perception. This is the ability to understand the full extent of objects, even when they are partially or fully occluded. The dataset includes diverse object categories and scenarios, such as objects that are both partially or fully occluded, and those that may be out of the frame in video sequences. The authors benchmark several state-of-the-art tracking methods and introduce fine tuning strategies that improve amodal tracking performance slightly.
weaknesses: 1. The main contribution of the paper is building a dataset for occluded objects. However, in Table 1, NuScenes has more Occlude boxes and tracks. Besides, the annotation FPS of TAO-amodal is only 1 fps, which is much smaller than previous datasets.


2. Although this is a paper primarily contributing a dataset, it is better to propose a method related to the most critical point of this paper (occluded objects) in order to effectively convey the importance of this work. The proposed fine-tuning method is too simple. The technique innovation is limited. 


3. How can the quality of the annotations in the submitted dataset be ensured? There should be some quantitative metrics to demonstrate the accuracy of the annotations.


4. The TAO-Amodal training set is relatively small, which may restrict the ability to train models from scratch and may require pretraining on modal datasets.

5. What are the primary obstacles preventing end-to-end multi-object tracking (MOT) models from achieving optimal performance on the TAO-Amodal benchmark? What factors primarily contribute to this difficulty?

6. Despite fine-tuning and data augmentation, the performance improvements (2.1% in tracking and 3.3% in detection) are modest, indicating that current systems still struggle with occlusions.

7. Another important issue is the impact of the performance on normal object detection/tracking. It would be problematic if we only obtain minor performance improvements on amodal targets, but gain worse results for normal objects.

8. The experiments conducted in the paper primarily focus on a single dataset (TAO-Amodal). Broader testing across multiple datasets could provide more compelling results.

9. In the experimental section, further Analysis of Data Annotation Uncertainty is needed. The paper does not thoroughly explore the specific impact of uncertainty on evaluation outcomes, which warrants deeper investigation.

10. The compared methods are somehow out-of-date. Thus, the experimental results are not convincing.

11. More details about the baseline are needed, such as network architecture configurations.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 2
questions: 1. Could the authors provide more detailed analysis of how performance varies across different object categories?
2. The paper would benefit from more discussion about the annotation uncertainty in heavily occluded cases and its impact on evaluation metrics.
soundness: 3
strengths: 1. The paper introduces the first large-scale amodal tracking benchmark that encompasses a diverse range of object categories, which is a significant advancement in the field. This benchmark addresses a critical gap in current datasets and tracking evaluations, as it provides a platform to assess and improve tracking algorithms under occlusion scenarios. 
2. The authors have developed a comprehensive annotation protocol that includes rigorous quality control measures, ensuring the reliability of the dataset. They have thoughtfully considered both in-frame and out-of-frame occlusions, which are essential for a holistic evaluation of tracking algorithms. 
3.The dataset is constructed on a large scale, with 332k boxes annotated across 2,907 videos, providing a substantial dataset for training and evaluating tracking models. It covers an impressive 833 object categories, making it one of the most diverse datasets available. The quality of the annotations is exceptionally high, with over 99% of the annotations meeting the quality standards set by the authors, which is crucial for the development and testing of robust tracking algorithms.
summary: This paper introduces TAO-Amodal, a large-scale benchmark dataset for amodal object tracking, built upon the existing TAO dataset. The work addresses an important gap in computer vision by providing annotations for tracking objects under heavy occlusions and out-of-frame scenarios across 833 diverse categories in 2,907 videos. The authors evaluate current trackers and propose simple but effective finetuning schemes to improve amodal tracking performance.
weaknesses: 1. The training set appears to be relatively small compared to the validation and test sets. Could the authors explain the rationale behind this dataset split? It would be useful to discuss how the smaller training set might impact model performance. Additionally, considering options like data augmentation could help maximize the utility of the available training data and potentially improve model robustness.
2. A more detailed analysis of failure cases, such as full occlusion,  would be advantageous, as it would help to better understand the limitations of the models and potential areas for improvement. Including metrics or visualizations to highlight these limitations would add depth to the analysis and offer valuable insights for future research.
3.To maintain the state-of-the-art in tracking research, the authors are encouraged to incorporate the latest tracking methodologies, such as Hybrid-SORT [1] and MeMOTR [2], into the study.

[1] Yang, M., Han, G., Yan, B., Zhang, W., Qi, J., Lu, H., & Wang, D. (2023). Hybrid-SORT: Weak Cues Matter for Online Multi-Object Tracking. AAAI Conference on Artificial Intelligence.
[2] Gao, R., & Wang, L. (2023). MeMOTR: Long-Term Memory-Augmented Transformer for Multi-Object Tracking. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), 9867-9876.