Prototypical Cross-Attention Networks for Multiple Object Tracking and Segmentation

Abstract: Multiple object tracking and segmentation requires detecting, tracking, and segmenting objects belonging to a set of given classes. Most approaches only exploit the temporal dimension to address the association problem, while relying on single frame predictions for the segmentation mask itself. We propose Prototypical Cross-Attention Network (PCAN), capable of leveraging rich spatio-temporal information for online multiple object tracking and segmentation. PCAN first distills a space-time memory into a set of prototypes and then employs cross-attention to retrieve rich information from the past frames. To segment each object, PCAN adopts a prototypical appearance module to learn a set of contrastive foreground and background prototypes, which are then propagated over time. Extensive experiments demonstrate that PCAN outperforms current video instance tracking and segmentation competition winners on both Youtube-VIS and BDD100K datasets, and shows efficacy to both one-stage and two-stage segmentation frameworks. Code and video resources are available at http://vis.xyz/pub/pcan.


Reviews and author responses:

 The rebuttal has addressed my concerns, and all the reviewers agree on the good performance of the proposed method and its contribution to the community, so I keep my initial rating. Hope the authors can make the suggested edits so the technical details can be conveyed accurately.

 Thanks for the feedback, my concerns were addressed well and I kept my rating. I left a few more comments in the finalized review! 

This paper provides a Prototypical Cross-Attention Networks (PCA) for Multiple Object Tracking and Segmentation (MOTS). Two prototypical cross-attention modules are adopted for frame-level and instance-level features to retrieve information from past frames. By applying prototypical cross-attention, computational cost has been reduced. Experiments on two datasets demonstrate the effectiveness of the proposed method.
  

### Originality: 
The prototypical cross attention using clustering is new for MOTS. It reduces computational complexity for obtaining spatial temporal information compared to spatial temporal attention.


### Quality:
The method is technically sound. However,  many designs in the paper are not validated. 
For instance, the impact of the number of cluster N is not validated. The impact of each module to the tracking performance is not clear.


### Clarity:
The paper is badly written. Many crucial details are missing. In figure1, the aggregate feature $\bar{y}_t$ is used for classification, detection, tracking, and segmentation. However, there are no descriptions in the paper to elaborate the processes on $\bar{y}_t$. How to construct positive and negative instance prototypical?  Training details are also missing in the paper. 

### Significance: 
The proposed method outperforms SOTA one-stage methods on two MOTS datasets.
 1. This paper proposes PCM to reduce computational cost for online tracking and segmentation. However, 15 fps is much slower than 28fps in [18]. Authors should report FLOPS for compared methods. 

2. I'm confused by Figure 1, 2 and 3.  The shapes of frame embedding in Figure 1, 3 are the same, but are different from Figure 2. 
  The frame embedding is passed to PCM in Figure 1 and is passed to the Multiple-level temporal aggregation module in Figure 2.  
However, the frame embedding is passed to TSM for mask prediction in Figure 3. The TSM module is also not mentioned in the text. 

3. What is the number of mixtures in GMM (N) during training? What is the temporal length $t$ during training? Many important training details are missing. The impact of N is not validated. Impacts of modules should be summarized in a table.

4. Instance-level prototypes only have two mixtures (foreground and background). Are all instances belonging to the foreground prototypical?  What do prototypes stand for? How to infer a tracking mask with $\bar{y}_t$ and $\bar{k}$?


 Thank to authors for providing the detailed response. All my concerns are well addressed, so I changed my score to accept paper.

 Thank you for providing the detailed rebuttal. They are satisfactory so I moved my score from 6 -> 7. Thank you!

The paper focuses on making transformers for multi-object-tracking-and-segmentation (MOTS) more efficient, where typically all pixels from the past frames have to "transform" with all current pixels. The trick in the paper is to group past pixels into prototypes (clusters) and hence reduce complexity. 

The authors claim contributions in:

- Improve transformers's efficiency for dense segmentation by operating at a per cluster (prototype) level rather than per pixel for past frames.

- Applied the method to MOTS.

- Further improvement by introducing foreground and background prototypes.

- Experiments and above SOTA performance.

I agree with the novelty assessment. I am not an expert in MOTS, without looking up additional comparison, this work seems to do well on Berkeley deep drive (BDD) and Youtube-VIS, although some of the numbers in the paper do not agree with what are in original papers (see below).  I am borderline on this paper, leaning acceptance, based on sufficient novelty and good experimental results compared to SOTA. My concerns are a bit in the ablation and the mismatch between cited SOTA numbers and what they are from papers.

Sufficient novelty: The novelty of grouping pixels into clusters and use them as unit for the transformer seems like a very simple idea to improve the efficiency. I was honestly quite surprised that I couldn't find a work that has done this before. Assuming that the idea is indeed new, it should be sufficient for this venue.

Sufficient experiments against SOTA: The authors performed experiments on both the BDD and the Youtube-VIS dataset and showed strong results against SOTA. My only comment is that the VisTR numbers quoted in this paper don't match those in the original publication (e.g. 40.1 AP). Please elaborate the difference.

Questions:

Ablation: The motivation of the work is to improve the efficiency by grouping pixels together. Is there any experiments showing how much accuracy we lose by doing this? If not feasible, how about trivial grouping such as a simple tiling of the input images?

Nits:
- For equations, there shouldn't be a point after each equation.

 Since the model keeps all prototypes from all past frames, a limitation is that it won't scale to infinite time. However, a simple fix is to "forget" some prototypes after a while.

Yes on societal impact, no comment.

This paper builds upon a mask propagation family of methods that leverage memory mechanism to decode current-frame segmentation mask(s) (assuming the first-frame mask is given, either specified via human or predicted by a network). In a nutshell, these methods will need to attend to all pixels stored in the memory (that effectively represents a spatiotemporal volume), leading to high computational cost (quadratic wrt. the image size). This paper proposes a very interesting alternative: instead of computing cross-attention maps between query and the spatio-temporal volume (memory), attending to WxHxT pixels, the memory will be clustered into a fixed (pre-defined) number of components, modeled as Gaussians. As a consequence, attention will be computed over cluster representatives (means), i.e., N-prototypes. This yields a significant reduction in the complexity (attending to N-prototypes instead of WxHxT pixels) and (as Tab. 5 shows) doesn’t have a negative impact on the final performance (works even slightly better). The experiments conducted on Youtube-VIS and BDD100k dataset confirm that the presented method is effective and outperforms the state-of-the-art, and design decisions are well-justified via well-designed ablation studies. 
  
Pros

--------------

* This paper is well-written and motivated, technically sound, and presents a very nice solution to a relevant problem. Methods that leverage Spatio-temporal memory are highly competitive and very promising research direction in the field of tracking and segmentation, and the remedy for the memory issue proposed in this paper could lead to winder adaptation of this family of methods! 
* The paper presentation is (mostly) fine. Certain parts could be connected better and certain explanations could be refined (e.g., it should be explained sooner how the learned clustering will be utilized to reduce the complexity; also, 3.3.2 is a bit verbose), but I think that is doable within a few additional passes/iterations over the paper. 
* Experimental evaluation is convincing, even though the lack of experiments on the standard MOTS dataset (Voigtlander et al., CVPR'19) and VOS datasets (Perazzi et al., CVPR'16) is rather surprising. I especially appreciate the thorough ablation studies, clearly demonstrating how the performance improves with the size of the memory bank (wrt. the temporal dimension) and demonstrating that the prototypical cross-attention model does not hurt the performance. 
* This paper makes a big step towards the applicability of memory-based methods for tracking and segmentation-related tasks, and I am very excited about the prospects of that! 

Cons

--------------


* While I am fairly convinced with the experimental evaluation (i.e., all claims are sufficiently supported experimentally), I was quite surprised to see that no experiments were conducted on the MOTS dataset- why? This is surprising, as this is the first MOTS dataset with a public test set containing a significantly larger number of entries compared to three baselines reported in Tab 2. If the model could not be trained due to a small amount of training data, it would be nice to discuss that. 
* Given that the proposed method tackles mask propagation through the volume, why is not this method evaluated on the video-object segmentation datasets (DAVIS, YouTube-VOS)?
* Presentation-wise, there is a big jump between 3.1 and 3.2; while 3.1 nicely summarises issues with the prior memory-based methods, 3.2 jumps directly into clustering. I feel that a sentence or a paragraph that would provide the intuition on how the prototypes will be used to reduce the complexity would be well in place. 
* If I am not mistaken, this method needs to be applied independently to each mask (either given in the first frame of the clip or detected on the fly in case of MOTS). This should be made very very clear and preferably listed as a disadvantage of this method (correct me if I misunderstood this). 
* It should be made more clear what aspect of MOTS task this paper tackles precisely: i.e., only mask propagation and not detection.  


Additional questions:

* Looking at the temporal window size (for memory), the performance did not saturate with T=32. Why didn’t you try bigger temporal windows?
* What is N (#prototypes) in practice, and how was it determined? How does varying N impacts the performance? 


Justification

--------------

Overall I am excited about the contribution of this paper its potential impact on the community. I am certain that the community will also appreciate these findings. Given that this paper is informative, presents an interesting (and novel) solution to a relevant problem, and backs up all claims well experimentally, I think it should be accepted to NeurIPS proceedings. 


Post-rebuttal Comments and Justification

--------------

I would like to thank the authors for a good rebuttal, my doubts were addressed well. I also read comments from my colleagues and I see we have a consensus that this is a good paper that will be valuable to the community. 

I would suggest adding the number of prototypes ablation to the main paper (I see that my colleagues also agree on that). I still think the evaluation on MOTS dataset would make sense, the standard approach is to compare to vision-only based methods. I find a direct comparison to the first (and as simple as possible) TrackR-CNN baseline more meaningful compared to the BDD100k based discussion. 

Overall, I think this paper will open the door for a wider applicability of methods that explicitly leverage external memory to aid data association and keep my original rating.   Yes, the authors did address potential negative social impacts but could do a better job at addressing limitations (see my comments). 

 We thank the reviewer for acknowledging that our method is new, technically sound, and validated by different benchmarks with SOTA performance and reduced complexity. We address all the questions and comments below.

**Q1. The FLOPs and FPS for the compared method SG-net [18].**

We compare PCN with SG-Net on Youtube-VIS using ResNet50 as below. Note that the FPS of SG-Net [18] is 23 instead of 28. SG-Net has no space-time memory reading operations, and it focuses on joint optimization of different task heads, reducing box proposal number and better mask prediction in sub-regions. These contributions are complementary to ours and they can be integrated to further reduce the computational cost of our PCN. 

| Method | AP | FLOPs | FPS  |
| ---- | ---- | --- | --- |
| SG-Net (CVPR' 21) | 34.8 |  69.6 G  |  23  |
| PCN | 36.1 | 92.7 G | 15 |

**Q2a. The confusion on shape embedding in Fig. 1, 2 and 3.**

The shape of frame embeddings in Fig. 1, 2 and 3 are all $C \times H \times W$. In Fig. 1 and Fig. 3, it is drawn in 2D because the figure is a simplified illustration of the whole pipeline, where we show the feature with high resolution ($H \times W$) of multiple frames at different time steps. In Fig. 2, the frame embedding is drawn in a 3D style as it illustrates the frame-level prototypical cross-attention in a detailed manner. 

**Q2b. Different frame embeddings usages.**

The input frame embedding passed to Prototypical cross-attention in Fig.1 corresponds to “Reference Frame Feature” and “Current Frame Feature” denoted in the bottom of Fig. 2. These do not correspond to the “Reconstructed Frame Feature”, which is passed to the Multi-level aggregation. In Fig. 3, the current aggregated frame feature (we use the current frame to aggregate the past frame feature with our frame-level PCN) is used for the TSM module for final segmentation output. 

We regret that there is a small error in Fig. 3, which might cause confusion. Multi-level temporal aggregation, as shown in Fig. 2, is performed before the TSM module in Fig. 3. We will correct this in the next revision by adding a corresponding aggregation block in Fig. 3.

**Q2c. The details of the TSM module.**

The TSM module is composed of 4 layers of 3x3 convs and 1 deconv layer. It takes the current temporally aggregated frame feature, initial detected mask, and fine-grained instance attention maps (produced by instance prototypes mentioned in Q4a) as input, and generates the final mask. Fig. 3 shows its structure. 

**Q3a. The impact of the number of clusters/prototypes N.**

Table 6 analyzes the impact of the number of instance-level prototypes. We selected 30 positive and 30 negative ones for each tracklet, specified on L30 of the supplementary material, since the results saturate for larger numbers.

For the number of frame-level prototypes, we used 64 during training and testing. The results on YouTube-VIS below show that the precision saturates for larger numbers of prototypes.

| Frame Prototypes Num | 8 | 16 | 32 | 64 | 128 |
| --- | --- | --- | --- | --- | --- |
| AP |  32.6 |  33.1  | 33.9 | 34.2 | 34.1 |
| AP$_{50}$ |  52.8 | 53.3 | 53.5 | 53.7| 53.8 |

**Q3b. The temporal length T during training.**

During training we use a temporal length of T=10 frames. These are sampled without replacement from a larger temporal interval of size 16 to further increase diversity during training.

**Q3c. Summarizing the impact of modules in a single table.**

We summarize the impact of the frame-level and instance-level prototypical attention on YouTube-VIS in a single table below (results are found in Table 1, 3, and 6 in the paper). Both PCN modules bring significant gains in final performance. 

| Method | AP | AP$_{50}$ |
| --- | --- | --- |
| Baseline |  32.5 |  53.0  |
| Baseline + Frame-level PCN |  35.4 | 53.8  |
| Baseline + Instance-level PCN |  33.9 | 54.1  |
| Baseline + Frame & Instance-level PCN |  36.1 | 54.9  |


**Q3d. More training details.**

More training details besides Section 4.1 (paper) is in Section 2 of the supplementary file.

**Q4a. What do prototypes stand for? How to construct the positive and negative prototypes?**


The prototype condenses a set of similar feature vectors in the spatio-temporal memory into a single cluster. Formally, the prototype represents the mean vector $\mu$ of a Gaussian cluster in the Gaussian Mixture Model (GMM) (Eq. 2 in the paper). The prototypes are computed by fitting the GMM using the EM algorithm, as mentioned in L113-119. The instance-level positive and negative prototypes are produced by enforcing GMMs to separately focus on the foreground and background regions of the extracted features for each object. This is done by constructing separate GMMs for the regions inside and outside the initially detected object mask (see L169-175). 

We construct instance-level prototypes for each tracklet independently. Each consists of 30 positive mixture components from the instance itself, and 30 negative mixture components from its surrounding background. Each positive prototype attends on specific parts of the tracklet (such as the head of the person in Fig. 3) based on appearance similarities. The negative prototypes capture valuable information about the background appearance (such as the sky in Fig.3), which can greatly alleviate the segmentation process (shown in Table 7). For easier understanding, the corresponding instance attention weights produced by these prototypes are visualized in Fig. 3 of the paper, and Fig. 2-6 of the supplementary file. 


**Q4b. The process of computing $\bar{y_t}$.** 

The detailed process on producing the aggregated feature $\bar{y_t}$ is illustrated on L155-L165 of the paper. L43-L47 and Fig. 1 of the supplementary file provide more details.


**Q4c. The process of inferring the tracking mask from $\bar{y_t}$ and $k$.**
 
$\bar{y_t}$ (in Fig.2) denotes temporal aggregated frame feature at time $t$, while $k$ (in Fig. 3) denotes the instance-level positive prototypes ($k^+$) and negative prototypes ($k^-$). To produce the final segmentation mask, we concatenate the current aggregated frame feature $\bar{y_t}$, initial mask detections, and instance-level attention maps (produced by $k^+$ and $k^-$). These are then passed to the TSM module (discussed in Q2c) for final segmentation. 

 We thank the reviewer for acknowledging that our work has sufficient novelty, experiments, and above SOTA performance. We will incorporate all feedback and address the questions below. 
 
**Q1. Differences in VisTr results.**

We reported the result of VisTr as 34.4 AP, which is provided in the arXiv v1/v2 version of VisTr (https://arxiv.org/pdf/2011.14503v1.pdf). This was the result available during the CVPR peer review period. Note that the citation in our paper specifies the v1 arXiv version (L354 of paper). The new arXiv verison v3/v4 was submitted in late March and April. The result was changed without ​​explicitly mentioning reasons. Specifically, the mentioned 40.1 AP is also not consistent with the 35.6 AP reported in the official CVPR camera ready version. Note that the latter result was made available after the NeurIPS submission deadline. Our approach, when also using ResNet-50, achieves 36.1 AP. Importantly, our approach is a causal (i.e. online) method, whereas VisTr exploits future frames and is therefore not applicable to online applications such as self-driving. We will update the VisTr result to 35.6 AP, which is reported in the official published CVPR 2021 version:

https://openaccess.thecvf.com/content/CVPR2021/papers/Wang_End-to-End_Video_Instance_Segmentation_With_Transformers_CVPR_2021_paper.pdf


**Q2. The influence of clustering on the final performance.**

In Table 5 of the paper, we compared the prototypical cross-attention to standard attention mechanisms. Our PCN not only achieves high efficiency but also outperforms Non-local attention by 0.6 AP. As mentioned in L110-112, our PCN not only reduces computational complexity, but can also provide a more robust representation of past features by deprecating noisy and spurious information.

**Q3. Limitation when storing infinitely many past prototypes.**

Although this is a common issue faced by space-time memory reading methods, PCN efficiently reduces the storage workload by condensing high-resolution memory into prototypes. This allows our approach to utilize a longer temporal memory. As suggested, the limitation of growing memory can be solved by simply removing old prototypes.


 We thank the reviewer for the valuable feedback and for acknowledging that our work is a solid contribution. We address all comments and questions below.
 
**Q1. How to achieve the balance between discriminability and robustness. - “The person wearing white shirt and black pants example”.**

GMM clustering is not a simple average pooling over each tracklet feature. We construct multiple instance prototypes for each tracklet (30 positive ones) and its surrounding background regions (30 negative ones) by generative appearance modeling. Each positive prototype attends to a specific part of the tracklet, such as the head of the person in Fig. 3 and the front part of the car in Fig. 2 of the supplementary file, based on feature appearance similarities and cluster distribution.

In the mentioned example, “a person wearing a white shirt and black pants”, these two parts will be represented by different prototypes in our model. Thus, there is no averaging of features performed between these two regions. We also extensively visualize the instance attention weights produced by each instance prototype in Fig. 3 of the paper, and Fig. 2-6 of the supplementary file.  

We obtain a balance between discriminability and robustness by selecting the number of GMM components. Its influence on final performance is analyzed in Table 6 and in the response to Q7 of reviewer TiXB. Furthermore, a good trade-off between discriminability and robustness is also partly ensured during training of our PCN, since the feature spaces for the GMM clustering are learned by directly maximizing the final segmentation quality.


**Q2. Correction of Eq.4.**

This is indeed a typo. The $N$ in Eq. 4 will be replaced by $H \times W$ in our next revision.

**Q3. The difference of using L2 distance and cosine similarity in Eq.5. Why use dot product instead of L2 in Eq. 7?**

In Eq. 5, the L2 distance allows us to perform GMM-based clustering by seeking the set of means and covariances that maximizes the likelihood of the observed data. In particular, the resulting maximization step of the EM algorithm corresponds to minimizing the Euclidean distance (L2) between the observations in each cluster and their cluster centroids. Using cosine distance is not suitable here as it only considers dot product (direction) similarities of two vectors, while the GMM clusters vectors that are located closely in feature space.

In Eq.7, it computes the pairwise similarities with non-negative affinities by applying a softmax operation for feature aggregation. Since the d-dimensional input feature vectors are L2-normalized before computing affinities, maximizing cosine similarity is equivalent to minimizing the squared Euclidean (L2) distance. We will clarify this in the revised version.

**Q4. Explanation on the feature alignment process.**

In Line 154, the past frame features are clustered into prototypes (as keys) and reconstructed using current frame features (each pixel as queries) by prototypical cross-attention (Eq.6 and Fig. 2). Past features, encoded as prototypes, are therefore matched to similar features in the current frame. This can be seen as a spatial alignment (or registration) of past features to the current frame.

**Q5. The meaning of subscript $j$ in section 3.3.2.**

$j$ denotes the $j$th positive instance prototype of each tracklet or the $j$th negative prototype of the corresponding background. 

**Q6. The number of positive and negative prototypes.**

In our final configuration, we use 30 positive prototypes to represent the object instance itself and 30 negative ones to encode the surrounding background (see L30 of the supp. and Table 6 in the paper for analysis). 

**Q7. How to get the initial mask prediction.**

We utilize the single-frame mask detection method (one-stage SipMask on Youtube-VIS and the two-stage Mask R-CNN on BDD100K) to compute the initial masks (see L202-L203 and Section 2 of the supplementary).

**Q8. How to select the foreground and background features of each object? Why are they contrastive and how to update the background features?**

After obtaining the initial detected masks, we define foreground as the object mask region, and background as the remaining regions in the object bounding box. The instance-level positive and negative prototypes are produced by fitting two independent GMMs (Eq. 2 of the paper) to represent/attend these two regions separately. Thus, the contrastive property is guaranteed. This property is visualized in Fig. 3 of the paper, and Fig. 2-6 of the supplementary file. We update both the foreground and background prototypes of each tracklet using Eq. 8 in a moving average manner.

 We thank the reviewer for acknowledging that our work is well-motivated, technically sound, and validated with convincing evaluation. We will incorporate the valuable feedback and address all concerns and questions below.

**Q1: Experiments on KITTI-MOTS.**
 
We evaluated PCN on two large-scale MOTS/VIS benchmarks, where BDD100K covers the self-driving scenario and Youtube-ViS has more diverse object categories.

BDD100K Segmentation Tracking and KITTI-MOTS are both self-driving benchmarks. However, BDD100K is 6 times larger (3,0817 vs 5,027 training images,  480K vs 26K instance masks) with richer and more diverse driving scenarios (shown in the supplementary video). Note that many published methods on KITTI-MOTS, such as Vip-DeepLab、EagerMOT and MOTSFusion, use 3D bounding boxes, LIDAR point clouds, or optical flow (PointTrack). In contrast, our method only relies on RGB images.

For additional comparison, we train our PCN on the KITTI-MOTS training set. Our initial results, with no hyperparameter tuning and window size 8, on val set also show significant improvements compared to the TrackR-CNN (also a two-stage tracker based on Mask RCNN) on the benchmark.

| Method | Car-MOTSA | Ped-MOTSA | Car-MOTSP | Ped-MOTSP |
| --- | --- | --- | --- | --- | 
| PCN | 89.6 |  66.4 | 88.3 | 76.1 |
| PCN w/o frame prototypes |  87.3 | 65.3 | 86.9 | 75.0 | 
| PCN w/o instance prototypes |  87.8 | 65.8 | 87.1 | 75.5 | 
| TrackR-CNN | 87.8 | 65.1 | 87.2 | 75.7 |

**Q2. Video object segmentation datasets (DAVIS, YouTube-VOS)**

VOS and MOTS/VIS are different problems. We address the MOTS/VIS task, which requires joint detection, classification and tracking of objects. In contrast, VOS aims to propagate the mask of an object, which is given in the first frame. However, our PCN module is a generic component, which can be integrated into popular cross-attention based VOS methods, such as STM [23]. This is indeed an interesting future work.

**Q3: The transition jump between section 3.1 and 3.2.**

This will be revised according to the reviewer’s suggestion.

**Q4: Does the method need to be applied independently to each mask?**

PCN is employed at both frame-level and instance-level. The frame-level PCN does not require the object masks, and it is only performed once for each frame. The instance-level PCN digests instance-specific information for each object, including the initial single-frame mask detection. However, this operation is efficiently parallelized over all object instances in each frame by simply stacking them along the batch dimension. 

**Q5: What aspects of the MOTS task this paper tackles precisely?**

The aggregation of temporal information allows us to tackle all aspects of the MOTS/VIS task. The frame-level PCN can improve detection, classification, and association by aggregating temporal information in an earlier stage. The instance-level PCN then focuses on fine-grained temporal information for each instance in order to achieve a more accurate segmentation. Fig. 4 and the supp. video qualitatively show that our approach provides more consistent detections, classifications, and object identities, while generating more accurate segmentation masks.


**Q6: Why not use a temporary window size larger than 32?**

The longest video length of annotated frames in the Youtube-VIS dataset is 36. We found that a bigger window size of 36 does not provide additional benefits. 

**Q7: The number of prototypes and its influence.**

Table 6 analyzes the impact of the number of instance-level prototypes. We selected 30 positive and 30 negative ones for each tracklet, specified on L30 of the supplementary material, since the results saturate for larger numbers.

For the number of frame-level prototypes, we used 64 during training and testing. The results on YouTube-VIS below show that the precision saturates for larger numbers of prototypes.

| Num. Frame Prototypes| 8 | 16 | 32 | 64 | 128 |
| --- | --- | --- | --- | --- | --- |
| AP |  32.6 |  33.1  | 33.9 | 34.2 | 34.1 |
| AP$_{50}$ |  52.8 | 53.3 | 53.5 | 53.7| 53.8 |



This paper tackles multiple object tracking and segmentation. The whole framework is based on previous cross attention methods, and the main selling point is to deal with the complexity of the attention computation by constructing prototypes in the feature space. In addition to frame-wise prototypes, instance prototypes are also computed to refine the segmentation. Good performance is shown on two major benchmarks Youtube-VIS and BDD100K. Also, a good trade-off between efficiency and accuracy is reported in Table 5. It is a marginal yet solid contribution, but there are still many issues that need to be clarified such that the message here can be accurately delivered.  the key idea of this paper is to reduce the computational cost of attention-based tracking and segmentation by cross attention between the query feature and a set of prototype features. An additional procedure is to compute the attention within a specific instance. It is a natural extension of traditional cross-attention methods, and the novelty is a bit limited.
However, the experiments, including both the comparison to the other methods and the ablation study, demonstrate good performance on two major datasets.

there are no major flaws in the proposed method given the goal is to reduce computation during the attention procedure.  Since the prototypes are averaged features, how does the proposed method achieve the balance between discriminability and robustness? for example a person wearing a white shirt and black pants, the average could be anything, how do you ensure that the pixel-wise feature in the current frame can attend to the right features? Moreover, there are many descriptions that are not clear, which makes it difficult to appreciate the details of the method. please check the following:

- in eq. 4, the summation is over N values, which is equivalent to the number of prototypes. it is weird that the number of features to be clustered is equal to the number of clusters.
- in eq. 5, the similarity is measured by the L2 distance between the feature and the prototypes, how much difference does it make if a cosine similarity is used here?
- line 154, what do you mean by "essentially aligns the condensed feature information in frame \hat{t} with the current frame"? it is a linear combination of the value features from time \hat{t}, specifically, what information is been aligned here?
- in eq. 7, why do you use dot product here instead of L2? any normalization of the features during dot product?
- what does the subscript j mean in the notation K^+_tj in section 3.3.2? does it mean instance id?
- line 175, how many prototypes do you have for the foreground and background? 
- line 178, for each object, how do you know which set of foreground and background features you need to attend?
- line 179, how do you get the initial mask detection?
- line 181, when you update the features, how do you select the background features?
- line 167, why is this contrastive foreground and background representation? is there an explicit learning procedure that encourages the foreground feature and background to be different?
 I think so. there is a whole paragraph addressing the issues related to societal impact.

Meta-review: All four reviewers recommend acceptance of this paper. The final ratings are: 7, 7, 6, 7.

Overall, the reviewers appreciated the interesting take on reducing the computational cost of memory-based mask propagation methods by means of matching to a small set of prototypes. The cost-accuracy tradeoff is acknowledged as very strong. The ablations are informative and convincing.

However, several reviewers comment that the presentation and description of the method should be improved to do full justice to the approach. As noted by Reviewer TiXB, the ablations on the number of prototypes is highly relevant and should be added to the paper if at all possible.


Decision: accept