Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes to address the problem of multiple-object tracking and segmentation (MOTS) by creating a space-time memory to store useful information from previous frames that can be accessed with an attention module. To keep the computational complexity manageable, information is summarised in "prototypes" represented by a Gaussian Mixture Model.

Large-scale experiments are performed on the YouTube-VIS and BDD-100K benchmarks.
main_review: **Strengths**

I think that the main strength of the paper is to show the feasibility of applying a fairly low-cost attention module to the problem of MOTS. It's interesting to see that using prototypes in a space-time memory is a viable option for this type of problem.

**Weaknesses**

I list my main concerns below. For specific points, with references to line numbers, please see the next section.

* The paper does not reference relevant literature on a very similar problem: self-supervised learning from video (which is generally evaluated with single-object tracking and segmentation). In particular, methods like "MAST: A Memory-Augmented Self-supervised Tracker" also consider a memory and address similar issues to the ones discussed in this paper.
* I did not find any details on what is the loss used by the proposed method.
* Evaluation is done on benchmarks validation seta. This is fine, but hyperparameter choices should be detailed to show that are not chosen ad-hoc for the set used to report the results.
* Since one of the main strengths is the computational advantage with respect to other attention strategies, authors should also show how the computational complexity of the proposed methods fairs against non attention-based approaches (from Table 1 and 2)
* The presentation could be improved.

**Specific points**
* [L15-L16] MOTS and VIS have minor differences and respective benchmarks cover different scenarios. It would be good to detail these (perhaps in the experimental section). Also, in L67 the wording changes slightly and contradicts L15-16.
* [L23] Methods that take into account longer time spans do exist (MAST is the first that comes to mind, but I am sure there are  more).
* [L37] "Noise-reduced". Seems difficult to prove that this features are actually "noise-reduced"? What does it mean? With respect to what?
* [L37] The sentence seems disconnected from the sentence in the previous page.
* [L47] The list of contributions reads more like a breakdown of the paper structure rather than the actual contributions.
* [L57] "Regress". Predict? 
* [L58] Not sure to understand "predicts a linear combination of mask bases".
* [L90] Is the proposed method also doing *detection*, or is it using existing detectors for that? It is not clear from this sentence.
* [L110] For the sake of clarity, it would be important at this point to better explain what can be an "item" in the memory and at what granularity (e.g. frame-level, instance-level, both, ...). As the text currently stands, it becomes clear only from L137.
* [L114] I would remove the use of qualifiers like "principled", "elegant", and so on in the text. It might well be true, but the text reads definitely better without them.
* [L124] What is the value of sigma? How did the authors choose it?
* [Fig3 caption] "Memorial" means "intended to commemorate someone or something". Not the appropriate usage here.
* [L148] Unclear what "parallel" means here. Do they use two different sets of weights?
* [L170] The fact that the masks are provided by a pre-trained detector could be made clearer earlier on.
* [Sec 3.3.1] Why rewriting the formulae of the previous page here? Could presentation be organised a bit differently to avoid that?
* [Sec 3.3.2] Many emphatic words that to me should be avoided: "compact and robust", "empower", "rich and dynamic", "naturally learn", "rich and valuable information". It's not super important, but I would stick to a more factual and dry writing style. 
* [L185] What is the value of lambda, how did the authors chose it?
* [Sec 3] I could not find an explanation of how the loss is computed.
* [L212] Authors should explain what 1-stage and 2-stage frameworks are.
* [Table 3] This could be shown as a plot, and it would be good to go beyond 32 to show where things start to break.
* [Table 4] Why not reporting also P4-P5 and P3-P4-P5?
limitations_and_societal_impact: The authors haven't addressed potential societal impacts, but I don't think this constitutes a problem for the method proposed. To respect the spirit of NeurIPS program chairs guidelines, a few lines could be dedicated to consider negative impacts such as empowering surveillance systems.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper concerns the problem of multi-object tracking and segmentation, using a prototype-based Cross-attention Network (PCN). The model first extracts temporal and instance-wise prototypes using a GMM clustering for all objects of interest from the past frames. Then, it uses those prototypes and their aggregation to detect, associate, classify and segment each object of interest in the current frame. The results have been evaluated on two datasets,   Youtube-VIS and BDD100k, and have been compared against tracking and/or video segmentation frameworks.
main_review: 1- Clarity: The paper is well written, and the idea is clearly explained. The technical section can be followed easily. However, the application focus of the proposed technique is not very clear to me. The main claim in the paper is that it is a framework for multi-object tracking (MOT) and segmentation. Considering the dataset used for the evaluation, I am not convinced that the framework is suitable for the MOT/MOTS problem (See the extra notes at the end of my comments). 

2- Contribution: the paper does not offer any new theoretical contribution. It is rather a pure application paper using built upon the existing and well-explored machine learning ideas (cross attentions, prototypical generations and contrastive learning) with some tweaks and minor extensions to address a challenging practical problem (i.e. Multi-object tracking and segmentation paper). While the application papers are also well acknowledged by the NeurIPS research community, but such papers require substantial experimental evaluation on the existing benchmark datasets to validate the efficacy of their proposed system. Without this, I would be hesitant to vote for the acceptance of such a paper in this venue. This would be my major concern about this paper elaborated Next.

3- Experiments and datasets:  The experiments are tested on two datasets one could be assumed suitable for MOTS problem (BDD100k), but the other one is mainly for video object segmentation (VOS) problems. Considering the main claim of the paper about addressing MOTS problem and taking this into account that this is an application paper, I am not sure why the superiority of this approach has not been tested on well-established MOTS benchmark such as MOTSChallenge and KITTI (MOTS).

4- Some missing details: I didn't find how the approach deals with track initiation and termination and occlusion handling in this framework.  


Extra notes: The challenges in the MOT problem is very different from what is in visual object tracking (VOT) or video object segmentation (VOS).  In MOT, the goal is to track many similarly looking instances (often with the identical semantic class, e.g. pedestrians, Cars ), while in VOT or VOS, the objects of interest are often semantically well distinguishable. Otherwise, similar-category objects are well separated (no need to deal with challenges in MOT). The MOT methods need to be robust in fine-grain visual similarities and they require reliable data association, track management techniques, while VOS and VOT require more general but robust visual representation.  There is neither experimental evidence nor theoretical insight this framework can be robust for all these problems

    
limitations_and_societal_impact: Yes, the authors have well addressed the limitation and potential negative social impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a new prototypical cross-attention network for online multi-object tracking and segmentation. Specifically, the proposed method learns a set of foreground and background prototypes from the past frames. The learned prototypes are merged with the current frame embedding via cross-attention. Several experiments conducted on Youtube-VIS and BDD100K datasets show that the proposed method performs favorably against the existing methods.
main_review: 1. The idea of the proposed method is interesting. 
2. Extracted from the past frames, the prototypes in the frame-level and instance-level are effective to enhance the feature embeddings efficiently.
3. The ablation studies are conducted to demonstrate the effectiveness of the frame and instance prototypes.
4. The prototypical cross-attention module uses much less memory than the standard transformers.

limitations_and_societal_impact: 1. In Table 2, the method is evaluated on the BDD100K val set. What about the performance on the test set? Besides, how did you obtain the results of the compared methods? The results are different from the reported scores in Table 2 of the work [24].
2. The running speeds of the compared methods are not reported in Table 1.
3. The authors should summarize how to select some important parameters in the network in Section 4.1, e.g., the number of prototypes, and the update rate for prototypes.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper considers the problem of multiple object tracking and segmentation.  The overall framework is a memory-based cross-attention architecture, using temporally previous reference features to correlate and then enhance the current predictions. The main contribution is two-fold. First, they propose to use GMM to compute several clustering centers for the memory features as the "prototype memory", so space and time cost of cross-attention is largely reduced, and using longer temporal reference scope is then possible. Second, they propose an instance-level memory to contrast foreground and background and also allow moving average updating of the template.  Experimental results show promising tracking accuracy.
main_review: Appreciate the authors for their effort! Overall this is a technically solid paper, with a clear motivation (save the space and time cost of memory & attention), presents a complete solution, and conducts extensive experiments to validate the effectiveness of the proposed method.  The idea makes sense and the results seem good. However I have the following concerns, if they are properly addressed I would consider raising my rating.

1. Considering using attention-based models and space-temporal memory has already been a most popular framework for MOTS/VIS,  I believe the biggest contribution of this work is that the authors propose the concept of "memory prototype" to reduce the size of memory and thus enable longer temporal reference scope. However it is not shown at what cost the improvement of accuracy is achieved. An elephant in the room is that almost all implementation details of the GMM clustering are missing, e.g. how often to perform the clustering,  and how long one clustering step takes, how to choose the GMM component number $N$, and how the performance will vary with different $N$.

2. Regarding temporal feature aggregation. I notice that in the proposed method each reference frame is first aligned to spatially match the target frame, and then a weighted sum is used to aggregate reference features along the temporal dimension in each spatial location.  This makes sense, but the necessary should be clarified because there exists a simpler alternative: concatenating all the reference features (from multiple reference frames) into a *single* reference feature set and apply cross attention between this single reference feature set and the target features. In fact, several existing methods[a,b] adopt this strategy, and the benefit is that no explicit temporal aggregation is needed. 

3. I am a bit confused that why both frame-level and instance-level memory are needed. It looks like they have the exact same formulation, and the only difference is that the instance-level one supports moving average updating for the foreground template. Is it possible to use instance-level memory only?

4. Regarding tracklet feature propagation and updating, Eq (8), how do you ensure the temporal consistency of the GMM components? For instance, it is possible that the component $j$ in timestep $t-1$ does not correspond to the component $j$ in timestep $t$. Do you do association here?

5. Again, the comparison in Table 5 is not fair. It makes less sense to compare FLOPS, could you compare the runtime directly, between the proposed method (including the runtime of GMM) and Non-local attention?

6. In the supplementary video, I see several visualizations but found them usually too short (2-5 sec.). If it is possible, could the authors provide more results of longer video sequences?  It is important to see how a tracker performs in longer videos.

7. Minor things:

    - L#48 "a MOTS approach" -> "an MOTS approach"

    - L#146, should $k_{\hat{t}i}^{\mu}$ be $k_{\hat{t}j}^{\mu}$?

    - L#171, "GMM (2)": does it mean GMM (Eq. 2)?

a. MAST: A Memory-Augmented Self-Supervised Tracker. https://arxiv.org/pdf/2002.07793.pdf

b. Space-Time Correspondence as a Contrastive Random Walk. https://arxiv.org/pdf/2006.14613.pdf
limitations_and_societal_impact: I do not see any negative societal impact of this work.
review_title: null

Meta-review: The paper received borderline negative reviews, three "below threshold" and one "above threshold". The main concerns were:

1. missing lit review on self-supervised methods.
2. choice of hyperparameters, since evaluation is only on the validation set.
3. missing evaluation on the BDD100k test set.
4. limited theoretical contribution. application paper, but not enough experiments.
5. missing details, clarity.

The authors provided a detailed response. The reviewers were only partially satisfied with the response. In particular, reviewers noted that: 1) low novelty and insight - the paper is combination of existing techniques, with no particular insight gained; 2) discussion of the relationship with self-supervised models was not sufficient in the paper or the response; 3) although the memory is saved, the computation cost is not fully reported or analyzed; 4) other missing details.  In the end, all reviewers were negative. The AC agrees and also recommends reject.
Decision: Accept (Spotlight)