Prototypical Cross-Attention Networks for Multiple Object Tracking and Segmentation
Authors: Lei Ke
Xia Li
Martin Danelljan
Yu-Wing Tai
Chi-Keung Tang
Fisher Yu
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: multiple object tracking and segmentation
video instance segmentation
efficient cross-attention networks
space-time memory

Abstract: Multiple object tracking and segmentation requires detecting, tracking, and segmenting objects belonging to a set of given classes. Most approaches only exploit the temporal dimension to address the association problem, while relying on single frame predictions for the segmentation mask itself. We propose Prototypical Cross-Attention Network (PCAN), capable of leveraging rich spatio-temporal information for online multiple object tracking and segmentation. PCAN first distills a space-time memory into a set of prototypes and then employs cross-attention to retrieve rich information from the past frames. To segment each object, PCAN adopts a prototypical appearance module to learn a set of contrastive foreground and background prototypes, which are then propagated over time. Extensive experiments demonstrate that PCAN outperforms current video instance tracking and segmentation competition winners on both Youtube-VIS and BDD100K datasets, and shows efficacy to both one-stage and two-stage segmentation frameworks. Code and video resources are available at http://vis.xyz/pub/pcan.

Prototypical Cross-Attention Networks for Multiple Object Tracking and Segmentation
Lei Lei Kel.2
Xia Li'
Martin Danelljan'
Yu- Yu-Wing Tai Chi-Keung Tang²
Fisher Yu'
'ETH Zürich
PKKUST
SKuaishou Technology
(1keab, citan)/ ust. hk, Sxia.li,mane martin jtwp:aimaaaainaaeellaaaaaaaaaaea jamelljan) ee, www..e..ee................... ch
yuxingogmail com, i0yf.io
Abstract
Multiple object tracking and segmentation requires detecting, tracking, and seg- menting objects belonging to a set of given classes. Most approaches only exploit the temporal dimension to address the association problem, while relying on sin- gle frame predictions for the segmentation mask itself. We propose Prototypical Cross-Attention Network (PCAN), capable of leveraging rich spatio-temporal in- formation for online multiple object tracking and segmentation. PCAN first distills space-time memory into a set of prototypes and then employs cross-attention to retrieve rich information from the past frames. To segment each object, PCAN of contrastive foreground
adopts prototypical appearance module to learn and background prototypes. which are then propagated over time. Extensive exper- iments demonstrate that PCAN outperforms current video instance tracking and segmentation competition winners on both Youtube- VIS and BDD100K datasets, and shows efficacy to both one-stage and two-stage segmentation frameworks. Code and video resources are available at http: /vvi..vii//// httvvvvi.s//ppu///e
Introduction
Multiple object tracking and segmentation (MOTS), also known as Video Instance Segmentation (VIS), is important problem with many real-world applications, including autonomous driving [10, 26] and video analysis [4, 46]. The task involves tracking and segmenting all objects within a video from given set of semantic classes. We witnessing rapidly growing research interest MOTS
thanks to the introduction of large scale benchmarks [46, 50, 37J. State-of-the-art methods [46, 5, 37, 29] for MOTS mainly follow the beekiing-b--deecctinn paradigm, where objects are first detected and segmented in individual frames and then associated over time.
Although methods based the popular tracking-b--detectton by-detection philosophy have shown promising results, temporal modeling is limited to the object association phase [46, 5, 22] and only between two adjacent frames [37, 18]. On the other hand, the temporal dimension carries rich information about the scene. The information encoded in multiple temporal views an object has the potential of improving the quality of predicted segmentation, localization, and categories. However, effectively and efficiently leveraging the rich temporal information remains a challenge. While sequential modeling has been applied for video processing [40, 41, 9, 28, 12], these methods generally operate directly on the high-resolution deep features, requiring large computational and memory consumption, which greatly limits their use.
We propose Prototypical Cross-Attention Module, termed PCAM, to leverage temporal information for multiple object tracking and segmentation. As illustrated in Figure the module first distills spatio- temporal information into condensed prototypes using clustering based on Expectation Maximization. The resulting prototypes, composed of Gaussian Components, yield rich and generalizable yet compact representation of the past visual features. Given a deep feature embedding of the current
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Memory
Posales
Paspaare
Mehyarrl
III
Frame Prototypes
Instance Prototypes
Previous Frames
Protolypical Croes-Altention
Feature Extractor
Memory Altention
Classification
Dotection
Segmentation
Query
Association
Prediction
Figure 1: We propose Prototypical Cross-Attention Network for MOTS, which first condenses the space-time memory and high-resolution frame embeddings into frame-level and instance-level prototypes. These are then employed to retrieve rich temporal information from past frames by our efficient prototypical cross-attention operation.
frame, PCAM then employs prototypical cross-attention read relevant information from prior frames.
Based the noise-reduced clustered video features information, we further develop a Prototypical Cross-Attentin Network (PCAN) for MOTS, that integrates the general PCAM at two stages in the network: on the frame-level and instance-level. The former reconstructs and aligns temporal past frame features with current frame, while the instance level integrates specific information about each object in the video. For robustness object appearance change, PCAN represents each object instance by learning sets of contrastive foreground and background prototypes, which are propagated an online manner. With a limited number of prototypes for each instance or frame, PCAN efficiently performs long-range feature aggregation and propagation in video with linear complexity. Consequently, our PCAN outperforms standard non-local attention [40] and video transformer [41] on both the large-scale Youtube- VIS and BDD 100K MOTS benchmarks.
Our main contributions summarized follows: We introduce the PCAN module for efficiently utilizing long-term spatio-tempone video information. (ii) We develop a MOTS approach that employs PCAN on frame and instance-level. (iii) We further represent the appearance each video tracklet with contrastive foreground and background prototypes, which are propagated over time. (iv) We extensively analyze our approach. Our PCAN outperforms previous approaches on the challenging self-driving dataset BDD100K [50] and the semantically diverse YouTuube VIS dataset [46].
2 Related work
Video instance segmentation (VIS) Existing VIS methods [46, 2, 21] widely adapt the two- stage paradigm of Mask R-CNN [11] and its variants [13, 15] by adding an additional tracking branch. Thus, their typical pipelines first detect regions of interest (Rols) and then use the instance features after RolAlign to regress object mask and associate cross-frame instances. More recent works [5, 18, 22, 48] employ one-stage instance segmentation method, e.g. the anchor-free FCOS detector [34]. which predicts linear combination of mask bases [3] as its final segmentation. The aforementioned approaches make very limited use of temporal information to enhance the quality of the segmentation, instead relying on single image-based mask prediction, or only model short-term temporal correlation between two consecutive frames [18, 30]. In the context of long-term temporal association, the offline method VisTr [41] adapts vision transformer [6] for VIS, but suffers from a huge computational burden and memory consumption due to the dense pixel-level attention operations over long sequences. Compared to these methods, our PCAN temporally aggregates and propagates the prototypical features with both the long-term benefit and linear complexity.
Multiple Object Tracking and Segmentation (MOTS) Similar to VIS, MOTS methods [37, 27, 29] mainly follow the tracking-by-dee -eddeccctionn paradigm. Objects are first detected and segmented, followed

---

by association between frames. Track R-CNN [37] integrates temporal context feature from two neighboring frames using 3D convolutions. TrackFormer [25] performs joint object detection and tracking by recurrently using Transformers, while Stem-Seg [1] adopts short 3D convolutional spatio-temporal volume to learn pixel embedding by treating segmentation as a bottom-up grouping. In contrast, our approach clusters appearance features in long spatio-temporal volume with explicit foreground and background prototypes that are updates online. Besides, the mixture Gaussian components in instance appearance module equips PCAN stronger modeling ability compared to instance-level average pooling [33, 49] or single Gaussian model [51, 14].
Temporal attention models Video understanding usually requires long-range sequential modeling of relations between spatio-temporal locations. Recently, datention-basee approaches, such as non-local attention [40, 39, 28, 12] and transformers [8, 35, 16]. have been successfully adopted in video classification and action recognition. These tasks [23, 32, 43] involve dense pixel-level attention, leading to quadratic complexity in the sequence length, thus making them excessively expensive for long sequences. Improved temporal attention models mainly include double attention mechanism [7] on image recognition with global-local decomposition, and clustered attention Transformer [38] for language sequence modeling. Besides, recent prototypical methods [19, 45] use the EM algorithm for single-image semantic segmentation few-shot learning [33]. Unlike these methods, our PCAN
uses compact prototypical representation both for temporal feature aggregation and compact instance
appearance feature propagation.
3 Method
We propose an approach for Multiple Object Tracking and Segmentation. Given video sequence, the goal is to detect, track, and segment objects from predefined set of object categories. Specifically, we consider the online setting, where the predictions only depend on current and past frames.
3.1 Traditional Cross-Attention
To utilize the rich temporal information to improve the segmentation prediction, recent approaches [28, 12] have employed cross-attention. We consider past spatio-temporal information encoded in memory M, consisting of deep features of size x W x T x C. The memory encapsulates valuable information about the past appearances and predictions of objects and background in a scene. To attend to the memory, the information is first separately embedded into key k" and value y.M feature vectors. The keys are used to address relevant memories whose corresponding values are returned. The standard memory reading process is a non-local operation computed as the weighted sum,
1 HxWXT
Σ
exp(k? k2 >
Vi Z
where k denotes query key map, which is predicted from the current frame. Further, and j are the index of each query and the memory location, and Z, Σ, exp(k? k))) is the normalizing factor.
Although proven effective, the standard attention operation (1) is known suffer from poor computa- tional and memory scaling properties [20]. In particular, since all queries are matched to all keys, it experiences quadratic scaling ((HW) of computations in the spatial size HW of the feature map. This is particularly problematic for segmentation tasks, where fine- the-grained high-resolution information desired to improve the quality of the predictions.
3.2 Prototypical Cross-Attention
To address the aforementioned limitations of the standard cross-attention, we introduce the proto- typical cross-attention to first condense sets of high-resolution feature vectors in the past frames. Our approach is based on a clustered memory M.. We call these clusters prototypes, since they correspond representative items in the memory. While clustering effectively reduces the number of items in the memory, it also serves to deprecate noisy information, leading to a more generalizable and robust representation of the memory.
To employ an attention mechanism, similar to (1), we require a clustering of the memory that generates principled continuous and differentiable clustering assignment function. We therefore
3

---

Cross-atteation
Reconsinuttd Framc Fouture Yt
protelypes
protetypes
Attention Sum
N
Pioel-level embedaing 6(O)
Multi-l evel Temponil
Appregation
Openiiio
Assignment Probubiiity
MXW
y;
Claseriiation
K-1
Key enbeldinn k
Value embedding iy
Key enbbelding
Value endoalding
Detection
Memary M
Scymenmation
Cocvs
Concs
CHANWW
Hacdboes RTS
Memory
Current Frame Feature
Current
Figure 2: Overview of our frame-level prototypical cross-attention. For a frame i in the memory we first perform GMM-based clustering achieve the key ky and value V prototypes. Given the key encoding k, of the current frame, we attend to the prototypes to generate the reconstructed feature yı- which are then aggregated temporally and fused with the current value encoding Vt.
cluster the keys in the memory by fitting a Gaussian Mixture Model (GMM),
exp (-
p(k)
N 2 pkk j).
pkk=== = (2TO²)
2 | - kg
(2)
Here, N denotes the number of Gaussian mixtures, D) is the feature dimension of the keys. We use constant variance parameter o² and uniform cluster priors p(= =j) t. where denotes the latent cluster assignment variable. The component means k" represent the prototype keys in the memory. We generate the clustering (2) using the standard Dnnectation-Maximizzaee
algorithm.
The GMM allows to compute soft cluster assignment by evaluating the posterior probability of the latent assignment variable z. Using Bayes rule, the probability of a key value k to be assigned to the jth prototype derived as,
p(= j(k) = =
exp
23
k
E p(k|2 l)p(z =)) E exp
2 k
k(ll))
The resulting cluster assignment thus be written SoftMax operation, where the corresponding logits are provided by the negative cluster distance | - kj I|² scaled with temperature of 20
Since the clustering is performed in the key space of the memory, we next retrieve the corresponding value prototypes. To this end, we employ the key cluster assignment probabilities in (3) to compute the values for each memory prototype,
HxW Σ p(== M
For attending to our clustered memory, we first predict the key encodings k0 of the query image. We then read from the clustered memory by computing the average over the value prototypes v. weighted with the cluster assignment probabilities.
N
Σ p(= p(==
1
Σ exp
1
kg v4.
(5)
Z,
202
The final attention operation has much similarity with the original dot- cross attention (1). Note that the key-query similarity in our approach is measured by Euclidian distance instead of dot-product. Importantly, our formulation (5) attends to reduced of N prototypes, while the
original attention (I) requires attending to the full spatio-temporal memory of size H x W x T.

---

3.3 Prototypical Cross-Attention Network
Here, we propose the Prototypical Cross-Attention Network (PCAN) for MOTS by integrating our prototypical cross-attention module into both the frame-level and instance-level. The former aims to align and aggregate temporal frame features stored in memory, while the latter is for propagating the instance appearance features over time and produce instance cross- attentonnn maps to help segmentation. Besides, we also design a prototypical instance appearance module to represent each video tracklet with contrastive mixture foreground and background prototypes.
3.3.1 Frame-level Prototypical Cross-Attention
In Figure 2, prototypical cross-attention first produces prototypes by fitting a Gaussian mixtures model (2) to the feature in the memory. To provide further flexibility when dynamically updating the memory M, we first perform frame-wise clustering for each reference frame feature at index to compute the N key prototypes 1- and retrieve the corresponding value embeddings {v J=1
using (4) for each memory frame independently. The key and value features are predicted using two parallel convolutional layers.
Frame-wise prototypical memory attention Given the query key encoding ky of the current frame 4, we perform prototypical cross-attention to each memory frame independently using our formulation (3) as,
Yu
1 Zí
exp (T20 IKk -
20
v
Z
exp
20
k
KHIP) (6)
Note that the index refers to a spatial coordinate in the current frame. The resulting feature map Yi can intuitively be seen as a projection of features from frame to the current frame. This projection essentially aligns the condensed feature information in frame with the current frame.
Temporal feature aggregation Since frame-wise attention does not fuse temporal information, we perform a temporal aggregation. The temporal information y; in (6) from different frames are fused linear combination, weighted by the feature similarity with the current frame. Specifically, the temporally aggregated representation is obtained as
exp(yss Yi.)
wHy
WA
exply= Yos)
i=1
Note that f = t in the sum refers to the value embedding ya vi extracted from the current frame. The contribution of each frame i is thus weighted by the similarity to this current frame prediction using the attention weights wi. This strategy ensures that incorrect or dissimilar regions suppressed when computing the final aggregated feature embedding yt. To handle object with large-scale variation and produce more fine- -reggriieed instance mask prediction, we further extend temporal aggregation multi-level using different levels of the extracted FPN features, as detailed in the supplementary material.
3.3.2 Instance-level Prototypical Cross-Attention
Contrastive foreground and background representation In additional to the condensed frame-level representation, for more accurate segmentation results, we further encode each tracked object with compact and robust appearance prototypes. To further empower our proposed attention mechanism, we utilize the initially detected object mask to identify each foreground instance. We then separately model the extracted foreground and background features using GMM (2). We denote the resulting foreground prototypes as k,, and background prototypes as kzj. The former thus focuses on the appearance of the specific object, creating rich and dynamic appearance model. When employed in our prototypical cross- cross-attention framework (Section 3.2), it provides tne-grained attention from localized prototypes that naturally learn to focus specific parts of views of the object, as visualized in Fig. 3. Furthermore, the background prototypes k,,, capture valuable information about the background appearance, which can greatly alleviate the segmentation process. For each object instance attend to the foreground and background prototypes separately using (3). The results
5

---

Thee?
Manorid Franc Exbeidis
[-T
Maneril Frans Endochdnn
Canas Frazse Enlosking
|
Peadive
Negalive
FSM
Ineticce Attrnszn
Trregrenn
Backggeandd
Faceurazal
Hadkgpordd
Toregrend
Backymerd
Fiaal Madi Profiction
Figure 3: Our instance-level prototypical attention with foreground and background prototypes and temporal propagation. The foreground/backgrounn attention maps from (bottom) demonstrate the localized and discriminative appearance representation. Temporal Segmentation Module (TSM) takes the current frame, initial mask, and instance attention maps as input and generates the final mask.
concatenated together with the initial mask detection to the Temporal Segmentation Head (TSM) for final prediction, as illustrated in Figure
Tracklet feature propagation and updating To effectively model the object appearance change and preserve the most relevant information, we design a recurrent instance appearance updating scheme. From the first video frame where object appears, the accumulated prototypes k kú, for the instance are propagated the subsequent frames and updated with new appearance prototypes k,j. k,, using an update rate À as,
kij - A)K[-1.5 + Akit.
kiy = (1 A)Kī-15 + Akij.
(8)
Figure also reveals the consistency of the attended region of specific prototype j.
4 Experiments
Here, we present comprehensive evaluation and analysis of our approach. Experiments performed on two large scale datasets, namely YouTube- VIS [46] and BDD100K [50].
4.1 Experiment setup
Youtube-VIS YouTube- VIS-2019 [46] dataset contains 2,883 high quality videos with 131k anno- tated object instances belonging to 40 diverse categories. The task is simultancously classifying, segment and track object instances belonging to these categories. The evaluation metrics for this task are adaptation of the Average Precision (AP) and Average Recall (AR) of image instance segmentation.
BDD100K We also evaluate the large-scale tracking and segmentation dataset of BDD100K [50], which is a challenging self-driving dataset with 154 videos (30,817 images) for training. 32 videos (6,475 images) for validation, and 37 videos (7, 17484 images) for testing. The dataset provides 8 annotated categories for evaluation, where the images in the tracking set are annotated per 5 FPS with 30 FPS frame rate. We adopt the well-established MOTS metrics [37] to our task.
Implementation details We implement PCAN based on two different existing MOTS approaches. For Youtube-VIS we adopt ResNet with FPN pre-trained on COCO as the backbone, and build our segmentation tracker on the one-stage segmentation model [5]. Both the instance and frame cross-attention is built on the extracted FPN features. Our model is trained with initial learning rate 0.0025 on 4 GPUs using SGD, and executes with speed of 15.0 FPS on ResNet-50. Similar to [46, 22, 18], we use the input size x60x640 for training. On BDD100K, we build PCAN by extending the two-stage MOT method [29] with our temporal segmentation modules. We follow the same training strategy of QDTrack-mots [29]. More details can be found in supplemental material.

---

Table Comparison with state-of-the-an on the YouTube- VIS validation set. Results are reported in terms of mask accuracy (AP) and recall (AR), Asterisks denote concurrent works on arXiv.
Method
Backbone
Type
Online
AP
APso
AP7s
AR1
AR10
VisTr" [41]
ResNet-50
Transformer
35.6
56.8
37.0
35.2
40.2
OSMN [47]
ResNet-50
Two-stage
23.4
36.5
25.7
28.9
31.1
FEELVOS [36]
ResNet-50
Two-stage
26.9
42.0
29.7
29.9
33.4
SERT [42]
ResNet-50
Two-stage
26.1
42,9
26.
27.8
31.3
WaskTrack R-CNN [46]
ResNet-50
Two-stage
30.3
51.1
32.6
31.0
35.5
STEm-Seg [1]
ResNet-50
One-stage
x
30.6
50.7
33.5
31.6
37.1
SipMask [S]
ResNct-50
One-stage
V
32.5
53.0
33.3
33.5
38.9
STMask [18]
ResNet-50
One-stage
33.5
52.1
36.9
31.1
39.2
SG-Nct" [22]
ResNct-50
One-stage
34.8
56.1
36.8
35.8
40.8
PCAN (Ours)
ResNet-50
One-stage
36.1
54.9
39.4
36.3
41.6
STMask [18]
ResNet-101
One-stage
36.3
55,2
39.9
33.7
42.0
SG-Net" [22]
ResNet-101
One-stage
36.3
57.1
39.6
35.9
43.0
PCAN (Ours)
ResNet-101
One-stage
37.6
57.2
41.3
37.2
43.9
Table 2: State-of-the-art comparison on the BDD100K segmentation tracking validation set. I: ImageNet. C: COCO. S: Cityscapes. B: BDD100K. "-fix" means adopting the pretrained model from the BDD100K tracking set, fixing the existing parts, and only training the added mask head.
Method
Pretrained
Online
mMOTSAf
mMOTSP1
mIDFt
ID sw.
mAPt
SortloU
I, I.C.S
10.3
59.9
21.8
15951
22.2
MaskTrackRCNN [36]
I,C,
12.3
59.9
26.2
9116
22.0
STEm-Seg [1]
I,C,
x
12.2
58.2
25.4
8732
21.8
QDTrack-mots [29]
I,C,
V
22.5
59.6
40.8
1340
22,4
QDTrack-mots-fix [29]
B,
V
23.5
66.3
44.5
973
25.5
PCAN (Ours)
I, B
27.4
66.7
45.1
876
26.6
4.2
State-of-the-Art Comparison
We compare approach with the state-of-the-art methods the aforementioned large-scale MOTS/VIS benchmarks Youtube-Ve VIS and BDDI00K, where PCAN outperforms all existing meth- ods without bells and whistles, and shows efficacy both one-stage and two-stage segmentation frameworks, We follow the official metrics of each benchmark to evaluate our model.
Youtube-VIS VIS The results of Youtube-VIS benchmark is in Table where PCAN achieves the best mask AP of 36.1% 6% using ResNet-50 and 37.6% using ResNet- 101 respectively, while being an online method. Our approach consistently surpasses most recent SOTA methods, including STMask [18] and SG-Net [22] by significant margin. These methods only conduct temporal modeling between two adjacent frames for feature correlation. Compared to baseline SipMask [5], single-image based segmentation with object centerness association, PCAN improves the mask AP from 32.5% to 36. 1,, which shows the effectiveness long-term temporal modeling in helping object tracking and segmentation.
BDD100K Table 2 shows our results BDDI00K tracking and segmentation benchmark, where PCAN outperforms the strong baseline methods MaskTrackRCNN [46] and QDTrack-mots [29]. Our approach achieves large advantage in mMOTSA. with over points gain and around 10% ID switches decrease. MOTSA measures segmentation as well as tracking quality, while ID Switches measure the performance of identity consistency. The significant adyancements demonstrate that our method with prototypical cross-attention enables more accurate pixel-wise object tracking by effectively exploiting temporal information.
4.3 Ablation study and analysis
We conduct detailed ablation studies on Youtube- VIS validation set, where we investigate the effect of our proposed prototypical cross-attention components for MOTS during training and testing.
Effect of frame-level prototypical cross-attention module study the importance of temporal information amount, conduct ablation study models with different input temporal window

---

Table 3: Results of varying temporal memory
Table 4: Effect of multi-layer prototypical fea- ture fusion with tube length 4 on You YouTube- VIS.
length in our PCAN on YouTube- VIS.
Length
AP
AP
APTs
AR
AR
FPN Layer
AP
AP
AP7s
AR1
AR10
32.5
53.0
33.3
33.5
38.9
P3
30.8
$1.7
32.0
32.6
37.0
2
33.7
53.8
35.3
33,9
39,5
P4
32.0
51.5
34.1
32.6
37.2
4
33.9
54,0
36.8
34.1
40.0
P5
32.9
52.1
35.9
33.2
38.6
34.2
53.7
37.6
34.4
40.3
P3-P4
33.1
52.3
35.6
33.6
38.5
16
34.6
53.7
38.3
40.5
P3-P5
33,9
54.0
36.8
34.1
40.0
32
35.4
53.8
39.1
35.9
41.0
Table 5: Comparison with non-local attention [39] and transformer [6, 41] on YouTube-VIS,
Prototypical Cros-Attention
Non-local Attention
Transformer (Multi-Head Self-Attention
Length
AP
FLOPP(B) Memory(M)
FL.OP((B)
Memory(M)
FLOPs(B)
Memory(M)
33.7
5.8
323
33.2
24.3
2497
24.6
103.8
5321
33.9
12.0
652
33.3
49.1
4763
25.8
387.2
9844
34.2
23.7
1419
33.6
99.6
9631
28.3
1413.3
18762
vlo Irame
PCAM
We franne
PCAM
wio Instoncc
PCAM
We instance
PCAM
Figure Qualitative impact of our PCAM on YouTube- -IS.. Mask colors encode object identity. Our frame-level PCAM (second row) helps provide consistent detections and preserve identities compared to the baseline (first row). The instance-level PCAM (fourth row) provides more accurate masks, while further improving identity consistency compared to not employing our module (third row).
lengths in Table 3. A temporal length of 1 thus means that no prior temporal information guidance is used during video instance segmentation. By varying the frame length from to 32, the mask AP increases from 32.5% 35.4%, which reveals that richer temporal information with multiple views of segmented object indeed brings more gain to model performance. For the number of frame-level prototypes, we used 64 during training and testing. The results on YouTube- VIS in Table show that the precision saturates for larger numbers of prototypes.
Effect of multi-layer temporal aggregation Since perform temporal feature aggregation the extracted FPN features, to help deal with objects with partial occlusion and large-scale variation, we also study the effect of using different levels of the extracted FPN features. In Table 4, select the FPN feature map from P3-P5 layers for (excluding P6 and P7 due to impractical computation cost), and perform prototypical temporal aggregation each FPN layer. We find that multi-layer information also important to final model performance.
Computation and memory efficiency In Table 5 analyze different attention mechanisms. Compared to standard space-time memory reading using non-local attention [39, 28] or recent popular transformer [41, 6] with multi-head self-attention layer, the prototypical cross-attention with condensed prototypes not only enjoys high accuracy advantage, but also largely reduces the memory consumption and computation amount. For input tube length the prototypical memory consumption less than 10% the transformer with negligible FLOPs computation due to the small number of representative prototypes in (5).
Effect instance-level prototypical appearance module We analyze the instance-level prototypical cross-attention module, which represents each video tracklet using the contrastive prototypes. In

---

Table Ablation study on number of instance- level prototypes on YouTube-VIL
Table 7: Ablation on instance-level EM feature propagation and updating on YouTube VIS.
Pos. Proto. Number
Neg. Proto. Number
AP
APsc
version
AP
so
32.5
53.0
32.4
52.3
No instance prototype propagation
33.5
53.2
32.1
52.4
Using initial instance prototype
33.0
52.8
Update momentum 0.2
34.3
53,8
5
33.1
52.6
Update momentum 0.5
34.0
53.6
54.1
50
33.6
53.8
Table Ablation on number of frame- level prototypes on YouTube- VIS.
Table Results of varying EM iterations for our PCAN on YouTube- VIS.
Proto. Number
APs0
Iteration number
AP
AP7o 76
AR;
AR
8
32.6
52.8
33.3
53.4
35,8
33.2
38.8
16
33.1
53.3
2
33.7
53.9
36.4
33.6
39.3
32
33.9
53.5
33.7
54.1
36.5
33,9
39.5
64
34.2
53.7
33.9
54.0
36.8
34.1
40.0
128
34.1
53.8
8 33.6
53.6
36.1
33.7
39.3
Table 6, we study the influence of instance prototype number and the effect foreground brrroounndd-akrrouuddd contrasting. Using both positive and negative prototypes improves AP from 32.5% to 33.9%. Compared to the single prototype representation, the GMM demonstrate a stronger appearance modeling ability. We further find that the performance saturates when the number is larger than 60. In the Figure 6 and supplementary file, we provide additional instance cross-attention maps visualization to highlight the various attended regions.
In Table 7, we investigate the effectiveness of instance prototype (including the both positive and negative ones) propagation in an online manner, and compared it with using the instance prototype the initial frame or current frame. We find that updating object prototypes recurrently with momentum of 0.2 improves video segmentation AP of 1.3%.
Influence of EM iteration number We study the influence of EM iteration number T during condensing pes and the results are shown in Table 9. Using temporal memory length 4, we iterations from 6, and the improvement starts to saturate when T 2 We use the same iteration number during training and test.
find that the accuracy gains of PCAN increase with
Ablation study on KITTI-MOTS We also train PCAN on the KITTI-MOTS [37] training set and conduct ablations on the instance and frame PCAMs. Table 10, PCAN with window size on val set also shows significant improvements compared to the TrackR-CNN [37] (a two-stage tracker based on Mask R-CNN) on the benchmark. Note that many published methods on MOTTI-MOTS such Vip-DeepLab [31 EagerMOT [17] and MOTSFusion [24]. 3D bounding boxes, LIDAR point clouds, or optical flow (PointTrack [44]). In contrast, our method only relies on RGB images.
Qualitative analysis In Figure 4, we showcase qualitative ablation results of PCAN on Youtube-VIs, Compared to the baseline, we see that our model results more consistent segmentation and better tracking using prototypical -ates-attention module. We also provide visual results on BDD100K in Figure 5, where PCAN produces robust tracking and segmentation results even under large object appearance change (first row) or low illumination (second row). In the 3rd row, PCAN has limitations handling missing detections (the person in the first frame) with limited appearance information under extreme lighting, and produce tracking errors in the second frame when visible parts of the same car is totally different across frame and with low appearance similarity.
Cross-Attentin Visualization In Figure 6, we visualize instance-level prototypical cross-attention of the interested car for both the corresponding foreground and background regions on three continu- ous frames on BDD 100K, where the attended region of each object prototype reveals the implicit unsupervised temporal consistency. More visualization cases on instance and frame cross-attention maps and relevant analysis in the supplementary file.
Societal impact PCAN has high potential impact in important applications, such as transportation, sports analysis, and self-driving vehicles. However, this powerful technology can be deployed in human monitoring and surveillance as well which raise ethical and privacy issues. Potential negative impact be avoided by enforcing strict and secure data privacy regulation such as the GDPR.

---

Table 10: Ablation study of PCAN on KITTI-MOTS [37] validation set.
Method
Car-MOTSA
Ped-MOTSA
Car-MOTSP
Ped-MOTSP
SrackR-CNN [37]
87.8
65.1
87.2
75.7
PCAN w/o frame PCAM
87.3
65.3
86.9
75.0
PCAN w/o instance PCAM
87.8
65.8
87.1
75.5
PCAN (Ours)
89.6
66.4
88.3
76.1
Figure 5: Qualitative results of our method on BDD100K. PCAN produces robust tracking and segmentation results under large motion and appearance changes (1st row) and heavy traffic in low-light conditions (2nd row). In the 3rd row, PCAN misses detection (the person to the left in Ist frame), and produces tracking errors (2nd frame) when it covers totally different regions of the with low appearance similarity. Zoom for better view. Video results are in the suppl. file.
Foreground
Background
Foreground
Rackground
Foreground
Background
Input video frames
Predictions
Attention Sum!
Instance Attention Maps
Figure 6: Instance cross-attes attenttttnttonn maps visualization for the car specified by the red dotted bounding box on BDD100K. We select the first four gnreground/backgroon one focuses on specific car sub-regions with implicit unsupervised temporal consistency over time. proper technology management education, and having an open dialogue among various stakeholders on how such technology should be deployed and regulated.
prototypes as example, where each
Conclusion
We present PCAN, new online method for MOTS. PCAN first distills the space-time memory into a frame-level and instance-level prototypes, followed by cross-attention retrieve rich information from the past frames. In contrast to most previous MOTS methods with limited temporal consideration, PCAN efficiently performs long-term temporal propagation and aggregation, and achieves large performance gain on the two largest MOTS benchmarks with low computation and memory cost. We validate the efficacy PCAN on both the existing one-stage and two-stage trackers. We believe PCAN will significantly benefit more video understanding tasks in the future.
10

---

Acknowledgments and Disclosure of Funding
This research is supported in part by the Research Grant Council of the Hong Kong SAR under grant no. 16201818 and Kuaishou Technology.
References
[1] Ali Athar, Sabarinath Mahadevan, Aljoša Okep, Laura Leal-Taixé, and Bastian Leibe. Stem-seg: Spatio- temporal embeddings for instance segmentation in videos. In ECCV, 2020.
[2] Gedas Bertasius and Lorenzo Torresani. Classifying, segmenting, and tracking object instances in video
with mask propagation. In CVPR. 2020.
[3] Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee. Yolact: Real-time instance segmentation. In
ICCV, 2019.
[4] Sergi Caelles, Kevis-Kokinn Maninis, Jordi Pont-Tuset, Laura Leal- Laixe Daniel Cremers, and Luc Van Gool. One-shot vidco object segmentation. In CVPR, 2017.
[5] Jiale Cao, Rao Muhammad Anwer, Hisham Cholakkal, Fahad Shahbaz Khan, Yanwei Pang, and Ling Shao. Sipmask: Spatial information preservation for fast image and video instance segmentation. In ECCV, 2020.
[6] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV. 2020.
[7] Yunpeng Chen, Yannis Kalantidis, Jianshu Li. Shuicheng Yan, and Jiashi Feng. ne-nets: Double attention networks. In NeurlPS, 2018.
[8] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2020.
[9] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for vidco recognition. In ICCV, 2019.
[10] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012.
[11] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. ICCV, 2017.
[12] Ping Hu, Fabian Caba, Oliver Wang. Zhe Lin, Stan Selaroff, and Federico Perazzi. Temporally distributed networks for fast video semantic segmentation. In CVPR, 2020.
[13] Zhaojin Huang. Lichao Huang, Yongchao Gong, Chang Huang. and Xinggang Wang. Mask scoring r-cnn. In CVPR, 2019.
[14] Joakim Johnander, Martin Danelljan, Emii Brissman, Fahad Shahbaz Khan, and Michael Felsherg. generative appearance model for end-to-end video object segmentation. In CVPR, 2019.
[15] Lei Ke, Yu-Wing Tai, and Chi-Keung Tang. Deep occlusion-aware instance segmentation with overlapping bilayers. In CVPR, 2021.
[16] Lei Ke, Yu-Wing Tai, and Chi-Keung Tang. Occlusion-aware video object inpainting. In ICCV, 2021.
[17] Aleksandr Kim, Aljoša OSep, and Laura Leal- Taix'Taxx' e. Eagermot: 3d multi multi-object tracking via sensor fusion. In IEEE International Conference on Robotics and Automation (ICRA), 2021.
[18] Minghan Li. Shuai Li, Lida Li. and Lei Zhang. Spatial feature calibration and temporal fusion for effective
one-stage video instance segmentation. In CVPR, 2021.
[19] Xia Li, Zhisheng Zhong. Jianlong Wu, Yibo Yang. Zhouchen Lin, and Hong Liu. Expectation- maximizatiiinn attention networks for semantic segmentation. In ICCV, 2019.
[20] Yingwei Ii, Xiaojie Jin, Jieru Mei, Xiaochen Lian, Linjie Yang, Cihang Xie, Qihang Yu, Yuyin Zhou, Song Bai, and Alan Yuille. Neural architecture search for lightweight non-local networks. In CVPR, 2020.
[21] Chung-Ching Lin, Ying Hung, Rogerio Feris, and Linglin He. Video instance segmentation tracking with modified vae architecture. In CVPR, 2020.
[22] Dongfang Liu, Yiming Cui, Wenbo Tan, and Yingjie Chen. Sg-net: Spatial granularity network for one-stage video instance segmentation. In CVPR, 2021.
[23] Xiankai Lu, Wenguan Wang, Danelljan Martin. Tianfei Zhou, Jianbing Shen, and Van Gool Luc. Video object segmentation with episodic graph memory networks. In ECCV, 2020.
[24] Jonathon Luiten, Tobias Fischer, and Bastian Leibe. Track to reconstruet and reconstruct track. IEEE Robotics and Automation Letters, 183:11003-1810 2020.
[25] Tim Mcinhardt, Alexander Kirillov, Laura Leal-Taixe, and Christoph Feichtenhofer. Trackformer: Multi- object tracking with transformers. arviv preprint arXiv:2101.02700 2021.
[26] Anton Milan, Laura Leal-Taixé, Ian Reid, Stefan Roth, and Konrad Schindler. Mot16: benchmark for multi-object tracking. arXiv preprint 16Xiv:1603,00000 00831, 2016.
[27] Anton Milan, Laura Leal-Taixé, Konrad Schindler, and lan Reid. Joint tracking and segmentation of multiple targets. In CVPR. 2015.
[28] Scoung Wug Oh, Joon- Young Lee, Ning Xu, and Seon Joo Kim. Video object segmentation using time memory networks. In ICCV, 2019.

---

[29] Jiangmiao Pang, Linlu Qiu, Xia Li, Haofeng Chen. Qi Li. Trevor Darrell, and Fisher Yu. Quasi-dense similarity learning for multiple object tracking. In CVPR, 2021.
[30]
Jiyang Qi, Yan Gao. Yao Hu, Xinggang Wang. Xiaoyu Liu, Xiang Bai, Serge Belongie, Alan Yuille, Philipp Torr, and Song Bai. Occluded video instance segmentation. arXiv preprini urXi::2102.015500 2021. Styuan Qiao, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Vip-deeplab: Learning visual perception with depth-aware video panoptic segmentation. In CVPR, 2021.
[31]
[32] Hongje Scong. Junhyuk Hyun, and Funtai Kim. Kemeliized memory network for video object segmentation. In ECCV, 2020.
[33] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NeurlPS,
2017.
[34] 7hi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: Fully convolutional one-stage object detection.
In ICCV, 2019.
[35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz Kaiser, and Illia Polosukhin. Attention all you need. In NewrlPS, 2017.
[36] Paul Voigtlaender, Yuning Chai, Florian Schroff, Hartwig Adam, Bastian Leibe, and I Liang-Chich Chen.
Feelvos: Fast end-to-end embedding learning for video object segmentation. CVPR, 2019.
[37] Paul Voigtlaender, Michael Krause, Aljosa Osep, Jonathon Luiten, Berin Balachandar Gnana Sekar, Andreas Geiger, and Bastian Leihe. Mots: Multi-object tracking and segmentation. In CVPR, 2019.
[38] Apoory Vyas, Angelos Katharopoulos, and François Fleuret. Fast transformers with clustered attention.
NeurIPS, 2020.
[39] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018.
[40] Xiaolong Wang and Abhinav Gupta. Videos as space-time region graphs. In ECCV, 2018.
[41] Yuqing Wang. Zhaoliang Xu. Xinlong Wang. Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. arXiv preprint arXiv:2011. 1.450v.., 2020. [42] Nicolai Wojke, Alex Bewley, and Dietrich Paulus. Simple online and realtime tracking with a deep association metric. In IEEE international conference on image processing (ICIP), 2017.
[43] Haozhe Xie, Hongxum Yao, Shangchen Zhou, Shengping Zhang, and Wenxiu Sun. Efficient regional memory network for video object segmentation. In CVPR, 2021.
[44] Zhenbo Xu, Wei Zhang, Xiao Tan, Wei Yang, Huan Huang, Shilei Wen, Errui Ding, and Liusheng Huang. Segment as points for efficient online multi-object tracking and segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
[45] Boyu Yang. Chang Liu, Bohao Li, Jianbin Jiao, and Qixiang Ye. Prototype mixture models for few-shot semantic segmentation. In ECCV, 2020.
[46] Linjie Yang. Yuchen Fan, and Ning Xu. Vidco instance segmentation. In ICCV, 2019.
[47] Linjie Yang. Yanran Wang, Xuehan Xiong, Jianchao Yang, and Aggelos K Katsaggelos. Efficient video object segmentation via network modulation. In CVPR, 2018.
[48] Shusheng Yang. Yuxin Fang. Xinggang Wang. Yu Li, Chen Fang, Ying Shan, Bin Feng, and Wenyu Liu. Crossover learning for fast online video instance segmentation. In ICCV, 2021.
[49] Zongxin Yang. Yunehao Wei. and Yi Yang. Collaborative video object segmentation by foreground- background integration. In ECCV, 2020.
[50] Fisher Yu, Haofeng Chen, Xin Wang, Wengi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In CVPR, 2020.
[51] Chi Zhang, Guosheng Lin, Fayao Liu. Rui Yao, and Chunhua Shen. Canet: Class-agnostic segmentation networks with iterative refinement and attentive sho- learning. In CVPR, 2019.
12


Reviews:
[{"review_id": "_UUMBwRhwWZ", "reviewer": "Reviewer_nMFp", "summary": "This paper proposes a new prototypical cross-attention network for online multi-object tracking and segmentation. Specifically, the proposed method learns a set of foreground and background prototypes from the past frames. The learned prototypes are merged with the current frame embedding via cross-attention. Several experiments conducted on Youtube-VIS and BDD100K datasets show that the proposed method performs favorably against the existing methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The idea of the proposed method is interesting. \n2. Extracted from the past frames, the prototypes in the frame-level and instance-level are effective to enhance the feature embeddings efficiently.\n3. The ablation studies are conducted to demonstrate the effectiveness of the frame and instance prototypes.\n4. The prototypical cross-attention module uses much less memory than the standard transformers.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. In Table 2, the method is evaluated on the BDD100K val set. What about the performance on the test set? Besides, how did you obtain the results of the compared methods? The results are different from the reported scores in Table 2 of the work [24].\n2. The running speeds of the compared methods are not reported in Table 1.\n3. The authors should summarize how to select some important parameters in the network in Section 4.1, e.g., the number of prototypes, and the update rate for prototypes.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "JA-6atTmye4", "reviewer": "Reviewer_q7JA", "summary": "This paper proposes to address the problem of multiple-object tracking and segmentation (MOTS) by creating a space-time memory to store useful information from previous frames that can be accessed with an attention module. To keep the computational complexity manageable, information is summarised in \"prototypes\" represented by a Gaussian Mixture Model.\n\nLarge-scale experiments are performed on the YouTube-VIS and BDD-100K benchmarks.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\nI think that the main strength of the paper is to show the feasibility of applying a fairly low-cost attention module to the problem of MOTS. It's interesting to see that using prototypes in a space-time memory is a viable option for this type of problem.\n\n**Weaknesses**\n\nI list my main concerns below. For specific points, with references to line numbers, please see the next section.\n\n* The paper does not reference relevant literature on a very similar problem: self-supervised learning from video (which is generally evaluated with single-object tracking and segmentation). In particular, methods like \"MAST: A Memory-Augmented Self-supervised Tracker\" also consider a memory and address similar issues to the ones discussed in this paper.\n* I did not find any details on what is the loss used by the proposed method.\n* Evaluation is done on benchmarks validation seta. This is fine, but hyperparameter choices should be detailed to show that are not chosen ad-hoc for the set used to report the results.\n* Since one of the main strengths is the computational advantage with respect to other attention strategies, authors should also show how the computational complexity of the proposed methods fairs against non attention-based approaches (from Table 1 and 2)\n* The presentation could be improved.\n\n**Specific points**\n* [L15-L16] MOTS and VIS have minor differences and respective benchmarks cover different scenarios. It would be good to detail these (perhaps in the experimental section). Also, in L67 the wording changes slightly and contradicts L15-16.\n* [L23] Methods that take into account longer time spans do exist (MAST is the first that comes to mind, but I am sure there are  more).\n* [L37] \"Noise-reduced\". Seems difficult to prove that this features are actually \"noise-reduced\"? What does it mean? With respect to what?\n* [L37] The sentence seems disconnected from the sentence in the previous page.\n* [L47] The list of contributions reads more like a breakdown of the paper structure rather than the actual contributions.\n* [L57] \"Regress\". Predict? \n* [L58] Not sure to understand \"predicts a linear combination of mask bases\".\n* [L90] Is the proposed method also doing *detection*, or is it using existing detectors for that? It is not clear from this sentence.\n* [L110] For the sake of clarity, it would be important at this point to better explain what can be an \"item\" in the memory and at what granularity (e.g. frame-level, instance-level, both, ...). As the text currently stands, it becomes clear only from L137.\n* [L114] I would remove the use of qualifiers like \"principled\", \"elegant\", and so on in the text. It might well be true, but the text reads definitely better without them.\n* [L124] What is the value of sigma? How did the authors choose it?\n* [Fig3 caption] \"Memorial\" means \"intended to commemorate someone or something\". Not the appropriate usage here.\n* [L148] Unclear what \"parallel\" means here. Do they use two different sets of weights?\n* [L170] The fact that the masks are provided by a pre-trained detector could be made clearer earlier on.\n* [Sec 3.3.1] Why rewriting the formulae of the previous page here? Could presentation be organised a bit differently to avoid that?\n* [Sec 3.3.2] Many emphatic words that to me should be avoided: \"compact and robust\", \"empower\", \"rich and dynamic\", \"naturally learn\", \"rich and valuable information\". It's not super important, but I would stick to a more factual and dry writing style. \n* [L185] What is the value of lambda, how did the authors chose it?\n* [Sec 3] I could not find an explanation of how the loss is computed.\n* [L212] Authors should explain what 1-stage and 2-stage frameworks are.\n* [Table 3] This could be shown as a plot, and it would be good to go beyond 32 to show where things start to break.\n* [Table 4] Why not reporting also P4-P5 and P3-P4-P5?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors haven't addressed potential societal impacts, but I don't think this constitutes a problem for the method proposed. To respect the spirit of NeurIPS program chairs guidelines, a few lines could be dedicated to consider negative impacts such as empowering surveillance systems.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "Gv1R7M2UoX", "reviewer": "Reviewer_x3ez", "summary": "This paper considers the problem of multiple object tracking and segmentation.  The overall framework is a memory-based cross-attention architecture, using temporally previous reference features to correlate and then enhance the current predictions. The main contribution is two-fold. First, they propose to use GMM to compute several clustering centers for the memory features as the \"prototype memory\", so space and time cost of cross-attention is largely reduced, and using longer temporal reference scope is then possible. Second, they propose an instance-level memory to contrast foreground and background and also allow moving average updating of the template.  Experimental results show promising tracking accuracy.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Appreciate the authors for their effort! Overall this is a technically solid paper, with a clear motivation (save the space and time cost of memory & attention), presents a complete solution, and conducts extensive experiments to validate the effectiveness of the proposed method.  The idea makes sense and the results seem good. However I have the following concerns, if they are properly addressed I would consider raising my rating.\n\n1. Considering using attention-based models and space-temporal memory has already been a most popular framework for MOTS/VIS,  I believe the biggest contribution of this work is that the authors propose the concept of \"memory prototype\" to reduce the size of memory and thus enable longer temporal reference scope. However it is not shown at what cost the improvement of accuracy is achieved. An elephant in the room is that almost all implementation details of the GMM clustering are missing, e.g. how often to perform the clustering,  and how long one clustering step takes, how to choose the GMM component number $N$, and how the performance will vary with different $N$.\n\n2. Regarding temporal feature aggregation. I notice that in the proposed method each reference frame is first aligned to spatially match the target frame, and then a weighted sum is used to aggregate reference features along the temporal dimension in each spatial location.  This makes sense, but the necessary should be clarified because there exists a simpler alternative: concatenating all the reference features (from multiple reference frames) into a *single* reference feature set and apply cross attention between this single reference feature set and the target features. In fact, several existing methods[a,b] adopt this strategy, and the benefit is that no explicit temporal aggregation is needed. \n\n3. I am a bit confused that why both frame-level and instance-level memory are needed. It looks like they have the exact same formulation, and the only difference is that the instance-level one supports moving average updating for the foreground template. Is it possible to use instance-level memory only?\n\n4. Regarding tracklet feature propagation and updating, Eq (8), how do you ensure the temporal consistency of the GMM components? For instance, it is possible that the component $j$ in timestep $t-1$ does not correspond to the component $j$ in timestep $t$. Do you do association here?\n\n5. Again, the comparison in Table 5 is not fair. It makes less sense to compare FLOPS, could you compare the runtime directly, between the proposed method (including the runtime of GMM) and Non-local attention?\n\n6. In the supplementary video, I see several visualizations but found them usually too short (2-5 sec.). If it is possible, could the authors provide more results of longer video sequences?  It is important to see how a tracker performs in longer videos.\n\n7. Minor things:\n\n    - L#48 \"a MOTS approach\" -> \"an MOTS approach\"\n\n    - L#146, should $k_{\\hat{t}i}^{\\mu}$ be $k_{\\hat{t}j}^{\\mu}$?\n\n    - L#171, \"GMM (2)\": does it mean GMM (Eq. 2)?\n\na. MAST: A Memory-Augmented Self-Supervised Tracker. https://arxiv.org/pdf/2002.07793.pdf\n\nb. Space-Time Correspondence as a Contrastive Random Walk. https://arxiv.org/pdf/2006.14613.pdf", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not see any negative societal impact of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "EFDwrsAWxn", "reviewer": "Reviewer_h7Rc", "summary": "This paper concerns the problem of multi-object tracking and segmentation, using a prototype-based Cross-attention Network (PCN). The model first extracts temporal and instance-wise prototypes using a GMM clustering for all objects of interest from the past frames. Then, it uses those prototypes and their aggregation to detect, associate, classify and segment each object of interest in the current frame. The results have been evaluated on two datasets,   Youtube-VIS and BDD100k, and have been compared against tracking and/or video segmentation frameworks.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1- Clarity: The paper is well written, and the idea is clearly explained. The technical section can be followed easily. However, the application focus of the proposed technique is not very clear to me. The main claim in the paper is that it is a framework for multi-object tracking (MOT) and segmentation. Considering the dataset used for the evaluation, I am not convinced that the framework is suitable for the MOT/MOTS problem (See the extra notes at the end of my comments). \n\n2- Contribution: the paper does not offer any new theoretical contribution. It is rather a pure application paper using built upon the existing and well-explored machine learning ideas (cross attentions, prototypical generations and contrastive learning) with some tweaks and minor extensions to address a challenging practical problem (i.e. Multi-object tracking and segmentation paper). While the application papers are also well acknowledged by the NeurIPS research community, but such papers require substantial experimental evaluation on the existing benchmark datasets to validate the efficacy of their proposed system. Without this, I would be hesitant to vote for the acceptance of such a paper in this venue. This would be my major concern about this paper elaborated Next.\n\n3- Experiments and datasets:  The experiments are tested on two datasets one could be assumed suitable for MOTS problem (BDD100k), but the other one is mainly for video object segmentation (VOS) problems. Considering the main claim of the paper about addressing MOTS problem and taking this into account that this is an application paper, I am not sure why the superiority of this approach has not been tested on well-established MOTS benchmark such as MOTSChallenge and KITTI (MOTS).\n\n4- Some missing details: I didn't find how the approach deals with track initiation and termination and occlusion handling in this framework.  \n\n\nExtra notes: The challenges in the MOT problem is very different from what is in visual object tracking (VOT) or video object segmentation (VOS).  In MOT, the goal is to track many similarly looking instances (often with the identical semantic class, e.g. pedestrians, Cars ), while in VOT or VOS, the objects of interest are often semantically well distinguishable. Otherwise, similar-category objects are well separated (no need to deal with challenges in MOT). The MOT methods need to be robust in fine-grain visual similarities and they require reliable data association, track management techniques, while VOS and VOT require more general but robust visual representation.  There is neither experimental evidence nor theoretical insight this framework can be robust for all these problems\n\n    ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have well addressed the limitation and potential negative social impact of their work.  ", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received borderline negative reviews, three "below threshold" and one "above threshold". The main concerns were:

1. missing lit review on self-supervised methods.
2. choice of hyperparameters, since evaluation is only on the validation set.
3. missing evaluation on the BDD100k test set.
4. limited theoretical contribution. application paper, but not enough experiments.
5. missing details, clarity.

The authors provided a detailed response. The reviewers were only partially satisfied with the response. In particular, reviewers noted that: 1) low novelty and insight - the paper is combination of existing techniques, with no particular insight gained; 2) discussion of the relationship with self-supervised models was not sufficient in the paper or the response; 3) although the memory is saved, the computation cost is not fully reported or analyzed; 4) other missing details.  In the end, all reviewers were negative. The AC agrees and also recommends reject.
Decision: Accept (Spotlight)