STOW: Discrete-Frame Segmentation and Tracking of Unseen Objects for Warehouse Picking Robots
Authors: Yi Li
Muru Zhang
Markus Grotz
Kaichun Mo
Dieter Fox
Venue: CoRL 2023 Poster (2023)
Keywords: Unseen Object Instance Segmentation
Unsupervised Multi Object Tracking
Zero-shot
Discrete Frames

Abstract: Segmentation and tracking of unseen object instances in discrete frames pose a significant challenge in dynamic industrial robotic contexts, such as distribution warehouses. Here, robots must handle object rearrangements, including shifting, removal, and partial occlusion by new items, and track these items after substantial temporal gaps. The task is further complicated when robots encounter objects beyond their training sets, thereby requiring the ability to segment and track previously unseen items. Considering that continuous observation is often inaccessible in such settings, our task involves working with a discrete set of frames separated by indefinite periods, during which substantial changes to the scene may occur. This task also translates to domestic robotic applications, such as table rearrangement. To address these demanding challenges, we introduce new synthetic and real-world datasets that replicate these industrial and household scenarios. Furthermore, we propose a novel paradigm for joint segmentation and tracking in discrete frames, alongside a transformer module that facilitates efficient inter-frame communication. Our approach significantly outperforms recent methods in our experiments. For additional results and videos, please visit \url{https://sites.google.com/view/stow-corl23}. Code and dataset will be released.

STOW: Discrete-Frame Segmentation and Tracking of Unseen Objects for Warehouse Picking Robots
Li University of Washington
Muru Zhang University of Washington
Markus Grotz
Kaichun Mo NVIDIA
Dieter Fox University of Washington and NVIDIA
University of Washington
Abstract: Segmentation and tracking of unseen object instances in díscrete frames pose significant challenge in dynamic industrial robotic contexts, such as distribution warehouses. Here, robots must handle object rearrangement, in- cluding shifting, removal, and partial occlusion by new items, and track these items after substantial temporal gaps. The task further complicated when robots encounter objects not learned in their training sets, which requires the ability segment and track previously unseen items. Considering that continuous observa- tion is often inaccessible in such settings, our task involves working with discrete set of frames separated by indefinite periods during which substantial changes to the scene may occur. This task also translates to domestic robotic applications, such as rearrangement of objects on table. To address these demanding chal- lenges, we introduce new synthetic and real-world datasets that replicate these industrial and household scenarios. We also propose a novel paradigm for joint segmentation and tracking in discrete frames along with a transformer module that facilitates efficient fnter-frame communication. The experiments we conduct show that our approach significantly outperforms recent methods. For additional results and videos, please visit website. Code and dataset will be released.
Keywords: Unseen Object Instance Segmentation, Unsupervised Multi Object Tracking, Zero-shot, Discrete Frames
Introduction
Object segmentation and tracking, key percep- tion task for robotic picking, is particularly impor- tant in warehouse environments, where millions of commodity items are organized daily on warehouse shelves for storage and categorization, as shown Figure Future intelligent robots must acquire strong perception capabilities to help human workers stow and fetch items from these shelves. These pabilities include detecting objects with diverse ge- ometries in cluttered scenes and tracking them while other items are being added or picked up. Despite considerable research progress in this area, re- mains notoriously difficult to detect and track un- known objects in highly cluttered environments.
A densely packed shelf environ- ment. The shelf holds objects from wide array of categories. During the stowing pro- cess, human operator may obscure the cam- era's view and rearrange the objects within the bin. The robot's 's task is to pick a specific object as directed by the given order index of the object's placement in the bin.
Figure 1:
Researchers commonly address this task using a tegment-then-tracck paradigm [1, 2], which executes the two procedures sequentially. During segmentation, each frame is handled as an independent
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

image on which advanced unseen object instance segmentation methods are applied [3, 4. 5, 6]. The subsequent tracking step leverages varied techniques [7, 8, 9, 10] to group masks of the same object across frames. However, this approach has inherent limitations. Segmentation methods struggle to resolve ambiguities because they cannot utilize information from other frames to enhance segmenta- tion within each frame. Tracking lacks alternatives when segmentation fails, since its success heavily relies on consistent object appearance or location across consecutive frames. These drawbacks limit the paradigm's 's effectiveness when there are crowded scenes and discrete frames.
Our task resembles video instance segmentation (VIS), which involves segmenting and tracking objects in videos, with the leaderboard predominantly occupied by simultaneous segmenting and tracking methods [11, 12, 13]. This similarity suggests the potential adopt their methods to realize end-to-end solution in our task. However, their methods, mainly designed for videos with continuous frames, display subpar performance when faced with significant object movements between frames, prominent challenge in our task.
We therefore introduce STOW, Discrete-Franc Segmentation and Tracking of Unseen Objects for Warchouse Picking Robots, a new framework for addressing challenges in our context. STOW con- sists of new paradigm to jointly perform segmentation and tracking and a novel module, called the multi-frame attention layer, that facilitates efficient inter-frame communication. It succeeds in simuitancously achieving high segmentation accuracy, high tracking accuracy, and high robust- ness to the sim-to-real gap. Remarkably, even when trained exclusively on synthetic images, our method significantly surpasses baseline on real data and live robot experiments.
In summary, our main contributions are: (1) Task formulation for unseen object instance segmen- tation and tracking in discrete frames as well as realistic synthetic data generation and real dataset data collection and manual labeling for bins in the shelf and tabletop environments, facilitating search in this domain (2) A new paradigm to perform joint segmentation and tracking in discrete frames, along with a new module, i.e, multi-frame attention, that efficiently communicates infor- mation across frames (3) Experiments conducted on real data and on working robot to verify our network's 's superior performance
Related Works
Unseen object instance segmentation. In computer vision, traditional object instance segmentation requires prior knowledge about the objects. In contrast, our work targets potentially unseen objects without such knowledge. Previous efforts, such as UCN [4], UOIS [3]. and MF [5], addressed unseen object discovery in single-frame images using varied strategies, e.g., RGB-D feature embed- dings and metric learning loss. A recent model, SAM [6], also demonstrates robust segmentation across a wide variety of objects by training on a large amount of data. While these works focus on segmenting and tracking in single-frame images, our research extends it to multiple frames.
Video object segmentation. Like the video object segmentation task [14, 15, 9], we track unseen objects in the test set. However, the VOS task assumes accessibility an object's mask in one frame, which is not applicable in our task. For video object instance segmentation, previous works adhere to the tracking by detection paradigm, e.g., [16] and similarly [17, 18, 19], and often address the problem of multi-object tracking (MOT), i.e., the estimation of bounding boxes and identities of objects in consecutive RGB image streams. MOT tasks usually focus on tasks in traffic scenes and objects like people [20, 21] and vehicles.
Video Instance Segmentation. Our task diverges from existing VIS (Video Instance Segmentation) datasets [22, 23] in two main aspects. First, while VIS employs closed-set category approach for detection/segmennatiion. our open-set problem adds complexity by recognizing instances regardless of class, as opposed to relying on learned patterns. Second, unlike VIS's focus on continuous, limited-changes video sequences, our dataset emphasizes tracking amid drastic changes between frames, making it ill-suited for benchmarking with VIS datasets.
Unsupervised multi-object tracking. The unsupervised video instance segmentation task intro- duced in DAVIS 2019 [24] bears similarity to Video Instance Segmentation (VIS), with focus on

---

open-set category objects akin to our task. However, our approach diverges in two key aspects: (1) despite targeting unseen objects, DAVIS 2019 predominantly includes humans, vehicles, and animals, contrasting with the warehouse objects our study emphasizes, and (2) akin to VIS, neces- sitates objects to exhibit continuous movement, thereby avoiding an ill-defined task.
Image co-segmentation. Our task is similar to object co-segmentation [25], which extracts recur- ring objects from an image pair or set of images. While the goal of co-segmentation is to identify shared objects in scene, we focus on instance segmentation and do not allow similar objects of the same class. Distinguishing between an object instance and an object class is a crucial requirement for industrial warchouses.
Temporal Attention. In temporal attention, multi-frame method aligns with but uniquely stands out from prior works. Context R-CNN integrates per-Rol features from various frames to enhance current detections. Unlike its static approach and two-phase method, we embed temporal attention within each block, updating all frame object queries post each temporal attention. Meanwhile, [26] proposed module called Alignment-guided Attention (ATA) which apply temporal attention on patches with similar features through bipartite matching. Unlike ATA's ID fixed-size patch focus, our technique employs all object queries across all images, capturing varied masks and accessing broader information.
3 Problem Formulation
In this work, we introduce the novel task of segmenting and tracking unseen objects given a series of input dis- crete image frames. Though challenging, this task has broad applications in the field of robotics and is partic- ularly useful in warchouse environments, such as that shown in Figure 1. Scenes involving warehouse shelves can be exceedingly packed and cluttered, consisting of a vast assortment of items, including some that have not been previously encountered. Moreover, there may be temporal gaps between successive snapshots of the scene, during which human workers may place new objects, robots may retrieve existing items, and some objects may undergo changes in pose.
Input
Output (masks)
T
Figure 2: Task Illustration: The left col- umn presents the images inputted into our network, while the right column showcases the expected segmentation and tracking outcomes. Identical colors indicate the same object.
Formally, we formulate the problem as follows. The input to the task is sequence of images I = {I, ... Ir I, [HxWxCr }. where H and W are the height and width of the images, respectively, and C₁ represents the number of channels, i.e., 3 for RGB images and 4 for RGB-D inputs. The task involves detecting, segmenting. and tracking Ky object instances that appear in these input images, where Ky may not be known beforehand. The output of the task a set of binary object instance masks, M, = (M,, M2,..., M Mi (0, #x 1,2, ... Kz ). corresponding to each input image I, E I. Figure 2 shows the problem setting under consideration.
4
Method
uses query-based transformer architectures [27, 13] for object detection and segmentation, which we describe in Sec. 4.1. To enable the tracking of object instances across dis- crete image frames, we introduce the learning of additional object embeddings for tracking (Sec. 4.3) as well as multi-frame attention layer to distinguish between identical or distinct object instances (Sec. 4.2). Sec. 4.4 discusses training specifics and loss functions.
Our system (Figure 3)
4.1 Backbone Architecture
Following [27, 13], the backbone of our network consists of three components: (1) a ResNet-based image encoder, (2) transformer-based object query decoder, and (3) several prediction heads tasked with determining object properties, such the likelihood of object existence and object masks.
3

---

Backbooe
Fhithct N/.5
BAWSS
HAWAC
K
Dense Feature map
Pixol Embedhing
Pixel Eirbediing
Mask Prodiction
Malti Pern----mmee Atto.
Cros Atin.
Aetf
Mask Embedding
FEN
B.W.
Pred
Scote
Inpuil Framus
Initial Queries
Object
Object
Queriees
Tokens
Object Embedding
Transformer Decoder
Predictor
Figure 3: As outlined by a dashed rectangle, our transformer decoder ingests dense feature maps converted from input frames and produces object tokens for each image. These tokens predict con- fidence scores, mask embeddings for mask prediction, and object embeddings for association. We also introduce novel "multi-frame attention" layer, which attends object queries from all frames.
4.2 Multi-Frame Attention
ResNet-based image encoder. We use ResNet-50 [28] to transform every input frame I (f XC 1, ., T") into a dense low-resolution feature map F; E 1R $ x xCr, Here, CF denotes the channel dimension of the output dense feature map, while S 32 is the down-sampling ratio used in this work.
Transformer-based object query decoder. We employ a DETR-like transformer decoder [29] that takes the produced dense feature map F; as input and learns to decode a set of N₂ object tokens E CC,, as the outputs. Each object token contains the latent information neces-
{ql, q.. q N₂ sary for tasks such as classification estimation, mask prediction, and tracking in discrete frames. Our transformer decoder consists of L = 10 transformer blocks; each block contains one cross-attention layer, one self-attention layer, one feed-forward layer, and one novel multi-frame attention layer that correlates object features across different image frames (Sec. 4.2).
Prediction head for object masks. To get a per-pixel segmentation mask Mi E {0, Hx for each object token q. we first two-layer multilayer perceptron (MLP), which maps input object token qf to mask embedding of E RC:; we then employ a multi-scale deformable attention transformer module (MSDeformAttn) [30] to convert the dense feature map F, to pixel embedding map P.- E * xC. Here, C. denotes the channel dimensions used for the mask and pixel embeddings. We calculate the dot product between the object embedding e, and the pixel embedding Pt in order to obtain the mask prediction Mi at reduced resolution of x 4 bilinear upsampling operator is applied to map Mj back to the original image resolution for the final mask prediction Mf E (0, 1} Hx
W. Subsequently,
Prediction head for object existence scores. Taking the object token qf as input, we leverage simple linear layer to estimate an object existence likelihood score 8t [0, 1]. In the context of unseen object instance segmentation, we are not concerned with precise target object categories, nor do we have access to this knowledge. This characteristic simplifies the task into a binary classifica- tion, the objective of which is to estimate the confidence score of whether each segment corresponds appropriately to an object.
4.3 Object Embedding for Tracking
In addition to the predicted object existence score and its segmentation mask in each frame, add new prediction head for object tracking embedding to enable the association of object tokens belonging to the same object from different input frames. Specifically, we employ a two-layer MLP to learn a mapping from the input object query qf to another object embedding used for tracking rf E C.C. Here, C, represents the number channels of the object embedding vector.
During the inference phase, implement an associator group object tokens with similar object embeddings. Specifically, sequentially traverse each frame in the sequence. For each frame,

---

we retain only those object tokens q' whose confidence scores surpass predefined threshold 8gcare- For each input sequence, we maintain a trajectory bank, T, that encompasses the trajectories of all observed objects. Each trajectory in the trajectory bank T E ' consists of object tokens q' that are considered to belong to the object Subsequently, we compute the similarity score between these selected object tokens and the previous trajectory using the following equation:
Sim(q', T") max( )), for q/ E T
(1)
In this equation, R. is the function that transforms the object token q¹ into the object embedding r'.
After this step, we use the Hungarian algorithm [31] to search for an optimal bipartite matching Ô from all possible bipartite matchings P that can maximize the overall similarity between object tokens q and the trajectory bank T:
N₂
i arg max DEP Σ Sim(q', (
(2)
We initialize the trajectory bank, T, with an adequate number of false alarm tokens, TFA, each of which exhibits a constant similarity Smatch to all object tokens. The hyper-parameter Smatch can false alarm threshold in matching. Any predicted object tokens assigned to
also be interpreted tokens from TAA are assumed not to match any existing trajectory; thus, new trajectory is opened for them.
Our tracking method also enables the possibility of handling multiple identical objects in the same scene. Specifically, object tokens corresponding identical objects are likely to be recognized due to the expectation that object embeddings near to them.
The previous modules we introduced work independently for each frame without any cross-frame information exchange. To facilitate efficient communication between frames, we introduce a new component, the multi-frame attention layer, into the transformer decoder.
The fulti-frame attention layer is extension of the self-attention layer, which operates on object queries from single frame; it attends to object queries from all accessible frames. To illustrate, we denote the intermediate object queries after each feed-forve meed-forwrrd network as Xf. where and t indicate the index of transformer blocks and frames, respectively. A standard self-attention layer (with a residual path) computes the following (we omit the normalization term /dz here for simplicity):
SelfAut(Xf) (X+) = softmax ([tttmax(((())) (J)(X)) JK (x)) + X/.
In contrast, our multi-frame attention layer computes:
MultiFrameAAtm X softmax(Jo(X)) fx(())² }ʃv (X1) + X;-
(4)
Here, X, represents the set of object queries from all frames X; {X) for t = 1,,,...,.77 The functions fo. fK, and fv are linear transformations that convert X₁ or X into C-dim space.
T..
The multi-frame attention layer is positioned at the end of each transformer decoder block, which repeated L times in our network. Therefore, output object queries can incorporate the dense feature map of the current frame update their prediction after communicating with object queries from other frames. Furthermore, the multi-frame attention layer is computationally efficient since it attends only to object queries from all frames, typically around 100 queries per frame vs the 1200 queries per frame used in Mask2Formerr video [12] for images with size 640 x 480. Such efficiency lets it process long-term input sequences and large-scale images.
4.4 Training and Losses
Our model adopts the same loss function as utilized in Mask2Former [27] for classification and mask prediction. This includes the softmax cross-entropy loss, denoted as Letasss for classification, along with binary cross-entropy loss, denoted as Lee, and the Dice loss, denoted as Laice, for mask prediction. To address tracking loss, we employ the contrastive loss Lcounn used in DCN [32] along with the softmax loss (also referred to the n-pair loss InfoNCE loss) from CLIP [33]. For an object token of object tokens paired with the same object o; in different frames g.' of f are treated
5

---

as positive samples and pushed closer, while tokens assigned to different objects or backgrounds are treated as negative pairs and pushed away. See the supplementary material for more details.
Consistent with the approach DETR [29] and Mask2Former [27], we leverage the Hungarian algorithm [31] to establish bipartite matching between the predicted object tokens and ground truth that minimizes the overall loss. Notably,
exclude the tracking loss from the Hungarian algorithm's s computation given that Lsoftmaz is influenced by the object tokens contributing to this loss. The final loss is Liout = Aclass Celass + Xeece + Xdice Eaice + Avonna + Asottmax Ssoftmax
Experiments
We assess our methodology in two typical environments bin in a shelf and tabletop- both are representative settings in a multitude of warehouse and domestic applications. For each setting, we generate corresponding synthetic data for the training phase and collect and annotate real-world data for evaluation. We also integrate our approach into a bin-picking robotic system for practical experimentation.
both of which
We train our models separately distinct synthetic datasets for the shelf and table environments. Each run exclusively uses one dataset and is evaluated against the corresponding real-world test set. Inference takes around 0.4 seconds for 15-frame sequence on an RTX 2080Ti. Additional training details are in the supplementary material.
5.1 Dataset and Evaluation
Synthetic data. We construct mesh models for shelf and table bins using textures from the CCO dataset [34], and object meshes from the Google Scanned dataset [35], consisting of over 1000 models. Excluding 70 with isolated parts, we utilize 900 objects for the training set and 100 for validation. Objects are randomly rotated and positioned to avoid collision, with no heavy occlusion the shelf environment. total, we generate around 10,000 sequences for the shelf, each with at least 2 packed frames, and 2,000 sequences for the tabletop, each containing 15 frames.
collect and manually label 44 sequences with 220 images for the shelf scenario and 20 sequences with 280 images for the tabletop scenario. Our dataset includes over 150 diverse objects. These range from relatively simple objects, such as boxes and bottles, to more complex ones, like transparent water bottles enclosed in plastic bags. In each sequence, we progressively add objects until either the bin is full or around 10 objects are placed on the table. Object rearrangement could occur between any two frames, leading to significant changes in object location and appearance.
Real-world data. For real-world scenarios,
Evaluation metries. We adopt the evaluation method of the VIS challenge [22], modified version of the MS-COCO metric [36]. In video instance segmentation, each object is represented by series of masks, and the Intersection over Union (loU) is calculated at the level of these mask sequences. To construct the Ciecision-Recall (PR) curve, the confidence threshold is systematically varied, with each threshold yielding a distinct data point on the curve. The area under the PR curve provides the Average Precision (AP).
In the context of this study, if not further specified, AP@0.5 denotes the average precision at an IoU threshold of O.5. Similarly, AP@ all represents the mean AP calculated over multiple IoU thresholds, specifically from 50% to 95% in 5% increments.
5.2 Results and Analysis
Baseline methods. We benchmark our approach against three state-of-the-ar Video Instance Seg- mentation (VIS) methods: MInVIS [13], Mask2Former-video [12], and VITA [11]. To ensure an equitable comparison, all methods utilize ResNet-50 as the backbone, and the hyperparameters (such as batch size, maximum iterations, and the number of sampled frames) are standardized to match ours. Hence, all methods are trained on an identical number of images.
Qualitative results. As depicted in Table our method notably outperforms existing VIS tech- niques, yielding an approximate 10% improvement in the shelf environment and a 20% increase in the table environment, even without using the multi-frame attention layer. Interestingly, MinVIS

---

Shelf
Tabletop
Method
AP@all
AP@0.5
AP@all
AP@0.5
MinVIS
6.3
21.2
0.7
0.0
Mask2Former Video
35.0
66.1
27.7
56.7
VITA
42.7
70.1
26.6
55.0
STOW (Ours)
55.6
81.3
49.7
75.4
Table 1: Comparison between our method and leading video instance segmentation methods. Net-
works are trained on synthetic data and tested on real, unseen data. All use ResNet-50[28] and train on an identical number of images.
PPP
Figure 4: Visualized results for the shelf environment. Masks with the same color and index are associated and predicted as the same object by the network.
exhibits subpar performance in our task. We speculate that this is due its reliance on the proximity between object tokens as measure of similarity: this method presupposes that the changes between frames minimal and that objects maintain similar locations and appearances. However, these conditions do not consistently hold in our task, potentially explaining the subpar performance.
Quantative results. We also show visualization results on a subset of real test data in Figure and Figure 5. Our approach adeptly handles frame changes amid various types of noise. Despite significant movement and rotation between frames, the network successfully segments and tracks broad array of object categories. It efficiently segments and continues tracking any new objects that introduced. Figure 5 also demonstrates the method's robustness against backgrounds: it avoids predicting them objects even though walls are not included in the training set. Supplementary materials contain additional results that contrast our approach with other methods.
5.3 Ablation Study
Frame attention. We evaluate the performance of our method with and without the cross-frame attention module on both the shelf and table scenes using ResNet-50 and Swin-T backbones. The cross-frame attention module yields consistent performance improvements across all configurations, as shown in Table 2.
Sim2Real gap. We evaluate our methods the synthetic validation set, which contains ob- jects not included in the training set, and com- pare it to the number we tested on the real test set. Results, shown in Table 3, reveal that other methods get results that are relatively to ours for the synthetic set, but their performance drops dramatically when we evaluate on the test set. This implies that they have difficulties solving the sim2real gap.
multi frame
shelf
tabletop
AP@all
AP@0.5
AP@all
AP@0.5
51.8
78.7
68.5
55.6
81.3
49.7
75.4
Table 2: With and without the multi-frame tention layer. The left column denotes whether we incorporate multi-frame attention in this ex- periement. All other hyper-parameters remain the same. Use of the frame attention layer boosts both shelf and tabletop environment performance by ~5%.
5.4 Real Robot Applications
We integrated our visual perception technique
into an autonomous shelf-picking system [37]. The system's multi-component software architecture is managed by state machine. The system setup uses UR16e industrial robot situated in front of industrial warchouse shelf filled with objects. Within the robotics community, it's standard to
7

---

Figure 5: Visualized results for the table environment. Masks with the same color and index are associated and predicted as the same object by the network.
address perception problems by combining unseen instance segmentation methods with other estab- lished techniques, as seen [1],[38, [39], [40]. Other than VITA [11] Table 1, we also combine UCN [4], staple in unseen instance segmentation, with SIFT [7], renowned keypoint extraction method, reflecting the conventional solution for this challenge. The center of the mask used the grasping point for suction cap. Our evaluation protocol employs fixed set of diverse items, stowed at specific locations and orientations within the bins, to ensure reproducibility and comparability of results since performance can fluctuate with different item configurations and inherent system stochasticity.
We testing each method with 50 trials across different levels of difficulty, involving over 100 objects. For UCN [4]+SIFT [7], the perception success rates stand at 56%. With our STOW method, this rate increase to 76%.
method
synthetic AP@all
real
AP@0.5
AP@all
AP@0.5
MinVIS[13]
0.7
0.0
M2F-V[12]
71.6
83.7
27.7
56.7
VITA[1I]
69.4
81.9
26.6
55.0
Ours
74.1
89.3
49.7
75.4
Limitations
Table 3: Ablation study on solving the sim2real gap. After training on synthetic tabletop training set, separately evaluate each method on syn- thetic tabletop validation set and a real tabletop test set; note that objects in the synthetic valida- tion set are not included the synthetic training set.
Our method, while effective, has limitations in handling highly cluttered environments and complex objects. False positives and negatives occur object detection, especially in intricate settings. The segmentation process can result in over- or under-segmentation due to complex object boundaries and textures. In object track-
ing, we sporadically encounter mistracking incidents and occasional failures to distinguish between multiple objects. Refer to the supplementary material for detailed analyses and examples of these limitations.
Conclusion
In this paper, we introduce the task of segmenting and tracking unseen objects in discrete frames which is widely used in robotics tasks but under investigation. We formulated the problem and collected both synthetic and real datasets. We also propose a novel paradigm for joint segmentation and tracking, incorporating multi-frame attention for better inter-framt communication. Even when trained solely on synthetic data, our method adeptly handles clustering and large movements in real-world sequences. Our innovative approach excels in segmenting and tracking within both shelf and tabletop settings, surpassing tfte--ofthhe-a techniques with 10%-20% improvement in AP in real-world scenarios and more than 20% success rate in robot experiments.

---

Acknowledgments
This research is funded by the UW Amazon Science Hub as part of the project titled, "Robotic Manipulation in Densely Packed Containers." We would like thank Dr. Michael Wolf from Ama- zon for valuable discussions. We further would like to thank our students Sanjar Normuradov and Soofiyan Atar
References
[1] A. Goyal, A. Mousavian, C. Paxton, Y.-W. Chao, Okorn, Deng, and D. Fox. Ifor: Iter- ative flow minimization for robotic object rearrangement. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14787-14797, 2022.
[2] Y. Lu, N. Khargonkar, Z. Xu, C. Averill, K. Palanisamy, K. Hang, Y. Guo, N. Ruozzi, and Y. Xiang, Self-supervised unseen object instance segmentation via long-term robot interaction. arXiv preprint arXiv:2302.03799 2023.
[3] C. Xie, Y. Xiang, A. Mousavian, and D. Fox. Unseen object instance segmentation for robotic environments. IEEE Transactions on Robotics, 37(5):1343 135-1139999999,, 2021.
[4] Y. Xiang, C. Xie, A. Mousavian, and D. Fox. Learning rgb-d feature embeddings for unseen object instance segmentation. In Conference Robot Learning, pages 461-470. PMLR, 2021.
[5] Y. Lu, Y. Chen, N. Ruozzi, and Y. Xiang. Mean shift mask transformer for unseen object instance segmentation. arXiv preprint arXiv:2211.11673 2022.
[6] A. Kirillov, E. Mintun, N. Ravi, Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-.. Lo, al. Segment anything. arXiv preprint arXiv:2304.02643. 2023.
[7] D. G. Lowe. Distinetive image features from scale-invariant keypoints. International journal of computer vision, 60:91-110, 2004.
[8] Z. Teed and J. Deng. Raft: Recurrent all all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceed- ings, Part Il 16, pages 402-419. Springer, 2020.
[9] H. K. Cheng and A. G. Schwing. Xmem: Long-term video object segmentation with an ackinson-shiffrin memory model. In Computer Vision-ECCV 2022: 17th European Con- ference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVIII, pages 640-658. Springer, 2022.
[10] J. Yang, M. Gao, Z. Li, S. Gao. F. Wang, and F. Zheng. Track anything: Segment anything meets videos, 2023.
[11] M. Heo, S. Hwang,
Oh, J.-Y. Lee, and Kim. Vita: Video instance segmentation via object token association. arXiv preprint arXiv:2206.0444 04403, 2022.
[12] B. Cheng, A. Choudhuri, Misra, A. Kirillov, R. Girdhar, and A. G. Schwing. Mask2former for video instance segmentation. arXiv preprint arXiv:21 /2. 10764, 2021.
[13] D.-.. Huang, Z. Yu, and A. Anandkumar. Minvis: A minimal video instance segmentation framework without video-based training. arXiv preprint arXiv:2208.022 02245, 2022.
[14] N. Xu, L. Yang, Y. Fan, D. Yue, Y. Liang, J. Yang, and T. Huang. Youtube-vos: large-scale video object segmentation benchmark. arXiv preprint arXiv.1809.03327. 2018.
[15] J. Pont-Tuset, F. Perazzi, S. Caelles, P. Arbeláez, A. Sorkine-Horung and L. Van Gool. The 2017 davis challenge video object segmentation. arXiv preprint 4rXiv:1704.00675, 2017.

---

[16] A. Bewley, Z. Ge, L. Ott, F. Ramos, and B. Upcroft. Simple online and realtime tracking. In 2016 IEEE international conference on image processing (ICIP), pages 3464-3468, 2016.
[17] N. Wojke, A. Bewley, and D. Paulus. Simple online and realtime tracking with deep as- sociation metric. In 2017 IEEE international conference on image processing (ICIP), pages 3645-3649. IEEE, 2017.
[18] J. Luiten, I. E. Zulfikar, and B. Leibe. Unovost: Unsupervised offline video object segmen- tation and tracking. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2000-2009, 2020.
[19] Garg and Goel. Mask selection and propagation for unsupervised video object segmenta- tion. Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pages 1680-1690. 2021.
[20] L. Zheng, L. Shen, L. Tian, S. Wang, J. Wang, and Q. Tian. Scalable person re-identification: A benchmark. In Proceedings of the IEEE international conference computer vision, pages 1116-1124, 2015.
[21] A. Wu, W.-S. Zheng. H.-X. Yu, S. Gong, and J. Lai. Rgb-infrared cross- moos-modalitty person re- identification. In Proceedings of the IEEE international conference on computer vision, pages 5380-5389, 2017.
[22] L. Yang, Y. Fan, and N. Xu. Video instance segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 5188-5197, 2019.
[23] J. Qi, Y. Gao, Hu, X. Wang, X. Liu, X. Bai, S. Belongie, A. Yuille, P. H. Torr, and S. Bai. Occluded video instance segmentation: A benchmark. International Journal of Computer Vision, 2300)))202,20399 2022.
[24] S. Caelles, J. Pont-Tuset, F. Perazzi, A. Montes, K.-K. Maninis, and L. Van Gool. The 2019 davis challenge vos: Unsupervised multi-object segmentation. arXiv:1905.00737,, 2019.
[25] C. Rother, T. Minka, A. Blake, and V. Kolmogorov. Cosegmentation of image pairs by his- togram matching-inccoporati a global constraint into mrfs. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 06), volume 1, pages 993- 1000. IEEE, 2006.
[26] Y. Zhao, Z. Li, X. Guo, and Y. Lu. Aligmeen--guide temporal attention for video action recognition. Advances in Neural Information Processing Systems, 35:13627-13639, 2022.
[27] B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar. Masked-attention mask trans- former for universal image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1290-1299, 2022.
[28] K. He, Zhang, Ren, and Sun. Deep residual learning for image recognition. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[29] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end ob- ject detection with transformers. In Computer Vision -iconnECCV222 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pages 213-229. Springer, 2020.
[30] X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai. Deformable detr: Deformable transformers for end-to-end object detection, arXiv preprint arXiv:2010.04159, 2020.
[31] H. W. Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.
10

---

[32] P. R. Florence. Dense visual learning for robot manipulation. PhD thesis, Massachusetts Institute of Technology, 2020.
[33] A. Radford. J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, al. Learning transferable visual models from natural language supervi- sion. In International conference machine learning, pages 3748-8763. PMLR, 2021.
[34] ambientCG. Creative Commons Zero (CCO). atttp::/ambieett
2023.
[35] L. Downs, A. Francis, N. Koenig, B. Kinman, R. Hickman, K. Reymann, T. B. McHugh. and V. Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In 2022 International Conference on Robotics and Automation (ICRA), pages 2553- 2560. IEEE, 2022.
[36] T.-Y. Lin, M. Maire, Belongie, Hays, P. Perona, D. Ramanan, P. Dollár, and L. Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Swirzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740- 755. Springer, 2014.
[37] M. Grotz, S. Atar, Y. Li, Torrado, B. Yang. N. Walker, M. Murray, M. Cakmak, and Smith. Towards robustly picking unseen objects from densely packed shelves. In Work- shop on Perception and Manipulation Challenges for Warehouse Automation, 2023. URL
http: / www.ancbcchhhc..mm..m s3. www.aaaaaassss..............aaas www. rss23. html.
[38] A. Gouda, A. Ghanem, and C. Reining. Dopose-6d dataset for object segmentation and 6d pose estimation. 2022 21st IEEE International Conference on Machine Learning and Applica- tions (ICMLA), pages 477-483, 2022. URL https:ttt/./c......t http:::/apl..eematticccee.ee
www.mnttisccoolo.ooo... .org/
CorpusID: :
[39] J. Chen, M. Sun, T. Bao, R. Zhao, L. Wu, and Z. He. 3d model based zero-shot pose estimation pipeline. ArXiv, 0bs/22055.19944 2023. URL https: / /api..p../// semant weschcclss .rrg/ CorpusID 258960481.
[40] M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox. Contact- -antact-graspnet: Efficient 6-dof grasp generation in cluttered scenes. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 13444 IEEE, 2021.
[41] R. Florence, I.. Manuelli, and Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. arXiv preprint arXiv:1806.087555 2018.
A Dataset Detail
A.1 Synthetic Data
We built a synthetic dataset using high-quality household models from the GoogleScanned dataset[35] with two typical settings: a) Shelf and b) Tabletop.
Shelf environment. In shelf environments or other bin-based object arrangements, the objects are akin to books and constrained ensures that each object is guaranteed to have at least one visible face, but it also leads to significant occlusion among objects. The camera is positioned at the front of the bin to capture images of the scene, subject to random perturbations in the location that inject noise into the data.
orientation. This scheme
Given that each bin contains maximum 3 to 5 objects, segmentation and tracking tasks become trivial if the scene contains fewer than objects. To address this issue, image frames are generated only when the bin is nearly full. We leverage approximately 900 objects sourced from the Google Scanned datasct, resulting in training set of approximately 9000 image pairs. We use the remaining

---

Figure 6: Some objects used during the evaluation. Objects vary greatly in shape and physical properties, with some being partially transparent or wrapped in a bag.
100 objects to generate approximately 1000 image pairs for the test set. Each image pair may exhibit the introduction of new object in addition to existing objects undergoing flipping operation or relocation with certain probability.
Tabletop environments. Generating datasets of objects placed on a table requires different set- tings given the absence of walls and typically larger surface area than in a bin-based object ar- rangements. As result, we adopt an alternative strategy for dataset generation. Specifically, each sequence consists of 15 images, with the first 10 images incrementally introducing new objects while shuffling existing objects between frames. No new objects added in the final 5 frames, though the shuffling of existing objects persists. Due to the random placement of objects the table, instances of full occlusion may occur in certain frames and subsequently reappear in subsequent frames.
To construct our training and testing datasets, we utilize 900 objects sourced from the Google Scanned dataset, producing 2000 sequences for the training set, with the remaining 100 objects used generate 500 sequences for the test set.
A.2 Real Data
As we did for the synthetic evaluation, split the evaluation into shelf and tabletop environments, the most common real-world scenarios encountered. To evaluate our method on challenging real- world scenarios, we need a large variety of objects; Figure depicts some of the objects used during the tablefop evaluation. For shelf environments, we use an Azure Kinect RGB-D sensor, and for tabletop ones we use an Intel Realsense D455 camera. Camera distance ranges from to 1.5 meters. Each time an object is placed on the table or in new bin, new image captured. Objects can be rearranged maximize space utilization as they are placed in the scene. After all the objects are placed in the scene, also displace the objects for more refined evaluation. Camera images are manually labeled using the interactive segmentation of the object tracking framework XMem [9]. We collected and annotated more than 280 images with more than 150 different objects for the tabletop scenario and 220 images for the shelf scenario.
B Training and Inference Details
B.1
Training Details
We set the maximum number of iterations to 16k using an initial learning rate of le-5, which was then dropped by 0.1 after 14k iterations. The number of classes is set to since we are aiming to handle unseen objects. For the shelf dataset, we trained our network with a batch size of 32 and leveraged frames from each sequence; for the table dataset, we set the batch size 8 and randomly selected frames from each sequence. To enhance the diversity of our dataset, applied random color jittering and rotation to the input before feeding it to the network. The training process was executed on single NVIDIA A-40 GPU and took approximately 13 hours.
12

---

During the training phase, we excluded the initial predicted object embedding, which was directly generated from the query feature. Additionally, when handling negative queries, we adopted a more selective approach by considering only queries whose IoU with any ground truth was lower than 0.6 rather than regarding all unmatched queries as negatives. This was motivated by lack of clarity regarding which patches truly represent objects in unseen object settings (in contrast close-set settings).
B.2 Associator
We show below an example of code demonstrating how associate object tokens from frame with the trajectory bank built in previous frames. In implementation, score 0.6 and match 0.2 (similarity ranged in [-1,
def sesociate_one_ttteee frame (traj dbject_tokens_cur___ delta_score, delta_track):
object_token [x for in queries_this_frrne frame if scorc"" delta_score]
num_trackers 1en (traj_bank)
Ng = (en tokknss
similarity torch. (num trackers+Ng, num numppredy *delta +delta_trace track
Extract ob embedding from current frame's s object tokens obj_embed torch.stack cbj_embed) for x in object_tokens))
Compute similarity between object embedding of trajectory and current frame'ss s object tokens
for traj_idx, traj in enumerate (traj_bank): trai_obi_embed' sim = trai__ii_eeaeeeeeeee @ ebi_embed iiniaarityltrajiiaa
forch.stace [x[' [xl[oojjeebedd ] for x in traj))
sim.max (dim=0) [0]
Perform Hungarian matching to find bipartite matching which have hightest similarity traj indices, obj token indices = bungarian_ natching/-similarien
Update tracker for traj_idx, token_idx in zip(traj_indices, obj_token_indices)
if traj_idx > num_trackers:
# if it is not matched with any existing trajectory
braj_bank.appere ([object_tokens [token_idx]])
else:
traj_bank [traj_idx] append (object_tokens [token_idx]] return bank
B.3 Loss
We keep the loss function that Mask2Former used for classification and mask prediction, which means binary cross entropy and dice loss for mask prediction and softmax cross entropy loss for classification.
For the object embedding head, we also two losses: contrastive loss and softmax loss (or n-pair loss and InfoNCE loss).
13

---

Contrastive Loss. from [32].
We use contrastive loss modified from DCN[41] with hard-negative scaling
mattces (Q)
Σ
D(a 9:2)
(5)
N Mattches
Nasnctes
1
monn-mtthhes (Q)
Σ
(M, M - D(Li 9 (45)
(6)
Nhard-neg Nemmennctss
((Q) mantches (Q) + - mon-matches (Q),
where
Nard negatives
Σ
1(M - D 922 0).
Here, Q denotes all object tokens from images, and qi" denotes the object tokens assigned to object o₂ in frame MM the margin parameter used to ensure that non-matched pairs have distance of least M apart. The distance function D is the cosine distance function, in UCN [4], which is defined as:
Da (2) 1 - r'
Here, r¹ f(a') [f('') is the object embedding of object token i, which is computed by first forwarding the query to a linear layer and then normalizing it unit vector. To expedite the training process, selectively incorporate subset of negative queries to contribute to the contrastive loss, thereby enhancing its efficiency.
An illustration of the contrastive loss is shown in Figure 7. Assuming that three frames are sampled from a sequence during training, the contrastive loss will computed between all frames. In the figure, matched pairs are denoted by dark gray and apply loss according to Equation 5, while non- matched pairs are denoted by light gray and apply loss according to Equation 6.
Softmax Loss
We also modified the N-pair/InfoNCE loss used in CLIP[33].
explr? rf ee))
Luuttmax (t)
Σ
(10)
KEQ iEO(I) Ejeq, exp(re rt)e")
gootmmax
Σ
Lxottmax(1) (t),
(11)
IE1,-.,T
where Q denotes all positive queries, O(t) denotes all objects in frame and Q denotes all queries in frame t. This is also shown in Fig. 7-b, where the label of each row corresponds to the index of the query assigned to the same object. If there are identical objects in the same frame, the softmax loss should be extended by copying all queries in this frame times, each time keeping only one query for that object. This allows queries containing the same object to be converted into multiple query sets, each consisting of only the target object.
Thus, the final tracking loss can be represented as
Luack
Acontra Ceontra + Agoftmax Cootmax
(12)
C Detailed Analysis
C.1 Sim-to-real Gap
Results are shown Table 4. The experiment is conducted in the tabletop environment with the same setting as in main manuscript. We see the following from
(1) The Image AP results from both the synthetic validation set and real test demonstrate that VIn outperforms Mask2Former Video and VITA. This indicates that the approach of
14

---

nagatia spertexx
manthed paits
sampled negattie
ignored parr
label
-1
(8)
(b) (OjaaiiinN Loss
Figure 7: Tracking loss. In this example, three frames are sampled from a sequence, denoted with different border colors. Object tokens that match the objects in the images represented by blue square, an orange triangle, and red circle. The hexagon denotes background object tokens that do not match to any objects. (a) the contrastive loss is computed between all frames, where matched pairs (dark gray) apply loss using Equation 5, non-matched pairs (white) apply loss using Equation 6, and ignored pairs (light gray) not contribute to the loss. (b) the n-pair/InfoNCE loss is computed over all positive queries and queries from each frame. Equivalent to using softmax
cross-entropy while setting the label of the index of queries assigned to the same object.
syn, video AP AP@all
syn., image AP
real, video AP
real, image AP
method
AP@0.5
AP@all
AP@0.5
AP@all
AP@0.5
AP@all
AP@0.5
UCN [4]
67.3
82.9
52.4
86.6
MinVIS [13]
0.3
2.6
82.4
91.8
0.7
0.0
54.5
72.7
Mask2Former Video [12)
71.6
83.7
72.8
82.6
27.7
56.7
38.9
57.3
VITA [1]]
69.4
81.9
70.6
80.2
26.6
55.0
41.4
63.0
Ours
74.1
89.3
87.6
95.3
49.7
75.4
80.1
Table 4: Evaluation of SOTA VIS methods the unseen object instance segmentation task. "syn" indicates evaluation on synthetic tabletop dataset; "real" denotes evaluation on a real-world table- top dataset. The evaluation metrics include "video" and "image" for assessing performance in dif- ferent contexts (as described in subsection B.1). We see that MinVIS exhibits superior performance in detection, while Mask2Former Video and VITA excel in matching. Remarkably, our proposed method harnesses the strengths of both approaches, surpassing all evaluated methods in overall per- formance.
Mask2Former Video and VITA, which utilizes single object token predict object masks across entire sequence, is less effective than using distinct object tokens for each frame. Consequently, it is difficult for object tokens to efficiently manage discrete frames with considerable movement and appearance variations.
(2) significant decline apparent the Image AP and Video AP results on the real test set for Min This suggests suboptimal tracking performance when applying object tokens directly.
(3) Notably, our method experiences less dramatic drop in performance, evidenced by the Video AP results on both the synthetic validation set and the real test set. This suggests that our method is more adept at managing the simulation-to-reali gap.
D Failure Cases
The efficacy of our method is sometimes compromised in scenarios characterized by crowded scenes or significant alterations object appearance, as illustrated Figure 10. We categorize these short- comings into two principal groups: segmentation failures and tracking failures.
Segmentation failures arise from issues such as:
Under-segmmentation
This occurs when objects have similar colors or lack clear borders, leading blending of distinct entities.
Over-segmentatiin:: In this case, single object is erroneously identified multiple entities due to recognition failures.
15

---

1
Figure 8: Results from different methods on the tabletop dataset. Methods ordered from top to bottom: MinVIS, Mask2Former- Video, VITA, and Ours (STOW).
Detection Here, an object entirely missed, leading to its absence in the segmented output.
Tracking failures, on the other hand, include:
Mismatch: This involves incorrect associations between objects across frames confusion arising from similar-looking distractors.
Mistrack: In these instances, the algorithm fails to consistently identify the same object, resulting in tracking inconsistencies.
In both categories of failure, the complexity of scene compositions and variations in object appear- ances are pivotal factors that undermine the performance of our tracking method. We are devoted to exploring advanced strategies to mitigate these limitations, aiming for enhanced robustness in diverse and dynamic environments.
16

---

Figure 9: Results from different methods on the bin dataset. Methods ordered from top to bottom:
Min VIS, Mask2Former-Videe VITA, and Ours (STOW).
Tabletop
Shetf
ADT
Figure 10: Failure Cases Illustrated. In the Tabletop settings (top 2 rows), we observe under- segmentation (top left), over-segmentation (top right), mismatch (bottom left), and mistrack (bot- tom right). In the Shelf settings (bottom two rows), the failures include a combination of under- segmentation and failure detect new objects (top left), failure track and mismatch (top right), failure to detect (bottom left), and failure to segment (bottom right).
17


Reviews:
[{"review_id": "wvLH2aYSNpU", "reviewer": "Reviewer_Bc1q", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This method presents an architecture for object segmentation and tracking across frames. While prior methods include video-based object tracking assuming minimal movement between frames, this method makes no such assumptions. It leverages attention across time along with bipartite matching of embeddings into the past, in order to track objects, including those that disappear and reappear, over time.", "summary_of_recommendation": "Overall, the paper is solid. However, I have some concerns about the claimed novelty and performance compared to some simpler pipelines, which I hope the authors will address in the rebuttal.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "- Is it possible to move the architecture figure a bit earlier in the paper? It is useful to refer to it as the architecture is being described.\n- How is the multi-frame attention different from past papers using temporal attention?\n- Why is a different baseline used for real-world physical experiments compared to the perception-only experiments?\n- How does this do when objects are lying down vs standing upright? Is the method able to consider this dramatic variation in pose as belonging to the same object?\n- Is there any explanation for why your method does better transferring to real compared to other methods?\n- In related work, co-segmentation was mentioned but the difference between that and this method is not too clear.\n- What is the performance difference between a simpler method mentioned in the review above (not end-to-end) versus this end-to-end approach.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper overall is quite strong and presents a novel approach to tracking objects across frames with significant differences. I believe the paper has the following strengths. \n\n- Very well written and clear figures.\n- Solid proposed architectural solution to the problem.\n- A good set of experiments and ablations that demonstrate the performance of the method.\n- Real robot experiments using the method.\n\nI have the following suggestions/comments for the authors as well.\n\n- I was wondering whether the multi-frame attention is truly a novel contribution of this work. What is unique about the multi-frame attention layer compared to temporal attention from past papers, with just a few examples below.\n    - https://arxiv.org/abs/1912.03538\n    - https://arxiv.org/abs/2210.00132\n- I’m curious as to how this would compare to a straightforward/naive approach of using a frozen segmentation network (e.g. SAM) combined with some pre-trained image embedding on the masked objects from earlier frames and some bipartite matching with dot products.\n- The examples shown in the video all seem quite straightforward. I’m wondering if there are simpler methods (like the point above) that could work equally well for this. Is there really a need for an end-to-end architecture for this problem? If so, can you demonstrate success on examples where a simpler method wouldn’t work?", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "q7Ir7CTOdfu", "reviewer": "Reviewer_izjm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces a transformer based architecture for segmentation and tracking that enables tracking across discrete image frames. This allows the system to work in packed and cluttered scenes such as warehouse shelves, and also handles unseen objects between frames. Compared to other state of the art video segmentation models, this methods improves performance for shelf and tabletop synthetic datasets. The paper also introduces a synthetic and real dataset.", "summary_of_recommendation": "I am recommending a weak accept. The overall system and architecture seem sound - while this work does not seem to contain any major advances, there is some novelty in the architecture that shows better results compared to other methods.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "* A more thorough study on failure cases may be interesting, to better understand where methods like this may fail. E.g. Are there scenes that are too crowded for proper segmentation/tracking?\n* Will the code and dataset be released?\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n* Overall the presentation and contributions are clear.\n* The architecture introduced in the paper is interesting and makes sense for this segmentation and tracking task.\n* Ablation studies show that the multi-frame attention helps for shelf and table scenes.\n* The synthetic + real dataset introduced seem useful for these sorts of industrial warehouse tasks.\n* Real robot results including videos.\n* Failure cases described in appendix.\n\nWeaknesses\n* It seems that results are only reported on the datasets constructed for this task. It would also be interesting to compare to other methods using other datasets, even not specific to robotics.\n* While some results are mentioned in the real world robot setting, there seems to have been only two test cases for evaluating the method for real robot tasks.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "dB20VWGE1KP", "reviewer": "Reviewer_ySeC", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents STOW, a dataset and model architecture for tracking unseen objects over time. The dataset consists of scenes from industrial and household scenarios. The model architecture is transformer based image model. Experimental results show improvement over baselines.", "summary_of_recommendation": "I think the paper is well written and has some novel model architecture contributions. However, I'm unable to see a difference between STOW and a low-frame rate version of VIS. If there are more differences between the datasets than frame rate, the authors should better explain the difference. At the very least, I think VIS deserves a paragraph in related work. Otherwise, the should include downsampled versions of VIS datasets in their model comparisons.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "What are the differences between this dataset and VIS?\nWhy wasn't this model architecture benchmarked on existing VIS datasets?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is well-written and easy to understand. The authors do a good job of providing background information on different datasets and problems related to instance segmentation. The method is well-described and includes some novel components, such as the multi-frame attention.\n\nHowever, I'm having trouble identifying a significant difference between this problem setting and video instance segmentation (VIS), which serves as a popular benchmark for segmenting videos and comes with a few public datasets (YouTube-VIS 19, YouTube-VIS 21, OVIS). The introduction says \"... these methods are mainly designed for videos with continuous frames. As a result, they display subpar performance when faced with significant object movements between frames, a prominent challenge in our task.\" Based on this, my understanding is that the dataset used in this paper is a low frame rate version of the VIS task. This observation seems to be supported by the fact that all the baseline comparisons are made against VIS models.\n\nIf the only difference between this dataset and VIS is the frame rate, then I believe this model should be compared to downsampled VIS datasets, which are comparatively much larger and more diverse. By evaluating the proposed model on larger datasets, it would better support the claim of its superiority at lower frame rates. If the proposed model architecture does perform better under lower frame rates, it would be an important finding that could be strengthened with evaluations on larger datasets.", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}]

Author rebuttal:

Meta-review: **Strengths**:

- The paper is well-written and easy to understand.

- The architecture introduced in the paper is interesting and novel, e.g. the multi-frame attention, and makes sense for this segmentation and tracking task.

- A good set of experiments and ablations that demonstrate the performance of the method.

- The synthetic + real dataset introduced seem useful for these sorts of industrial warehouse tasks.

- Real robot experiments using the method.

**Weaknesses**:

- Results are only reported on the datasets constructed for this task. It would also be interesting to compare to other methods using other datasets, even not specific to robotics.

- Lack of a section on limitation discussion.


Post-rebuttal: The authors have made great responses to the concerns from the reviewers. Additional results also clarify the difference between STOW and a low-frame rate version of VIS. The authors are encouraged to publish the code and dataset with the final version. Also please revise the paper in line with the rebuttal discussion for the camera-ready submission.
Decision: Accept (Poster)