Learning Segmentation from Point Trajectories
Authors: Laurynas Karazija
Iro Laina
Christian Rupprecht
Andrea Vedaldi
Venue: NeurIPS 2024 spotlight (2024)
Keywords: unsupervised segmentation
motion segmentation
point tracking
Primary area: machine_vision

Abstract: We consider the problem of segmenting objects in videos based on their motion and no other forms of supervision. Prior work has often approached this problem by using the principle of common fate, namely the fact that the motion of points that belong to the same object is strongly correlated. However, most authors have only considered instantaneous motion from optical flow. In this work, we present a way to train a segmentation network using long-term point trajectories as a supervisory signal to complement optical flow. The key difficulty is that long-term motion, unlike instantaneous motion, is difficult to model -- any parametric approximation is unlikely to capture complex motion patterns over long periods of time. We instead draw inspiration from subspace clustering approaches, proposing a loss function that seeks to group the trajectories into low-rank matrices where the motion of object points can be approximately explained as a linear combination of other point tracks. Our method outperforms the prior art on motion-based segmentation, which shows the utility of long-term motion and the effectiveness of our formulation.

Learning segmentation from point trajectories
Laurynas Karazija, Iro Laina, Christian Rupprecht, Andrea Vedaldi
Visual Geometry Group
University of Oxford Oxford, UK
(laurynas iroo
, chrisr, (orodaald@@rrbootta ac
Abstract
We consider the problem of segmenting objects in videos based on their motion and no other forms supervision. Prior work has often approached this problem by using the principle of common fate, namely the fact that the motion of points that belong to the same object strongly correlated. However, most authors have only considered instantaneous motion from optical flow. In this work, present way to train a segmentation network using long-tew tong-term point trajectories as a supervisory signal to complement optical flow. The key difficulty is that long-tern motion, unlike instantancous motion, is difficult to model - any parametric approximation is unlikely capture complex motion patterns over long periods of time. We instead draw inspiration from subspace clustering approaches, proposing a loss function that seeks group the trajectories into low-rank matrices where the motion of object points can be approximately explained as a linear combination of other point motion-based segmentation, which
tracks. Our method outperforms the prior shows the utility of long-term motion and the effectiveness of our formulation.
Introduction
Segmentation, the task of delineating and isolating distinet objects, is a fundamental problem in computer vision. Much of the current approaches are supervised, relying on expensive manual annotations. Attempts to approach this task without supervision have largely relied on manual heuristics or exploited the rich semantics of self-supervised feature extractors. Video data, however, offers an additional option as it contains motion, which can be exploited for an additional inductive bias. Such approaches which posits that elements that move together are more likely to belong together.
rooted in the principle of common fate from Gestalt psychology [66],
Motion information is usually captured by optical flow. Flow is attractive as it arises from low- level visual properties and can provide signal before scenes are parsed and objects are discovered. Furthermore, optical flow estimators, such as RAFT [60] or FlowFormer [24], can be trained purely on synthetic artificial data, transferring to real-worke scenes with remarkable accuracy and without manual annotation. This has led many consider optical flow critical modality to discover and learn objects from video data by learning to attribute and explain the motions of objects.
Optical flow, however, only describes the instantancous motion of the scene, which can create blindspots: not all objects are necessarily in motion at all times. Similarly, groups of objects might coincidentally move together. Recent advances in point tracking [14, 15, 22, 28] offer an alternative form of motion information. Point trackers "lock on" to set of query points and describe their position and visibility over the course of the whole video. This provides long-term motion information. Like optical flow estimators, point trackers are trained on synthetic data. However, unlike optical flow, point trajectories describe only sparse set of points.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

In this paper, we ask whether the long-term motion information obtained in point trajectories is beneficial. To that end, we explore how to supervise image segmentation networks using motion information with point trajectories. At a glance, this presents several problems. Firstly, point trajectories are time-varying 2D point clouds, and combining them with image-based networks not straightforward. Furthermore, the evolution of long-term object motion is too complex, even in the simplest cases. Our main insight is that the motion of points belonging to the same object should be well correlated. We thus propose a loss function that encodes this intuition by seeking to explain groups of points as combinations of other points in the group. With our method, a segmentation network predicts objects in the scene, inducing grouping of trajectories that are eurrently visible. The loss function then assesses how well such a grouping explains the long-tern motion. While point trajectories describe motion over longer time, they are limited by the number of points tracked, which is often much less than the number of pixels. We thus propose to train using both trajectory-based loss and optical flow-based loss and show that spatially sparse but longer-time motion information synergises with spatially dense optical flow.
Discovering objects using point trajectories has a long history in computer vision. Our approach is inspired by ideas of subspace clustering, which assume that data comes from distinct subspaces and seek to reconstruct membership information of data points. This has previously been applied to the problem of motion segmentation [17, 38]. These approaches, however, are sensitive to noise and either rely on specialised optimisation procedures to recover graph of trajectory relationships [32, 48] or use manual instead [32. 48]. Normalised cuts or spectral clustering are then used to group the trajectories. However, the need for affinity matrix limits the number of trajectories that can be used due to quadratic memory requirements. Furthermore, "densification" is still required to extend trajectory clusters to the whole image. By construction, these approaches can process only single sequence at a time. Our proposal instead trains an image segmentation network directly end-to-end using a dataset of videos while supporting a large number of trajectories.
In summary, our work makes the following contributions. (1) We propose a loss function that enables training any image segmentation architecture using point trajectories as a source of supervision. (2) We investigate our proposed loss in a principled way in a simulated setting, showing the feasibility of approach. (3) We apply such loss in per-sequence optimisation, outperforming other subspace clustering baselines. (4) We use our loss to train a single network on a dataset of vidcos for the task of video object segmentation, demonstrating strong results. (5) We show how our proposed loss formulation obtains better performance than alternatives.
2 Related work
Unsupervised video object segmentation. Video object segmentation (VOS) aims to label pixels of objects in video. Current VOS benchmarks [35, 51, 53] usually define the problem as binary separation or salient object segmentation. The task is usually approached in two ways: cuni-supervised and unsupervised VOS. cemi-supervised methods require initial frame annotations and aim to propagate them to the rest of the video [7]. Unsupervised VOS aims to discover object(s) of interest without the initial targets [18, 25, 36, 42, 52, 61]. This however does not differentiate methods based on data used to train them. Most of the traditional research in semi- or unsupervised VOS relies on annotations during training. Our approach, in contrast, does not rely on any manual annotations to learn Some authors explore related unsupervised video instance segmentation [65] task without any annotations, object-centric learning appraoches [2, 58, 73], some of which make use of flow [29] and depth [57].
Motion segmentation. A closely related task video object segmentation motion segmentation, which aims to extract the main moving objects in a video. The practical difference between these two tasks more difficult to delineate as the same benchmark datasets are often used. Early works modeled the scenes as layers [8, 27]. which later works accomplish using slot-attention mechanism [13, 34, 69]. Flow mixture models accounted for multiple motion patterns [26, 62], and corrections were introduced for rotating cameras [3, 4]. Later works [10, 46, 47] considered parametric flow models fit to explain the scene. AMD [40] employs single model with separate appearance and motion 'pathways'. Other works train flow-only models by generating synthetic data, which generalise well to real videos [33, 67]. An alternative line of work adopts a more generative approach, training an inpainter networks to predict optical flow [70, 71]. Several authors [37, 59]
2

---

adopt a multi-stage self-labelling [65] approach for motion segmentation: initial masks are estimated using an optical flow-based approach, followed by DINO-based refinement and CRF post-processing to generate pseudo-labels and train a final segmentation network.
Trajectory-based motion segmentation. Trajectory-based motion segmentation has also been explored. Older works consider data of multiple trajectories and employ non-negative matrix factoriza- tion and related decomposition methods [9, 11, 17, 19, 55, 68]. This line of work primarily operates by defining affinity between pairwise trajectories in single video setting. In [6, 30, 31, 49, 50], heuristic graphs are constructed between trajectories, considering increasingly complex motion models, and employing specialised solvers to solve the optimisation problem. However, due to the specialised optimisation procedures and tight coupling with trajectory estimation methods, this line of work has received less attention than deep methods that exploit optical flow similarly to RGB frames.
Subspace clustering. A specific kind of trajectory-based technique is subspace clustering ap- proaches, which rely on the self-expressive property of the data. They can largely be summarised [21] as solving constrained optimisation problem arg mine | D11 + X0(C') for some dataset D Ædx= of n points in d dimensions. C is matrix of coefficients, which expresses the data and can be represented as a linear combination of other points. Given a solution for the coefficient matrix, it is transformed into an affinity matrix for spectral clustering. The approaches mainly differ in the second term of the objective and specialized methods to solve the optimisation problem. SSC [17] define ((C)) as I norm. LLR [38] use nuclear norm instead, while LSR [41] uses instead l2 regularisation. [41, 43, 64] combines 1, 12, and nuclear norms. Under some strong assumptions [2 l.. these approaches enjoy some theoretical guarantees. However, they are difficult to scale in practice as the number of points n grows, as C' is n X n. Additionally, the secondary step of spectral clustering also limiting and difficult to tune. Instead, we take inspiration from these approaches and propose a way to supervise the network directly using the self-expressive property of point trajectories.
3 Method
Our goal is to solve the video segmentation task in an unsupervised manner: given a video, want to segment out the objects that are moving independently within it. A video is sequence of frames I, E R#W x3 each of which is an RGB image defined on the lattice Ω = vec(f1..... H) f1,..., w1) R #W*1, To segment the objects, self-supervise neural network that takes as input each frame I, in turn, and outputs a corresponding segmentation mask (Zz,) = M, E [0, H x where K is the number of possible segments we expect observe in the video. Segmentation matrix entries softly assign each pixel to one of K possible segments.
The challenge is how to supervise the network & without labels, utilising only the video itself as training material. The key inductive principle that we propose to use is that physical points that belong to the same object tend to have highly correlated motion, often called principle of common fate. When these points are projected to pixels, they result in corresponding highly correlated apparent motions, which we can measure using techniques like optical flow and point tracking. Therefore, we propose to supervise the network & from an analysis of apparent motion extracted automatically from the video using off-the-shell components.
Motion can be measured at two temporal scales. Optical flow extracts instantaneous motion, mea- suring the 2D velocity of the 3D points found at each pixel in each video frame. Point tracking extracts long-term motion, estimating the 2D location of a certain number of 3D points throughout the video's 's duration. These two sources of information are complementary. Optical flow is dense, easy to extract, and easy to model to discover correlations within it; however, by considering different times in isolation, it ignores most of the correlations that exist in the data. Tracks are sparse, more difficult to extract and harder to model, but potentially contain information ignored by optical flow.
Prior works such as [10] have studied how to model optical flow for segmentation. Here, motivated by a new generation of high-quality point trackers [14, 15, 22, 28], we aim at developing the machinery necessary to use track information as well. From this analysis, we construct losses which assess the quality of the predicted mask M, given the video itself. Next, we introduce two such losses, one for optical flow from prior work, and a new one based on point tracking.
3

---

3.1 Learning from optical flow
First, we describe the case of optical flow. Because optical flow instantaneous, we can fix our attention on specific frame I and corresponding mask M, dropping for now the time index t. The optical flow F "2xx for this image associates a 2-dimensional flow vector to each of the H W pixels. Each flow vector can be understood as the velocity of the pixel.
Let M₁ E TR x1 be the binary matrix for segment k; obtained by extracting the k-th column of M. Let F2 = M& @ denote the Hadamard (element -wise) product between the mask and flow vectors, broadcasting the mask along the rows.
Assuming that the object is rigid, the optical flow can be approximated as a linear parametric model of 2D coordinate embeddings (see [1] for overview). Following [10], consider -ixeeimensional quadratic embedding kernel emb( [z. y)) = [x, x², y. y². zy, 1] E R for pixel coordinates [z, y] E Ω and associate to each region & corresponding set of 12 parameters 0g E R Optical flow vectors within a region should be expressible as a linear combination of these six basis functions.
We then consider all pixels embeddings stacked in single matrix F2- Ms @ emb(S2) where the product with the soft mask ensures that the embeddings are "active" only the corresponding pixels are. The optical flow vectors in the region are then approximated as
F& ~ FA = E2Ox where
02 = (E) Ex) Fk.
(I)
where 0g is obtained via least square. We can use the residual of this approximation as a measure of how well the mask My fits the data:
(y(M = - -
Fg
Intuitively, this considers the correlation of pixel motion in the spatial sense: how pixel coordinates determine its motion based motion parameters 6μ.
3.2 Learning from trajectories
Having covered optical flow, move now developing analogous loss for tracking. We write P E R2TxN for the track matrix, with one trajectory per column. With slight abuse of notation, we write (P); E E2xN for indexing rows corresponding point locations at some time t. To connect pixel-wise masks and sparse points, we use a sampling operation (-)), writing T (Mc, (P).) Mr e [0, 1] for mask values at point locations at an appropriate time. Furthermore, we denote by
PA P Mr the masked version of the trajectory matrix, selecting the columnsstrajectories that belong to object & with obvious broadcasting of the mask values.
Unlike optical flow, trajectories are too complex to be modelled using a small set of fixed basis functions. Instead, we posit that the set of trajectories should be low-rank trajectories belonging to the same object should be explained well by linear combination of some small number of trajectories. We illustrate this intuition in Fig. using a 2D example.
This assumption results in a factorization of Ps using singular value decomposition (SVD) as Ps = U where (U,, Ex, V₂²) = SVD(P:). As P& should be low-rank, we can thus form an approximation using truncated SVD, by considering only first r components. We write [Uk]r to denote such truncation. With this, we obtain the loss
Croatty (M(P) = Σ P - [x),
Since truncated SVD offers optimal decomposition for the error above, lowering this loss amounts to making PA as close as possible to rank r, i.e., by grouping trajectories into Ps that do not increase its rank, and should come from rigid objects.
As we show in Section 5.3, we found an alternative formulation of this idea works better. Note the rank r matrix has the r-th and all later singular values as O. We can optimise singular values higher than r-th to be close 0 (ignoring Ug and Vt). Thus, for trajectories, we formulate loss simply as: min(27.N)
(M Σ Σ
o (Pk))
where (.(Pk) is the i-th singular value of PR. We assume < min (2n,

---

1
11=3
62
01
Pt
P4
02
12
2t Malt cably 4elt
Figure 1: Illustrative 2D example for the low-rank nature of Pk. A triangle undergoes rigid rotation over three frames. As the rate of rotation not constant, the flow vectors and point positions are difficult to model. However, the point p is part of the triangle and can be expressed as a combination of the three vertices an appropriate time. Thus, the last column of Ps. linearly dependent, and Ps is rank deficient. Any points in the triangle could be included in Py without increasing its rank.
Meaning of decomposition. We show that under certain simplifying assumptions, the decomposi- tion in (3) is exact and models time-varying camera motion and object geometry as two terms. We consider a simple case of a rigid body motion observed through perspective camera. For points on the object, can consider only the relative motion between the camera and the object and attribute it all to the camera for simplicity.
Given (stacked) camera projection matrices W2 E 3T points X3 E R R in homogenous coordinates that remain at constant projective depth d RN from the camera the whole sequence, we note the following equation [23]]
Pr W,Xx diag(d))
(5)
where P& R is Ps homogenous coordinates. Both W. and X₃ diag(d) can be recovered by considering a truncated SVD rank W, |U>4||Ea and Xg diag(d) [
The trajectory matrix factorises into the time-varying camera matrices and object geometry. As the depth not constant in the real-world setting, this decomposition is approximate and suggests the following alternative loss:
Spr ΣIP. W,Xx diag -d)
where W., and X₂ diag(d) -1 are obtained via SVD above.
Choice of r. Setting r correctly is important. Intuitively, it captures the degrees of freedom present in the trajectory data the number of trajectories that sufficient to form basis. From the analysis above, we saw that rank r = 4 corresponds to assuming constant depth and perspective camera. However, higher is needed tolerate changing depth and tracking errors [12, 23]. Similarly, all motion is rigid in real-world videos, which also requires increasing r. We empirically determined = 5 to yield good results.
3.3 Training segmenter using flow and trajectories
The losses above require optical flow trajectories P, and masks M, obtained using segmentation network & (Z) = M.. This suggests simple procedure of training a segmentation network given dataset of videos, which we summarise in Fig. 2. We precalculate optical flow for each frame and obtain a set of point trajectories for each video using off-the-shelf pretrained networks. For training. we consider triples of (I, F, P); for each frame i, where for trajectories P. we take trajectories for which the points are visible in the image Z. This can be accomplished by making use of visibility output point trackers or calculating trajectories by querying points in each frame. We use bilinear sampling for T(-) to obtain mask values at trajectory coordinates.
predictions
Temporal smoothing.
We include a temporal smoothing loss, which matches mask predictions between two frames offset by At using the predicted trajectories:
C = ( (I,), (Pt):) - (I (P:)
where I, is the 1-th frame and are trajectories associated with 1-th frame. We write the final loss as: L = X + XeC, + Cr, where 'y de, X, balance the contribution of the different loss terms.
5

---

Linear Parametric Model
Estimator
Optical Flow
Segmentation Network
Region
Point
Tracker
Part Tejjetor
Trajectory
Figure 2: Overview of our approach. We self-supervise segmentation network, i.e., without access to mask annotations, using both short-term motion information (optical flow) and long-term motion (point trajectories). We design a loss function that encourages the segmentation network to cluster regions where trajectories form low-rank-r groups, which should align well with objects. Off-the-she methods are used to estimate optical flow and point trajectories given dataset of videos.
Choice of k. Following prior work [10], we set number of predicted masks, to be higher than the maximum number of objects in the scene account for potential parallax and non-rigid motion. In the binary segmentation case, we recover two components by considering the average appearance feature of each component and solving for the normalised cut on graph with k nodes.
Feasibility study
Our proposed trajectory loss (4) enables training a segmentation network using trajectory data. We first show the feasibility of the proposed cost function in controlled setting, without actually training Φ. To this end, we consider a synthetic scene from the MOVI-F Kubric [20] dataset for which we obtain ground-truth trajectories for every point and ground-truth object segmentation masks. We explore the loss landscape of the proposed formulation by corrupting the segmentation masks along several principled axes and studying the effect of such corruptions on the trajectory loss.
First, we consider a random alteration of mask pixels, which we refer to as mask noise. We control the amount of mask noise using 7) such that 0.0 corresponds no pixels changed and 1.0 corresponds to completely random masks. Along this axis, we test whether our loss favours predictions with lower noise. Second, we consider structural alterations, namely under/over seder-segmentttion, we merge object masks with the background at random. To simulate over- segmentation, we randomly split the existing object mask into two parts in the middle along either the y-axis. We represent this type of mask corruption using integers. Negative values indicate the number objects removed, while positive values correspond new objects generated from existing ones. Such structural corruption investigates whether the loss can correctly identify the number of moving objects. Finally, we consider the "softness" of the predicted masks by transforming masks into logits and increasing the temperature in the softmax operation. This tests whether the loss will prefer low-entropy values. We leave further details of the corruption procedure to Appendix D.
mnder/over-segmeetatioo segmentation. To simulate
The results of these analyses are shown in Figure 3. All three plots show the loss value as function of structural corruption. The trajectory loss decreases as the noise and temperature of the masks are reduced, as seen in the first two plots. The third plot also shows that such solutions are preferred in combination. Furthermore, we observe that the loss values are lower when the correct number of segments is detected, and this holds even in the presence of noise or when masks are more uniform. Note, however, that segeeneggettttionn penalised less than under-segmentattin, i.e., missing moving objects leads to higher value of the loss than, c.g., splitting object into several components.

---

60
60
Maxe rnder//oe segronsbetion
Xlak mrder Joory angreenation
Figure 3: Feasibility analysis of Lt. Using synthetic sequence (left), we vary the amount of noise n injected into the mask, the temperature т of the mask logits and plot the loss value as function of the mask under/over segmentation. The plots show that the loss is reduced in low-noise, low-entropy settings and penalises both over- and seddentsgmmnnn
Experiments
In this section, we evaluate our approach for unsupervised motion segmentation and compare it with simple baselines and prior subspace clustering methods. Next, we compare our method with state-of-the-art methods for unsupervised video object segmentation across several datasets in binary segmentation setting. We finish with ablation experiments of our approach.
Datasets. We consider four primary datasets in this study. We use the synthetic MOVi-F variant of the Kubric [20] dataset with ground truth trajectories for comparison with subspace clustering- approaches. We adopt this setting to eliminate noise in point trajectories as previous methods are sensitive to We report the adjusted Rand index (ARI) as the main metric, measuring how close clustering is to the ground truth up to the permutation of cluster identities, where I is perfect match, and 0 means roughly random assignment. We also report FG-ARI, i.e., ARI only on foreground pixels (determined by ground truth masks), which identifies how well different objects are separated.
We also evaluate our approach on real- world datasets: DAVIS 2016 [53]. SegTrackv2 (STv2) [35], and FBMS [51], which are popular benchmarks for video object segmentation. Following standard practice [69, 70], foreground objects STv2 and FBMS are consolidated. We report the Jaccard (T) score, computed using Hungarian matching between predicted and ground truth segmentations.
Implementation. For the experiments on real-world datasets, optical flow is estimated using RAFT [60] and point trajecto- ries using CoTracker [28]. Trajectories are computed within a context window f 20 around each frame, with reflec- tion padding around video boundaries, resulting chunks of 2f + = 41 frames. To reduce the effect of noisy predic- tions, we also filter trajectories along the time dimension using an average filter with window size of 11. For the experiments on MOVi-F, small U-Net [56] is trained as the segmentation network, starting from random initialisation. For faimess of comparisons on DAVIS, STv2 and FBMS, we use the same architecture as in [10] - MaskFormer with DINO backbone. We specify further details in Appendix E.
Table I: Comparison of our LRTL trajectory- based formulation with prior methods.
MOVi-F
Method
ARIf
FG-ART
K-Means
15.26
42.53
SSC [17]
11.12
39.21
LRR [38]
7.47
37.36
LRTL (Ours)
46.07
65.76
5.1 Comparison to trajectory-based methods
In Table 1, we compare our low-rank trajectory Joss (LRTL) with prior subspace clustering approaches per-video optimisation setting. Subspace clustering operates similar intuition to our proposed trajectory loss by grouping of trajectories that should be linearly dependent. We also consider K- means clustering of trajectories as a simple baseline. For fair comparisons, we train our segmentation model optimising only the trajectory loss (C,). We use l = 25 components for each video and train for 5000 steps. This is comparable to the computation requirements and steps of other methods. For K-means, SSC [17] and LRR [38], search for optimal set of hyperparameters and the number of components reporting the best results. Our approach shows significantly stronger performance than simple K-Means and subspace clustering approaches.
7

---

Table Unsupervised video segmentation on DAVIS, SegTrackv2, and FBMS. Where possible,
we report results without widely applicable post-processing (e.g., CRF) or indicate results in grey.
Inf. Input
Input Resolution
Motion Est.
DAVIS
STv2
FBMS
Method
RGB
Motion
Method
J1
J
J
Single-sequence methods
FTS [52]
LDOF [5]
47.8
47.7
CUT [31]
LDOF [5]
55.2
54.3
57.2
DS [72]
240 426
RAFT [60]
79.1
72.1
71.8
Ponimatkin et al. [54]
X
480 848
ARFlow [39]
80.2
74.9
70.0
OCLR [67] (test ft.)
480 848
RAFT [60]
80.9
72.3
69.8
Single-stage end-to-end methods
OCLR [67]
RAFT [60]
67.6
65.4
DivA [34]
/
/
128 x 224
RAFT [60]
72.4
64.6
60.9
Meunier et al. [45]
X
/
128 224
RAFT [60]
73.2
55.0
GWM [10]
X
128 224
RAFT [60]
79.5
78.9
78.4
Multi-stage methods
RCF [37]
x
480 848
RAFT [60]
80.9
76.7
69.9
LOCATE [59]
848
ARFlow [39]
80.9
79.9
68.8
RAFT [60]
LRTL (Ours)
/
x 192 352
82.2
81.2
79.6
CoTracker [28]
Table 3: Alternative losses proposal. Other variants do not match the performance of our formulation.
Table 4: Ablation of loss terms. All loss terms synergise to improve performance.
Loss
DAVIS (J
Loss
DAVIS (J f)
freeta (3)
11.1
X AeLi del + AeL.
Leper (6)
18.2
71.9
Liootss (3)
14.6
81.7
trucks-as-flow
65.3
Jcly + ALC. + XrCT
82.2
Ours L. (4)
71.9
5.2 Unsupervised video object segmentation
We compare to recent methods on the unsupervised video object segmentation task without first-frame prompting post-processing. In this setting, we train a single network on the benchmark datasets for binary video segmentation. We compare with single-sequence methods that perform optimisation for each sequence/video individually. Additionally, we benchmark Kataset-wide single-stage end-to-end methods where training is performed over multiple videos simultaneously, training network in an end-to-end manner. We also compare with multi-stage methods that train and re-train several networks. We report our results on standard benchmarks in Table While the closest prior work relies on multiple stages of training, pseudo-labelling applying CRF, and retraining, our end-to-end trained method shows better performance at lower resolutions. We attribute this to the effectiveness of our approach in incorporating long-term motion information.
In Fig. 4, we show qualitative results of our approach and compare with RCF [37]. a state-of-the multi-stage approach. Our network trained with both flow and trajectory losses yields segmentations with noticeably better boundaries despite operating at lower resolution. Notably, our formulation also effectively avoids segmenting shadows and water ripples of the swan, which are difficult to separate based on instantaneous motion alone.
5.3 Ablations
Alternative losses. We have explored several alternative formulations of the trajectory loss in our approach and present the analysis Table Losses based on full SVD reconstruction fail to train network sufficiently. Sper performs the best out of these, likely DAVIS contains several scenes

---

ROBB
GT
Ours
RCF
Figure 4: Qualitative comparison of our results on DAVIS with RCF which uses higher resolution and multi-stage training. Our method contains slightly better boundaries, does not segment shadows and separates water ripples from the swan.
with panning camera tracking rigid object an approximately constant distance, which matches the assumptions. Increasing or decreasing the rank of the approximation performs worse. We also consider track-as-flow loss, where trajectories P are treated optical flow by subtracting positions from adjacent times. Then, for T frames, Eq. (2) can be applied. We find that such a formulation still trajectory- based formulation (Eq. (4)).
underperforms in comparison
We believe our formulation provides better results than the above for two possible reasons. First, by minimising higher-than-r singular values, we are not strictly enforcing assumptions like rigidity. Second, our loss formulation is more numerically stable as it requires only gradients w.r.t. to the singular values. As we seek to drive them close zero, the matrices Ps become increasingly ill-conditioned as the training progresses. Additionally, gradients w.r.t. U and V depend on inverse singular values ター1 [63]. which become numerically unstable they are approaching zero. On the other hand, Σ IN o (U P V) does not have this problem.
Influence of losses. In Table 4, we consider the method with only the flow loss component and only the trajectory loss component. We find that our vajectory-based loss improves flow-only performance. Using only trajectory-based loss shows weaker performance than just optical flow, likely due to only sparse set of points and noise introduced by estimating positions for occluded points. Ablating temporal smoothing loss slightly lowers performance as well.
Linitations. While we have demonstrated the effectiveness of learning segmentation from long- term motion, there is potential for further improvements in leveraging point trajectories. First, while modern trackers predict reasonable positions for occluded points, naturally, these predictions are less accurate. Thus, more explicit handling of occlusions and tracking noise would likely help. Second, we currently only use trajectory estimates from nearby frames for training. This means that sometimes track the same point multiple times, which could be avoided with caching trajectories. While we handle non-rigidity using over- wver-eegmenattion, extending this principle to video with multiple non-rigid objects is important feature direction.
Conclusion
We have introduced a principled method train an image segmentation network using long-term motion information expressed as point trajectories. Our trajectory loss formulation follows the principle of common fate and aims to group trajectories into low-rank matrices, representing the idea the motion of points belonging to the same object can be roughly explained as combination of other points. Using synthetic data we have shown that such loss should prefer low-noise and low-entropy solutions as well identify the correct number of moving objects. In comparison with other methods,

---

our loss formulation has shown superior performance compared to subspace clustering baselines on synthetic data and achieved state-of-the-an results on unsupervised video object segmentation benchmarks when combined with optical flow-hased loss.
Acknowledgements L. K. is supported by is supported by EPSRC AIMS CDT EP/S024050/1. L., C. R. and A. V. are supported by ERC-CoG UNION 101001212 and EPSRC VisualAI EP/T02852/11
References
[1] Gilad Adiv. Determining buree-dimensiona motion and structure from optical flow generated by several moving objects. IEEE transactions on pattern analysis and machine intelligence, (4):384-401, 1985. 4
[2] Görkay Aydemir, Weidi Xie, and Fatma Guney. Self-supervised object-centric learning for videos. Advances in Neural Information Processing Systems, 36, 2024.
[3] Pia Bideau and Erik Learned- Miller. It's moving! a probabilistic model for causal motion segmentation in moving camera videos. In European Conference on Computer Vision, pages 433 449, Springer, 2016. 2
[4] Pia Bideau, Aruni RoyChowdhury, Rakesh R Menon, and Erik Leaaned-Millrr The best of both worlds: Combining cnns and geometric constraints for hierarchical motion segmentation. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 508-517 2018.
[5] Thomas Brox and Jitendra Malik. Large displacement optical flow: descriptor matching in variational motion estimation. IEEE transactions on pattern analysis and machine intelligence, 33(3):500-513, 2010.
[6] Thomas Brox and Jitendra Malik. Object segmentation by long term analysis of point trajectories. In Computer Vision VisionEECVV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part V 11, pages 2822295.20 Springer, 2010.
[7] Sergi Caelles, Kevis- Kokiisi Maninis, Jordi Pont-Tused Tust,Tuset, Laura Leix- Daniel Cremers, and Luc Van Gool. One-shot video object segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 221-230, 2017. 2
[8] Jason Chang and John W. Fisher III. International Conference Computer Vision, pages 161-168, 2013.
layered tracking with latent flow, 2013 IEEE
[9] Anil M Cheriyadat and Richard J Radke. Non-negative matrix factorization of partial track data for motion segmentation. In 2009 ieee 12th international conference computer vision, pages 865-872. IEER, 2009.
[10] Subhabrata Choudhury, Laurynas Karazija, Iro Laina, Andrea Vedaldi, and Christian Rupprecht. Guess What Moves: Unsupervised Video and Image Segmentation by Anticipating Motion. In British Machine Vision Conference (BMVC), 2022. 2, 3, 4, 6, 7,8, 8, 14, 17, 18
[11] Joao Costeira and Takeo Kanade. A multi-body factorization method for motion analysis. In Proceedings of IEEE International Conference on Computer Vision, pages 1071-1076. IEEE, 1995. 3
[12] Joao Paulo Costcira and Takeo Kanade. A multibody factorization method for independently moving objects. International Journal of Computer Vision, 29:159-179, 1998. 5
[13] Shuangrui Ding, Weidi Xie, Yaho Chen, Rui Qian, Xiaopeng Zhang, Hongkai Xiong, and Qi Tian. Motion-inductive self-supervised object discovery in videos. arXiv preprint 2.Xi::2210.00211 2022. 2
[14] Carl Docrsch, Ankush Gupta, Larisa Markeeva, Adria Recasens Continente, Kucas Smaira, Yusuf Aytar, Joao Carreira, Andrew Zisserman, and Yi Yang. Tap-vid: A benchmark for tracking any point in video. In NeurlPS Datasets Track, 2022. 1,3
[15] Carl Doersch, Yi Yang, Mel Vecerik, Dilara Gokay, Ankush Gupta, Yusuf Aytar, Joao Carreira, and Andrew Zisserman. Tapir: Tracking any point with per-frame initialization and temporal refinement. arXiv preprint arXiv:2306.0800 006327 2023, 1, 3, 14, 16
[16] Carl Doersch, Yi Yang, Dilara Gokay, Pauline Luc, Skanda Koppula, Ankush Gupta, Joseph Heyward, Ross Goroshin, João Carreira, and Andrew Zisserman. Bootstap: Bootstrapped training for rrakkig-any--pin arXiv preprint arXivv2402.0084/ 00847, 2024. 14
[17] Ehsan Elhamifar and René Vidal. Sparse subspace clustering: Algorithm, theory, and applications. IEEE transactions pottern analysis and machine intelligence, 35(11):2765-778 2013. 2, 3, 18
10

---

[18] Alon Faktor and Michal Irani. Video segmentation by non-local consensus voting, In Proceedings of the British Machine Vision Conference. BMVA Press, 2014.
[19] Matthieu Fradet, Philippe Robert, and Patrick Pérez. Clustering point trajectories with various life-spans. In 2009 Conference for Visual Media Production, pages 7-14. IEEE, 2009.
[20] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo, Charles Hemmann, et Kubric: A scalable dataset generator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3749-3761
2022. 6, 7, 16, 17, 18
[21] Benjamin David Hacffele, Chong You, and Rene Vidal. A critique of self-expressive deep subspace clustering. In International Conference on Learning Representations, 2020.
[22] Adam W Harley, Zhaoyuan Fang, and Katerina Fragkiadaki. Particle video revisited: Tracking through occlusions using point trajectories. In European Conference on Computer Vision, pages 59 75. Springer, 2022. 1, 3,3,1 16
[23] R. I. Hartley and A. Zisserman. Multiple View Geometry Computer Vision. Cambridge University Press, ISBN: 0521540518, second edition, 2004.
[24] Zhaoyang Huang. Xiaoyu Shi, Chao Zhang, Qiang Wang. Ka Chun Cheung, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Flowformer: A transformer architecture for optical flow. In European Conference on Computer Vision, pages 668-685. Springer, 2022.
[25] Suyoe Jain, Bo Xiong, and Kristen Grauman. Fusionseg: Learning to combine motion and appearance for fully automatic segmention of generic objects in videos. arXiv preprint arXiv:1701.03333 2017. 2
[26] Allan D. Jepson and Michael J. Black. Mixture models for optical flow computation. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 760-761, 1993. 2
[27] N. Jojic and B.J. Frey. Learning flexible sprites in video layers. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Puttern Recognition. CVPR 2001, volume pages I-I, 2001.
[28] Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: is better to track together. arXiv preprint arXiv:2307.07635, 2023. 1, 3, 7, 8, 14, 16, 17
[29] Laurynas Karazija, Subhabrata Choudhury, Iro Laina, Christian Rupprecht, and Andrea Vedaldi. Unsuper- vised multi-objec object segmentation by predicting probable motion patterns. Advances in Neural Information Processing Systems, 35:2128 2141,,11112111 2022. 2
[30] Margret Keuper. Higher-order minimum cost lifted multicuts for motion segmentation. In Proceedings of the IEEE International Conference Computer Vision, pages 4242-4250, 2017.
[31] Margret Keuper, Bjoern Andres, and Thomas Brox. Motion trajectory segmentation via minimum cost multicuts. In Proceedings of the IEEE international conference on computer vision, pages 3271-3279.
2015. 3,8 8
[32] Margret Keuper, Siyu Tang, Yu Zhongjie, Bjoern Andres, Thomas Brox, and Bernt Schiele. A multi-cut formulation for joint segmentation and tracking of multiple objects. arXiv preprint arXiv:1607.06317
2016. 2
[33] Hala Lamdouar, Weidi Xie, and Andrew Zisserman. Segmenting invisible moving objects. In BMVC, 2021.
[34] Dong Lao. Zhengyang Hu, Francesco Locatello, Yanchao Yang, and Stefano Soatto. Divided attention: Unsupervised multi-object discovery with contextually separated slots. arXiv preprint arX1v:2304.01430 2023.
[35] Fuxin Li, Tacyoung Kim, Ahmad Humayun, David Tsai, and James M. Rehg. Video segmentation by tracking many figure-ground ground segments. 2013 IEEE International Conference on Computer Vision, pages 2192-2199, 2013. 2,
[36] Siyang Li, Bryan Seybold, Alexey Vorobyov, Alireza Fathi, Qin Huang, and C-C Jay Kuo. Instance embedding transfer to unsupervised video object segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2018. 2
11

---

[37] Long Lian, Zhirong Wu, and Stella X Yu. Bootstrapping objectness from videos by relaxed common fate and visual grouping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14582-14591, 2023. 2, 15
[38] Guangean Liu, Zhouchen Lin, Shuicheng Yan, Ju Sun, Yong Yu, and Yi Ma. Robust recovery of subspace structures by low- www-rank representation. IEEE transactions on pattern analysis and machine intelligence, 35
(1):171-188) 1847711144,0 2012. 2,
[39] Liang Liu, Jiangning Zhang, Ruifei He. Yong Liu, Yabiao Wang. Ying Tai. Donghao Luo, Chengjie Wang. Jilin Li, and Feiyue Huang. Learning by analogy: Reliable supervision from transformations for unsupervised optical flow estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020. 8
[40] Runtao Liu, Zhirong Wu, Stella Yu, and Stephen Lin. The emergence of objectness: Learning zero-shot segmentation from videos. Advances in Neural Information Processing Systems, 34, 2021. 2
[41] Can-Yi Lu, Hai Min, Zhong- Qiu Zhao, Lin Zhu, De-Shuang Huang, and Shuicheng Yan. Robust and elficient subspace segmentation via least squares regression. In Computer VCsi 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part VII 12, pages 347-360, Springer, 2012. 3, 18
[42] Xiankai Lu, Wenguan Wang. Chao Ma. Jianbing Shen, Ling Shao, and Fatih Porikli. See more, know more: Unsupervised video object segmentation with co-attention siamese networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 3623-3633 2019. 2
[43] Dijun Luo, Feiping Nie, Chris Ding, and Heng Huang. Multi-subspace representation and discovery. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2011, Athens, Greece, September 5-9, 2011, Proceedings, Part II 22, pages 405 420. Springer, 2011. 3
[44] Nikolaus Mayer, Eddy 1lg. Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4040-4048, 2016. 17
[45] Etienne Meunier and Patrick Bouthemy. Unsupervised space-time network for temporally-consisn segmentation of multiple motions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22139 22142221181111 June 2023, 8
[46] Etienne Meunier and Patrick Bouthemy. Unsupervised motion segmentation in one go: Smooth long-term model video. arKiv preprint 23X00:2310.01040, 2023. 2
[47] Etienne Meunier, Anais Badoual, and Patrick Bouthemy. Em-driven unsupervised leaminng for efficient motion segmentation. CoRR, abs/2201.00 002074, 2022.
[48] Peter Ochs and Thomas Brox. Object segmentation video: a hierarchical variational approach for turning point trajectories into dense regions. In 2011 international conference on computer vision, pages 1583-1590. IEEE, 2011. 2
[49] Peter Ochs and Thomas Brox Higher order motion models and spectral clustering. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 614-621. IEEE, 2012. 3
[50] Peter Ochs, Jitendra Malik, and Thomas Brox. Segmentation of moving objects long term video analysis. IEFE transactions on pattern analysis and machine intelligence, 36(6): 187- 2013. 3
[51] Peter Ochs, Jitendra Malik, and Thomas Brox. Segmentation of moving objects by long term vidco analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36: 187--200, 2014. 2,7
[52] Anestis Papazoglou and Vittorio Ferrari. Fast object segmentation in unconstrained video. In Proceedings of the IEEE International Conference Computer Vision (ICCV), December 2013. 8,8
[53] F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine-Horuung A benchmark dataset and evaluation methodology for video object segmentation. In Computer Vision and Pattern Recognition, 2016. 2, 7
[54] Georgy Ponimatkin, Nermin Samet, Yang Xiao, Yuming Du Renaud Marlet, and Vincent Lepetit. A simple and powerful global optimization for unsupervised video object segmentation. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 1992-5903, 2023. 8
12

---

[55] Shankar R Rao, Roberto Tron, René Vidal, and Yi Ma. Motion segmentation via robust subspace separation in the presence of outlying, incomplete, or comupted trajectories. In 2008 IEEE conference on computer vision and puttern recognition, pages 1-8. IEEE, 2008.
[56] Olaf Ronneherger, Philipp Fischer, and Thomas Brox, U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assiiteed intervention, pages 234-241. Springer, 2015. 7.
[57] Sadra Safadoust and Fatma Güney. Multi-ohject discovery by low-dimensional object motion. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pages 734-744, 2023. 2
[58] Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple unsupervised object-centric learning for complex and naturalistic videos. Advances in Neural Information Processing Systems, 35:18181 18196, 2022.
[59] Silky Singh, Shripad Deshmukh, Mausoom Sarkar, and Balaji Krishnamurthy. Locate: Self-supervised object discovery via flow-guided graph-cut and bootstrapped self-training. In British Machine Vision Conference (BMVC), 2023. 2,8
[60] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In European Conference on Computer Vision (ECCV), 2020. 1, 7. 8, 17
[61] Pavel Tokmakov, Cordelia Schmid, and Karteek Alahari. Learning to segment moving objects. Int. J. Comput. Vision, 127(3))282-301, mar 2019. ISSN 0920-5691.
[62] Philip H. S. Tom. Geometric motion segmentation and model selection. Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences, 356:1321 1340, 1998. 2
[63] James Townsend. Differentiating the singular value decomposition. 2016. URL https://j-toons: github. httpapers//aaaaeeoooppte svd dee-deeiiittt derivative. pdf. 9
[64] René Vidal and Paolo Favaro. Low rank subspace clustering (lrsc). Pattern Recognition Letters, 43:47-61. 2014. 3
[65] Xudong Wang, Ishan Misra, Ziyun Zeng, Rohit Girdhar, and Trevor Darrell. Videocutler: Surprisingly simple unsupervised video instance segmentation. arXiv preprint arXiv:2308. 14710, 2023. 2,3, 15
[66] Max Wertheimer. Experimentelle studien uber das sehen von bewegung. Zeitschrift fur psychologie, 61: 161-165, 1912.
[67] Jun Xie, Weidi Xic, and Andrew Zisserman. Segmenting moving objects via an object-centric layered representation. ArXiv, abs/2207 002006, 2022. 2, 8
[68] Jingyu Yan and Mare Pollefeys. A general framework for motion segmentation: Independent, articulated. rigid, non-rigid, degenerate and non degedegeeer In Computer Vision-ECCV 2006: 9th European Confer- ence on Computer Vision, Graz, Austria, May 7-13, 2006, Proceedingy. Part IV 9, pages 90-106. Springer. 2006. 3
[69] Charig Yang. Hala Lamdouar, Erika Lu, Andrew Zisserman, and Weidi Xie. Self-supervised video object segmentation by motion grouping. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7177-7188, 2021. 2, 7, 17
[70] Yanchao Yang, Antonio Loquercio, Davide Scaramuzza, and Stefano Soatto. Unsupervised moving object detection via contextual information separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019. 2,
[71] Yanchao Yang, Brian Lai, and Stefano Soatto. Dystab: Unsupervised object segmentation via dynamic- static bootstrapping. In Proceedings of the IELE/CVF Conference Computer Vision and Pattern Recognition, pages 2826-2836, 2021. 2
[72] Vickie Yc, Zhengqi Li, Richard Tucker, Angjoo Kanazawa, and Noah Snavely. Deformable sprites for unsupervised video decomposition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2657-2666, June 2022.
[73] Andrii Zadaianchuk, Maximilian Seitzer, and Georg Martius. Object-centric learning for real-world videos by predieting temporal feature similarities. Advances in Neural Information Processing Systems, 36, 2024.
2, 15
[74] Yang Zheng, Adam W. Harley, Bokui Shen, Gordon Wetzstein, and Leonidas J. Guibas. Pointodyssey: A large-scale synthetic dataset for long- term point tracking. In ICCV, 2023. 14
13

---

Table 5: Influence of the rank of the trajectory matrix used in loss function (4).
Table 6: Influence of k. the number of predicted components before merging.
r
DAVIS (J 1)
DAVIS
76.0
78.0
4
79.6
3
82.0
5
82.2
4
82.2
6
80.9
5
72.8
Table 7: Influence of context length of the tra- jectory matrix.
Table 8: Influence of trackers used to estimate point trajectories.
Context length
DAVIS (J 1)
Tracker
DAVIS (J f)
10
79.1
TAPIR [15]
73.4
15
81.0
PIPs++ [74]
74.9
20
82.2
BootsTap [16]
76.8
30
80.8
CoTracker [28]
78.9
Supplementary material
In this supplementary material, we consider additional ablations of our approach (Appendix B), include further results (Appendix C), and provide the implementation details (Appendix E). Accom- panying this supplementary material, we include videos of our results on DAVIS and SegTrackv2 datasets. We also include an example video visualising sample of trajectories that the model receives as input. The code and models will be released upon acceptance.
A Broader impact
Segmentation is component in very large and diverse spectrum of applications in healthcare, image processing, computer graphics, surveillance and more. As with many technologies, the application can be good bad. In this paper, explore how to train model to perform segmentation in unsupervised manner. This has the positive benefit of removing manual labour requirements to obtain annotations, which might also eventually apply to bad actors. We, however, consider the immediate real-world impact beyond the research community of our work here limited as unsupervised systems still show lower performance than supervised counterparts.
B Additional ablations
Rank of trajectory matrix. In Table 5, we vary the rank of the trajectory matrix used in the trajectory loss (Eq. (4)). As previously mentioned, the choice of rank reflects the degrees of freedom in the system and controls implicitly the assumptions about the types of motion and cameras used to capture sequences. At r = 3 and 4, we observe slight impact on the performance in comparison to 5. = 5 appears be the optimal setting, which is what used in our main experiments. At r = 6, the performance drops again, likely as it becomes sufficient to group and explain simple motions together.
Number of segments k. In Table vary k, the number of masks predicted by our method, before merging. As in prior work [10], the k = appears to be the optimal setting. The performance drops beyond this point it becomes difficult to group objects.
Influence of context window length T. In Table we vary the length of the context windows (f) and thus 7 for our method when considering trajectories. We find increasing the context window helps slightly. However, the performance starts to drop afterwards. We hypothesise that this due to difficulty predicting sensible point trajectories for points that move outside of the frame and become invisible, as DAVIS contains many videos where the camera tracks the main subject. Though several values of this setting are viable.
14

---

Table Alternative network architectures for segmentation.
Table 10: Comparison with appearance-only methods.
Network
DAVIS (J 1)
Method
DAVIS (J 1)
UNet
80.6
VidcoCutLER [65]
67.2
MaskFormer + Swin-Tiny
81.2
VideoSAUR [73]
17.5
MaskFormer DINO
82.2
LRTL (Ours)
82.2
Source of tracks. In Table 8, we experiment with different trackers to obtain tracks. We consider TAPIR', PIP++2 and BootsTap" along Co Due to the limitation of some options (PIPs++ not predicting visibility) and inherent noise in invisible tracks for TAPIR, we lowered the context window to 15. We also do not consider tracks from adjacent frames this seems to lower performance other trackers. Finally we did not use EMA in these experiments. We observe that CoTracker performs the best while other trackers show slightly weaker results. We hypothesise this is due to CoTracker estimating reasonable trajectories for occluded points, which are included in the matrix Ps. Some trackers, c.g., TAPIR, are restricted to predicting points within the frame, thus providing extremely noisy estimates in scenes where objects move outside the frame.
Alternative networks. As our proposed loss function is network- architk-ccc agnostic as it only requires mask prediction. Thus, any network which predicts masks or has mask-like representation could be used. In Table 9, we experiment with changing the segmenter architecture in the DAVIS benchmark. This shows that we can swap different network architectures with relative ease and obtain similar results.
Inference speed. Here provide the inference time comparison using different networks as average FPS during DAVIS evaluation. For MaskFormer DINO configuration, we measure 3.3 FPS, while with UNet we measure 6.4 FPS. Note that since our contribution is loss function, it is network architecture agnostic. Using it does not affect inference time; only the choice of network architecture does. We matched the architecture with prior work for the best comparisons.
Comparison with appearance-only works. Finally, we include comparison to unsupervised methods that consider only appearance during learning. In Table 10, we provide a comparison of VideoCutl .ER [65] and VideoSAUR [73] on DAVIS using the same merging strategy for combining multiple predictions to a binary segmentation as in our method.
Our method shows significant advantage. We observe that VideoCutLER has trouble segmenting instances from crowds in the background. VideoSAUR has imprecise object boundaries which severely impacts performance when measurred using Jaccard score.
C Additional results
C.1 Qualitative results on SegTracky2
In Fig. 5, we provide additional qualitative results from our approach on the SegTrackv2 dataset. We compare with the state-of-the-art multi-stage Relaxed Common Fate (RCF) approach [37]. Our method correctly identifies more parts of the objects and has better boundaries.
D Parametric mask alterations
In this section, we show the effect the parametric ground truth mask alterations used to study the trajectory loss in section 4.1. The purpose of these alterations is to disturb ground truth masks in
Coode and models available https: https::///tttttthg http://gtttuu.ooggggggeee soou/goooglee wtt.://oooleddeiinn///oo cense.
under Apache-2.0 li-
"odee models available https://githu: cww/ wwt./ooo/aaaaaeeeee com//haareyypippo under MIT license.
Code and models available https: gitb... com/ google- aeoepiind/tapne under Apache-2.0 li-
cense.
15

---

ROB
GT
Ours
RCF
Figure 5: Qualitative comparison of our results on SegTrackv2 with RCF which uses higher resolution and multi-stage training. Our method contains slightly better boundaries and segments more whole objects.
1) 0.0
17 0.25
1] 0.5
0.75
1 1.0
Figure 6: Example noise mask alteration. The parameter is the probability of assigning a mask pixel at random.
controlled way to enable studying the effect this has on the loss. For this purpose, we use synthetic data from MOVi-F sequences of the Kubric [20] dataset suite, which is the same data that is used to train CoTracker [28]. TAPIR [15], PIP [22] and similar. We consider three types of alterations:
The first kind of alteration is random noise. With probability n, set each mask pixel to random class sampled from (0, K), where K = 20 in this case. When O, thus, there is alteration. When 1 0.5, around half of the mask pixels (in expectation) are assigned randomly. Fig. 6 shows the effect of rj in practice.
The second kind of alteration consider is structural change meant to approximate http/unnde-semmmeattne
For ander-segmentation. we change the mask regions correspond- ing to the whole object the background. Fig. shows this in effect. For over-segmentation. we split an existing component randomly along an axis passing through the object centre and parallel to either the or y-axis at random. Fig. 9 shows this in effect. We parameterise this alteration with integers s, where a positive number controls the number of components split, and negative numbers correspond to the number components
the background.
The third kind of alteration is temperature. Its purpose is to model how the entropy of the categorical distribution modelled by the segmentation network might affect the loss. For this, we increase the temperature r in softmax operation softmad coftmax//// (1/+) for logits calculated from the input mask, which results in increasingly "soft" masks.
16

---

8 3
Figure 2 = =-15
Figure 8: Example mask structural mask alteration showing modification used to approximate under- segmentation. The parameter controls the number of objects set to the background.
8=1
2
8 3
Figure 9: Example structural mask alteration modelling over-segmentation. The parameter controls the number if objects split into two random.
The three types of alteration composed generate synthetic prediction mask that be used to investigate how the trajectory loss behaves as the mask changes. We use 25 trials to estimate the value of the loss for given configuration of the parameters 8, 8,7,, [, T.
E Implementation details
Here, we further specify the configuration and implementation details used in our experiments.
E.1
Extracting flow
We use RAFT [60]* to extract optical flow pretrained on FlyingThings3D [44]. We follow the methods used to extract flow in previous work [10, 69]. Namely, we consider pairs of frames with distance in time of either or 2, both in forward and backward directions for DAVIS and SegTrackv2. For FBMS, we consider distances of 3 and 6 due to lower motion setting in the dataset. The optical flow is extracted before training.
E.2 Extracting trajectories
We use CoTracker [28] to extract point trajectories. CoTracker is trained MOVi-F Kubric [20] datasets. We use CoTracker v2 We query at every 4th-pixel coordinate for each frame to extract point trajectories. At 480 854 resolution for DAVIS, this results in 25k points for each frame. When tracking, we find it beneficial to inject auxiliary query points. For this, we define two additional query grids with stride of 32, querying frame seven frames in the future and the past (or less if the video boundaries). This generates around 2k additional points, which we do not use for training. When processing videos of heterogeneous resolutions, resize the input 480 854 to maintain the same number of points.
'Code and models available httpa////wppp//.ooooooo////////a license.
under BSD-3
*Code and models available attp:s//ggthhbcoo/eeeeberrrreree tracker under non- comercial license.
17

---

E.3 Training hyperparameters
For the segmentation network, we use the same model architecture as in [10] MaskFormer with DINO backbone. We feed images at 192 x 352 resolution. We also use random horizontal flipping predict k 4 components, which, in the case of binary
augmentation. The network is trained segmentation, are then merged into two following [10]. We train using Adam optimiser, with learning rate of 1.5e-4, weight decay of 0.01, batch size of 8, and a linear learning warmup schedule for 1500 iterations. We train for 5000 iterations. 6 We use an Exponential Moving Average (EMA) with the decay power of 2/3 with a warmup of 1500 iterations and update every 10 steps to help stabilise the training. On SegTrackv2 we instead used decay power 4/5 as the dataset is considerably smaller than others. We set A = 0.03, A 5 10 5, and X, 0.1 all experiments, which yields loss values in similar numerical range. For the temporal smoothing loss, we use Al = 5.
E.4 MOVi-F experiments
When conducting experiments on the MOVi-F dataset (Sec. 4.2), we consider ground-truth trajectories obtained from modified rendering script [20]. We normalise the trajectories to the [0, 1] range based on image width and height.
For K-Means, we consider the trajectories with the initial position at T = 0 subtracted, thus clustering offsets from the initial position.
For SSC [17]. we translate the method to Python following the original implementation in Matlab' We use the ADMM variant, which found to give better results. We set the 100 and kept the rest of the hyperparameters unchanged. To transform the coefficient matrix into graph adjacency, found that simple symmetrisation yielded slightly better results than the proposed method that additionally normalised and filtered values. We report results for this method using the optimal number of clusters for spectral clustering.
For LRR [41], we similarly translate the method to Python following the original implementation in MatlabS. We use À 0.2. Additionally, we found it beneficial to reduce & 1..11 and use larger number of iterations (10k) than proposed. Similarly to SSC, we experimented with different ways to transform the coefficient matrix to adjacency, including automatically determining the number of clusters based on the block diagonal structure. We found, however, that using simpler symmetrisation with optimal numbers of clusters determined by oracle gave the best results.
When considering our trajectory loss, we parameterise the masks with a small randomly initialised Unet [56] predicting 25-way segmentation, which we optimize using Adam W optimizer.
Note that K-Means, SSC and LRR baselines cluster trajectories rather than segmenting the image. To map back to the image domain and obtain segmentation masks, we repeatedly apply the method for each frame within a sequence, considering the trajectory for each pixel. This enables the most direct way to establish segmentation of the images through significant additional computation effort. An alternative could be to consider sequence trae-trajcctories jointly; however, approaches like SSC and LLR do not scale well to such a large number of trajectories. For our trajectory loss, optimisation can be performed per sequence and, we show in our real-world experiments, datasct-wide.
'We estimate about 3 hours to train model using A6000 GPU (peak GPU memory 25GB). We estimate around 100 GPU hours to train models for the results here
7 "oode available bect:://ww... vision. jhu. edu/ code/
Cooee available https://sites: http://ttts.ogglll...ee.o www./ot//eeeeeeo.oooooooooooewee t guangcanliu/
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have confirmed the viability of our loss formulation in controlled simu- lated settings, per-sequence optimisation settings with tracking noise and in real-world settings. We also considered alternative formulations of the trajectories and found them to underperform.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See Section 5.3.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
19

---

Answer: [NA]
Justification: the paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We list all relevant details in Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
20

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We do not include code at the time of submission but commit to releasing the code and models later time.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to rum to reproduce the results. See the NeurIPS code and data submission guidelines (https: / /nips. wctpublic//aeeccccccccccccccecct hwwolccgguides//// /Coo//eoeeeeeeeee cumbbmisssinn ssiissolioiyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We include brief summary of key details Section and complete information in Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We do not include confidence intervals when reporting main experimental results due the computational burden of doing so. We report to intervals in our feasibility study.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We give the computation cost of a single experiment and estimate total GPU hours required train models for the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:/ //nerrip/../eeeeeeeeee/ee cttpcccccppcceeeeeecceccccccccct wtwehhhhhhiiieeennnieeeeeeee
Answer: [Yes]
Justification: We make use of publicly available and open-source code and models, respect- ing individual licenses.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix A.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
22

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: As our key proposal a method for learning segmentation using trajectory data, we do not foresee our models trained for benchmark datasets requiring safeguards as their use is limited due to the small dataset scale.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the models we build upon which are released on open licenses permitting such use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
23

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
24

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "fGqW9QxIQb", "reviewer": "Reviewer_8Nte", "summary": "The paper tackles video object segmentation by incorporating into the loss function not only instantaneous optical flow information but also long term pixel tracking information.  Raft was used for optical flow and CoTracker was used for long term pixel tracking in the experiments.  The experiments show a marginal improvement in performance when combining the two information sources in the loss function.", "questions": "1. I found the related works were like a laundry list.  You divided the categories into unsupervised video object segmentation, motion segmentation, trajectory based motion segmentation and subspace clustering.  That is find however your focus is only video object segmentation, why is that and how can you address the other problem areas?\n2.  I would imagine if we had 3d scene flow, by perhaps combining monocular depth and optical flow would result in good results without long term tracking?\n3.  why not incorporate appearance information as well? \n4.  Appearance information for segmentation in the examples would suffice, it would be interesting to focus on cases where appearance info is not sufficient for segmentation and we require motion information.", "limitations": "I am not sure that the examples actually illustrate that motion segmentation is necessary for these cases.  I would focus on cases where appearance information is not enough.\nCan this system deal with a moving camera or does the camera have to be static?\nHow well does the system work under occlusion?\nDifferent motions of the object of interest will results in different performance, perhaps diving into this analysis would be informative.\nBoth sources of info, optical flow and long term pixel tracking info are based on 2D info, the projection of 3D info.  This has limitations.  The paper should have explored different object movements.  It does state that non rigid objects when dealing with multiple objects is an issue however an in depth exploration for a single non rigid object would be informative.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper flows quite well, it addresses that video object segmentation is the problem space, the focus is on loss function, Figure 2, the layout appears clear as well.  There are a handful of datasets and comparing methods used in the experiments.", "weaknesses": "Table 2 where the experimental results are presented lists a collection of methods categorized into different groupings.  Perhaps these groupings and methods could be better discussed in the lit review, it appears that the categories in the lit review do not correlate nicely and I do not know the difference of these methods unless I look at the references and read the papers myself.\nThe improvement is incremental.  IT is expected that there would be some improvement however what cases do we actually get the improvement in, a bit of more depth in the analysis would make this a better paper.\nI assume that the camera is static?, correct? if not, perhaps making this clearer would help.\nI have no idea how long the long term point trajectories were, perhaps analyzing this would help.  Also depending on the trajectories, were there occlusions or other interesting factors that contribute to the loss function would be interesting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for constructive comments and suggestions. We reply to concerns below.\n\n> Table 2 where the experimental results are presented lists a collection of methods categorized into different groupings. Perhaps these groupings and methods could be better discussed in the lit review.\n\nWhile our literature review is structured based on related sub-fields, the methods in Tab. 2 are arranged based on the approach to the task: using single-video optimization (L95), training a network in multiple stages (L89) or single-stage training. For clarity, we will summarise the groupings in the respective section.\n\n> The improvement is incremental.\n\nWe disagree with this assertion. Firstly, we propose a principled loss to derive an objectness signal from a novel modality. Secondly, our proposal shows effective improvement across several benchmarks where improvements have been historically difficult. For example, after OCLR [66] has first reached a Jaccard score of 80.9 on the DAVIS benchmark, follow-up works [54,37,59] could only approach or match it for several years, showing the difficulty of the task.\n\n> I assume that the camera is static?, correct? if not, perhaps making this clearer would help. How well does the system work under occlusion? Different motions of the object of interest will results in different performance, perhaps diving into this analysis would be informative.\n\nThe camera does not need to be static (and in almost all scenes it is not); in fact, our loss explicitly models a time-varying camera (see L185). Similarly, we make no assumptions about occlusions or object motion.\n\nIn general, the method is robust: the included result videos  contain complex camera motion, \"zooming\" (e.g., dance videos), strong occlusions (e.g., bicycle or dog scenes), self-occlusions and non-rigid motion. Our method works well in quantitative studies and qualitative examples. \n\n>  I have no idea how long the long term point trajectories were, perhaps analyzing this would help.\n\nThe default trajectory length is 41 frames (L267). We studied the effect of trajectory length in Tab. 7 (Appendix). Longer tracks suffer from increasing tracking errors over longer periods. On the other hand, very short tracks might not be sufficiently informative. Our results in Table 7 support this.\n\n> Also depending on the trajectories, were there occlusions or other interesting factors that contribute to the loss function would be interesting.\n\nAs can be seen in the example tracking visualization video from the supplement, occlusions (in blue) are very common. Point trackers estimate the position of tracked points even when they are occluded, and we make use of this prediction in our loss. We also experimented with different trackers (L566-568, Tab. 8). We found that TAPIR and BootsTAP, which are about as good as CoTracker for visible points, are less reliable for occluded points, which hinders performance.\n\n> That is find however your focus is only video object segmentation, why is that and how can you address the other problem areas?\n\nThe difference between video object segmentation and motion segmentation is that the latter only segments objects in the frames where they are moving, whereas the former segments objects as long as they move at some point. As we note in L80-82, the difference between these two tasks is relatively small, and in fact, often, the same benchmark data is used to assess both. In our case, the network is trained to segment the object given a single image as input at a time, and so it should be regarded as solving video object segmentation as it cannot perceive motion directly. However, it is easy to reuse it for motion segmentation by checking if the segmented region actually moves in a given frame, e.g., by measuring optical flow.\n\n> I would imagine if we had 3d scene flow, by perhaps combining monocular depth and optical flow would result in good results without long term tracking?\n\nThis is an interesting idea, but we are not aware of works that have done so yet, at least for video object segmentation. One challenge is that it is difficult to obtain robust 3D predictions (see, e.g., the extensive literature on video depth estimation). Should this information be available, our formulation can be extended trivially to 3D trajectories simply by replacing 2D points with 3D ones. In principle, by avoiding camera projection, these 3D trajectories should be statistically \"simpler\", so a lower value of the parameter $r$, e.g., 3 or 4, should suffice.\n\n> Appearance information for segmentation in the examples would suffice, it would be interesting to focus on cases where appearance info is not sufficient for segmentation and we require motion information.\n\nOur method does not ignore appearance. \nFirst, motion is used as a learning signal to train a segmentation network that takes as input an _image_, and is thus appearance-based. Second, our approach also uses appearance indirectly, as optical flow and point trajectories are predicted from RGB data.\n\nIn any case, the main goal of our paper is to show how point trajectories can be used to derive an \"objectness\" signal to train a segmentation network. We propose a new loss to do so. We note that our new loss could be combined with other losses that capture complementary sources of supervision, but doing so is orthogonal to our main goal.\n\nFor completeness, and as suggested by other reviewers, we do compare with SoTA video object segmentation methods that are appearance-based, i.e., VideoCutLER and VideoSAUR, and obtain better results than them. This also suggests that appearance-based approaches are not sufficient for this data yet (as the gap is considerable). \n\n> Both sources of info, optical flow and long term pixel tracking info are based on 2D info, the projection of 3D info.\n\nPerspective projection indeed complicates the analysis. As we state L198, we increase $r$, to generalize to wide range of scenarios in the real world data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lOtW3LlMkN", "reviewer": "Reviewer_SSEv", "summary": "This paper proposes a model to process long-term motion and short-term motion simultaneously to achieve motion-based segmentation. Specifically, motivated by subspace clustering, this work proposes a loss function that enables training a neural network to learn motion grouping from both optical flows and point trajectories. It outperforms the previous method in the unsupervised video segmentation task. The qualitative comparison also shows obvious improvement, giving a clearer boundary.", "questions": "Why does solely using long-term loss get worse performance than solely using optical flow loss (7 percent drop in Table 4)? Though the paper gives a short explanation that it is due to the sparse set of points and noise, long-term motion also has its advantage like it's more stable than short-term information.", "limitations": "As mentioned in the weakness, the principle the paper proposed is reasonable, but seems like it does not fully support the motivation and the ultimate goal of the task. More analysis and experiments are needed to show the right practice when applying the proposed to real-world videos.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The motivation and method explanation seems to be clear. The paper writing is easy to follow.\n2. Using a simple sample to introduce the low-rank intuition is convincing and reasonable. Based on this core idea, other smoothing losses and regular loss from optical flow make learning more effective.\n3. Experiments show the strength of the proposed strategy. A comprehensive ablation study has been performed to illustrate the impact of each factor.", "weaknesses": "1. As mentioned in the limitation, the paper's principle assumes that the object is rigid. However, the task that this paper works on not only includes rigid objects -- it's a general video segmentation task. Then it seems that the low-rank theory can not extend to a general setting. And why not consider local rigid like ARAP loss? (SpatialTracker)\n2. Do not give some corner cases or failure cases, especially for non-rigid objects. I hope to see some corner cases like multiple objects, where they behave similarly in the short term but different in the long term. Then it can better demonstrate the motivation of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that the reviewer finds the motivation and method clear, easy to follow, convincing and reasonable, with strong results and comprehensive ablation. \nWe thank the reviewer for constructive comments and helpful suggestions.\n\n> the paper's principle assumes that the object is rigid. However, the task that this paper works on not only includes rigid objects -- it's a general video segmentation task. Then it seems that the low-rank theory can not extend to a general setting.\n\nWe *do not* assume objects are rigid. As we state on L202, relaxing $r$ enables supporting non-rigid objects too.\n\n> And why not consider local rigid like ARAP loss? (SpatialTracker).\n\nUsing an ARAP loss would require additional depth input and training an additional encoder network to embed trajectories. In SpatialTracker, this is done using RGBD data for the trajectory prediction task.\nAdditionally, similar to other adjacency matrix methods, the scaling of pairwise trajectory loss is quadratic (L111): SpatialTracker uses 256 trajectories during training using 8xA100, while we can train using 70k trajectories on a single GPU. \n\n> Do not give some corner cases or failure cases, especially for non-rigid objects. I hope to see some corner cases like multiple objects, where they behave similarly in the short term but different in the long term. Then it can better demonstrate the motivation of the paper.\n\nWe included additional visualisations in the Rebutall PDF showing predicted components as well as the motion of the frame. \n\nNote that our trained network does not observe motion during inference. Because of this, it is robust to lack of motion for all horses in the 2nd sequence (Fig. R1), or similar flow patterns of people in the 3rd sequence (Fig. R1), or even noisy optical flow in the 2nd sequence (Fig. R2). It has learned to separate instances or parts. We show predicted components to highlight that the network learns to predict instances and sometimes limbs as separate masks as they might not move together.\n\nWe also include some failure cases in Fig. R2, such as missing very small objects and learning to identify and segment shadows as a component.\n\n> Why does solely using long-term loss get worse performance than solely using optical flow loss (7 percent drop in Table 4)? long-term motion also has its advantage like it's more stable than short-term information.\n\nOptical flow carries less information than long-term tracking, but it has some advantages. First and foremost, it can be computed at full resolution (even CoTracker, which is efficient, cannot cope with more than 70k tracks, which are fewer than the number of image pixels). This is extremely beneficial when the task requires pixel-level accuracy, such as segmentation. Second, instantaneous motion is easier to predict and model than long-term motion. Hence, when optical flow is sufficient to segment the objects, it generally allows it to be done robustly and easily. Long-term tracking is thus complementary to optical flow.\n\n> As mentioned in the weakness, the principle the paper proposed is reasonable, but seems like it does not fully support the motivation and the ultimate goal of the task. More analysis and experiments are needed to show the right practice when applying the proposed to real-world videos.\n\nThe main goal of our paper was to establish whether long-term trajectories are useful source of \"objectness\" for training segmentation networks and to propose a principled way to accomplish this. We have established a positive answer showing improvement over prior art in a series of benchmarks. Additionally, we analysed and showed why alternative implementations of this idea do not work as well.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PmHiQGPkQy", "reviewer": "Reviewer_yFh7", "summary": "This paper proposes a novel loss function that allows training image object segmentation\nmodels based on object motion in videos. Motivated by recent work on self-supervised\nlearning of segmentation using optical flow, the authors propose to use longer point\ntrajectories as additional self-supervision signal. Related to subspace clustering, the\nproposed loss function encourages to predict segments whose trajectories can be well\nexplained by few basis trajectories. The predicted segments are merged into a binary\nsegmentation mask and evaluated on standard, real-world segmentation benchmarks.\nPrevious methods based only on optical flow are consistently outperformed, demonstrating\nthe effectiveness of the proposed method.", "questions": "- How do the predicted segments look like before merging? Visualization would help to\n  better understand the capabilities and limitations of the proposed method.\n\n- The principle of common fate is not cited in the paper, a reference to the literature\n  on Gestalt psychology would be appropriate (e.g., Wertheimer 1912).\n\n- How well does the proposed method perform on MOVi when estimating trajectories using\n  RAFT and CoTracker? This would allow for better judging how much the proposed method\n  could be improved in the future by using more accurate trajectory estimation methods.", "limitations": "The authors address limitations of their work in a dedicated paragraph. Their\ndiscussion is brief but adequate in my view.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Unsupervised learning of segmentation is an important problem. Several recent methods\n  approached this task using optical flow as a self-supervision signal, extending this\n  line of research to trajectories is a well-motivated idea.\n- The mathematical motivation of the loss is very well explained. Without having a deep\n  mathematical background, I could follow the derivation of the loss function without\n  issues.\n- Standard benchmarks and modelling components are used for evaluation, which makes it\n  easy to compare the proposed method to previous approaches.", "weaknesses": "1. It is not described clearly enough what kind of segmentation task is targeted. From\n   the introduction and method section it seems to me that multi-object segmentation is\n   adressed, only at the very end of the method section it is mentioned that the\n   predicted segments are merged into a binary segmentation in some cases.\n   - To my understanding the task is multi-object segmentation for MOVi and binary\n     segmentation for all other datasets. This should be clearly stated in the\n     experiment section.\n   - It should be stated in the introduction and method section more clearly that the\n     main task is binary segmentation.\n\n2. The proposed method is not compared to models that do not use optical flow for\n   self-supervision. It would be interesting to see how the proposed method compares to\n   other self-supervised segmentation approaches. For example\n   - CutLER ([Wang et al. 2023](https://arxiv.org/abs/2301.11320)) and VideoCutLER ([Wang et al. 2023](https://arxiv.org/abs/2308.14710))\n   - DINOSAUR ([Seitzer et al. 2023](https://www.amazon.science/publications/bridging-the-gap-to-real-world-object-centric-learning)) and VideoSAUR ([Zadaianchuk et al. 2023](https://proceedings.neurips.cc/paper_files/paper/2023/hash/c1fdec0d7ea1affa15bd09dd0fd3af05-Abstract-Conference.html))\n   \n    The masks predicted by these models could be merged to obtain a binary segmentation\n   in the same way as for the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for thoughtful comments and suggestions and are happy that they recognized our work as well motivated, well explained, and easy to compare.\n\n> To my understanding the task is multi-object segmentation for MOVi and binary segmentation for all other datasets. This should be clearly stated in the experiment section. [...] the main task is binary segmentation.\n\nCorrect! While this is already noted on L261, we will make this clear throughout in the final version.\n\nThe reason why several experiments focus on binary segmentation is to allow comparison to relevant prior art which used video benchmarks with binary masks, as these are far more common. The \"over-segmentation and merging\" approach is also common in the literature, so we also adopt it to be more comparable to prior works.\n\nHowever, as shown in the MOVi experiment, our loss is indeed more general and not limited to binary segmentation.\n\n> The proposed method is not compared to models that do not use optical flow for self-supervision. It would be interesting to see how the proposed method compares to other self-supervised segmentation approaches. For example\nCutLER (Wang et al. 2023) and VideoCutLER (Wang et al. 2023)\nDINOSAUR (Seitzer et al. 2023) and VideoSAUR (Zadaianchuk et al. 2023)\nThe masks predicted by these models could be merged to obtain a binary segmentation in the same way as for the proposed method.\n\nIn the table below, we provide a comparison of VideoCutLER and VideoSAUR on DAVIS using the same merging strategy for combining multiple predictions to a binary segmentation as in our method:\n\n| Method | DAVIS |\n| -------- | -------- | \n| Ours     | **82.2**     | \n| VideoCutLER | 67.2 |\n| VideoSAUR | 17.5 |\n\nOur method shows a significant advantage. We observe that VideoCutLER has trouble segmenting instances from crowds in the background. VideoSAUR has imprecise object boundaries which severely impacts performance when measurred using Jaccard score.\n\nWe also note that VideoCutLER and VideoSAUR are trained on much larger video datasets as they, ultimately, attempt to derive a learning signal from DINO features computed across adjacent frames. The goal of our paper is to study whether point trajectories can be used to train a segmentation network, which is a complementary rather than an alternative approach: these forms of supervision could be combined.\n\n> How do the predicted segments look like before merging? Visualization would help to better understand the capabilities and limitations of the proposed method.\n\nWe include the visualization of segments before merging them in the rebuttal PDF, Figures R1 and R2. Mostly, different entities or parts of entities (legs, heads, other limbs) are segmented out. This suggests that the model learns that both individual instances and articulated parts of the bodies are likely to have coherent motion.\n\n> The principle of common fate is not cited in the paper, a reference to the literature on Gestalt psychology would be appropriate (e.g., Wertheimer 1912).\n\nThank you, we will include a suitable references.\n\n> How well does the proposed method perform on MOVi when estimating trajectories using RAFT and CoTracker? This would allow for better judging how much the proposed method could be improved in the future by using more accurate trajectory estimation methods.\n\nWe repeat the experiments in Tab. 1 using CoTracker-estimated trajectories for our method and show the results as \"Ours (CoTracker)\" below. Note that our experiments on MOVi-F used only the trajectory loss (no optical flow) to match the same input modalities employed by other methods.\n\n\n| Method | ARI | FG-ARI |\n| -------- | -------- | -------- |\n| K-Means     | 15.26     | 42.53     |\n| Ours (GT traj.) | 46.07 | 65.76 |\n| Ours (CoTracker) | 38.20 | 57.85 |\n\nThe estimated (imperfect) tracks impact performance but it still exceeds that of simple K-Means and prior methods. This shows that improving the tracks can further improve results achieved by the loss.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4CoctVqn8z", "reviewer": "Reviewer_ceYj", "summary": "This paper introduces a method for training a segmentation network using long-term point trajectories as a supervisory signal to enhance optical flow. It proposes a novel loss function aimed at grouping these trajectories into low-rank matrices, allowing the motion of object points to be approximately represented as a linear combination of other point tracks. The proposed approach surpasses previous methods in motion-based segmentation, demonstrating the value of long-term motion and the effectiveness of the new formulation.", "questions": "1. Does the method proposed in this paper work on other segmentation networks as well, and would additional experiments on other segmentation networks help demonstrate the generality of the proposed loss function?\n2. How is the training time and inference speed of the method proposed in the paper? It would be better to include some quantitative comparison experiments.\n3. What is the specific contribution of this paper to Subspace Clustering? \n4. The Flow Estimator and Point Tracker are both frozen during training in this work. Is it possible to also update them during training to leverage the information in the dataset?", "limitations": "The authors have acknowledged some limitations of their work. I suggest the authors could further describe the limitations in the generalizability of their approach.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The introduction describes the problem in more detail when introducing the issue.\n2. The structure of the article is good.\n3. The experimental results of the method proposed in this paper show a significant improvement.", "weaknesses": "1. The main contribution of this paper is the proposal of two losses, but the loss seems to be effective in the experiments of other segmentation methods.\n2. The contribution of the paper in Subspace Clustering is not described clearly.\n3. The resolution of Fig 3 is relatively low.\n4. There is a lack of comparison in terms of inference speed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are happy the reviewer has found our work detailed, well-structured and showing significant improvement. We thank the reviewer for their thoughtful questions and suggestions. We reply to each comment below. \n\n> The contribution of the paper in Subspace Clustering is not described clearly. [...] What is the specific contribution of this paper to Subspace Clustering?\n\nWe do not consider our loss function a contribution to the area of Subspace Clustering. The main question we explore is how to supervise segmentation networks with the motion information contained in point trajectories (L37).\n\nThe reason why we discuss Subspace Clustering is because our loss is inspired by it, and because Subspace Clustering has historically been applied to clustering motion trajectories. However, as we show in our experiments, general Subspace Clustering algorithms such SSC or LRR might not be suited to training neural networks.\n\n\n> The resolution of Fig 3 is relatively low.\n\nThank you; we shall update the size of the picture for the final version.\n\n\n> Does the method proposed in this paper work on other segmentation networks as well, and would additional experiments on other segmentation networks help demonstrate the generality of the proposed loss function?\n\nYes, the proposed loss function is network-architecture agnostic as it only requires mask prediction. Thus any network which predicts masks or has mask-like representation could be used. In our experiments, we used (a) a scaled down version of a UNet network when considering single-sequence optimization (L270), and (b) the same network architecture as used in prior work (MaskFormer + DINO) in video segmentation (L273).\n\nIn the table below, we experiment with changing the segmenter architecture in the DAVIS benchmark. This shows that we can swap different network architectures with relative ease. \n\n| Network | DAVIS | \n| -------- | -------- | \n| MaskFormer + DINO| 82.2 |\n| MaskFormer + Swin-Tiny | 81.2 |\n| UNet | 80.6 |\n\nIn these results, we did not further optimize batch size, learning rate and other parameters that typically depend on the type/architecture of neural network trained (except training UNet for 60k iterations rather than 5k as it is randomly initialized), so these numbers likely underestimate the actual performance that could be obtained with such networks after hyperparameter tuning.\n\n> There is a lack of comparison in terms of inference speed. [...] How is the training time and inference speed of the method proposed in the paper? It would be better to include some quantitative comparison experiments.\n\nIn the table below, we provide the inference time comparison using different networks as average FPS during DAVIS evaluation. Note that since our contribution is a loss function, it is network architecture agnostic. Using it does not affect inference time; only the choice of network architecture does. We matched the architecture with prior work for the best comparisons. Below, we show inference speed for different network choices.\n\n| Network | Inference Speed |\n| -------- | -------- | \n| MF + DINO| 3.3 FPS  |\n| UNet | 6.4 FPS |\n\n\nAs expected, with smaller networks, the inference time is reduced.\n\nA good question is how much of a computation burden our trajectory loss imposes. We measure time spent in the loss function during the forward pass as 0.3s or about 14% of the total batch time, which is small. \n\nAs we reported in Appendix E.3, training a network for our experiments takes about 3 hours. With our proposed losses, one can train a chosen segmentation network quickly on the benchmark data. \n\n> The Flow Estimator and Point Tracker are both frozen during training in this work. Is it possible to also update them during training to leverage the information in the dataset?\n\nThis is an interesting idea! As the task under consideration is completely unsupervised, we would not recommend continuing to train flow and point-tracking networks while training the segmenter. That is because there is a trivial degenerate solution: if no motion is predicted (e.g., by ignoring the input) by flow and/or point tracker then the segmentation network can predict arbitrary segmentation as well with minimal loss, thus the training would likely collapse.\n\nHowever, one could alternate training segmenters and motion predictors on data where point annotations are available. In this work, we concentrate on establishing how trajectory data can be used as an \"objectness\" signal.\n\n> I suggest the authors could further describe the limitations in the generalizability of their approach.\n\nWhile we showed good results in quite general real-world videos, as mentioned in L324, extending our loss to multi-object segmentation in videos where objects undergo non-rigid articulated motion to the same effectiveness is not trivial. This is, ultimately, because the precise number of objects is unknown, so recovering the appropriate number of instances from that might have been seperated into parts is not straightforward. This could be overcome by introducing an appearance-based loss across groups of predicted masks, effectively encouraging the network to predict $n$ objects each of up to $k$ parts by predicting $n \\times k$ total components.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pmBrXW3gAL", "reviewer": "Reviewer_Nk1m", "summary": "The authors propose a loss function that seeks to group the trajectories into low-rank matrices where the motion of object points can be approximately explained as a linear combination of other point tracks. Experiments on the synthetic MOVi-F variant of\nthe Kubric dataset  and the real datasets DAVIS 2016, SegTrackv2 and FBMS show that the proposed method outperforms single-sequence methods, single-stage end-to-end methods and multi-stage methods.", "questions": "No questions", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1) The authors address key issues in the field and the contribution is original even if somewhat incremental.\n2) The proposed method is detailed and reproducible.\n3) Experiments are relatively well conducted on synthetic and real datasets showing the superiority of the proposed method.", "weaknesses": "About the presentation, please clearly state a name/acronym to the proposed method and replace \"ours\" by it in the comparison tables.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are happy the reviewer has found our work to be original, detailed, reproducible, addressing key issues, and with well-conducted experiments. We also thank the reviewer for suggestions to improve the clarity of our work.\n\n> About the presentation, please clearly state a name/acronym to the proposed method and replace \"ours\" by it in the comparison tables.\n\nWe will update the text and tables using the tag \"LRTL (Ours)\" to denote our *Low-Rank Trajectory Loss*.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the Reviewers for their thoughtful comments and suggestions. We are happy they found our presentation clear, well-flowing, well-motivated, convincing and reasonable, our results strong and our experiments comprehensive. We reply to each comment individually.

To aid replies, we also provide a Rebutall PDF that contains two figures (R.1, R.2), which we reference. The figures show more examples, including failure cases, from FBMS sequences alongside motion and predicted components. We shall include these figures as well as additional results and discussion presented in the reply in the final paper.

Meta-review: All of the reviewers are in favor of accepting this paper, to varying degrees. They appreciate the novelty of the loss function incorporate long-term temporal information to complement short-term optical flow, and the thorough experiments. There was a robust response to the reviews from the authors that alleviated concerns and contributed to acceptance of the paper.
Decision: Accept (spotlight)