Moving Off-the-Grid: Scene-Grounded Video Representations
Authors: Sjoerd van Steenkiste
Daniel Zoran
Yi Yang
Yulia Rubanova
Rishabh Kabra
Carl Doersch
Dilara Gokay
Joseph Heyward
Etienne Pot
Klaus Greff
Drew A. Hudson
Thomas Albert Keck
Joao Carreira
Alexey Dosovitskiy
Mehdi S. M. Sajjadi
Thomas Kipf
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Self supervised learning
point tracking
representation learning
Primary area: machine_vision
TL;DR: We propose an "off-the-grid" representation model which learns from video and binds tokens to scene elements and tracks them over time consistently via self-supervised learning.

Abstract: Current vision models typically maintain a fixed correspondence between their representation structure and image space.
Each layer comprises a set of tokens arranged “on-the-grid,” which biases patches or tokens to encode information at a specific spatio(-temporal) location. In this work we present *Moving Off-the-Grid* (MooG), a self-supervised video representation model that offers an alternative approach, allowing tokens to move “off-the-grid” to better enable them to represent scene elements consistently, even as they move across the image plane through time. By using a combination of cross-attention and positional embeddings we disentangle the representation structure and image structure. We find that a simple self-supervised objective—next frame prediction—trained on video data, results in a set of latent tokens which bind to specific scene structures and track them as they move. We demonstrate the usefulness of MooG’s learned representation both qualitatively and quantitatively by training readouts on top of the learned representation on a variety of downstream tasks. We show that MooG can provide a strong foundation for different vision tasks when compared to “on-the-grid” baselines.

Moving Off-the-Grid: Scene-Grounded Video Representations
Sjoerd van Steenkiste" Daniel Zoran* Yi Yang", Yulia Rubanova', Rishabh Kabra², Carl Doersch", Dilara Gokay², Joseph Heyward', Etienne Pot Klaus Greffe, Drew A. Hudson Thomas Albert Keck
Joao Carreira", Alexey Dosovitskiy i.3 Mehdi S. M. Sajjadi?, Thomas Kipf*.2 Google Research, Google DeepMind, Inceptive
Abstract
Current vision models typically maintain a fixed correspondence between their representation structure and image space. Each layer comprises set of tokens arranged "on-the-grid," which biases patches or tokens to encode information a specific spatio(-tempora) location. In this work we present Moving Off-the-Grie (MooG), a self-super video representation model that offers an alternative approach, allowing tokens to move "off-the-grid to better enable them to represent scene elements consistently, even they move across the image plane through time. By using combination of cross-attention and positional embeddings we disentangle the representation structure and image structure. We find that a simple self-supervised objective-nee next frame rredictionttriined on video data, results in a set latent tokens which bind to specific scene structures and track them as they move. We demonstrate the usefulness of MooG's learned representation both qualitatively and quantitatively by training readouts on top the learned represen- tation variety of downstream tasks. We show that MooG can provide strong foundation for different vision tasks when compared to "on-the-grid" baselines!.
Introduction
Learning visual representations of the physical world is at the of computer vision. Recent years have seen a surge of vision models that address this problem via self-supervised learning [5, 8, 23, 40]. By leveraging objectives such contrastive learning [5, and masked image modelling [23], great strides have been made towards learning useful representations from image data. The vast majority of these methods use convolutional networks [35], vision transformers [14, 54] or combination thereof [4]. This choice of architecture comes to no surprise, as it inherently reflects the structure of the underlying datasets: images (typically) represented as grids of pixels, which are conveniently and efficiently processed using 2D convolutions and patch-based heuristics. This grid-based processing, however, leads to an inherent entanglement between the representation structure and image structure. In other words, specific tokens or feature vectors of the representation are encouraged to capture the contents of specific image location, instead of binding the underlying content of the physical scene.
This issue is particularly apparent when processing video: when there is motion in the scene, either by ego-motion object motion, the contents of the scene will move across the image plane and as such the representation (i.c. in terms of what is encoded where) will change accordingly. However,
Project page: attps://moog paper- .err.githht github, io/. "Equal contribution, Work done while at Google.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

many down-streen stream scene understanding tasks require observing how individual objects (or object parts) change their configuration over time, even when other factors like camera motion translate the objects around the image plane. In this case, a representation that preserves correspondences between meaningful scene elements and representational elements is likely preferred.
As a consequence, many works targeting object-centric tasks such as object detection [4, 36], tracking [25, 33, 38J, or segmentation [34], have adopted specialized architectural components that learn object-based representations: representations that are lifted from the image grid to bind to individual objects. These representations, however, are specialized to object-centric tasks and either need to be learned with detailed supervision [4, 34, 38] or have difficulty scaling to diverse real-world raw video data [19,
In this paper we propose a transformer-hased video model that learns representations that are "off-the- grid" (OTG) in self-supervised manner, providing consistent features that bind to underlying scene elements, and tracking them as they move through time. Our method, Moving Off-the-Gria (MooG), makes extensive use of cross-attention to learn a latent set of tokens that decoupled from the image grid: tokens are updated via cross-altention when a new input frame arrives, and decoded back into images via cross-attention. MooG can process videos of arbitrary length by iteratively updating the representation as new frames are observed.
In summary, contributions are follows:
We introduce Moving Off-the-Grid (MooG), novel transformer-basee recurrent video representa- tion model that is capable of learning OTG representations via a simple next-frame prediction loss.
We qualitatively demonstrate that the OTG representation of MooG binds to different parts of the scene and tracks its content under motion, whereas grid-based representation fails to do so.
Finally, we demonstrate how this representation facilitates a variety of downstream vision tasks, including point tracking, monocular depth estimation, and object tracking. Our approach outper- forms self-supervised grid-based baselines, such as DINO [5, 40J, and performs competitively with commann-speciff approaches, such TAP-Net [12] and TAPIR [13] for point tracking.
2 Related Work
Transformer architectures [54] for visual tasks have gained substantial traction in the machine learning community in recent years. Starting with methods such as the self-attention architecture applied to CNN feature maps by Zambaldi et al. [62], the Image Transformer [41] and later popular approaches
such as the Vision Transformer (ViT) [14], the vast majority of this class of methods operates on grid of image features (e.g. patches or CNN feature maps), all the way from pixels to the final output of the transformer. This choice of representation, while extremely successful on wide range of tasks, naturally couples representations to spatial 2D locations image space.
The predominant approach for decoupling internal model representations from the image grid by using cross-attention, where one set of tokens is updated based on the value of another set of tokens. In particular, object-centric tasks such as detection [4, 64], tracking [33, 38, 63]. and instance segmentation [9, 34] have found widespread adoption of this architectural principle to learn individual object tokens that are detached from the image grid, both in supervised methods such as the Detection Transformer (DETR) [4] or Group ViT [59], and unsupervised methods such as Slot Attention [36, 57] or CLIPpy [46]. Especially when extended to multi-view observations [27, 48] and representation allows for consistent representation of individual objects across views and frames in a video. In contrast to these approaches, our method does not assume a one-to-one mapping between OTG tokens and objects, but instcad assigns large set of latent tokens that can flexibly bind to any part of a scene, such small surface elements, without committing to any particular notion of an object.
video [19, 33, 38, 65], this mee-token-per-ofiee
The Perceiver [28] is closely related to our work: it uses a large set of latent tokens, updated via cross-attention from visual inputs, to support a range of downstream tasks. While the original Perceiver model primarily focuses on end-to-end classification tasks, PerceiverIO [29] extends this framework to use pixel-based objectives or predict other modalities (such as audio). A single time step of our model can be seen as variant thereof: we similarly use cross-attention to map to a latent set of tokens, and decode targets (such as pixels, point tracks, etc.) similarly using cross-attention. This type of cross-attes aoossntttntion decoder is also used in Scene Representation Transformers [49], which

---

predict
predict
predict
docodo
recied state
corrected slate
correcled slate
prodicted stoto
procieted
encode
encode +
lime
MooG recurrent, transformer- based, video representation model that can be unrolled through time. MooG learns a set of "off-the the--hiiddrd latent representation. The model first predicts predicted state based on the previous model state and observation. The current observation is then encoded and cross-attended to using the predicted state as queries to produce a correction to the prediction. When training, the predicted state is decoded with cross-attention using pixel coordinates as queries in order to reconstruct the current frame. The corrected state is used as input to the predictor to produce the next time step prediction, and so on. The model is trained to minimize the pixel prediction error. By decoupling the latent structure from the image grid structure the model is able to learn tokens that track scene content through time.
Figure 1:
have recently been applied to video modeling [50, 51J. Different from these works, our approach processes video in recurrent fashion, encouraging tokens to stay consistently attached to the same scene element independently of how the camera or the content of the scene moves.
separate set of tokens with either back-and-fort cross-attention, such as RIN [26] and FTT [7]. or pure self-attention, with extra tokens simply appended to the original set of briid-baedd tokens. The latter category includes approaches such Ada [61] and Register Tokens [10]. In our work, we solely update tokens by -ooss-attending into grid-based representations, without "writing back" into the grid-based representations.
Several recent works
related line work explores autoregressive prediction of visual tokens for self-supervised represen- tation learning [2, 18]. Different form this line of work, MooG uses recurrent architecture to enable consistent binding of latent tokens to scene elements (as opposed to using fixed image patches).
Naturally, most explicit 3D approaches for vision are "off-the-grid"" such as architectures operating on top of point clouds [44, 45], and methods for 3D rendering such as 3D Gaussian Splatting [30, 37] and particle-bas based neural radiance fields [56, 60]. In contrast, our method does not associate explicit 3D coordinates with individual OTG tokens but instead learns dinen-iinenionnl vector representations. Outside of the field of computer vision, off- -ie--he--rddd representations are the predominant representation used to model the physical world various scales, e.g. terms of particle-based representations for atomistic systems (one vector per atom) [20, 32, 52] mesh-based representations for macroscopic physical systems [42], where representations are anchored surface elements.
Method
Moving Off-the-Grid (MooG) is self-supervised transformer model for representation learning from video. In Section 3.1 we describe its model architecture, which enables learning of scene grouuddd video representations. To obtain predictions for various vision tasks, connect readout modules to MooG's OTG representation, which we describe in Section 3.2.
3.1 Learning self-supervise OTG video representations
We design MooG as recurrent model that can process an arbitrary number of video frames, while keeping a consistent scene-grounded OTG representation of the video. MooG takes as input sequence of observed frames [X,)f= X, E #x and iteratively encodes them into a set {2]]r=
of latent tokens. We separate the latent state into corrected states {>ā
1-11> z E RXxD which
obtained by encoding input frames, and predicted states {2P)T [t=1 2t E R RXDD which the

---

model's 's internal prediction for what will be observed at the next time step. This recurrent processing an important part of MooG as it allows individual tokens to consistently track elements of the scene through videos of arbitrary length and "aatiicipate" where a scene element will be observed next.
MooG's training objective is next frame prediction given the previous frame and model state. The model is comprised of three main networks: predictor P which predicts the current predicted state from the previous corrected state, a decoder 'D which decodes the predicted state reconstruct the current frame and a corrector C which encodes the current frame and attempts to correct prediction errors made by the predictor'. We now describe, in order of operation, each component role and inner workings. We refer to Figure 1 for an overview of the model's 's structure and to Appendix C for details. For comparison, we depict a typical "on-the-grid baseline model in Figure 2.
Predictor
The predictor takes the previous corrected state zi-1 and produces the predicted state for the current time step z
Figure 2: For comparison
to MooG we here depict classic "on-the-grid" model where tokens in the latent state are inherently tied to specific pixel locations.
of = 2 + P(xqv= = =f
(I)
The role of the predictor is to predict the current state based on previous observations, before the current time step's inputs are observed. The predictor network itself P is a simple self-attention transformer net- work [54]. Note that the initial corrected state zi is initialized to random Gaussian noise (with zero mean and o 10-4) and is not learned. This choice of initialization comes from the need to break the symmetry among the state's tokens, as well as preventing "specialization" of tokens and maintaining permutation symmetry.
Corrector The role of the corrector is to convey information from the current observation X, and use it to update the predicted state zf to form the corrected state 2f for the current time-step. The resulting corrected state should contain the new information obtained from the observation. The image is first encoded using a convolutional neural network & with an added Fourier positional embedding to its output and linearly projecting the result to produce feature grid F2 E H''WW' xD Here H' and W' are the resulting spatial dimensions after accounting for striding. This feature grid is then attended to with a cross-attention transformer C using the current predicted state 2f as initial queries to obtain the state update:
o = =| + F 2f'), where F. = E(x))
(2)
is important to note that the corrected state does not receive its own individual loss and is only used to provide better estimate for the predictor in order to predict the next step. In this sense, the separation between the corrector transformer and predictor is somewhat artificial. It is, however, crucial that the image is decoded only from the predicted state- decoding the current frame from the corrected state reduces the problem simple auto-encoding and hurts representation quality considerably.
Decoder The decoder takes the current predicted state z! and decodes it into an RGB image of arbitrary resolution. Decoding is done using cross-attention where queries are embedded pixel coordinates P and keys and values come from the predicted state 2 We utilize the same architecture for arbitrary pixel-based readouts (described in Section 3.2); see Figure 3 for a schematic depiction. Note that the states that are learned are comprised of tokens that are OTG and offer no direct correspondence to the spatial image grid, which necessitates this design of our decoder. At training time we decode X,, sub-sampled version of the target image for efficiency:
X. = D(kv = P)).
To allow for efficient training, we decode only randomly selected subset of pixels each training iteration, which reduces computational demands significantly. For details see Appendix C.
Of special note are the attention weights of the decoder network D as they allow us to understand the relationship between a specific spatial position in the image and specific tokens in the latent representation. We analyze this relationship in Section 4.1.
"This is reminiscent of Kalman Filter, albeit with implicitly learned dynamics and variance estimations.

---

Visbiity
MLP
MLP
Transformer (crose- avas only)
Embed
Transformer (cross-iin unl)
MLP
Transformer (cross- winh only)
Spatial queries (suboampled, random offset)
Points (or boxes)
[2P. 2/]
[R... 2:9]
[2:P, 2x
(a) Pixel readout (RGB, depth).
(b) Recurrent readout (points, boxes).
Figure 3: Readout decoders overview: for grid- readouts (e.g. pixels), we use simple per-frame cross-attention architecture with spatial coordinates queries, whereas for set-based readouts (points, boxes), we adopt a recurrent readout architecture.
Loss and training We use simple L2 training loss image pixels. For cach frame decode the predicted state =f into sub-sampled output image X,, sub-sample the input image X₁ at the same pixel locations and calculate the per-frame loss Le:
T = (x2 X2).
Note that this is next-frame prediction loss as the predicted state depends only on the previous frame and model state. During training we unroll the model over frames, initializing the state with random Gaussian noise (as described above) and equally weighting all frames for the loss. The final self-supervised prediction training loss averages the Le loss across frames and pixels.
Readout decoders for downstream tasks
We qualitatively assess properties of the learned representation in Section 4.1. To make quantitative assessment, we propose general readout decoders that support a variety downstream tasks. We distinguish between two types of readouts: grid-based readouts (e.g. RGB depth pixels) and tracking-based readouts (e.g. 2D points or object tracking). To produce the desired output, all readouts read from the OTG tokens contained in the predicted and corrected states for given timestep. See Figure for schematic overview.
Grid-based readouts For dense grid-based readouts, reuse the same decoder architecture as we use for the pixel decoder: individual spatial locations are queried using their (s. y) location in transformer that solely rross-attention. For computational efficiency, query using subsampled spatial grid with a random offset to avoid overfitting on a particular subset of pixels.
Recurrent, query-based readouts For tracking-based readouts that require keeping track of con- tent in the video after starting from an initial location, we adopt a more sophisticated design similar the corrector- preedcct component of MooG. Given of N queries @1 E R N (c.g. points or boxes) and a sequence of observed frames {Xx)V=1 the task is to predict all future readout targets {q.} for 2...7. We associate latent encoding Ye Nx,, with every target at time step t. We first encode the queries using positional encoding followed by an MLP to obtain the readout latents ye. Latents are processed by a corrector, followed by predictor. Different from MooG, the corrector is implemented
transformer which solely cross-attends into the inputs (here: MooG states [=f, z²²)) without self-attention between readout latents, to avoid interaction between them. Likewise, the predictor is implemented solely by an MLP, i.e. without any self-attention between readout latents. To read out target values, we apply learnable MI.P head to the (corrected) target latents yr for each output type, eg. coords, visibility, etc.

---

GT
Prediction
Moo
Token
Grid Taken
Figure 4: Qualitative analysis of MooG trained on natural videos, shown here are every frames of the original 36 frame long sequence. From top to bottom: Ground truth frames, predicted frames, example MooG token attention map super-imposed on the ground truth frames, example token attention from the recurrent grid-based based baseline (see text for details). As be seen the model is able to predict the next frame well, blurring when there is fast motion or unknown elements enter the scene. The MooG attention map indicates that the visualized token tracks the scene element binds to across the full range of motion. In contrast, the grid-based token attention map demonstrates how these tokens end up being associated with specific image location that does not track the scene content. Please see the supplementary material (and website) for other representative examples.
Experiments
4.1 Self-supervised Training and Qualitative Results
We begin by qualitatively investigating properties of the learned OTG representation. We trained MooG with 1024 512-dimensional OTG tokens natural videos from the Ego4D dataset [21] and Kinetics700 dataset [6] using only the self-supervised prediction loss in (4). The model is trained on randomly sampled sub-sequences of frames, and we observe how it learns to predict next frames very well, achieving a PSNR of 25.64 (on the evaluation set) after 500K steps of training. For evaluation sequences the model has not been trained on and are much longer in duration. We first observe that the model has no trouble unrolling for much longer sequences than was trained on, and that the predictions made are inline with the actual ground truth frames (see Figure 4). When motion is fast or erratic the model produces blurry outputs, as would be expected from a deterministic model.
unroll the model on sequences of 36 frames from validation set these are
Cross-attention maps To understand the role of each token we focus the cross-attention weights of the decoder, which works by having x, y coordinates as queries that cross-attend into the repre- sentation in order to produce the pixel output (Section 3.1). Using the attention weights for each image location we can visualize how much each token is "responsible" for predicting a specific image location at any given time. We observe that tokens bind to the same image structure consistently through time: the attention maps of specific tokens track the image content at a particular location as it moves across the scene. representative example of this behavior shown in Figure 4, which observe consistently for different tokens and on different sequences. Note that an alternative strategy for the model could have been "tile" the image across space and make each token responsible for a specific spatial position, which is indeed is what we find the grid-based baseline tokens end up capturing (see Figure 4). For additional examples (and better viewing experience), refer the videos in the supplementary material'.
A more comprehensive way of visualizing the role of individual tokens is by taking the argmax attention weights at each image location and visualizing the result by colour coding according to the token index. We observe the model learns assign different parts of the scene different tokens, tiling the image while aligning well with image structure, not unlike super-pixels. This visualized
Also available at https: //moog///mooo//////o paper. saaer.githhn github.iii... 10/.

---

Figure 5: PCA of MooG tokens unrolled over batch of short sequences. The model was unrolled over batch of 24 sequences, 12 frames each. Predicted states from all time steps and batch samples were concatenated and PCA analysis was performed on the entire set jointly. We then reshape the projected set back to its original shape and use the arg-max token to visualize the result in image space (see text and Appendix for full details). Depicted are of the leading PCA components in RGB. Note the salient high-level scene structure (e.g hands) learned by the model.
in Figure 7 in Appendix D, supplementary material and website. If there is good binding between specific tokens and specific image structures we should expect the colour coded image to reflect the motion present in the underlying scene. Indeed, MooG does exactly that the model learns to devote specific tokens to specific scene structures and bind them consistently across time.
Principal Component Analysis To make sense of the content of the representation at a slightly higher level we can use PCA analysis and visualize the results. We unrolled MooG on batch of 24 sequences, 12 frames each and ran PCA on the concatenated set (across batch and time) of all tokens from the predicted state. We then project all tokens on the 64 leading PCA components and use the decoder attention weights to output 3 of the leading PCA components to image space. observe that many of the leading components capture the positional embedding and do not relate to sequence content. However, several others end up capturing interesting (semantic) scene structure. Figure shows the 3rd, 20th and 2 components in RGB space, from which it can be seen how tokens end up capturing scene content with similar projections for hands, for example, across several scenes. See website and appendix for more examples.
4.2 End-to-End Training and Quantitative Analysis
In the previous subsection we observed qualitatively how learned off-the-grid representations end up capturing meaningful elements of the scene. Here we study the quality of the learned representation quantitatively, focusing on three down-stream tasks: point tracking, depth prediction and object tracking. For each down-stream task we consider two different approaches for training readout head that reflect common use cases: (I) training on top the representations obtained from a frozen pre-trained MooG model and (2) training the readout decoder alongside MooG in end-to-end manner, i.e. by back- sack-propagating gradients into the model. MooG learned representations are quite local in nature due the simplicity of the loss and the short prediction horizon. As such do not expect it to learn abstract representations suitable for more high level tasks such as action recognition etc. We focus here low and mid level downstream tasks. Details available in Appendix C.
We focus on two classes of baselines in our comparison: (1) on-the-grid baselines derived from MooG, DINO [5, 40] VidcoMAE v2 [55], and (2) expert baselines that are more domain specifie. The
7

---

Table Down nownn-tream readout performance from frozen representations. For DINO and VideoMAE
v2 baselines we highlight what ViT configuration their encoder is based on: (S) ~20 parameters,
(B) ~80M parameters, (G) ~1000 parameters. MooG uses fewer than 35M parameters for encoder, corrector and predictor combined, which suggests a comparison to B-sized models.
MOVi-E
DAVIS
Waymo
Name
Points (tAJ)
Depth (LAbsRel)
Boxes (MoUU
Points (tAJ)
Boxes (MoU)
MooG
0.839
0.0359
0.793
0.687
0.730
Grid
0.769
0.0451
0.730
0.518
0.625
Grid Rec.
0.778
0.0443
0.734
0.559
0.629
DINOv1 (B)
0.518
0.0371
0.724
0.409
0.566
DINOv2 (B)
0.544
0.0370
0.738
0.402
0.559
VMAEv2 (S)
0.595
0.0567
0.700
0.365
0.567
VMAEv2 (B)
0.681
0.0458
0.736
0,434
0.611
VMAEv2 (G)
0.822
0.0311
0.793
0.720
0.708
former grid-based baselines have the same capacity as MooG which makes them directly comparable. We consider two variations: a simple auto-encoder with high capacity, where we have removed the corrector and predictor (named Grid); and a recurrent on-the-grid baseline where the corrector implements cross-attention between the output of the encoder (i.e. on-the-gric latents) and the corrected latents from the previous step. In both cases account for the absence of the predictor (and corrector) by adding self-attention transformer layers to the encoder. For DINO, we compute representations for each frame using the official pre-trained ViT-B/16 and ViT-B/14 checkpoints that are available for vl [5] and v2 [40] respectively. Similarly we use the publicly available checkpoints for VideoMAA v2 [55] for three different model sizes: S/B/G. Note that the available VidcoMAE v2 model size S and B are distilled from G, i.c. not trained from scratch (unlike MooG). The on-the-gre -ridd baselines (including DINO and VideoMAE v2) make use of the same readout decoders as for MooG.
We discuss expert baselines in the relevant paragraphs below.
Point tracking The task of point tracking requires tracking of individual 2D points (z, y) in video. This can be viewed as generalization of optical flow prediction, which similarly requires understanding of movement physical surfaces in a scene relative the observer. Intuitively, an OTG scene- grounded latent representation should align well with this task and support generalization.
We train MooG on Kubric MOVi-E [22] using point annotations computed in similar manner as in Doersch et al. [12]. For each video, we randomly sample clip of 8 frames to train on. We sample 64 points per frame and use the location of each point in the first frame as the query. To evaluate each model we report the average Jaccard (AJ) as in Doersch et al. [13], which evaluates both occlusion and position accuracy. Tables 1 and report results for MooG and on-the-grid baselines. can be seen how MooG learns representations that are better suited for this down-stream task as is evident from the considerable improvements over many of the baselines, especially the frozen setting (Table 1).
In addition to results on MOVI-E, Tables and 2 also report results for a zero-shot transfer setting to the real-world DAVIS dataset [43] using point tracks from Doersch et al. [12]. Interestingly, it can be seen how in the end-to-end setting (Table 2) the gap between the grid-based models and MooG decreases considerably, suggesting that while off-the-grid representations generalize better in the pre- training scenario, when optimized for a specific downstream task on-the-grid representations can still be competetive. We also compare to two expert Cecelines-TAA--A [12] and TAPIR [13c-which are designed specifically for point tracking and achieved state-of- performance when published. Both models use explicit cost volumes, i.c., exhaustive comparisons of query point's 's features with features on every other frame, and have steps which detect peaks in the similarity maps. Our model is more general and does not have explicit mechanisms such as these at the representation level. The readout mechanism is also quite general. All of these steps exploit the nature of point tracking as one-to-one feature matching problem, making it difficult to adapt the architecture to other problems. In Table we directly compare MooG (trained end-to-end using slightly larger encoder backbone) to these methods and report results for frames as well for the full sequence length. It can seen how on 8 frames, MooG rivals TAPIR's performance, despite being a less specialized approach.

---

Table 2: Down-stream readout performance trained in an end-to-end manner.
MOVi-E
Davis
Waymo
Name
Points (tAJ)
Depth (JAbsRell
Boxes (toUU
Points (tAJ)
Boxes (toUU)
MooG
0.886
0.0263
0.803
0.778
0.719
Grid
0.860
0.0264
0.775
0.644
0.615
Grid Rec.
0.902
0.0233
0.806
0.779
0.675
DINOv1 0,698
0.0381
0.728
0.578
0.557
DINOv2
0.732
0.0439
0.734
0.656
0.607
Monocular depth estimation Monocular depth estimation is well-studied computer vision task that requires estimating the distance of surface elements in the scene from the camera. To test whether a scene-grounded representation, i.e. a representation that consistently tracks surface elements the scene, facilitates the task of depth estimation,, train depth readout module.
We train MooG using the depth annotations available in Kubric MOVi-E [22] normalized using log(1 2). Similar to before, we randomly sample clip of 8 frames to train for each video. As our evaluation metric we report the mean of the absolute relative error (AbsRel), which is a standard metric in the literature [17]. Tables 1 and report results for MooG and the grid-based baselines. Though we observe considerable improvements from the representations learned by MooG in the frozen setting, the Grid baseline performs comparable in the end-to-end case. This isn't surprising given that monocular depth estimation a dense prediction task that can be learned well with on-the grid representations. However, the results in Table highlight how, when learning general representations that are not specific for single task, the representations learned by MooG are still favorable similar DINO and VidcoMAE v2 representations for this task when considering similar model sizes (VideoMAE G, having IB parameters and having been pre-trained on large scale data, performs slightly better than our much smaller model on some tasks).
As an alternative baseline, we compare to DPT [47] trained on the Waymo Open dataset [53], starting from pre-tran mreitrained ViT models as in Dehghami et al. [11]. All models trained end-to-end Table demonstrates how MooG is able to outperform these on-the- on-the-grid baselines in this setup.
Object tracking Tracking individual objects in a video not only requires spatio-temporal tracking of surface elements (as in point tracking), but also a broader semantic understanding of "objectness" to disambiguate object boundaries from surrounding scene elements.
We train MooG using the box annotations available in Kubric MOVi-E [22]. For each video, randomly sample a clip of 8 frames to train on and we make use of all available boxcs. As queries use the location of each box in the first frame, and we report the average IoU (excluding the first frame in the sequence for which GT is provided) as in prior work [19, 33]. Tables and 2 report results for MooG and the grid-based baselines. Similar to the point tracking results, it can be how overall MooG performs considerably better for object tracking compared to the on-the-grid baselines. Notice how, unlike for points, the box annotations focus specifically objects (excl. background).
As an -outio-ddittriuution evaluation we also reports results on the Waymo Open [53] dataset, where evaluate the model (and readout decoder) without additional training. Here observe that the representations learned by MooG are better suited for this task in both settings. We relate the performance of MooG a fully end-to-end supervised SAVi+ [19] model, by training and evaluating MooG on Waymo directly in the same set-up for 250K steps. In this set-up MooG achieves 0.667 IoU compared superssuee end-to- sod-to-end trained SAVi++ variant that is reported achieve 0.676 IoU in Elsayed et al. [19], despite only the readout decoder being supervised in MooG.
4.3 Analysis
Number of readout layers The default hyperparameters for our readout decoders are designed to be sufficiently expressive to learn a general mapping between latent representations and targets (e.g. point tracks). In particular, we purposely use multiple transformer layers, as is common in the literature [29, 49]. To determine to what extent the readout decoder capacity affects our results, repeat our end-to-end point tracking experiment, but using single layer point readout decoder.

---

Table 4: Depth comparison. End to end.
Table 3: Points comparison. End end.
Name
Waymo (LAbsRel)
Name
Davis-8 (tAJ)
Davis-full (tAJ)
MooG
0.094
MooG
0.824
0.510
DPT (ViT-L/16)
0.161
TAP-Net 0,687
0.392
DPT (ViT-E/14)
0.158
TAPIR
0.823
0.580
DPT (ViT-22b)
0.154
Figure 6 shows how using only single layer yields marginal improvement on MOVi-E, while on DAVIS-8 a slight drop in performance can be observed.
Number of tokens
advantage of having an "off-the aoff-the-grid" grid" representation that the number of tokens is independent of input frame resolution. Furthermore, because initialize the representation randomly, and because none of our model parameters depend on the number of tokens in the representation, can instantiate the model with different number of tokens at test time. Indeed, in Figure 7 we qualitatively show how MooG adapts to this change elegantly and is is still able to predict future frames well even with half quarter of the number of tokens used in training. The model makes tokens cover larger areas the scene to adapt for this change (these results are best view in video format). Quantitatively in Figure 6 it can be seen how increasing the number of tokens during training to 2048 or decreasing to 512 doesn't 't significantly affect MooGis performance.
5
Conclusion
While the vast majority of computer vision advances in the past decade can be attributed to successful "on-the- grid" architectures such as CNNs and Vision Transformers, the physical world ultimately does not live pixel grid. Instead of coupling the visual processing architecture to the architecture of the camera sensor (a pixel grid), we here propose to move visual representations off the image grid. Our MooG architecture allows representations to flexibly bind to scene surface elements and track the content of the scene as subject to motion. We demonstrated that this representation can serve as an effective alternative the established grid-based counterpart, facilitating tasks that require understanding of motion and scene geometry. The proposed model in this paper is still quite simple it is deterministic and ignores uncertainty inherent the prediction task, and uses a very simple L2 pixel loss as the objective. A possible next step is to introduce stochasticity into the model account for this inherent uncertainty, improving the representations and allowing for longer term prediction. The latter may help the model learn richer, higher-level features of the scene. We have observed that MooG struggles when applied more semantic downstream tasks and this can likely be explained by the simple deterministic nature of the prediction task and the short-term prediction horizon of the model.
Acknowledgments and Disclosure of o Funding
We would like to thank Lucas Beyer and Mathilde Caron for making their internal implementation of DINOv1 [5] and DINOv2 [40] available for comparison. Similarly, we would like thank Xinchen Yan for making VideoMAEE2 [55] available internally.
This work carried out at Google.
References
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. NewrIPS Deep Learning Symposium, 2016. 17, 18
[2] Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan Yuille, Trevor Darrell, Jitendra Malik, and Alexei A Efros. Sequential modeling enables scalable learning for large vision models. In Proceedings of the IEF.F/CVF Conference on Computer Vision and Pattern Recognition, 2024. 3
10

---

[3] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Londeemman-Mimae and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
http://githun www/ gomn//oooleaaa 18
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229, Springer, 2020. 1,2
[5] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self supe--upeeeiee vision transformers. Proceedings of the IEEE/CVF international conference computer vision, pages 9650-9660, 2021. 1, 1,
8, 10, 19
[6] Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A short note on the kinetics- 700 human action dataset. arXiv preprint arXiv:1907.06995 2019. 6, 20
[7] Ting Chen and Lala Li. Fit: Far-reaching interleaved transformers. arXiv:2305. 12689, 2023. 3
arXiv preprint
[8] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International conference machine learning, pages 15597-6077. PMIR, 2020.
[9] Bowen Cheng, Ishan Mísra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pages 1290-1299, 2022.
[10] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. International Conference on Learning Representations, 2024. 3
[11] Mostafa Dehghani. Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference Machine Learning, pages 7480-7512. PMLR, 2023. 9, 17, 20
[12] Carl Doersch, Ankush Gupta, Larisa Markeeva, Adria Recasens, Lucas Smaira, Yusuf Aytar, Joao Carreira, Andrew Zisserman, and Yi Yang. Tap-vid: A benchmark for tracking any point video. Advances Neural Information Processing Systems, 35:13610-13624 2022. 2,
16, 19
[13] Carl Doersch, Yi Yang, Mel Vecerik, Dilara Gokay, Ankush Gupta, Yusuf Aytar, Joao Carreira, and Andrew Zisserman. Tapir: Tracking any point with per-frame initialization and temporal refinement. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10061-10072, 2023. 2, 8, 16, 18
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. 1,2
[15] Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Rey- mann, Thomas McHugh, and Vincent Vanhoucke, Google scanned objects: high-quality dataset of 3d scanned household items. In 2022 International Conference Robotics and Automation (ICRA), pages 2553-2560. IEEE, 2022. 15
[16] Timothy Dozat. Incorporating nesterov momentum into adam. 2016. 18
[17] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from single image using a multi-scale deep network. Advances in neural information processing systems, 27, 2014.
9, 17
11

---

[18] Alaaeldin El-Nouby, Michal Klein, Shuangfei Zhai, Miguel Angel Bautista, Alexander Toshev, Vaishaal Shankar, Joshua M Susskind, and Armand Joulin. Scalable pre-training large autoregressive image models. arXiv preprint arXiv:2401.0854/ 2024. 3
[19] Gamaleldin Elsayed, Aravindh Mahendran, Sjoerd Van Steenkiste, Klaus Greff, Michael C Mozer, and Thomas Kipf. Savi++: Towards end-to-end object-centric learning from real-world videos. Advances in Neural Information Processing Systems, 0--28844-229555 2022. 2, 9, 15, 16, 17
[20] Justin Gilmer, Samuel S Schoenholz, Patrick Riley, Oriol Vinyals. and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pages 1263-1272. PMLR, 2017.
[21] Kristen Grauman, Andrew Westbury, Eugene Byme. Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995- 19012, 2022. 6, 20
[22] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, et al. Kubrie: A scalable dataset generator. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2022. 9, 15, 16, 17
[23] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[24] Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee. Flax: A neural network library and ecosystem for JAX, 2023. URL http://githut: com/ google e/flax. 18
[25] Georg Heigold, Matthias Minderer, Alexey Gritsenko, Alex Bewley, Daniel Keysers, Mario Lučié, Fisher Yu, and Thomas Kipf. Video owl-vit: Temporally-cooniitten open-world localiza- tion in video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13802-138111 2023. 2
[26] Allan Jabri, David Fleet, and Ting Chen. Scalable adaptive computation for iterative generation. In International Conference on Machine Learning. ICML, 2023.
[27] Allan Jabri, Sjoerd van Steenkiste, Emiel Hoogeboom, Mehdi S. M. Sajjadi, and Thomas Kipf. DORSal: Diffusion for Object-centric Representations of Scenes et al. International Conference on Learning Representutions, 2024.
[28] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning, pages 4651-4666 PMLR, 2021. 2
[29] Andrew Jaegle, Sebastian Borgeaud, Jean- Saanttaptsste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. International Conference on Learning Representations, 2022. 2.9
[30] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 12(4):1-14, 2023.3
[31] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations, 2015. 18
[32] Thomas Kipf. Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In International conference on machine learning, pages 2688-28 -697. PMLR, 2018. 3
12

---

[33] Thomas Kipf, Gamaleldin F. Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional Object-Centric Learning from Video. In International Conference on Learning Representations, 2022. 2, 9, 15, 17
[34] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan-Yew Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4015-40266 2023.
[35] Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence Jackel. Handwritten digit recognition with back-propagation network. Advances in neural information processing systems, 2, 1989.
[36] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances neural information processing systems, 1511111111138, 2020. 2
[37] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan, Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. In 2024 International Conference on 3D Vision (3DV), pages 800-809, IEEE, 2024.
[38] Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixe, and Christoph Feichtenhofer. Track- former: Multi-object tracking with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8844-8854, 2022.
[39] Thomas Norrie, Nishant Patil, Doe Hyun Yoon, George Kurian, Sheng Li, James Laudon, Cliff Young, Norman Jouppi, and David Patterson. The design process for google's training chips: Tpuv2 and tpuv3. IEEE Micro, 41(2):56-33, 2021. 18
[40] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. DI- NOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. 1, 2, 7, 8, 10, 19
[41] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International conference on machine learning, pages 4055-4064. PMLR, 2018. 2
[42] Tobias Pfaff, Meire Fortunato, Alvaro based simulation with graph networks. In International Conference Learning Representations, 2021.
and Peter Battaglia. Learning mesh-
[43] Jordi Pont-Tuset Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Corkine-Horuune and Lue Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint vrXiv1174.0067. 2017. 8, 16
[44] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017. 3
[45] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
[46] Kanchana Ranasinghe, Brandon McKinzie, Sachin Ravi, Yinfei Yang, Alexander Toshev, and Jonathon Shlens. Perceptual grouping in contrastive vision-language models. In Proceedings of the JEEE/CVF International Conference on Computer Vision, 2023.
[47] René Ranfil, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEF/CVF international conference on computer vision, pages 12179- 12188. 2021. 9, 20
13

---

[48] Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd Van Steenkiste, Filip Pavetic, Mario Lucic, Leonidas J Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. Advances in neural information processing systems, 35:9512-9524, 2022.
[49] Mchdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučié, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation transformer: Geometry- free novel view synthesis through set-latent scene representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6229, 2022. 2, 9, 18
[50] Mehdi SM Sajjadi, Aravindh Mahendran, Thomas Kipf, Etienne Pot, Daniel Duckworth, Mario Lučié, and Klaus Greff. RUST: Latent Neural Scene Representations from Unposed Imagery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17297-17306, 2023. 3
[51] Maximilian Seitzer, Sjoerd van Steenkiste, Thomas Kipf, and Mehdi S. M. Greff, Klaus Sajjadi. Videos. International
DyST: Towards Dynamic Neural Scene Representations on Conference on Learning Representations, 2024. 3
[52] Jonathan Shlomi, Peter Battaglia, and Jean-Roch Vlimant. Graph neural networks in particle physics. Machine Learning: Science and Technology, 2(2):02001,, 2020. 3
[53] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaík, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 2446-2454, 2020. 9, 16, 17
[54] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017. 1, 4, 17
[55] Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 14549-14560, 2023. 7, 10
[56] William F Whitney, Tatiana Lopez-Cuevara, Tobias Pfaff, Yulia Ruhanova, Thomas Kipf, Kim Stachenfeld, and Kelsey R Allen. Learning 3d particle-based simulators from RGB-d videos. In International Conference on Learning Representations, 2024. 3
[57] Yi-Fu Wu, Klaus Greff, Gamaleldin Fathy Elsayed, Michael Curtis Mozer, Thomas Kipf, and Sjoerd van Steenkiste. inverted-attention transformers can learn object representations: Insights from slot attention. In Causal Representation Learning Workshop at NeurIPS 2023, 2023. 2
[58] Ruibin Xiong, Yunchang Yang. Di He, Kai Zheng, Shuxin Zheng. Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In International Conference on Machine Learning, pages 10524-10533. PMLR, 2020. 17
[59] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 2
[60] Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-nerf: Point-based neural radiance fields. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 5438-5448, 2022. 3
[61] Fuzhao Xue, Valerii Likhosherstov, Anurag Arnab, Neil Houlsby, Mostafa Dehghani, and Yang You. Adaptive computation with elastic input sequence. International Conference on Machine Learning, pages 38971 138988. PMLR, 2023. 3
14

---

[62] Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, et al. Deep reinforcement learning with relational inductive biases. In International conference learning representations, 201 8.
[63] Fangao Zeng, Bin Dong, Yuang Zhang, Tiancai Wang, Xiangyu Zhang, and Yichen Wei. Motr: End-to-end multiple-object tracking with transformer. In European Conference on Computer Vision, pages 655-675. Springer, 2022. 2
[64] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: Deformable transformers for end-to-end object detection. In International Conference on Learning Representations, 2021.
[65] Daniel Zoran, Rishabh Kabra, Alexander Lerchner, and Danilo Rezende. PARTS: Unsuper- vised Segmentation With Slots, Attention and Independence Maximization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10439-10447, 2021. 2
A Limitations
Despite MooG's s simple and scalable design and our quantitative improvements over the on-the-grin baselines, there are number of limitations and open problems worth mentioning here.
The evaluations have used focused primarily on readout tasks that require tracking scene content (like objects or points) or capturing its geometry (like depth prediction). In contrast, there are many other possible downstream tasks might want video representation model support, including semantic segmentation, classification, or generation. Though in Figure 5 we have seen some preliminary evidence that MooG learns about structure in the scene that captures semantics, it is unclear to what degree these kinds of readouts are well-supported by OTG representations and compare on-the-grid alternatives.
Another potential limitation of OTG representations is the behavior of the tokens when scene content disappears or (re)appears. Indeed, it is reasonable assume that briddbased representation changes more gradually along the boundaries and provides a clearer expectation to the decoder terms of what content is encoded where. In contrast, OTG representation may latch onto entirely new scene elements as they (re)appear. This is general limitation of OTG representations, including prior approaches for learning slot-based objecttrepressentations [19, 33).
Finally, we have not investigated the scaling behavior MooG in-depth, both in terms of scaling the size of the model as well the amount of pretraining data.
B Broader Impact Statement
The focus of our work is training more capable video models for representation learning. These representations can be used for improving on down-stream tasks, as we have done for point tracking. depth estimation, and object tracking. There are many applications that could benefit from such improved capabilities, including in the domain of robotics and classic computer vision applications. As for many computer vision systems, these improvements may also transfer to applications with negative societal impact such as surveillance.
C Experiment Details
C.1 Datasets
MOVi-E The MOVi-E dataset is part of the MOVi benchmark that was introduced with the release
of Kubric [22], which is available under an Apache 2.0 license*. The MOVi-E dataset makes use of 380 high-resolution HDR photos backgrounds and 1028 3D-scanned everyday objects [15]. Each scene consists 10-20 static objects and between 1-3 dynamic objects that are tossed into the scene.
'nttps://githul http://tthhu..omggoooeeoeee hhap:/ogggeeeeeerrrr///tttt////a
15

---

The camera moves on a straight line with random constant velocity, whose starting point is sampled randomly half-sphere shell around the scene. The camera is always pointed towards the origin of the scene. The training set contains 97500 videos and the validation sets 250 videos, each of length 24.
Waymo Open Dataset The Waymo Open dataset [53] contains high-resolution videos (1280 1920 resolution). We follow the approach the same approach as in prior work [19] for processing the data, where we only use the video recorded from the front of the car, down-sampled to 128 x 192 resolution. The original dataset we use contains 798 training and 202 validation scenes that span 20 seconds each, sampled at 10fps. The Waymo Open dataset licensed under the Waymo Dataset License Agreement for Non-Commercial Use (August 2019)².
Kinetics700 v2020 The Kinetics dataset contains about 545,000 video clips of 250 frames at 25fps each. The videos depict people performing large variety of actions in diverse environments. We temporally sub-sample the clips randomly to the desired sequence length in training (8 frames in all experiments here). The videos are randomly cropped spatially and scaled to 128 x 128 pixels. Evaluation is done on the validation set which contains about 34,000 clips. The dataset was released under the Creative Commons license.
Ego4D Ego4D is a dataset of ego-centric videos. It has 24,000 long videos at 24fps which were sub-sampled into 2.4M 128 frames long clips. Of which further sub-sample 8 frames long clips during training. The videos are randomly cropped and down-sampled to 128 x 128 pixels. No other augmentation performed. The Ego4D dataset is used in accordance to the original dataset license
C.2 Training& & Evaluation
Training For our quantitative evaluation we primarily train MooG and grid-based baselines on videos from Kubric MOVi-E [22]. During training, replicate each video times to reduce bandwidth when data loading. For each video, we randomly sample a clip of 8 frames to train on, and apply random crop and color augmentations. For random crops, we ensure that crops span an area inbetween 0.3 and 2.0 times the starting image resolution, and have and aspect ratio that lies inbetween 0.5 and 2.0 times the starting resolution. After applying random crop augmentations to videos of 256 x 256 resolution, we resize the resulting crops to 128 x 128 resolution. For color augmentations, we randomly decide adjust the video brightness (up to a maximum of 32/ 22/255 relative change), saturation (between 0,6 and 1.4), contrast (between 0.6 and 1.4 times) and hue (up to a maximum relative change of 0.2) of the video with p = 0.8, followed by a p 0.2 chance of converting the video to grayscale.
On Waymo Open, we train on subsampled sequences of 16 frames using Inception-style random crop augmentations, where we ensure that at least 75% of the original frame is covered, before resizing back 192 1 128, followed by central crop 128 128.
Point Tracking On MOVi-E, we use point annotations computed in a similar manner as in Doersch al. [12]. We sample 64 points per frame from random grid having stride 4, while ensuring that at most 10% of the points cover a single object. We use the location of each point in the first frame as the query. and mask out points that are occluded throughout the entire sequence or occluded in the first frame (such that no query can be provided to the decoder). To evaluate each model we report the average Jaccard (AJ) as Doersch et al. [13], which evaluates both occlusion and position accuracy.
For evaluation on DAVIS [43] we use Vin--VidDDavis labeled Doersch et al. [12] for the DAVIS Validation set, consisting of 30 videos. We downsample 128 x 128 resolution. Since MooG is an auto-regressive architecture we adjust the query points to correspond to the location of the target points in the first frame. We evaluate TAP-Net and TAPIR on the same of points for fair comparison.
Monocular Depth Estimation On MOVi-E, use the depth annotations that are readily available for Kubric MOVi-E [22]. Following prior work [19], we transform depth values using log(1 + d). where d is the distance of a pixel the camera. As our evaluation metric we report the mean of
"nttpa://waymo. http///....///eeeeeeeeeeeee "nttps://ego4d data. wttpa/ooo/ppt//o///aa/e Ltwp:///eee---Lenss----reeer Draft. pdf
16

---

the absolute relative error (AbsRel), which is a standard metric in the monocular depth estimation literature [17]. We subsample the targets by a factor of 4 for evaluation.
On Waymo Open, we follow the procedure outline in Elsayed et al. [19] to obtain sparse depth values from LiDAR. Points in image space for which depth signal is available are masked out in the
metric and loss. For the DPT baselines, we sample random frames from the available training and evaluation videos for train and eval respectively. For evaluation we directly central crop to 128 128, and evaluate on 10 frames at the original resolution.
Object Tracking On MOVi-E we train MooG using the box annotations available in Kubric MOVi-E [22]. As queries we use the location of each box in the first frame (represented as Uminn Tmin: Umar, Tmaa), and we report the average IoU across the sequence (excluding the first frame in the sequence for which ground-truth is provided) as in prior work (19,33)
On the Waymo Open dataset [53] evaluate on sequences of 8 frames in the zero-shot transfer setting to compare to grid-based baselines. To compare to SAVI++, we train on Waymo open at resolution 192 x 128 following the procedure outlined in Elsayed al. [19] with random augmentations for sequences of 16 frames, and evaluate on sequences 10 frames. In both settings, we discard bounding boxes that cover an area of 0.5% during training an evaluation, and keep maximum of 10 boxes during training and for evaluation.
C.3 Model Details
C.3.1 MooG
Network Architecture The architecture of MooG for self-supervised training on video is divided into four components: encoder, corrector, predictor, and decoder, totalling approximately 35M parameters. To encode each frame, we use a convolution encoder as outlined in Table 5, followed by Fourier positional encoding using 20 Fourier bases, which we add to the encoder output features using a single dense layer as a projection. For comparing to domain-specific baselines (TAPIR, DPT, SAVi++, etc.) we also explore slightly stronger backbone, where we omit the striding in the last layer of the CNN and concatenate the positional encoding (as opposed to first projecting and then adding).
At timestep O, we initialize the latent representation having 1024 tokens of size 512 by drawing from standard normal multivariate Gaussian distribution scaled by a factor of le-4. The predictor predicts the state of the tokens for the next time step, which uses a 3 layer self-attention transformer as outlined in Table 7. The corrector updates the prediction based on the encoded observation, which implemented as a 2 layer transformer that uses both self-attention and cross-attention (Table 7), where queries are computed from the predicted tokens, and the key and values are computed from the encoded observation (32 x 32 patches). We apply Layer Normalization [1] ] to the output of the corrector, which found to be important for long-sequence rollouts. To decode the back to pixel space, we use a layer cross- coos-attention transformer as outlined in Table 7. Queries are computed from the coordinate grid (after concatenating a Fourier positional encoding with 16 bases), and keys and values from the predicted tokens.
For the down-stream readouts we make use of the same cross-attention Transformer backbone for each readout as seen in Table For spatial readouts (like depth), we follow the design of the pixel decoder where queries are computed from the coordinate grid (after concatenating Fourier positional encoding with 16 bases), and keys and values from the tokens (here using both the predicted and the corrected tokens). For tracking based tasks (like points and boxes prediction), we associate 312-dimensional latent state with each track that is initialized from the first-frame queries using Fourier positional encoding as before, followed by a two-layer MLP to project to the desired dimensionality. The transformer backbone in Table 6 is used to update these latent states at each step by ross-attending into the predicted and corrected tokens, using the latent states as queries. A per-latent predictor (implemented by a 2-layer MLP) acts as a predictor to initialize the tokens for subsequent steps.
For our transformer implementation, we mostly follow the standard design in Vaswani et al. [54], but using the pre-LN configuration as described in Xiong et al. [58]. We also include a few recent improvements based on Dehghani et al. [11]. In particular, we apply RMS norm to the queries and keys before computing the attention weights, and execute the cross- and self-attentions paths
17

---

Table Encoder CNN.
Table 6: Readout Transformer Backbone. XA: Cross-attention.
Features
Kernel
stride
SA: Self-Attentio..
64
3 3
Type
Type
Layers
QKV size
Heads
MLP size
128
3 x 3
x 1
128
3 3
1
Points
XA
3
64 x 8
8 2048
256
3 3x 3
2 2
Depth
XA
3
64 x 8
8
2048
256
3 3
1x
512
3 x 3
2 2
Boxes
XA
3
64 X
8
2048
Table 7: Transformer Layers. XA: Cross-attention. SA: Self-Attention.
Component
Type
Layers
QKV size
Heads
MLP size
Corrector
XA & SA
2
64 x 8
8
2048
Predictor
SA
3
64 x 4
4
2048
Decoder
XA
6 64 2
2048
in parallel (where applicable), but not the mlp path. Finally we apply another layer of Layer Normalization [1] to the output of the transformer.
Subsampled decoding Instead of decoding the full image each time-step during training we sub-sample the coordinate grid and embed the result. We first generate sub-sampled grid of pixel coordinates N4xwx2 where h H/S, w: W/S and S is the sub-sampling factor. In order to prevent over-fitting to specific coordinate grids we randomly offset G by random integer between O and S. We use Fourier positional embedding to embed G into a flattend embedding P E I (hhr) x and use the result as the initial queries in the multi-layer cross- attention transformer D, the output of which is projected to 3 dimensions and reshaped back to image size to form the decoded frame X, E R Using sub-sampling factor S > allows the model to decode only subset of the pixels in each frame reducing computational demands significantly.
Training We train MooG raw video data (see below for datasets used) for 1M steps using Adam with Nesterov momentum [16, 31] using a cosine decay schedule that includes a linear warm-up for 1000 steps, peak value of lc-4 and end value of le-7. Updates are clipped using maximum global norm of L.O, and we use 31 = 0.9, 82 = 0.95 inside Adam. We use a batch size of 128 for most of our experiments, and batch size of 256 for the comparison to domain--spcciio Tables & 4. On the Waymo Open dataset we train for 500K steps for end-to-end depth prediction and for 250K steps for end-to-end box prediction compare to SAVi++. Our MooG runs make use of 64 TPUv3 [39] chips having 32GiB memory, which each take about 48 hours for 1M steps. We implemented MooG JAX [3] using Flax [24].
baselines in
Our main training loss is a simple L2 reconstruction loss, which we compute for a subset of the pixels inspired by Sajjadi et al. [49]. Down-sampling is implemented via striding using factor of 8 and having random offset during training. For depth readouts we use a masked 1.2 loss (based on the given location) using the same Nown-sampling approach. Similarly,
availability of the depth signal for box readouts we use an L2 loss between the prediction and the normalized box coordinates. Finally, for point readouts, in addition to predicting their values (normalized image coordinates), we also predict whether they are visible and measure of certainty of the prediction; for the purposes of evaluation, we only output point as visible the model predicts (having confidence over 50%) that it is both visible and certain of the position. This set-up is identical to the one used in Doersch et al. [13] and we adopt the same combination of Huber and Sigmoid Binary Cross Entropy losses for these terms. We amplify the Huber loss that measures the prediction accuracy by factor of 1000 (note coordinates are normalized between 0 and 1) relative to the visibility and uncertainty losses. For points that have left the scene only use the visibility part of the loss.
18

---

C.3.2 Grid-based baselines
Grid (Rec.) We use the same approach to training and evaluating the grid-based baselines, which only differ in their network configuration. particular, the Grid baseline uses the same encoder described in Table 5 and decoder and readouts described in Tables 6 & 7. The key difference is that it does not include the corrector or predictor from Table 7, but treats the output of the encoder as the representation. To make up for the lost parameter count, augment the encoder with self-attention Transformer having 3 layers, QKV size of 64 x 8, heads, an MLP size of 2018 and a hidden size of dimensionality 512.
The grid- grid-based baseline with recurrence (Grid Recu:) also uses the same encoder described in Table 5 and decoder and readouts described in Tables 6 & 7. It does not make use of the predictor, but re-purposes the corrector as such. In particular, tokens for time-step are initialized from the output of the encoder (yielding similar amount of 1024 tokens of dimensionality 512) in an on-the-grin manner. The corrector uses these tokens as queries to cross-attend into the corrected tokens from the previous timestep, which implements the recurrence. To make up for the lost parameter count, augment the encoder with a self-attention Transformer having layers, QKV size of 64 x 8, 8 heads, an MLP size of 2018 and hidden size of dimensionality 512. The decoder reads from the output of the encoder (the tokens initialized on-the-grid) as is the case for the Grid baseline, while the other readout modules have access the corrected tokes as well (similar to in MooG).
To evaluate on DINO [5, 40] representations we make use of the official pretrained check- points available online?. We use the base ViT size, which exceeds MooG in the number of parameters. To evaluate on videos, we compute DINO features for each frame (after resizing to 224 x 224 and normalizing pixels the ImageNet value range) that are fed into the same readout decoders outlined in Table 6. We do not backpropgate the task-loss into the encoder in the frozen setting, while in the end-to-end setting we do.
DINO
To evaluate on VideoMAE v2 representations we consider 3 pub- licly available checkpoints the ViT-Small wvit_s_k710__ _ll_foomggiant and ViT-base crit_b_a710 _11_____f_________1 wwwm_rriaannnno variants, which contain 22M and 83M parameters respectively, as well as a ViT-giant variant wtt_gghhbbrrddp_111000 smaller variants were obtained by distilling the predictions of the ViT-giant model. We note that MooG contains approximately 35M parameters, which includes the pixel decoder. Another important difference to highlight is that the ViT-giant teacher network was finetuned for action recognition, while the ViT-small and ViT-base models were initialized from finetuned networks as well. To evalu- ate on videos, we first resize the video spatially to 224 x 224, after which we apply the VideoMAE v2 encoder to obtain feature representation. Next, we upsample the feature representation temporally by a factor of two to recover the original sequence length. The resulting representations are fed into the same readout decoders outlined in Table 6. We do not backpropgate the task-loss into the encoder.
VideoMAE
k71___k11__________ 1f) containing 1B params. The
C.3.3 Domain-specific Baselines
TAP-net For TAP-Net, we use the default released model from TAP-Vid paper [12], which encodes every frame (including query frame) at 256x256 resolution, runs through a TSM-ResNet backbone (with time shifting) and produces 32 x 32 feature map. The feature map then is used to compute the correlation volume and output predict location and occlusion flag. No uncertainty estimation is used here. The TAP-Net is trained on the Kubric MOVI-E dataset using 24 frames and 256 points per frame. There is no temporal processing in the model. Every frame is treated independently and only correlation volume is used for prediction.
TAPIR For TAPIR, we mainly use the online model variant for comparison due to the autoregressive nature of MooG and using query points that are always sampled from the first frame, again using the publicly-released model. The online TAPIR model use causal convolutions that only receive context feature from history for updating the current frame prediction during iterative refinement. For the
"Checkpoints be downloaded from / http:://ggggtu..c.///cceetttteee http:///w/..cceeeeeerrreaataaata and https: //github coon/facebookesseadde /dioov2 respectively.
'Checkpoints can be downloaded by following the instructions at .dd
19

---

backbone, it uses 18 18-layer ResNet (with strides 1. 2, 2. 1 and instance normalization) instead of TSM- ResNet, which is why it is frame independent but yields comparable performance. The backbone encodes 256 x 256 resolution and output two feature maps at 32 x 32 and 64 x 64 resolutions. The 32 32 feature map is used for global correlation volume same as TAP-Net, then the 64 x 64 is used with 32 x 32 together for iterative refinement. It runs 4 iterations for the refinement. The model is trained on the TAP-Vid Panning MOVI-E dataset from the paper, using 24 frames, 256 points per frame, and including the same color augmentations.
For completeness we have also computed the offline default TAPIR results. Indeed, we find that online TAPIR works reasonably close to the offline variant (obtaining 0.825 AJ on Davis-8 and 0.584 AJ on Davis-full), likely because all query points are all from first frame.
DPT To provide reference point to standard architectures employed for monocular depth predic- tion we make use of the Dense Prediction Transformer (DPT) [47] architecture. We follow a similar set-up as in Ranftl et al. [47], where we configure DPT with four reassemble and fusion blocks. Each block processes a 16 x 16 feature map from a pre-trained ViT at multiple spatial resolutions 4 x 4, 8 x 8, 16 x 16 and 32 x 32. We use feature dimensionality of 256 for each block and 128 dimensionality for the depth estimation head. Similar to Dehghnn et al. [11] we reuse the same ViT feature map at each stage. VIT features maps are obtained by upsampling the input frame to 224 x 224 for VIT-E and ViT-22b, which use a patch size of 14, and to 256 x 256 for ViT-L, which uses a patch size of 16. We use ReLU normalization on the output of DPT to ensure that it is non-negative, and adjust the value range by inverting the model prediction. The output resolution of DPT is at 224 x 224 and upsample the depth targets and mask accordingly. DPT is trained and evaluated on randomly sampled video frames of Waymo using the same crop augmentations as mentioned in the previous section.
C.4 Qualitative Experiments
For the qualitative experiments in Section 4 we trained MooG model on dataset mixture from Ego4D [21] and Kinetics700 [6]. The architecture used is identical to other models specified here. The model was train on 8 frame long sequences randomly subsampled from the longer sequences in the data. The model was trained with batch size of 256 from 1M steps and takes about 3 days on 8 3x5 8 TPU cores to train, though results are already quite good after a few hours of training.
To generate the attention maps in Figures 4 and 7 and the supplementary material, we first unroll the model on a test sequence to produce next frame predictions this can be done on sequence lengths much longer than the training sequence length. For each frame, we take each decoded pixel coordinate (remember these are the queries that are used as input in the decoder) and average the attention for each token across all decoder layers. This tells us how much attention a specific token contributes when decoding a specific image location at specific time step. We can visualize specific token attention maps through time as in Figure 4 or we can take the max across all token for specifie location and colour code the tokens to get more complete view of which token is most responsible for every image location across time 7 and videos in the supplementary material. Here the "grid-tokens" are obtained from the Grid Rec. baseline, trained the same manner as MooG.
PCA visualizations (Figure 5 were generated by unrolling the model on a batch of 24 sequences, each 12 frames long. We take the set of predicted tokens and concatenate all of them across the first dimension, resulting in 294, 912 x 512 matrix. We then perform PCA analysis with this data. taking the leading 64 components, resulting in a 294, 912 x 64 matrix which we reshape back the original unrolled size up to the channel dimension to size 24 X 12 X 1024 x 64. In order to visualize these in image space we use the argmax token for each image location. We observe (manually) that the first 20 or so components correspond to the positional encoding and contain no or very little scene relevant content. Many of the components are informative, however - corresponding to different elements in the scene, or properties such as colour, motion etc. In order produce the visualization in Figure we chose 3 of the "interesting" components and place them RGB channels, visualizing the result video.
20

---

0.6
Jaccard
Avg Jaccard
0.4
Tokees
Readout Layers
512
2048
0.0
MOVIE
DAVIS-8
Dataset
Dataset
Figure 6: We vary several key hyper--pramee pyrameearameters of MooG and report results for the end-to-end point tracking setup.
D Additional Results
Analysis We report the results for hyper- syper-parammetr study of MooG in Figure 6. Here MooG was trained end-to-end on MOVi-E together with the point tracking objective, identical to the setup for Table 2. We observe marginal improvement using single transformer layer in the point readout (Table 6) on MOVi-E, but decrease on DAVIS-8. 8. When changing the total number of tokens, we very slight improvement having additional tokens. We report qualitative results for changing the number of tokens at inference time in Figure 7.
We evaluated the effect of using different number of frames during training. We trained MooG end-to-end on MOVI with downstream tasks as in Table 2. Here we observed that training on 8 frames is significantly better than training on frames, which is in turn much better than training on 2 frames in the case of point the DAVIS-full evaluation
and 8 frames and evaluated on the point tracking and depth estimation
tracking. In particular, The differences between the three become more pronounced the longer the evaluation sequence is presumably the model learns longer term dynamics when training longer sequences, which improves its generalization ability to different sequence lengths. For depth evaluation we do not observe major difference between and 4 frames (but both much better than 2 frames), obtaining AbsRel error of 0.03, 0.03 and 0.19 respectively on MOVi depth.
obtain 0.51, 0.36, 0.16 AJ respectively.
Variance Many of the results reported for MooG are for single seed. To provide some indication of variance, we evaluated 3 seeds for the MooG variant reported in Table 3. We observe a standard error of the mean (SEM) of approx. 1 % (absolute) for all reported metries.
21

---

argmax attn.
blended
pred
ground truth
1024 tokens
8080
8000
512 tokens
6AA0
256 tokens
8000
Figure 7: MooG can be instantiated with a different number of tokens test time without retraining. Because the model architecture is independent of number of tokens and image resolution, we are free choose the number of tokens used at evaluation time. The model depicted here was trained with 1024 tokens. We instantiate the model with 256, 512 and 1024 tokens. As can be seen the model has no problem adapting to a different number of tokens, producing good predictions while tokens cover more area in the image as needed.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We extensively evaluate our approach in Section 4, both qualitatively and quantitatively and compare to relevant approaches in the literature.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
22

---

Answer:
[Yes]
Justification: Limitations are discussed in Section A.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
We do not provide any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: detailed overview of relevant implementation details concerning network architectures, optimization, datasets, and evaluation are described in Appendix C.
23

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-soure dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The data we use publicly accessible and we intent to release the main model code upon acceptance of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / mnips.cc/pbbli httpppuubucuicie...p/pppppp gutg//uieeeeeeeeee hot.e/Sumiiiisiiioooooo..ooo ssionPolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
24

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: An overview of all relevant details needed to understand the results provided in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Because of computational constraints we were not able provide multiple seeds for all results. Where possible, such as for the DPT baselines and DINO baselines we have reported results for 3 seeds. Further, have given an indication of the variance of MooG Appendix D, which is small.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Yes, details are provided in Appendix C.
25

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httpp////errp..ccpppiiiiiic
cc/ Teabblic/tthcscuieee
Answer: [Yes]
Justification:
We have reviewed the code of ethics and comply.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: a brief discussion is provided in Appendix B.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
26

---

Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Discussed in Appendix C.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URL.
The name the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets introduced.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
27

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing or research with human subjects was conducted.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This research does not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "QwBnKbcuVs", "reviewer": "Reviewer_MZrN", "summary": "This paper proposes a field-based method for video representation learning called MooG. Instead of propagating a discretized grid of features for every pixel location in the video, the method updates an arbitrary set of state tokens. These state tokens are used to parametrize a context dependent PerceiverIO-style “field”: a network that decodes the RGB values at a particular location (x, y) in the target frame conditioned on the “predicted” state tokens from previous frames and a position encoded (x, y) input. Since the state tokens parametrize a field, they are dubbed “off the grid”. The paper proposes to train such a model in a recurrent fashion using next frame prediction. Further, it describes methods to read-out the (potentially pre-trained) representations for downstream tasks. Results show qualitatively that the off-the-grid representations are “well behaved” through compelling visualizations and quantitatively work better than on-the-grid counterparts for a range of dense visual tasks, such as object tracking and depth prediction.", "questions": "-\tIs the paper the first to show token binding to task related activities emerging unsupervised in a field-based transformer? \n-\tAre the baselines chosen for point tracking (TAP-Net, TAPIR) and for depth prediction (DPT, etc) the state of the art results in the domains? (Given MooG is a general purpose video representation learning method, I agree it would not be fair to compete against the best task-specific engineered methods, but a discussion on limitation and deficit to state of art (if any) would be useful).", "limitations": "The limitations section is adequate and thorough (discussing disocclusions, occlusions, and alternate coarse / fine tasks possible on video), and future directions for improvement are also duly noted in Appendix A.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "-\tThe idea of combining recurrent state token estimation together with an off-the-grid field-like representation that is repeatedly queried across time and trained with next frame prediction is interesting and novel. \n-\tThe methods section is simple and intuitive and the writing is clear and concise. At important steps (such as 130 – 132) where there could be room for confusion (eg. decoding from predicted tokens vs correct tokens), the details are highlighted to elucidate the key elements that make the method work. \n-\tA wide range of experiments are shown for three important dense prediction tasks: monocular depth estimation, points tracking and boxes tracking. Baselines are chosen appropriately, representing widely used and ubiquitous recent methods in the respective fields. Quantitative experiments show that MooG is a powerful representation learning method, and that it outperforms baselines in the frozen setting, which should be the dominant paradigm to evaluate a video pre-training model. \n-\tThe qualitative visualizations are impressive and creative. That tracking (in the token attention sense) and segmentation features (in the PCA sense) emerge from using the limited token and off-the-grid inductive bias are exciting results. The paper shows many examples of these results.", "weaknesses": "-\tWhile appropriate baselines are chosen for all downstream evaluations, and comparison is made with on-the-grid representations that simply propagate features, baselines with architectures similar to MOOG have not been explored in a lot of detail. In the related work, the “off-the-grid” similarity between Perceiver IO and MooG is acknowledged, with the key difference being that MooG additionally implements recurrence of state tokens.  One baseline that would be a closest comparison would be to train a Perceiver IO like model, except with spatio-temporal queries: in particular, this would entail conditioning on time and position. This could be trained simply like an auto-encoder (essentially reducing to Perceiver IO for video), or with next frame prediction. \n- Papers that use similar architecture but specialized for domain-specific video prediction tasks are missing from the related works. While MooG is a general purpose representation learning method, a thorough discussion of some domain specific models would be important to highlight key differences between those papers and the potential general purpose nature of MooG representations. Some of these papers are already cited (eg. CONDITIONAL OBJECT-CENTRIC LEARNING FROM VIDEO, Kipf et al. 2022), but are missing deeper discussion on fundamental differences to MooG.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and constructive comments. We are pleased to hear that you found MooG interesting and novel, and recognized its considerable improvement over baselines in the frozen setting. We are glad that you found the writing and visualizations of high quality. \n\n**“One baseline that would be a closest comparison would be to train a Perceiver IO like model, except with spatio-temporal queries”**\n\nThank you for suggesting this baseline, which is indeed a very sensible one. In fact it is close to one variant of the model we experimented with in the early phases of the project. At that time, we experimented with a non-autoregressive version of our model that mapped all encoded frames in parallel to a joint set of latent tokens (a single set for multiple time steps) and finally replicated this set across time steps with added temporal position encoding for decoding.\n\nSuch an approach is in fact very similar to Perceiver IO for video (with some small design differences in encoder, decoder and position encoding). In preliminary experiments, we found that when evaluated on 8-frame DAVIS point tracking, both models performed roughly equally well. However, we faced challenges continuing the point tracking predictions of the parallel (Perceiver IO-style) model when moving beyond the 8-frame window that it was trained on. On the contrary, this was straightforward to achieve with MooG due to its autoregressive formulation.\n\n**“Papers that use similar architecture but specialized for domain-specific video prediction tasks are missing from the related works”**\n\nThank you for pointing this out. We will deepen the discussion of prior works in the related work section with regard to domain-specific approaches (such as SAVi), and highlight similarities and differences to MooG where relevant.\n\n**“Is the paper the first to show token binding to task related activities emerging unsupervised in a field-based transformer?”**\n\nTo the best of our knowledge, this paper is among the first to *explicitly study* this relationship in standard transformer architectures trained in a self-supervised fashion. That said, it is difficult to make a precise claim about this due to numerous (domain-specific) prior works such as SAVi and PARTS exploring architectures that display similar behavior at a coarser scale (i.e. object-level binding) and are at least “transformer-like”. It is also worth mentioning the recent body of work on 4D Gaussian splatting (e.g. Luiten et al., Dynamic 3D Gaussians, 2023), which aims to learn explicit 3D Gaussian representations that track elements of a dynamic scene, typically by using more explicit geometric constraints instead of generic transformer-based architectures.\n\n**“Are the baselines chosen for point tracking (TAP-Net, TAPIR) and for depth prediction (DPT, etc) the state of the art results in the domains? […] a discussion on limitation and deficit to state of art (if any) would be useful).”**\n\nTAPIR is very close to state of the art in the point tracking domain, and only outperformed by the very recent BootsTAP [1], which extends TAPIR by augmenting the pretraining dataset with unlabeled YouTube videos. The DPT architecture itself is commonly used in competitive monocular depth-estimation approaches such as the MiDaS line of work [2]. However, to achieve competitive performance, such approaches are usually pre-trained on multiple different depth estimation datasets and adopt more sophisticated losses to handle depth targets from varying sources in a single model. More recently, diffusion-based approaches have shown great promise for monocular depth estimation [3, 4], which move away from the DPT architecture. \n\nWe agree that it would be useful to contextualize these baselines a bit better relative to SOTA in the field, and we will update the paper accordingly.\n\n[1] BootsTAP: Bootstrapped Training for Tracking-Any-Point\n[2] MiDaS v3.1 -- A Model Zoo for Robust Monocular Relative Depth Estimation\n[3] Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation\n[4] Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MRkrgMv3oz", "reviewer": "Reviewer_r1sm", "summary": "The authors present a self-supervised video representation learning strategy. A grid-structure free feature extractor is trained using a next frame prediction objective. A corrector module extracts per-frame features. A predictor module predicts the next frame features. A decoder (with suitable grid free architecture) reconstructs frames. A sparse (for efficiency) reconstruction loss is applied on decoder output as the learning signal. \n\nThe method, MooG, appears to bind features to specific structures in the visual inputs and tracks these across frames. The authors present evaluations for tracking and depth estimation.", "questions": "See weaknesses", "limitations": "Ok", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. Interesting and novel idea for self-supervised representation learning from videos\n2. Clear explanation of methodology and through details", "weaknesses": "1. **Evaluations:**\nLearned representations are evaluated on a) niche tasks, b) with only image SSL baselines. Proposed method uses videos to learn unlike the baselines. Please compare a) on generic tasks (evaluate learned representations for classification, object detection, or segmentation) and b) compare against video SSL methods (on the identified tasks and generic ones).  \nIn fact, in introduction, the authors argue that tasks like object detection and segmentation need the kind of off-the-grid architectures being proposed. Evaluation on such tasks will strengthen the authors case. \n\n2. **Prior Video SSL works:**\nIt is unclear how these representations compare against video SSL methods [2,3,4,5]. Maybe try to apply some of these methods on the selected tasks to compare how MooG representations compare to them? Or evaluate MooG on tasks these are commonly used for.  \n\n3. **Related Works:**\nConsider discussing [1] which explores a similar idea of next frame prediction in a latent space to learn good representations. Also [6] which learns local features using language which shows DINO like grouping / tracking behaviour. \n\n\n[1] Sequential Modeling Enables Scalable Learning for Large Vision Models, CVPR 2024\n[2] Time Does Tell: Self-Supervised Time-Tuning of Dense Image Representations, ICCV 2023\n[3] Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video, ICLR 2024\n[4] Self-supervised Video Transformer, CVPR 2022\n[5] VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training, NeurIPS 2022\n[6] Perceptual Grouping in Contrastive Vision-Language Models, ICCV 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and constructive comments. We are pleased to find that you consider MooG to be an interesting and novel idea for self-supervised representation learning from videos, and that the presentation was clear to you.\n\n**“Please compare a) on generic tasks (evaluate learned representations for classification, object detection, or segmentation)”**\n\nThank you for suggesting this comparison. MooG was designed with spatio-temporal grounding in mind, without placing a focus on high-level semantic features, and we chose evaluation domains that would demonstrate this most clearly. Keeping this mind in, we chose point tracking, box tracking, and depth estimation as low-level tasks that require little semantic understanding of a scene, which aligns well with our goal of learning scene-grounded, temporally-consistent representations. In preliminary experiments we observed that features learned by MooG are not well-suited for semantic tasks such as action classification in combination with linear readout heads. This is potentially due to the local nature of the representation or the simplicity of the readout setup we tried. We are happy to include a discussion of these preliminary observations in the paper and make suggestions for improvements for future follow-up work.\n\n**“It is unclear how these representations compare against video SSL methods [2,3,4,5]. Maybe try to apply some of these methods on the selected tasks to compare how MooG representations compare to them?‘)”.**\n\nThank you for pointing this out. We have now added a comparison to VideoMAE v2 in the supplementary pdf attached to the main response, which is a more recent version of [5].\n\nWe consider representations from 3 [publicly available VideoMAE v2 checkpoints](https://github.com/OpenGVLab/VideoMAEv2/blob/master/docs/MODEL_ZOO.md): the ViT-small and ViT-base variants, which contain 22M and 83M parameters respectively, as well as a ViT-giant model (1B params). The smaller variants were obtained by distilling the predictions of the ViT-giant model. We note that MooG contains approximately 35M parameters, which includes the pixel decoder.\n\nThere are other differences that further make it difficult to compare, for example MooG was pre-trained on MOVI-E, while the VideoMAE models were trained on a superset of all Kinetics videos. MooG was trained purely in a self-supervised manner, while the VideoMAE v2 teacher network was finetuned for action recognition, and the ViT-small and ViT-base models were initialized from finetuned networks too.\n\nKeeping these differences in mind, we observe how MooG performs significantly better than the ViT-small and ViT-base sized models that offer similar parameter counts. Compared to the teacher ViT-giant model that has 30x more parameters, MooG performs very well: it is better on MOVi points and Waymo boxes, but worse on MOVi depth and DAVIS points. Both methods perform the same on MOVi boxes. We are hopeful that scaling MooG to 1B+ parameters and larger scale pretraining may lead to further improvements, though we leave this for future work as mentioned in our limitations section.\n\n**“Related Works: Consider discussing [1] which explores a similar idea of next frame prediction in a latent space to learn good representations. Also [6] which learns local features using language which shows DINO like grouping / tracking behaviour.”**\n\nThank you for pointing this out. We will make sure to contextualize our work further w.r.t. autoregressive next-frame prediction methods (e.g. regarding the reference you provided which uses a grid-based VQ-GAN representation [1]), and with perceptual grouping / object-centric methods in our related work section. In short: end-to-end object-centric grouping methods are indeed closely related (as alluded to in our related work section), but this line of work typically aims at grouping entire objects or well-defined parts into a single object – this includes methods such as GroupViT [7] and the paper you referred to [6]. For more general, fine-grained spatio-temporal tasks such as point tracking or monocular depth estimation, enforcing an object bottleneck (e.g. 1 token per object) may be too restrictive: we find that downstream task performance significantly increases by providing the model with more capacity in terms of number of “off-the-grid” vectors/tokens that it can use to encode the video. \n\n\n**References**\n\n\n[1] Bai et al., Sequential Modeling Enables Scalable Learning for Large Vision Models (CVPR 2024)\n\n[2] Salehi et al., Time Does Tell: Self-Supervised Time-Tuning of Dense Image Representations (ICCV 2023)\n\n[3] Venkataramanan et al., Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video (ICLR 2024)\n\n[4] Ranasinghe et al., Self-supervised Video Transformer (CVPR 2022) \n\n[5] Tong et al., VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training (NeurIPS 2022) \n\n[6] Ranasinghe et al., Perceptual Grouping in Contrastive Vision-Language Models (ICCV 2023)\n\n[7] Xu et al., GroupViT: Semantic Segmentation Emerges from Text Supervision (CVPR 2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j1XvkFDGtC", "reviewer": "Reviewer_RPGH", "summary": "This work discusses Scene-Grounded Video Representations. Compared with current vision models that make each layer consist of tokens organized in a grid-like fashion, the authors introduce Moving Off-the-Grid (MooG), a self-supervised video representation model that proposes an alternative approach. The novelties are: Introducing Moving Off-the-Grid (MooG), a novel transformer-based recurrent video representation model designed to learn off-the-grid (OTG) representations using a straightforward next-frame prediction loss; illustrating how this representation enhances a range of downstream vision tasks, including point tracking, monocular depth estimation, and object tracking.", "questions": "Please refer to the Weaknesses", "limitations": "The authors do some analysis and the work will not have any potential negative societal impact.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ This work allows individual tokens to consistently track elements of the scene through videos of arbitrary length and “anticipate” where a scene element will be observed next.\n+ The authors designed MooG to process an unlimited number of video frames while consistently maintaining a scene-grounded representation of the video.\n+ Tokens within the latent state are inherently associated with particular pixel positions. The model can obtain corresponding token representations based on different features of the image.", "weaknesses": "- In line 135, you said “It is, however, crucial that the image is decoded only from the predicted state—decoding the current frame from the corrected state reduces the problem to simple auto-encoding and hurts representation quality considerably.” So how to prove the effectiveness of the  Corrector module? \n- What’s the difference between MooG and Deformable DETR? Does the Corrector play a same role as the offset modal in Deformable DETR? Besides, do you treat the states as the tokens? \n- It seems that you used transformer (attention module) in Corrector and Predictor, and how does the algorithm perform in tracking tasks? What is its speed? Can it meet real-time requirements? How’s the performance in the Multi-object tracking field?\n- In table 3, the performance of TAPIR on Davis-full is better than MooG, so could you please provide more detailed explanations?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and constructive comments.\n\n**“how to prove the effectiveness of the Corrector module?”**\n\nThank you for your comment. Note that the corrector is the only part of the model that has access to the observation at the current time-step, i.e. to “correct” the prediction, and hence is the only channel for integrating new information. However, it also introduces a shortcut that bypasses the previous state when using the corrected state for reconstructing the observation. This is what necessitates both a predicted state (to be used for reconstruction that requires temporal prediction) and a corrected state (to be used for readouts that incorporate the current observation and is maximally informative). \n\n**“What’s the difference between MooG and Deformable DETR? Does the Corrector play a same role as the offset modal in Deformable DETR? Besides, do you treat the states as the tokens?”**\n\nOne of the key differences between MooG and Deformable DETR is that Deformable DETR utilizes sparse spatial sampling of deformable convolution on the 2D feature map (with explicit multi-scale features), while MooG models the image as an off-the-grid set of latent representations. There is no sparsity nor offset in the corrector in MooG, whose purpose is to “corrrect” the state predicted from the previous timestep based on the current observation. The corrector cross-attends to the feature maps and updates the set of latent features through transformer layers directly. MooG is easier to implement, it is mainly attention blocks, there is no need for any specialized kernel (while deformable DETR requires specialized [deform_attn_cuda.cu](https://github.com/fundamentalvision/Deformable-DETR/tree/main/models/ops/src/cuda)). \n\n**“It seems that you used transformer (attention module) in Corrector and Predictor, and how does the algorithm perform in tracking tasks? What is its speed? Can it meet real-time requirements?”**\n\nMooG is a very lightweight model, totalling approximately only 35M parameters during training (and fewer during inference). In particular, the transformers layers you mentionhave only 2 or 3 layers (predictor and corrector), while the conv-net is only 6 layers which should be no problem to run at real time on modern hardware.\n\nFurther, for readouts (like point/box tracking) the pixel decoder is not needed at inference time. The model performs well even when reducing the number of available tokens considerably (as shown in Figure 6 in Appendix C), which further improves speed.\n\n**“In table 3, the performance of TAPIR on Davis-full is better than MooG, so could you please provide more detailed explanations?”**\n\nThank you for pointing this out. Indeed, because of the auto-regressive nature of the readout module, error accumulates when unrolling readouts over long sequences. This is not an issue with the base representation (as the corrector receives continuous observations) but we have observed how the read-out module ends up drifting eventually, i.e. as the initial conditioning signal (box or point tracks in the first frame) becomes less informative.\n\nWe note how TAPVid and TAPIR are domain-specific SOTA approaches that leverage specialized architectures, such as explicit cost volumes, to mitigate such issues for a particular domain. They further have access to both past as well as future states, while MooG is a causal model with access only to past frames, i.e. it can be used for online tracking. MooG learns representations that are useful for a variety of downstream tasks, and we have not incorporated domain-specific improvements in the readout decoder. One interesting direction for future work might be how to make MooG representations perform competitively on one single domain by incorporating more specialized components in the readout decoder. We will update the draft to give more context to this comparison and why we do not necessarily expect to beat domain-specific SOTA methods with MooG.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MjgfDbe9Wn", "reviewer": "Reviewer_ykZT", "summary": "This work presents a video representation learning approach where tokens are decoupled from explicit grid locations in the video sequence.  Rather than simply extract patches to construct tokens and apply self-attention across blocks of frames, this proposed model predicts sub-pixel motion given a history of frames to advect latent features throughout the video.  Once those grid-aligned features are advected to the next frame to their non-grid aligned positions, cross attention can be applied between the grid aligned observed features and the advected features in order to correct the result.\n\nThe authors claim three contributions:\n* Introduction of MooG\n* Qualitative validation of the approach\n* Quantitative evaluation on a number of different downstream tasks", "questions": "I didn't get a good impression of what kind of motion dynamics the predictor properly supported.  It looked like there was some overshooting happening in the gifs, but it's not clear to me if this is a real issue.  Can this be clarified?\n\nOne ablation that would be useful is to change the backprop through time length.  It would be great if there was evidence that only 2 or 3 frames were enough to get the predictor to estimate dynamics.  Another question could be, what kind of dynamics are modeled with 2, 3, 4, 8, etc. frames and are there diminishing returns?  Can this be clarified?\n\nGiven that this model is deterministic and subject to over-blurring in the reconstruction task, have the authors explored other losses such as an LPIPS loss which might be more invariant to small misalignments?", "limitations": "looks good to me", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Idea seems like a fundamental one that the community will use.  It applies to a wide variety of tasks.  The concept is novel and general.  There's a lot of great qualitative experiments that help me understand that this method is implicitly learning a tracking representation.", "weaknesses": "I had a hard time understanding the point of the corrector for awhile.  If the predictor functions as intended, shouldn't a reconstruction loss applied to D(P(z)) be enough?  But after some more thought, I guess the intent isn't to build a an open loop video predictor (although I suppose the authors could evaluate it as such?), but instead the predictor is just a way to establish alignment between features in neighboring frames so that the cross attention of the corrector is more effective?\n\nI think the prior work could go farther back in history in order to frame this proposed prediction-correction framework as being related to old school recursive filters.  For example, I think if I were told how this is designed similar to a Kalman filter, then I'd have a much clearer picture of what is trying to be done.\n\nI don't think Fig 1 does enough to clarify the system.  I felt it was still necessary to read eqn 1 and 2 to understand what exactly was going on.  Some ideas include: (1) include arrows going up in the direction of image -> encode -> state and state -> decode -> image. (I originally read this as left-to-right top-to-bottom and realized I went the wrong direction once I saw encode and decode -- arrows would prevent this). (2) Make the relationship between prediction and correction clear.  It's currently presented as two colorful feature map sets, which doesn't communicate much.  Once again, if it's established early on what the predict-correct framework looks like, then this is obvious, but I failed to grok it on the first read through.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and constructive feedback. We were pleased to hear that you consider our contribution to be novel, general, and the ideas widely applicable. We appreciate that you recognize the strength of our qualitative experiments to help understand the method’s working.\n\n**“I had a hard time understanding the point of the corrector for a while. [...] the predictor is just a way to establish alignment between features in neighboring frames so that the cross attention of the corrector is more effective?”**\n\nThank you for pointing out this confusion. We will clarify this in the updated version of the paper. Indeed, as is mentioned in the paper, the separation between the corrector and predictor is slightly artificial if the predictor is not unrolled in an open-loop prediction set-up. Since this is a deterministic model, unrolling in an open loop would produce blurry predictions quite quickly, but this indeed an interesting future direction.\n\nThe purpose of the corrector is to integrate new information in the state based on the current observation. However, if we were to decode from this state it could create a shortcut that the model could exploit (the model could learn to ignore the temporal prediction component, i.e. the alignment between features in neighboring frames, and just learn to auto-encode via the corrector). This is why we decode from the predicted state, which forces temporal prediction. For readout of down-stream tasks, we additionally make use of the corrected state, which combines both the prediction and the current observation.\n\n**“I think the prior work could go farther back in history in order to frame this proposed prediction-correction framework as being related to old school recursive filters. For example, I think if I were told how this is designed similar to a Kalman filter, then I'd have a much clearer picture of what is trying to be done.”**\n\nThank you for pointing out the connection to Kalman filters and recursive filters. Indeed, there are some resemblances worth pointing out that we could comment on in the related work section. We will revise the text accordingly, which hopefully improves the overall presentation.\n\n**“I don't think Fig 1 does enough to clarify the system.”**\n\nThank you for pointing this out. We intend for the revised draft to make the connection between the corrector and the predictor clearer. Further, we have updated the main model figure with this in mind, which can be found in the supplementary pdf attached to the general response. Please let us know if you have any suggestions for further changes.\n\n**“I didn't get a good impression of what kind of motion dynamics the predictor properly supported. It looked like there was some overshooting happening in the gifs, but it's not clear to me if this is a real issue. Can this be clarified?”**\n\nWhat the predictor would learn exactly is difficult to quantify as it will depend on many factors - model capacity, training data and so on. We do observe that when there is fast motion the model tends to lose tracking and assigns a new token to the moving element. This makes sense considering that the amount of uncertainty grows with faster motion. Whether this is a real issue will similarly depend on the nature of the down-stream tasks. Evidently, for our current evaluation spanning several standard computer vision tasks, this is not a major issue. However, for more complicated dynamics the predictor could benefit from a more specialized architecture. \n\n**“One ablation that would be useful is to change the backprop through time length. It would be great if there was evidence that only 2 or 3 frames were enough to get the predictor to estimate dynamics. Another question could be, what kind of dynamics are modeled with 2, 3, 4, 8, etc. frames and are there diminishing returns? Can this be clarified?”**\n\nThank you for this suggestion. Generally speaking, we’d anticipate that a minimum of 3 frames is needed to make good predictions as acceleration information requires 3 different measurements - with 2 frames only velocity information is available. We agree that it would be interesting to ablate this hyper-parameter and we will commit to doing so in the updated version of the paper. \n\nPreviously we have explored training with a probabilistic “stop-gradient” that activated at random times through the sequence to effectively train on 3-4 frames long sequences but not overfit to a specific length. However as the model development continued, we no longer found this to be needed.\n\n**“Given that this model is deterministic and subject to over-blurring in the reconstruction task, have the authors explored other losses such as an LPIPS loss which might be more invariant to small misalignments?”**\n\nThank you for this suggestion. In fact, we have not explored LPIPS or other perceptual losses to mitigate this, though this would be an interesting direction for future work. Integration of perceptual losses into MooG is not straightforward as we decode a random subset of pixels at every time step for efficiency reasons, as pixels directly serve as queries in the transformer decoder. Future work could explore decoding (latent) patches instead of raw pixels to mitigate this issue, which would make the model compatible with perceptual losses. A related direction that is worth pursuing is to have the state be the outcome of a sampling process (eg. as in a diffusion model or VAE), which might also help address blurry predictions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their helpful comments.We are pleased to hear that the reviewers find our paper to be well-presented (MZrN13, RPGH11, ykZT08), interesting (MZrN13, r1sm12), and novel (MZrN13, r1sm12, ykZT08). Reviewers MZrN13 and ykZT08 positively highlight the quality of the experimental evaluation. Reviewer ykZT08 further notes that the “idea seems like a fundamental one that the community will use”. 

We address the concerns raised by each reviewer individually. Further, the supplementary pdf includes several additional baselines and updated results:

* We have included a comparison to DINOv1 and DINOv2 in the end-to-end setting, which was previously missing. It can be seen how both approaches perform considerably better to their frozen counter-parts, yet MooG performs considerably better in both settings.

* After submission, we discovered a bug in the “Grid Rec.” baseline where no gradients were being back propagated into the encoder. After re-running, we now observe how “Grid Rec.” consistently outperforms the “Grid” baseline without recurrence, as one might expect. Compared to MooG, in the frozen setting (which is the main paradigm of interest), “Grid Rec” continues to perform considerably worse than MooG. In the end-to-end setting, where we finetune the model on the down-stream task of interest, we now observe how “Grid Rec” and MooG perform comparably, suggesting that the added supervision can help balance out architectural differences.

* Based on reviewer r1sm12’s suggestions, we have included a comparison to VideoMAE v2 [1]. We consider representations from 3 [publicly available VideoMAE v2 checkpoints](https://github.com/OpenGVLab/VideoMAEv2/blob/master/docs/MODEL_ZOO.md): the ViT-small and ViT-base variants, which contain 22M and 83M parameters respectively, as well as a ViT-giant model (1B params). The smaller variants were obtained by distilling the predictions of the ViT-giant model. We note that MooG contains approximately 35M parameters, which includes the pixel decoder. There are other differences that further make it difficult to compare, for example MooG was pre-trained on MOVI-E, while the VideoMAE models were pretrained on a superset of all Kinetics videos. MooG was trained purely in a self-supervised manner, while the VideoMAE v2 teacher network was finetuned for action recognition, and the ViT-small and ViT-base models were initialized from finetuned networks as well. Keeping these differences in mind, we observe how MooG performs significantly better than the ViT-small and ViT-base sized models that offer similar parameter counts. Compared to the teacher ViT-giant model that has 30x more parameters, MooG performs very well: it is better on MOVi points and Waymo boxes, but worse on MOVi depth and DAVIS points. Both methods perform the same on MOVi boxes. We are hopeful that scaling MooG to 1B+ parameters and larger scale pretraining may lead to further improvements, though we leave this for future work as mentioned in our limitations section.

* Based on reviewers’ ykZT suggestion we have revised the main model figure, a draft of which is included. We will continue to make revisions based on reviewer input and any additional feedback is much appreciated. 


[1] VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking

Meta-review: This paper received consistently positive final ratings of Accept, Weak Accept, Weak Accept, and Weak Accept. The reviewers felt that this work introduced a clear and novel idea that will likely see significant use in the field. It addresses a clear shortcoming with previous video representation learning methods. There was some discussion about an error with the reported performance metrics ("After submission, we discovered a bug in the “Grid Rec.” baseline") but the reviewers felt that this wasn't a major concern given the strong performance for the frozen-features case. Congratulations! Please take the reviewers' thoughtful comments into consideration as you prepare the camera-ready copy.
Decision: Accept (spotlight)