SIRE: SE(3) Intrinsic Rigidity Embeddings
Authors: Cameron Omid Smith
Basile Van Hoorick
Chonghyuk Song
Vincent Sitzmann
Vitor Campagnolo Guizilini
Yue Wang
Venue: Accepted by TMLR ()

Abstract: Motion serves as a powerful cue for scene perception and understanding by separating independently moving surfaces and organizing the physical world into distinct entities. We introduce SIRE, a self-supervised method for motion discovery of objects and dynamic scene reconstruction from casual scenes by learning intrinsic rigidity embeddings from videos. Our method trains an image encoder to estimate scene rigidity and geometry, supervised by a simple 4D reconstruction loss: a least-squares solver uses the estimated geometry and rigidity to lift 2D point track trajectories into SE(3) tracks, which are simply re-projected back to 2D and compared against the original 2D trajectories for supervision. Crucially, our framework is fully end-to-end differentiable and can be optimized either on video datasets to learn generalizable image priors, or even on a single video to capture scene-specific structure -- highlighting strong data efficiency. We demonstrate the effectiveness of our rigidity embeddings and geometry across multiple settings, including downstream object segmentation, SE(3) rigid motion estimation, and self-supervised depth estimation. Our findings suggest that SIRE can pave the way towards self-supervised learning of priors over geometry and motion rigidity from large-scale video data.

Published in Transactions on Machine Learning Research (11/2025)
SIRE: SE(3) Intrinsic Rigidity Embeddings
Cameron Smith!
Basile Van Hoorick2
Chonghyuk Song
Vincent Sitzmann
Vitor Guizilini
Yue Wang
!University of Southern California
TTyota Research Institute *Equal Advising
"Massachusetts Institute of Technology
Reviewed on OpenReview: https://ppparrrii..nn/t/touuutup
Abstract
Motion serves as a powerful cue for scene perception and understanding by separating in- dependently moving surfaces and organizing the physical world into distinct entities. We introduce SIRE, self-supervised video-learning formulation that learns intrinsic rigidity embeddings to drive soft motion-based segmentation and rigid-body reasoning. Rigidity embeddings softly group pixels into rigid components, where pixels belonging to the same rigid object share similar embeddings. The core of our formulation is simple point-tracking reconstruction loss: our model estimates rigidity embeddings, which is used to lift off-the- shelf 2D point track trajectories into rigid SE(3) tracks, which are simply re-projected back to 2D and compared against the original 2D tracks for supervision. Crucially, our frame- work is fully end-to-end diffcrentiable and can be optimized either on vidco datasots to learn generalizable image-encoder priors, or even on single video to capture scene-specific struc- ture highlighting strong data efficiency. We demonstrate the effectiveness of our rigidity embeddings across multiple tasks, and our findings suggest that SIRE can learn powerful geometry and motion rigidity priors from video data with minimal supervision.
1 Introduction
Motion is powerful cue for organizing the structure of the physical world. In cognitive neuroscience, empir- ical studies support the Gestalt principle of common fale (Wertheimer, 1938; Spelke, 1990; Braddick, 1993), which refers to the tendency of the human visual system to group together elements based on synchronized movement. Prior works have been inspired by these foundational ideas and leveraged motion-based cues for learning object segmentation. Representative methods often structure their model to explain a video's optical flow using a set of unsupervised object masks (Flsayed et al., 2022; Kipf al., 2022; Yang et al., 2021). However, these works almost exclusively operate without 3D awareness and are therefore limited to synthetic scenes vidcos with simple motion.
In parallel, recent methods have emerged for self-supervised learning of geometry and camera pose estimation from the same off- the-sheelf optical flow supervision signal (Smith et al., 2024; 2023; Bowen et al., 2022). These methods are 3D-aware and operate on real-world scenes with more complex motion, but are generally limited namely,
to static scenes: they operate by explaining the optical flow via a single SE(3) transformation the camera motion. But dynamic videos importantly contain not just a single rigid body transformation; instead, there are a potentially unbounded number of them, in which the camera and dynamic scene content move independently.
We propose SE(3) Intrinsic Rigidity Embeddings (SIRE), which effectively combines these two lines of work to learn object-aware embeddings on real-world scenes featuring diverse motion. SIRE is 3D-aware and can learned from casual real-world videos. Our first innovation is to learn the underlying scene dynamics via SIRE, a representation that softly encodes the scene's distinct rigid groups, and second a simple 4D reconstruction loss which enables training these embeddings without direct supervision. A high-level overview is shown in Fig. 1.

---

Published in Transactions on Machine Learning Research (11/2025)
Video and Off-the-Shelf
SIRE Optimization via Gradient Descent
Per-Track SE(3) Trajectory, 3D Geometry, and Rigidity Embeddings
Point Tracks
3D Geometry
SE(3) Trajectories Rigidity Emb. (T. Φ)
Depth and Rigidity Embedding CNN
Per-Track SF(()) Least Squares Solve
Track 2D Reprojection Loss
SIRE learns intrinsic rigidily embeddings features that encode rigid-object priors directly from raw video and off- off-theshhelf 2D point tracks. An image encoder predicts per-pixel embeddings that softly group pixels into rigid bodics. A lightweight SE(3) solver lifts cach 2D track into an SE(3) rigid-body trajectory; those trajectories are re-projected to 2D and compared against the original tracks for supervision. Because the entire pipeline is end-to-end differentiable, STRE can be trained on large video collections to learn generalizable motion-aware priors, even fine-tuned on a single clip for scene-specific features.
Figure 1:
We conduct extensive experiments broad tasks to validate model's learned rigidity emboddings well intermediate geometry representations. Our experiments demonstrate that both dataset-widk and even per-video optimization of our rigidity embeddings produces strong semantic features useful for downstream segmentation tasks. Our findings suggest that this simple formulation can pave the way towards self- supervised learning of priors over geometry and object rigiditics from largo-scale video data.
2
Related Work
2.1 Self- Supervipe Motion- Based 2D Segmentation Learning
Using optical flow as a supervision signal for learning meaningful object features is promising: it naturally captures motion boundaries and can reveal distinct object structure by grouping regions that move rigidly together, without relying on explicit object labels. Understanding and decomposing scene motion from optical flow vidco has been widely studied in self-supervised settings. Most works in this line leverage slot- based model architectures which effectivel) explain the motion using N object groupings. SAVi (Kipf et al., 2022) learns compelling object-centric decompositions, but is empirically limited to simple synthetic scenes. Self-Supervised Video Object Segmentation by Motion Grouping (Yang et al., 2021) operates on real-world videos, but uses slot attention with just two slots, effectively achieving foreground bongrround-backgronndd segmentation, and empirically requires the background flow to be planar (via rotation-dominant camera captures).
These methods are not 3D-aware, meaning they cannot leverage full 3D scene structure: consider that the optical flow induced by a camera translating even through static scene induces complex 2D motion due perspective effects, but in 3D it is explained by single SE(3) motion. In contrast, our approach is 3D-aware, allowing it to generalize beyond videos with simple scene motion. Importantly, our method does not assume fixed number of rigid components or 'slots'. We demonstrate that our approach can produce compelling multi-body soft-segmentations, even in scenes with complex, non-rigid motions.
2.2 Self-Supervieed Penecturr-froo--eee
(SfM) Methods
Another class of approaches (Smith et al., 2023; 2024) use the same off-the-shelf optical flow and point tracks (Teed & Deng, 2020; Xu et al., 2022; Karaev et al., 2023; Doersch et al., 2023) as supervision, but leverage 3D-aware formulations and aim to Iearn compelling geometry and camera pose estimation from monocular videos of static scenes. FlowCam (Smith et al., 2023) and FlowMap (Smith et al., 2024) both demonstrate that it is possible to train self-supervised depth and camera pose estimation just from optical
2

---

Published in Transactions on Machine Learning Research (11/2025)
Per-Frame Network Estimates Depth and Rigidity Embedding
2) Per-Track Rigidity-Weightedd SE(3) Solving
2(a): Lift 2D tracks into 2(b): Extract Track Rigidities
2(0): Solve Per-Track SE(3)
3D Scene Flow
Video Frame and 2D Point Tracks
Rigidity Embedding
Depth
3D Scene
Track-Image Dot-Product
Track
Per-Track SE(3)
Estimate
Flow Field
Rigidities
Translation, Rotation
Image Encoder
Weighted
Least
Squares
SE(3)
Image
Encoder
Figure 2: A SIRE training forward pass. Given two frames and 2D point tracks, per-frame CNN first estimates rigidity embeddings as well as depth. We lift the 2D point tracks into 3D scene flow via the depth estimates. For cach point track, we extract rigidity map by comparing its rigidity embodding to all other point track rigidity embeddings, then solve for the SE(3) transformation on the global scene flow using the rigidity map as confidence weights the solver. Lastly, the per-track SF(3) trajectories are reprojected back to 2D and compared with the original 2D trajectory for supervision.
flow supervision. They introduce simple formulation for geometry and camera pose estimation by lifting the estimated depth and off-the-shelf 2D flow into 3D flow and solving for the camera pose; then, they supervise their estimation by comparing the off-the-shelf 2D flow to that induced by the predicted depth and camera motion.
These methods have delivered promising results but are critically restricted to modeling static scenes, where the observed optical flow can be explained by a single SE(3) motion (the camera pose). Our method builds upon FlowMap to inherit its simple formulation and strong performance, but we extend it to model dynamic scenes. At high level, extend it by solving for SE(() motion per 2D point track, instead of one global SB(3) camera transformation. This enables us to train unconstrained, unlabeled videos featuring non-rigid dynamics.
Other works have attempted to approximate the classical SfM pipeline (Schönberger & Frahm, 2016; Mur- Artal et al., 2015; Mur-Artal & Tardós, 2017; Campos et al., 2021; Rosinol et al., 2020) with deep learning counterparts, but these methods typically either have neural networks that directly estimate camera poses and are less accurate (Tang & Tan, 2018; Czarnowski et al., 2020; Zhou et al., 2018; Clark et al., 2018: Ummenhofer et al., 2017; Liu et al., 2019; Teed & Deng, 2018; Wang al., 2021b; Bloesch et al., 2018), are non-differentiable and approximate only subsets of the pipeline (Choy et al., 2016; Mishchuk et al., 2017; Luo et al., 2018; Ono et al., 2018; Sarlin et al., 2020; Zhao et al., 2022; Harley et al., 2022; Doersch et al., 2023).
2.3 Intrinsic Rigidity Embeddings
Intrinsic rigidity embeddings softly group points into rigid bodics: two points with similar embeddings are interpreted as belonging to the same rigid group. They were first introduced in RAFT-3D (Teed & Deng, 2021) to estimate piecewise-constant SE(3) maps for higher quality scene flow prediction. These embeddings are compelling because they a) avoid deciding how many rigid bodics are in the scene a-priori, thus supporting a potentially unbounded number of rigid bodies; and b) offer soft attention like gradients via their dot-product response formulation.
3

---

Published in Transactions on Machine Learning Research (11/2025)
RGB
Rigidity Embedding
Depth Estimate
Rigidity Maps
RGB
Rigidity Embedding
SE(3)
SE(3)
RGB
Rigidity Embedding
SE(3)
SE(3)
Rotation
Translation
Rotation
Translation
Figure 3: Results on the CO3D-Dogs Datasct. Here we plot estimated rigidity embeddings, inter- mediate depth estimates, and highlighted rigidity maps (top row). And while geometry estimation is not the primary goal of our method, we also plot accumulated 4D point clouds and color-coded SE(3) rotation and translation components (bottom row). Observe how SE(3) components within rigid bodies are often constant. Note that rigidity maps are estimated per-track and manually pick few representative ones here; consider how they yield semantically meaningful soft. segmentations without supervision. However, a main limitation of this original implementation is that ground-truth scene flow is required to train, which makes its usage largely restricted to simulated data. In this work, we instead show how intrinsic rigidity embeddings can be learned directly from video data using just off- ff-tth--shelf 2D flow.
2.4 Dynamic Scene Reconstruction
Until recently, reconstructing dynamic scenes was considered an outslanding task in computer vision. En- abled by recent breakthroughs in monocular depth and point-track estimation, some methods (Wang et al., 2024; Lei et al., 2024) have demonstrated strong 4D reconstructions on challenging dynamic scenes. Older works also leveraged monocular depth estimates and neural rendering (Kerbl al., 2023; Mildenhall al., 2020) but were less robust, were the depth and point track estimators they depended (Li et al., 2019;

---

Published in Transactions on Machine Learning Research (11/2025)
Method
Bear
Blackswan
Camel
Cows
Elephant
Flamingo
Goat
Lucia
Ours-Rig Burs-Rig.Emi Emb
0.834
0.590
0.729
0,818
0.753
0.745
0.798
0.670
Ours-Enc.
0.870
0.807
0.814
0.837
0.767
0.774
0.790
0.643
ImageNet Enc. DINOv2
0.659
0.179
0.103
0.123
0.253
0.083
0.217
0.944
0.930
0.915
0.923
0.917
0.797
0.885
0.861
Downstream Segmentation Results. We provide quantitative comparisons (TOU; higher is better) for downstream moving-object segmentation on several DAVIS scenes. We threshold the segmentation predictions at .5, resulting in a zero score for the 'ImageNet Enc! bascline on the Elephant scene. tNote that DINOv2 features are trained on large image datasets, whereas our results here are generated per-video.
Table 1:
Zhang et al., 2022; Bansal al., 2020; Fridovich-Kel et al., 2023; Broxton al., 2020; Li ct al., 2022b; Cao & Johnson, 2023; Song et al., 2023; Wang et al., 2022; Li et al., 2024a; Işik et al., 2023; Li et. al., 2022a; Weng et al., 2022; Du et al., 2021; Pumarola al., 2021; Park et al,, 2021a; Wang et al., 2021a; Park et al., 2021b; Xian et al., 2021; Li et al., 2023; Wu et al., 2024; Duan et al., 2024; Yang et al., 2023; 2024; Gao et al., 2022). These methods typically segment the scene into static background and dynamic foreground regions, reconstruct the background using classical SIM approaches, then re-integrate the foreground via re-rendering losses.
While these methods' results are indeed impressive, they are importantly cin-differentiable due to their multi- stage reconstructions and therefore can not be embedded in machine learning pipelines to learn generalizable priors over video datascts. Although our aims are different, note that our method is simple and fully differentiable, allowing us to train dolasel of videos to learn generalizable priors, paving the path towards large-scale video learning.
3 SE(3) Intrinsic Rigidity Embeddings
Our method, dubbed SE(3) Intrinsic Rigidity Embeddings (SIRE), takes in a dataset of videos with corre- sponding 2D point tracks and trains an image encoder to learn generalizable priors over object rigidity.
At a high level, our method trains a per-frame image encoder to predict M-dimensional rigidity embeddings F # as well as depth maps D E #xxx xx (though note that our aim is not to learn highly accurate geometry, geometry is just an intermediate property in our formulation). Given a video with T frames and point tracks, our training forward pass lifts the 2D point tracks E NxTx2 into SE(3) trajectories P E Nx7x6 via a lcast-squares solve using the estimated depth and rigidities, projects them back to 2D trajectories, and minimizes the difference with their original locations to supervise the depth and rigidity networks. We begin by describing how we can solve for a global SE(3) trajectory (temporarily assuming static scene) via the FlowMap formulation, then extend it to modeling dynamic scenes by solving for a potentially unique SE(3) trajectory per point track, and conclude with our complete formulation.
3.1 Solving for Global SE(3) Trajectories
Given a video with corresponding 2D point tracks p, our first goal is to lift these 2D trajectories into SE(3) trajectories P. If we assune temporarily that there is only one rigid body in the scene (i.e. the scene is static), FlowMap (Smith et al., 2024) showed that we can actually solve for the rigid transformation by lifting the 2D tracks into 3D scene flow via a depth estimate D, then solving for the SE(3) transformation which best explains the scene flow. Specifically, to solve for the SE(3) transformation Pa for point track from time É to 441, they use the Procrustes formulation argmin ||W (d.ck -pp Padar, K <<''pe,)))l P.tESE(3)
where Pat refers to all point tracks at time t, du refers to the depth of point track at time t, and W is a diagonal confidence matrix to down-weight bad correspondences (e.g. from incorrect flow sky regions). This weighted least-squares problem differentiable and can be solved via singular value decomposition (Choy et al., 2020; Smith et. al., 2023).
5

---

Published in Transactions on Machine Learning Research (11/2025)
RGB
SE(3)
SE(3) 4
Rig. Emb.
RGB
SE(3)
SE(3)
Rig. Emb.
RGB
SE(3)
SE(3) q
Rig. Emb.
Track Rigidity Maps
Per-Track Rigidity Maps
Per- Track Rigidity Maps
lighlighted Rigidity Maps
Highlighted Rigidity Maps
Hlighlighted Rigidity Maps
Figure 4: Rigidity Response Grid. Here we plot, for three scenes, the rigidity response grids where each cell contains rigidity map from the point track at that location to all other point tracks. While we only plot 16x16 grid here for space constraints, note in practice use 64x64 grids of point tracks. We also include (top) the per-track images of RGB, SE(3) rotation (Euler angles) and translation vector and rigidity embeddings, and (bottom) manually highlighted rigidity maps on the bottom row. The center example (bear) clearly shows that the largest component (blue) corresponds to the background (camera movement),, while point tracks on the bear's leg (red) form another distinct group (right leg movement). Note that these are results of per-scene optimizations without depth supervision.
3.2 Solving for Local SE(3) Trajectories
Dynamic scenes evidently feature more than just the camera motion consider scene with falling snow rumning water, where each observed point track might be induced by unique underlying rigid body; solving for global SE(3) trajectory is no longer appropriate. Assume for simplicity that we know which points belong to the same rigid bodies; that is, consider that for each point track we have binary mask R; over all other point tracks py: R& = tig((t,,,)[Vp)) where rig(p; Pj) is points and pj belong to the same rigid group and 0 otherwise.
We can use R; solve for Pa by ignoring all the scene flow vectors not in the same rigid group, multiplying the confidence weights by the per-track rigidity weights R₃:
argmin ||& (d.,K-1 . Preduty K Poe,)) P.. ESEE)(3))
(1)
3.3 Softly Parameterizing Per- Track Rigidity Masks via Intrinsic Rigidity Embeddings
Assuning known rigidity masks is impractical outside of simulation; instead, we need to estimate them. One simple and efficient way to do is via rigidity embeddings (introduced by RAFT-3D (Teed Deng, 2021)), pixcl-aligned M-dimensional feature map where two points are in same rigid body if they have similar embeddings. Formally, given two tracks p: and py and corresponding embedding features f. and fy, their rigidity is parameterized their cosine similarity, lower-bounded at O.. We can now define the rigidity mask

---

Published in Transactions on Machine Learning Research (11/2025)
Imput GT
Our Rig. Emb.
Our Eac.
ImageNet Enc.
DINO22
Figure 5: Downstream Segmentation Plots. We demonstrate that our method's embeddings are useful for downstream moving object segmentation by freezing features from model and baselines and training two-layer MLP for segmentation. We compare using features from rigidity embeddings, the last layer of our trained feature backbone, and the feature backbone before our training, Note these embeddings are trained from single-scene optimizations on each of these videos and without depth supervision. For each method, we visualize the PCA of their features (top) and their sogmentations (bottom).
for P; as:
mar (0.
Sa. S3
R2
(2)
3.4 Supervision via SE(3) Induced 2D Point Tracks
One problem remains: while we can robustly estimate 2D point tracks for in-the-wild videos, we do not nccessarily have corresponding ground-truth SE(3) trajectories, rigidity masks, and perhaps not depth estimates, to train our model. As illustrated in Fig. 2, to supervise our model's predictions, we lift each point track in frame t to 3D via the depth estimate, transport the point in 3D to the next frame t - 1 via the estimated SE(3) transformation, project back to 2D, and minimize the difference:
KPadaK Pa Pi4 11
(3)
3.5 Full SIRE Forward Pass
Given a video with intrinsics K and 2D point tracks {pa), the shared image encoder predicts per-frame depth D, and rigidity embeddings F, (Sec. 3). For cach track and adjacent pair (t, t+1), we unproject the 2D points 3D using D, and K and solve rigidity Procrustes problem to obtain the relative motion P E SF((3) via Eq. equation 1. Frame of reference: Note that depths defined in each camera-1

---

Published in Transactions on Machine Learning Research (11/2025)
Strat Strat Ead
Video
RGB
Accunulated antdd Point Clouds
Rigitity limbedting
Figure 6: Per-Scene Intermediate 4D Reconstructions Using Off-the-Shelf Depth. While we emphasize that geometry estimation or 4D reconstruction is not the primary aim of our method, here we plot. accumulated 4D point clouds and their corresponding unprojected rigidity embeddings after per-scene roconstruction using off-the-shelf monocular depth estimates.
coordinate frame and pi, maps camera-l to camera-(4+1), and motions between arbitrary times are obtained by chaining these relative poses from the start to end of the video sequence. Supervision is obtained by the SE(3)-induceh reprojection loss Eq. equation 3, which reprojects each point track from each timestep to each other timestep (dense re-projection in time) and compares projected 2D locations to the input 2D track locations.
4 Experiments and Results
4.1 Training SIRE
Our approach enables network optimization to be performed either per video, yielding a scene-specific net- work, or across a dataset of vidcos to train a generalizable prior. We demonstrate results in both settings to highlight that our method generates meaningful gradients and information even from a single video, and enables data-efficient training of priors over a dataset of videos. Results from Fig. and Table are op- timized using our dataset-wide prior (though we perform a short additional fine-tuning step per scene enhance the results in Fig. on top of our feed-forward estimates), and results in Fig. 3, Fig. 6, and Table 1 are optimized from-scratch, per video.
While depth supervision can be incorporated when available, the model can also operate in a fully self- supervised manner, relying solely on 2D point tracks for supervision to learn priors over objects and geometry. This makes particularly effective in woal-world scenarios when explicit. depth supervision unavailable or expensive. As learning geometry from a single dynamic vidco inherently ill-posed, our method can leverage depth supervision for more accurate per-scene reconstructions 4D estimations in this setting. Note that even in such challenging single-video cases without depth supervision, our 2D intrinsic rigidity embeddings are still compelling and useful. We only use ground-truth depth when explicitly indicated, such as in the 4D reconstructions of Fig. 6.
We leverage CO-Tracker (Karaev et al., 2023) for our 2D point track estimations, ML-PRO (Bochkovskii et al., 2024) and Video Depth Anything (Chen al., 2025) for any ground-truth depth (we use Vidco Depth Anything's estimates scaled to be metric via ML-PRO), GeoCalib (Veicht al., 2024) for estimates of intrinsics from each video, and Adam for optimization.
8

---

Published in Transactions on Machine Learning Research (11/2025)
4.2 Memory and Time Requirements
Our approach is computationally efficient, with rapid convergence and modest memory requirements. When optimizing per scene (~40 frames of video), convergence is achieved within minutes, with just few GB of VRAM. For full-scale training across multiple videos, we optimize on a single 48GB GPU, using a batch size 120 frames (10-frame videos with batch size of 12). The model trained for 30k iterations (~0.5 days), though we observe that convergence is largely achieved within the first 5k iterations, demonstrating the computo- and data-cfficiency of our method.
We evaluate SIRE's intrinsic rigidity embeddings and intermediale 4D reconstruction quality on multiple downstream tasks, including frozen feature segmentation estimation, self-supervised depth estimation, and SE(3) trajectory estimation (though we emphasize that these geometry reconstructions are intermediate byproducts of our method and not the primary objective). Our results suggest that our learned intrinsic rigidity embeddings contain semantically intuitive segmentation-like features of moving objects in videos and that our method can be used to learn powerful priors on large-scale video data. For results CO3D-Dogs, we pre-train our model on the entire CO3D-Dogs dataset, and for the DAVIS scenes, we fit our model from-scratch, per-video.
4.3 Downstream Feature Segmentation Training
To assess the quality of our learned rigidity em- beddings, we freeze features from either our rigid- ity embeddings or an intermediate layer of an im- age backbone and train a two-layer MLP on top of the frozen features for object segmentation. That is, we train SIRE per-video, then freeze the back- bone features or rigidity embeddings and separately train a small network on top of those frozen features in a downstrean ream segmentation lask. We evaluate on the DAVIS dataset (Caelles et al., 2019), which provides ground-truth moving object segmentations. We compare our features to those of the last. layer of the backbone before training (referred to "Ima- geNet Encoder"), the same layer of the backbone af- ter training our model ("Our Encoder"), our rigidity embeddings ("Our Rig. Emb."), and features from DINOv2 (Oquab et al., 2023; Fu et al., 2024). See Fig. 5 for segmentation results.
RGB
SE(3) t
SE(3) Clusters COLMAP
SE(3) Clusters SE(3) o
Figure 7: SE(3) Trajectories Though our method docs not explicitly estimate camera poses, we cluster the per-track SE(3) estimates and compare the esti- mated trajectories with COLMAP estimated poses, since presumably one of our estimated clusters should correspond to camera motion.
In Table 1, we report IOU and observe that our method achieves stronger segmentations compared to the prettrried network (He et al., 2015). Our backbone features after training contains slightly sharper features than our rigidity embeddings, but using denser point tracks will likely reduce this gap. While DINO features are more informative than ours, note that our method was trained per-video here, whereas DINO was trained on large image datasets. On the same DAVIS scenes, we further highlight the soft 2D segmentations that our model produces as intermediate per-track rigidity maps in Fig. 4. We also demonstrate in Fig. 6 that, with depth supervision, our model can learn compelling 4D reconstructions even on single scenes.
4.4 Supervsed Depth Estimation
We evaluate self-supervised depth estimation on the CO3D-Dogs (Sinha et al., 2023) dataset, comparing model against model ablations and FlowMap. We use MI-PRO (Bochkovskii et al., 2024) as ground-truth depth labels to compare our model against since the dataset does not provide ground-truth depth.
For the CO3D-Dogs dataset, we also mask our estimated rigidity weights with rigidity estimates from thresh- oldod Sampson distance on off-the-shelf optical flow, as used in prior works (Zhang et al., 2024; Ye et al., 2022). The Sampson distance effectively measures how well an optical flow vector is explained by the RANSAC-estimated median SF(3) transformation (likely the camera pose), and the error is thresholded
9

---

Published in Transactions on Machine Learning Research (11/2025)
such that only highll--confident vectors are marked static. During training, we set the rigidity weights to be constant (completely rigid) if a point is within the rigid mask, and the predicted value otherwise. We find that using these masks further constrains the learned geometry, but is not necessary when using off-the-shelf depth estimates. See the supplement for more details.
In Table 2, we quantitatively compare our depth estimates with those from a Flow-ap-trainndd model, and the following ablations of our model: a version without epipolar rigidity masks. See Fig. for examples of learned depth maps.
Her---yrranss-retrriiiie and version without
To train our model, we use the 1K videos from the CO3D-Dogs dataset. For evaluating depth estimation (above), we evaluate on the first frame of 100 videos, and for SE(3) trajectory estimation (below), we similarly evaluate trajectories on the first 30 frames of 100 videos.
4.5
SE(3) Trajectory Estimation
Our method does not explicitly estimate camera poses scparately; it estimates SE(3) trajectorics per-point- track. To evaluate our SE(3) with the dataset-provided COLMAP poses, we cluster the per-track SE(3) trajectories and report the minimum-loss trajectory from the clusters, assuming one corresponds to the cam- era motion. We report quantitative comparison (ATE) to COLMAP in Table 2 and visualization Fig. 7. We compare with the same model ablations and ablations as above for depth estimation, and also addition- ally compare with MegaSaM (Li et al., 2024b), a recent camera pose estimation method for dynamic scenes. Both our model and MegaSaM both estimate poses in real-time after pre-processing (point tracks in our case and depth maps in theirs). Note that we evaluate MegaSaM just poses because the ground-truth depth for evaluation comes from the same source that MegaSaM uses as input to its system. The fact that our poses are similar to COLMAP aligns with prior results from FlowMap, which has demonstrated COLMAP-comparable reconstructions, since our method reduces to Flo within each softly estimated rigid grouping.
Method
Depth (MSE)
Poses (ATE)
5 Conclusion
Full
0.6407
0.0044
No Pretrain
0.6441
0.0047
We have introduced STRE, a simple and differen- tiable formulation for learning priors over object rigidity from monocular video. Specifically, we show how to train intrinsic rigidity embeddings via sim- ple SE(3) reconstruction objective, self-supervised from monocular video and off-the-shelf point tracks. Through experiments we demonstrate that the in- trinsic rigidity embeddings our model learns contain mcaningful semantic information with high data- efficiency, paving the way towards learning priors over objects and rigid motion from either small-scale or large-scale video data. Note that our method in- deed has limitations: our method depends on accurate point tracks, which are known to struggle on fluids and sky regions, as well as intrinsios, and can struggle to learn accurate geometry on scenes with insufficient or excessive motion.
No Rig. Mask
0.7666
0.0088
FlowMap
0.8709
0.0466
MegaSaM
0.0104
Table 2: Generalizable Depth and Pose Estima- tion Results. We compare feedforward depth and SE(3) trajectory estimates from method and base- lines to those from MI-PRO (Bochkovskii et al., 2024) and COLMAP (Schönberger & Frahm,, 2016).
References
Aayush Bansal, Minh Vo, Yaser Sheikh, Deva Ramanan, and Srinivasa. Narasimhan. 4d visualization of dynamic events from unconstrained multi-view videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5366-5375, 2020. 5
Michael Bloesch, Jan Czarnowski, Ronald Clark, Stefan Leutenogger, and Andrew Davison. Codeslam- learning compact, optimisable representation for dense visual slam. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 2560-2568, 2568, 2018.
10

---

Published in Transactions on Machine Learning Research (11/2025)
Alcksci Bochkovskii, Amaäl Delaunoy, Hugo Germain, Marcel Santos, Yichao Zhou, Stephan R. Richter, and Vladlen Koltun. Depth pro: Sharp monocular metric depth in less than a second. ar arXie, 2024. URL https://arxxv.org///aa
02073. 8, 9, 10
Richard Strong Bowen, Richard Tucker, Ramin Zabīh, and Noah Snavely. Dimensions of motion: Monocular prediction through flow subspaces. In Proceedings of the International Conference on 3D Vision (3DV), 2022. 1
Oliver Braddick. Segmentation versus integration in visual motion processing. Trends in neurosciences, 16 (7):263-268) 1993. 1
Michael Broxton, John Flynn, Ryan Overbeck, Daniel Erickson, Peter Hedman, Matthew Duvall, Jason Dourgarian, Jay Busch, Matt Whalen, and Paul Debevec. Immersive light field video with a layered mesh representation. ACM Transactions on Graphics (TOG), 39(4):86-1, 2020. 5
Sergi Caelles, Jordi Pont-Tuset, Federico Perazzi, Alberto Montes, Kevis-Kokita' Maninis, and Lue Van Gool. The 2019 davis challenge on vos: Unsupervised multi-object segmentation. arXiv:1905.000 197000075007, 2019.
Carlos Campos, Richard Elvira, Juan J. Gómez Rodríguez, José M. M. Montiel, and Juan D. Tardós. Orb- slam3: An accurate open-source library for visual, visual inertial, and multimap slam. Transactions on Robotics, (7(6))1874 1890, 2021. 3
Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In Proceedings of the IEEE/CVP Conference on Compuler Vision and Pollern Recognition, pp. 130 141, 2023.
Sili Chen, Hengkai Guo, Shengnan Zhu, Feihu Zhang, Zilong Huang, Jiashi Feng, and Bingyi Kang. Video depth anything: Consistent depth estimation for super-long videos. arXim:2501.1200 2025.
Christopher Choy, Wei Dong, and Vladlen Koltun. Deep global registration. In Proc. CVPR, 2020. 5
Christopher B Choy, Jun Young Gwak, Silvio Savarese, and Manmohan Chandraker. Universal correspon- dence network. Advances in neural information processing systems, 29, 2016. 3
Ronald Clark, Michael Bloesch, Jan Czarnowski, Stefan Leutenegger, and Andrew J Davison. Learning solve nonlinear least squares for monocular stereo. In Proceedings of the Europeon Conference on Computer Vision (ECCV), pp. 284-299, 2018. 3
Jan Czarnowski, Tristan Laidlow, Ronald Clark, and Andrew J. Davison. Deepfactors: Real-time proba- bilistic dense monocular SLAM. Computing Research Repository (CoRR), 2020. 3
Carl Doersch, Yi Yang, Mel Vecerik, Dilara Gokay, Ankush Gupta, Yusuf Aytar, Joao Carreira, and Andrew Zisserman. Tapir: Tracking any point with per-frame initialization and temporal refinement. arXiv preprint XrXio:2306. 08637, 2023. 2,
Yilun Du, Yinan Zhang, Mlog-Xing Yu, Joshua B Tenenbaum, and Jiajun Wu. Neural radiance flow for 4d view synthesis and video processing. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14304-14314. IEEE Computer Society, 2021.
Yuanxing Duan, Fangyin Wei, Qiyu Dai, Yuhang He, Wenzheng Chen, and Baoquan Chen. 4d-rotor gaussian splatting: towards efficient novel view synthesis for dynamic scenes. In ACM SIGGRAPH 2024 Conference Papers, pp. 1 1-11, 2024. 5
Gamaleldin F. Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael C. Mozer, and Thomas Kipf. SAVi++: Towards end-to-end object-centric learning from real-world videos. In Advances in Neural Information Processing Systems, 2022. 1
Sara Fridovich-Keil, Giacomo Mcanti, Frederik Rahbærk Warburg, Benjamin Recht, and Angjoo Kanazawa.
K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12479-12485 2023.
11

---

Published in Transactions on Machine Learning Research (11/2025)
Stephanie Fu, Mark Hamilton, Laura E. Brandt, Axel Feldmann, Zhoutong Zhang, and William T. Free- man. Featup: A model-agnostic framework for features at any resolution. In The Twelfth International Conference on Learning Representations, 2024. URL https: httpps///pperrreeeeeeeerrtrrraae nettfoormmiddGkiiiinD00 9
Hang Gao, Ruilong Li, Shubham Tulsiani, Bryan Russell, and Angjoo Kanazawa. Monocular dynamic view synthesis: A reality check. Advances in Neural Information Processing Systems (NeurIPS), 35:33768- 33780, 2022.
Adam W. Harley, Zhaoyuan Fang, and Katerina Fragkiadaki. Particle video revisited: Tracking through oc- clusions using point trajectories. In Proceedings of the European Conference on Computer Vision (ECCV), 2022. 3
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015. URL https://arxiv.. wwtpriii.rr//bb//112....
9
Mustafa Işik, Martin Rünz, Markos Georgopoulos, Taras Khakhulin, Jonathan Starck, Lourdes Agapito, and Matthias Nießner. Humanrf: High-fidelity neural radiance fields for humans in motion. ACM Transactions on Graphics (TOG), 42(4):1-12, 2023. 5
Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rup- precht. CoTracker: It better to track together. In Proc. ECCV, 2023. 2,
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for rcal-time radiance field rendering. ACM Transactions on Graphics (ToG), 42(4):1-14, 2023. 4
Thomas Kipf, Gamalcldin F. Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional Object-Centric Learning from Video. In International Conference on Learning Representations (ICLR), 2022. 1,
Jiahui Lei, Yijia Weng, Adam Harley, Leonidas Guibas, and Kostas Daniilidis. MoSea: Dynamic gaussian fusion from casual videos via 4D motion scalfolds. arXio preprinl ur Xivx:4455.742.1 2024. 4
Ruilong Li, Julian Tanke, Minh Vo, Michael Zollhöfer, Jürgen Gall, Angjoo Kanazawa, and Christoph Lassner. Tava: Template-free animatable volumetric actors. In European Conference on Computer Vision, pp. 419 436. Springer, 2022a. 5
Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al. Neural 3d video syuthesis from multi-viow vidco. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5521-5531, 2022b.
Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8508 8520, 2024a. 5
Zhengqi Li, Tali Dekel, Forrester Cole, Richard Tucker, Noah Snavely, Ce Liu, and William T Freeman. Learning the depths of moving people by watching frozen people. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4521 4530, 2019. 4
Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neural dynamic image- based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4273 4284, 2023.
Zhengqi Li, Richard Tucker, Forrester Cole, Qianqian Wang, Linyi Jin, Vickie Ye, Angjoo Kanazawa, Alek- sander Holynski, and Noah Snavely. MegaSaM: Accurate, fast and robust structure and motion from casual dynamic videos. Xix preprint, 2024b. 10
12

---

Published in Transactions on Machine Learning Research (11/2025)
Chao Liu, Jinwei Gu, Kihwan Kim, Srinivasa G Narasimhan, and Jan Kautz. Neural rgb (I) d sensing: Depth and uncertainty from video camera. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11986-10999) 2019. 3
Zixin Luo, Tianwei Shen, Lei Zhou, Siyu Zhu, Runze Zhang, Yao Yao, Tian Fang, and Long Quan. Geodese: Learning local descriptors by integrating goometry constraints. In Procecdings of the European Conference on Computer Vision (ECCV), pp. 168 183, 2018. 3
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In Proceediiggs of the European Conference on Computer Vision (ECCV), pp. 405 421, 2020.
Anastasia Mishchuk, Dmytro Mishknnn Filip Radenovic, and Jiri Matas. Working hard to know your neighbor's margins: Local descriptor learning loss. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
Raul Mur-Artal and Juan D Tardós. Orb-slam2: An open-souree slam system for monocular, stereo, and rgb-d cameras. Transactions on Robolics, 33(5): 2017. 3
Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. Orb-slam: versatile and accurate monocular slam system. Transactions on Robotics, pp. 1147 1163, 2015. 3
Yuki Ono, Eduard Trulls, Pascal Fua, and Kwang Moo Yi. Lf-net: Learning local features from images. Advances in Newral Information Processing Systems (NeurIPS), 31, 2018. 3
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alazeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Xr preprint or Xiv:2304.07193, 2023. 9
Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla. Nerfics: Deformable neural radiance fields. In Proceedings of the IEEE/CVF
international conference on computer vision, pp. 5865 5874, 2021a. 5
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Brualla and Steven M Seitz. Hypernerf: A higher-dimension representation for topo- logically varying neural radiance fields. arXiv preprint arXix:2106. 13228, 2021b.
Albert Pumarola, Enric Corona, Gerard Moll, and Francesc Moreuo-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10318 10327, 2021. 5
Antoni Rosinol, Marcus Abate, Yun Chang, and Luca Carlone. Kimera: an open-source library for real- time metric-semantic localization and mapping- In Proceedings of the IEEE International Conference Robotics and Automation (ICRA), pp. 1689 1696. IEEE, 2020. 3
Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning fcature matching with graph neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pallern Recognition (CVPR), pp. 4938 4947, 2020.
Johannes Lutz Schönberger and Jan-Michael Frahm. Semetturefrom-Mootinn Revisited. In Proceedings of the IEEE Conference on Computer Vision and Pallern Recognition (CVPR), 2016. 3, 10
Samarth Sinha, Roman Shapovalov, Jeremy Reizenstein, Ignacio Roeco, Natalia. Neverova, Andrea Vedaldi, and David Novotny. Common pets in 3d: Dynamic new-view synthesis of real-life deformable categories. CVPR, 2023. 9
Cameron Smith, Yilun Du, Ayush Tewari, and Vincent Sitzmann. Flowcam: Training generalizable 3d radi- ance fields without camera poses via pixel-aligned scene flow. Advances in Neural Information Processing Systems (NeurIPS), 2023. 1, 2,
13

---

Published in Transactions on Machine Learning Research (11/2025)
Cameron Smith, David Charatan, Ayush Tewari, and Vincent Sitzmann. Flowmap: High-quality camera poses, intrinsics, and depth via gradient descent. In arXiv, 2024. 1, 2,
Liangehen Song, Anpei Chen, Zhong Li, Zhang Chen, Lele Chen, Junsong Yuan, Yi Xu, and Andreas Geiger. Nerfplayer: A streamable dynamic scene representation with decomposed neural radiance fields. IEEE Transactions on Visualization and Computer Graphies, 29(5):2732 2023. 5
Elizabeth S Spelke, Principles of object perception. Cognitive science, 14(1):29 56, 1990.
Chengzhou Tang and Ping Tan. BA-Net: Dense
bundle adjustment network.
arXiv preprint
arXiv:1806. 1806000700. 2018.
Zachary Teed and Jia Deng. Deepv2d: Video to depth with differentiable structure from motion. arXiv preprint Arxi 04605, 2018.
Zachary Teed and Jia Deng. RAFT: Recurrent all-pairs field transforms for optical flow. In Proceedings of the European Conference on Computer Vision (ECCV), 2020. 2
Zachary Teed and Jia Deng. Raft-3d: Scene flow using rigid-motion embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 3, 6
Benjamin Ummenhofer, Huizhong Zhou, Jonas Uhrig, Nikolaus Mayer, Eddy Tlg, Alexey Dosovitskiy, and Thomas Brox. Demon: Depth and motion network for learning monocular stereo. In Proceedings of the
IEEE Conference Compuler Vision and Pallern Recognation (CVPR), pp. 5038-5044 2017. 3
Alexander Veicht, Paul-Edouard Sarlin, Philipp Lindenberger, and Mare Pollefeys. GeoCalib: Single-image Calibration with Geometric Optimization. In ECCV, 2024. 8
Chaoyang Wang, Ben Eckart, Simon Lucey, and Orazio Gallo. Neural trajectory fields for dynamic novel view synthesis. arXiv preprint arXiv:2105. 05994, 2021a.
Liao Wang, Jiakai Zhang, Xinhang Liu, Fuqiang Zhao, Yanshun Zhang, Yingliang Zhang, Minye Wu, Jingyi Yu, and Lan Xu. Fourier plenoctrees for dynamic radiance field rendering in real-time. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 13524 13534, 2022. 5
Qianqian Wang, Vickie Ye, Hang Gao, Weijia Zeng, Jake Austin, Zhengqi Li, and Angjoo Kanazawa. Shape of motion: reconstruction from a single video. In orXiv, 2024. 4
Wenshan Wang, Yaoyu Hu, and Sebastian Scherer. Tartanvo: A generalizable learning-base based vo. In Confer- ence on Robot Learning, pp. 1761-1772. PMLR, 2021b. 3
Chung-Yi Yi Weng, Brian Curless, Pratul P Srinivasan, Jonathan T Barron, and Ira SaaeelmmccherSShizeemaa Humannerf: Pree-viewpoint rendering of moving people from monocular video. In Proceedings of the IEFE/CVF conference on computer vision and pattern Recognition, pp. 16210- 16210-16220, 2022. 5
Max Wertheimer. Laws of organization in perceptual forms. Psychological Research, 1938. 1
Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEFE/CVF conference on computer vision and pattern recognition, pp. 20310-20320, 2024. 5
Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free- viewpoint video. In Proceedings of the IEEE/CVP conference on computer vision and pottern recognition, pp. 9421 9431, 2021. 5
Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, and Dacheng Tao. Gmfllw: Learning optical flow via global matching. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8121-8130. 2022. 2
Charig Yang, Hala Lamdouar, Erika Lu, Andrew Zisserman, and Weidi Xie. Self-supervised video object segmentation by motion grouping. In ICCV, 2021. 1,2
14

---

Published in Transactions on Machine Learning Research (11/2025)
Zeyu Yang, Hongye Yang, Zijie Pan, and Li Zhang. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. arXiv preprinl arxiv:2310. 10642, 2023.
Ziyi Yang. Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. In Proceedings of the IREE/CVF conference computer vision and pattern recognition, pp. 20331-20341, 2024.
Vickie Ye, Zhengqi Li, Richard Tucker, Angjoo Kanazawa, and Noah Snavely. Deformable sprites for unsu- pervised video decomposition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
Junyi Zhang, Charles Herrmann, Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, Deqing Sun, and Ming-Hooe Haugn Yang. Monst3r: A simple approach for estimating geometry the presence of motion. arXiv preprint arziv:2410.00 00.0382, 2024.
Zhoutong Zhang, Forrester Cole, Zhengqi Li, Michael Rubinstein, Noah Snavely, and William T Freeman, Structure and motion from casual videos. In European Conference on Computer Vision, pp. 20 37. Springer, 2022.
Wang Zhao, Shaohui Liu, Hengkai Guo, Wenping Wang, and Yong-Jin Liu. Particlesfm: Exploiting dense point trajectories for localizing moving cameras in the wild. In Proceedings of the Europeon Conference on Computer Vision (ECCV), pp. 523 542. Springer, 2022. 3
Huizhong Zhou, Benjamin Ummenhofer, and Thomas Brox Deeptam: Deep tracking and mapping. Tn Proceedings of the European Conference on Computer Vision (ECCV), pp. 822 838, 2018. 3
15


Reviews:
[{"review_id": "YFZ61EONll", "reviewer": "Reviewer_KKcA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work introduces SIRE (SE(3) Intrinsic Rigidity Embeddings), which is a self-supervised video learning framework that learns pixel-wise rigidity embeddings and depth maps, which are then used to solve for and estimate local per-track SE(3) transformations. The key contribution lies in the use of rigidity embeddings to softly group points into rigid components, allowing the model to capture multiple independently moving rigid objects without supervision, as well as the loss function based on SE(3) induced 2D projection, which enables self-supervised training without the need for ground truth labels for trajectories, rigidity or depth. \n\nStrengths:\n- The model has a simple but effective training formulation, using an end-to-end differentiable pipeline based on reprojection error, which help avoids dependence on labels and heavy supervision.\n- The framework seems to be very data efficient, it shows good results both in dataset-wide and per-video optimization regimes, demonstrating flexibility.\n- Despite this apparent simplicity, the framework achieves competitive results on segmentation, depth, and pose estimation tasks.\n\nWeaknesses:\n- The paper provides almost no details about the network backbone, layers, or design choices, relying heavily on references for these details. I had trouble figuring out whether the embedding networks were CNNs or some other type of layer, as this is only mentioned in passing in one figure caption, and there are no further details provided about the layer. This harms the paper severely,  as it makes it difficult to understand for readers outside the narrow field and makes the work much more difficult to reproduce.\n- Similarly to the previous point, equations are introduced without properly explaining variables and which of them are learnable or not, and what their learning setup is in case they are learnable.\n- The ablation presented in the work is minimal, with only a few cases being shown (“No Pretrain,” “No Rig. Mask”) , making it unclear which other design choices contributed to the performance, such as embedding dimensionality, network depth, or supervision choices, etc.\n- The scope of the evaluations is fairly narrow, with benchmarks being restricted to DAVIS and CO3D-Dogs. More diverse datasets would better validate generality.", "claims_and_evidence_yn": "Yes", "claims_explanation": "In general, yes, the core claims are backed by quantitative comparisons, qualitative visualizations (rigidity maps, point clouds), and ablations that show the importance of induced rigidity embeddings and the flexible training procedure.\n\nHowever, the narrow scope of datasets makes it unclear whether the method generalizes beyond the tested settings. Evaluation on some other established datasets that include video data (e.g., LLFF, MipNeRF-360 among others) will make the results much more convincing.\n\nAdditionally, the lack of detailed ablations makes it difficult to assess which parts of the pipeline are actually contributing to the improved performance. This also makes it harder to guess whether the framework will be useful for other scenarios that were not explored in the paper.\n\nOverall the evidence is convincing but incomplete: strong enough to support the paper’s immediate claims, but not detailed enough to determine the wider applicability of the approach.", "audience_yn": "Yes", "audience_explanation": "The paper would be of interest to TMLR readers working in:\n- Self-supervised representation learning\n- 3D scene understanding and structure-from-motion\n- Video segmentation and motion analysis\n\nThe idea of intrinsic rigidity embeddings is appealing as a reusable prior, and the self-supervised nature is well in-line with current trends in large-scale video learning. That said, the paper could be more attractive if it provided, greater clarity on the architecture and implementation details, as it is currently really difficult for readers outside of the scene reconstruction community to understand. There is no mention of layer types, widths or depths for any almost part of the model. Furthermore, in equations such as the Procrustes formulation, some of the variables such as K are never explained. I assume this represents the camera intrinsics matrix, but it is entirely unclear whether this information is obtained as part of the datasets or also needs to be estimated with the model, and if it needs to be estimated no details are provided on how this is done.\n\nSo, while the audience may likely find it interesting, readability and impact of the paper is currently hampered by a lack in many technical details which needs to be improved with revisions.", "requested_changes": "1. Add architectural details (critical)\n    - Explicitly describe the encoder backbone, with basic details such as network type (CNN/ResNet/UNet), layers, embedding dimensionality, depth. An in depth architecture description with a graphical overview needs to be at least provided as a part of the supplement.\n     - Clarify exaclty how rigidity embeddings and depth maps are predicted, and also provide more details on how the camera intrinsics matrix is obtained. Do the embedding layers share weigths, are they independent, and other details relevant to reproducibility.\n\n2. Provide code (critical)\n   - I couldn't find any code or repository linked in the paper or as a part of the supplementary materials. This is essential for any ML paper to be published.\n\n3. Expand ablations (would strengthen work)\n   - Study embedding dimensionality, different encoder types, or training with/without depth supervision.\n\n4. Broader evaluations (would strengthen work)\n    - Test and compare on more diverse benchmarks (e.g., LLFF, MipNeRF-360 among others).\n\n5. Address typographical issues (critical)\n   - Page 5, last paragraph: errors in the Procrustes equation formatting, some variables contain commas between * and t, and others do not (e.g. p*t and p*, t+1)\n   - Page 6, last paragraph: \" two points are in same rigid body if they similar embeddings\" seems like a 'have' is missing there.\n   - A careful proofreading pass is recommended for other typos that may have been missed.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8bsIwVZoVL", "reviewer": "Reviewer_tFRj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces SIRE, a self-supervised framework that learns rigidity-aware embeddings from monocular video and off-the-shelf 2D point tracks. The method groups pixels into rigid components via embeddings and estimates per-track SE(3) trajectories, supervised by a simple reprojection loss. The key contributions are:\n\t•\tA fully differentiable and simple formulation for learning intrinsic rigidity embeddings, enabling both dataset-wide training and per-video optimization.\n\t•\tExtending prior FlowMap-style approaches from static scenes to dynamic videos, by solving per-track rather than global SE(3) motions.\n\t•\tDemonstrating that rigidity embeddings capture semantically meaningful motion segmentation cues and are useful for downstream tasks such as segmentation, depth estimation, and trajectory estimation.\n\t•\tEmpirical validation on CO3D-Dogs and DAVIS datasets, showing competitive performance to baselines despite operating in a self-supervised and data-efficient setting.\n\nStrengths:\n\t•\tConceptually simple and elegant framework; fully differentiable.\n\t•\tStrong efficiency and data efficiency, converging quickly per-video.\n\t•\tProduces embeddings with clear semantic association.\n\nWeaknesses:\n\t•\tHeavy reliance on point track quality; struggles likely under occlusion, fluids, or textureless regions.\n\t•\tEvaluation scope is somewhat narrow, focused on CO3D-Dogs and DAVIS, with limited demonstrations on multi-object dynamic scenes.\n\t•\tMissing implementation details, including architecture/initialization of the depth head and embedding head.\n\t•\tRequires per-video optimization during inference, limiting practicality compared to fully feed-forward alternatives.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors provide convincing evidence through both qualitative and quantitative experiments. The segmentation results demonstrate that rigidity embeddings capture motion-based structure. The depth and pose experiments validate the correctness of the SE(3) results. The ablation studies (e.g., without pretraining, without rigidity masks) support the value of the rigidity embeddings learning.\n\nSome aspects could be better substantiated, such as performance in more complex dynamic multi-object scenes and robustness under occlusion, but the presented evidence sufficiently supports the main claims.\n\nThe evaluation in Section 4.3 is a bit concerning. SIRE is trained per-video (closer to an overfitting setup), while the baselines such as ImageNet Encoder and DINOv2 are generic pretrained representations. Although the authors note this difference, the comparison still risks overstating SIRE’s advantage.", "audience_yn": "Yes", "audience_explanation": "The paper focuses on self-supervised learning, motion segmentation, and 3D scene understanding, all highly relevant to the TMLR topics. The proposed approach is simple, efficient, and general enough to influence research on self-supervised dynamic 3D learning from large-scale videos. Many in the audience will be interested in the idea of learning rigidity priors without explicit supervision.", "requested_changes": "- Clarify implementation details, including architecture/initialization of the depth head and embedding head. (P0)\n- Clarify the frame of reference for SE(3) trajectories (camera-centered, first frame, or another convention). (P0)\n- Expand evaluation to more diverse datasets with multiple independently moving objects (e.g., 100DoH, BONN, Stereo4D). Current experiments are limited in scope and may not stress-test the method’s generality. (P1)\n- Discuss robustness under occlusion and track dropouts. Since point tracks are the backbone of the approach, the paper should address how the method behaves when tracks are noisy. It would also be useful to provide qualitative examples of failure cases (e.g., scenes with fluids, textureless regions, or large occlusions). (P2)\n- Clarify why MegaSaM is evaluated only on poses and not depth. (P2)", "broader_impact_concerns": "No major ethical concerns arise from this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kX9uhlpgLB", "reviewer": "Reviewer_ucXD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper is proposed a self supervised method to learn depth and 3d motion of point tracks, focussing on learning rigid body embeddings. In other works [Flowcam], [Flowmap], [RAFT-3D], the concept is generally featured - but differs in the flavour. As such, the method uses the same ingredients as Flowmap and Flowcam, but adds a incremental improvement in creating 'rigidity' embeddings, which forms the subject of this paper. \n\nThe pipeline is very similar to Flowmap: \n1. Get matching points across video frames \n2. Predict and lift to 3d. \n3. Solve for relative pose using orthogonal procrustates (SVD)\n4. Project to 2d (lets say from cam 1 to cam 2) and minimize flow loss available from flow solver (cotracker). \n\nRigidity embeddings are used to weight the orthogonal procrustates algorithm, and are learnt implicitly. It differs from RAFT in that we use scene flow labels to supervise it, and it is learnt without labels as such.\n\nThe authors show that using rigidity embeddings help\n - motion segmentation: comparison is made with DINOv2 \n- depth and pose are better than with the Flowmap\n\nThese embeddings are usable in downstream tasks as a useful supervision cue.\n\nFlowcam: https://arxiv.org/pdf/2306.00180\nFlowmap: https://arxiv.org/pdf/2404.15259v1\nRAFT-3D: https://arxiv.org/pdf/2012.00726", "claims_and_evidence_yn": "Yes", "claims_explanation": "I like the general idea of learning interpretable embeddings that can help as object cues. It might be useful in tasks such as object detection and occupancy prediction too (I work in 3d vision for autonomous driving). \n\nThe solidity of the method is apparent. We build upon Flownet, but extend to learn object embeddings.", "audience_yn": "Yes", "audience_explanation": "This is a core computer vision and machine learning work. The TMLR audience will very much be interested.", "requested_changes": "Kindly elaborate on how the embeddings are used in a mathematical formulation. I would generally the entire algorithm to be depicted mathematically. I am also interesting in learning how well this method would perform in outdoor scenes (e.g. highway). Can you run an experiment?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)