FlowFeat: Pixel-Dense Embedding of Motion Profiles
Authors: Nikita Araslanov
Anna Ribic
Daniel Cremers
Venue: NeurIPS 2025 spotlight ()
Keywords: representation learning
optical flow
Primary area: applications
TL;DR: learning pixel-dense multi-task representations from apparent motion

Abstract: Dense and versatile image representations underpin the success of virtually all computer vision applications. However, state-of-the-art networks, such as transformers, produce low-resolution feature grids, which are suboptimal for dense prediction tasks. To address this limitation, we present *FlowFeat*, a high-resolution and multi-task feature representation. The key ingredient behind FlowFeat is a novel distillation technique that embeds a distribution of plausible apparent motions, or *motion profiles*. By leveraging optical flow networks and diverse video data, we develop an effective self-supervised training framework that statistically approximates the apparent motion. With its remarkable level of spatial detail, FlowFeat encodes a compelling degree of geometric and semantic cues while exhibiting high temporal consistency. Empirically, FlowFeat significantly enhances the representational power of five state-of-the-art encoders and alternative upsampling strategies across three dense tasks: video object segmentation, monocular depth estimation and semantic segmentation. Training FlowFeat is computationally inexpensive and robust to inaccurate flow estimation, remaining highly effective even when using unsupervised flow networks. Our work takes a step forward towards reliable and versatile dense image representations.

FlowFeat: Pixel-Dense Embedding of Motion Profiles
Nikita Araslanoy 12
Anna Sonnweber'
Daniel Cremers
TU Munich
MCML
Input
DINOV2 37
V-JEPA
VideoMAE 48
FeatUp 15
FlowFeat (ours)
Figure I: FlowFeat a versatile feature representation at pixel-level resolution. Embedding profiles of plausible motion, FlowFeat stands from existing techniques by offering excellent spatial precision coupled with temporal consistency. Here, we visualise (using PCA with three principal components) comparison of FlowFeat with the feature maps of the state-of-the-ee vision encoders.
Abstract
Dense and versatile image representations underpin the success of virtually all com- puter vision applications. However, state-of-the-arr networks, such as transformers, produce low-resolution feature grids, which are suboptimal for dense prediction tasks. To address this limitation, we present FlowFeat, a high-resolution and multi-task feature representation. The key ingredient behind FlowFeat is novel distillation technique that embeds a distribution of plausible apparent motions, or motion profiles. By leveraging optical flow networks and diverse video data, develop an effective self-supervised training framework that statistically approxi- mates the apparent motion. With its remarkable level of spatial detail, FlowFeat encodes a compelling degree of geometric and semantic cues while exhibiting high temporal consistency. Empirically, FlowFeat significantly enhances the representa- tional power of five state-of-the-art encoders and alternative upsampling strategies across three dense tasks: video object segmentation, monocular depth estimation and semantic segmentation. Training FlowFeat& is computationally inexpensive and robust inaccurate flow estimation, remaining highly effective even when using unsupervised flow networks. Our work takes a step forward towards reliable and versatile dense image representations]
Project wehsite: https:/tum-viisoon wwt.:iiiiiiiioiiiiiiii.e witthu.ii//towffaa io/flowfeat Code and pre-traieed models (Apache-2.0 Licensey: https:/ggitubbooowwuunnss//////r
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
The feature maps of state-of-the-art self-supervised encoders (e.g. 19 18 have drastically downsam- pled spatial resolutions (e.g. by a factor of 16), as illustrated Fig. While such downsampling improves the computational efficiency of deep networks, it compromises on the accuracy of dense prediction tasks, where spatial detail is crucial. Upsampling techniques. such as those based on bilateral filters [15], recover feature detail to impressive degree. However, bilateral upsampling incurs a tangible computational cost and struggles under challenging illumination scenarios (cf. Fig. third row). Alternatively, one could equip encoders with lightweight decoder module, such as DPT 41. However, training such decoders without human annotation is highly non-trivial. Building on this motivation, we present FlowFeat, multi-task pixel-level image representation obtained in label-efficient (or even label-free) manner.
Different from much of the existing work on representation learning, FlowFeat derives from dense motion patterns rather than the static appearance alone 10 18 37]. While FlowFeat is monocular model operating on a single input image at test time, it uses unlabelled videos for training to embed motion patterns into a pixel-level representation. Motion patterns are foundational to visual perception 33] they encode the compositional nature of visual scenes, encompassing both semantically and geometrically meaningful phenomena. However, as Fig. Iillustrates. video-based learning still fails to provide representations that are dense, versatile and effective [5 B 21].
As a step forward, we synergise state-of thettrr optical flow networks and real-world video data. On the one hand, modern optical flow networks produce dense motion estimates with outstanding accuracy, even in challenging settings 45 52 On the other hand, datasets of casual videos provide a treasure trove of motion and scene diversity [2 54). Combining both ingredients in joint learning framework, FlowFeat requires no manual annotation. Optical flow networks train predominantly on synthetically generated labels or even with self- ceffssueevviioo [35 44 video datasets derive from real-world benchmarks and require minimal curation (e.g. montage filtering). The key technical challenge is distilling the apparent motion in fashion accommodating its stochastic nature.
FlowFeat addresses this challenge with simple idea. We estimate the feature representation with a distribution of linear transformations. Intuitively, for a given image and flow estimate w.r.t. a randomly sampled counterpart, FlowFeat is trained admit a linear transformation approximating the flow. Specifically, every training iteration estimates lower bound of this transformation on-the-fly using a least-squares formulation. The statistical nature of this lower-bound approximation (due to sampling of the image pair) accommodates motion stochasticity and proves crucial for dealing with inaccurate flow and occasional static scenes. Consequently, the distribution of linear transformations allows FlowFeat to embed a distribution of plausible motion, or motion profiles [42)
Overall, our work presents two contributions. First, we develop an effective self supervised training framework that exploits the synergetic power of flow networks and large video datasets to embed motion profiles. Our framework is efficient at training time and can run comfortably within academic infrastructures. Second, we extensively evaluate the learned representation, FlowFeat, on three diverse tasks of dense prediction: video object segmentation (VOS), monocular depth estimation and semantic segmentation. Our analyses reveal a consistent benefit of FlowFeat across all tasks, exhibiting a compelling degree of temporal consistency and spatial detail. Furthermore, FlowFeat has appealing practical properties: (i) it is runtime- and label-efficient; (ii) it scales well with varying input resolution without the need for model fine-tuning, and (iii) it facilitates simple post- proseprocessing tasks, enhancing the quality of dense predictions without additional training.
2 Related Work
substantial effort towards unsupervised feature representations has focused on learning from large image sets 3
16 This development spans multiple axes of pursuit, such as model efficiency [[55], scalability I8 37 and framework architecture [12.. Although pre-training from image sets
II
dominates the research landscape in unsupervised learning, there have been natural extensions of image-based frameworks to learning from video data 13 48 However, it remains challenging to obtain spatio-ttmporal representations that are both dense (i.e. pixel-level) and temporally consistent
2 10 [5I. Central to learning spatio-temporal representations the design of the pre-text task. It constructs temporal palindrome
One prominent technique is cycle consistency [2] 27
43 51]

---

compute optical flor Floxtlt (frane, frane_ randon сrop transforus randon_crops
3
D
minA
A
two viers of the first frane
v1 crop(Irane, tf1)
v2 crop tf2)
F
ut uu crop(F, (rrpp dense leature mapa
Tep F
TAI
A +
11 x2 dec (encoder)
10 xi
U2
12 Eq. 4: teacher optinal
13 A" (x1, ui)
14 student predicted flev
15 u2 x2
3
Da
ming
w.r.t. dec 17 loss flov_loss (u2", u2)
Eq.
22.4" II
loss. backuard
19 EMA update
20 dec_ena. from
Figure 2: Embedding motion profiles: FlowFeat relies on the exponentially moving average (EMA) teacher model and learns reconstruct apparent motion with distribution of linear transformations. For given frame 4. we randomly sample its temporal counterpart 4r. A pre-trained network F computes optical flow F(t-ee). We generate two overlapping random crops of frame 4 and feed the resulting views and U2 to the teacher and the student networks, respectively. Obtaining the optimal linear transform A" on-the-fly with ridge regression in the teacher branch, we compute the reconstruction loss w.r.t. the flow crop U2 to update the student parameters 0 with gradient descent.
from a video sequence, ensuring consistency of a putative state in a forward and backward directions. Contrastive learning underpins another broad category of the research effort [39]. The main ideas are: constructing a reliable set of positive and negative samples [22]; combining learning on pixel, frame, and video levels [50] 53):; jointly representing a video clip with limited set of contrastive anchors [2]. Unlike these feature- based techniques, which have limited resolution, photometric- based learning, such colourisation, relies natural radiance-basse appearance [49]. Lai al. [28] leverage this technique video-based learning, reconstructing the target frame from previous frames observed in the CIELAB colour space.
Feature upsampling strategies, such as FeatUp [5] and LoftUp 20 are closely related our work. In contrast to bilateral upsampling [15 26), FlowFeat is more computationally efficient and has complementary properties to the low-resolution encoder features. Unlike contemporaneous work (0) leveraging SAM [25]. FlowFeat is label-efficient and can be trained in an unsupervised manner.
Representation learning by with motion estimation is not new the earlier works on trajectory clustering and
and traces back to
segmentation 7 14 29 [6). Training FlowFeat is efficient, since it does not require pairwise sampling [32] nor does it require object discovery [1. 38) Instead, FlowFeat learns directly from optical flow provided from off-the- networks with a distribution linear transformations. This approach takes primary inspiration from motion profiles, which model distribution of velocities at a given pixel [42]. Embedding motion profiles, FlowFeat enhances downstream accuracy the baseline representation across diverse tasks.
Embedding Motion Profiles
Linear maps for optical flow. To obtain pixel-level features enhancing the low-resolution rep- resentation of pre-trained encoders, we estimate apparent motion in real-world video sequences. Off-the-shelf optical flow models exhibit exceptional generalisation, despite being trained on synthetic
scenes [47 [52] or even with self-supervision 44) However, distilling motion estimates into a monoc-
ular model (in contrast to previous work (30), is highly non-trivial due to motion stochasticity train image representation Ho(l)
such that for any temporally neighbouring frame of I, there exists a linear operator on æ which approximates the optical flow w.r.t. that neighbour. Since we estimate the linear operator uniquely for each frame neighbour, the learned
Overcoming this issue,
Naively approximating optical flow with single linear layer unsurprisingly fails, verify Sec. 4.4
3

---

representation would embed statistical motion patterns for each input image I - an idea inspired by motion profiles 422
Given image I, and its temporal neighbour It- of resolution H x ((= N), we formulate the idea above with the following flow reconstruction objective (where || denotes "entry-wise" norm):
ming,4
Br,, [IIF
where Iv) E RNNx2 is the optical flow from pre-trained network 447 Ho((i) Nxx is our learned pixel-level feature representation and A E TRx is linear operator. Note that since Hy and A are both unknown, Eq. (1) an poo-p problem due to scale ambiguity TrerguityyTheeefrrree Therefore, we propose to compute the corresponding loss in two steps: (i) computing a lower-bound A with surrogate teacher network, while keeping H fixed; (ii) computing the gradient w.r.t. of the original network by swapping A into Eq. (1) as the lower-bound linear approximation.
Student-teacher framework. Fig. illustrates the framework and the corresponding training algorithm. Leveraging the mean teacher as the training model [46], our network HH := Dg o & comprises fixed (pre-trained) encoder & and a trained lightweight decoder Do. which outputs a dense feature representation of dimensionality d. The teacher model HEMA is equivalent to Ho with the exception of the decoder DEMA, which is an exponential moving average of Dg.
To construct the training batch, we sample two frames, 1 and 4y, where Ly' could be selected from temporal window around 4.- We first compute optical flow F(I,, Iv) E RRN x2 with a network
pre-trained on synthetic data [77] [52] or with self-supervision 44. Generating two overlapping
random crops of the first frame 4t we feed the corresponding views v1 and 0/2 to the teacher and student models, respectively. Using the teacher output, we solve least-qquares problem:
A argmin, A ||u1 FME (1. Al12
(2)
where i1 is the crop of the optical flow corresponding to view V1 In practice, solve Eq. (2) with ridge regression, which yields stable solutions in the presence of inaccurate flow estimates and improves training stability (ef. Sec. 4.4 for empirical results). Specifically, we solve
min ||u] - EMM (v1) Al12 + Al12
in each training iteration. Here, y is ridge hyperparameter fixed for all models. Setting T1 EMA (vi) to simplify the notation, the closed-form solution of Eq. (3) is naturally
A' = - (<I x + -1)-
Note that the first term has the feature dimensions, d d, fixed to d 128 in our experiments. Therefore, computing Eq. (4) has a negligible computational cost. In contrast to previous work [32. our framework remains computationally efficient regardless of the image resolution.
Fixing A", we now formulate the flow reconstruction loss w.r.t. the student parameters of Hg as
Lr.1 (w2, v2) = || - Ho (12) A|
The loss encourages the two overlapping crops of input frame to admit the same linear mapping A' from the features to optical flow, thereby promoting grouping of pixels with similar motion patterns. Note that for zero motion (i.e. static scenes) the solution is A = O, which yields zero gradient for the reconstruction term, effectively discarding such training samples in the learning process. As we also verify in the ablation study (ef. Tab. 3. ridge regularization and the robust L1 loss improve resilience of the framework to inaccuracies in the estimated target flow #1 and Ug respectively.
Focal gradient matching. Motion boundaries in optical flow are well-known to reveal semantic and geometric scene components. Therefore, we promote flow consistency at motion boundaries with an auxiliary second-ord aeder term implementing focal gradient matching:
c (uz, uz) = (1 Vava/ || 2
V
(6)
where u² : Ho(u)) A' and V, is the spatial gradient along the z-axis of the image plane. Equiva- lently, we compute the gradient for the y-axis and the corresponding term CV.
If A and H¹ are the solutions, so cA" and H' /e for any c 0.

---

Input
without Lx
with Lx
Ground Truth
0= 1100
o 0.1
0.05
Predictions
PeatUp
V.
FlowFeat
V.
Figure 3: Left: Focal gradient matching term Lv. The first row visualises the first three PCA components of FlowFeat trained with and without the gradient term. Observe sharper feature boundaries with the use of the gradient term. Additionally, we found benefit in modulating the gradient difference with a hyperparameter a, as defined in Eq. (6). The modulation with lower amplifies the effect of motion discontinuities (here, demonstrated for image gradients). Right: Qualitative examples VOS. FlowFeat reveals finer details of the semantic masks compared to existing upsampling strategies, such as FeatUp 15
Fig. 3 ilustratts the effect of the gradient matching loss. As we also demonstrate empirically in Sec. 4.4 the gradient loss results in sharper feature maps (see the top row in Fig. 3). Note that the focal term in Eq. (6) enables modulation of the gradient loss at motion discontinuities. As the two bottom rows in Fig. demonstrate, the hyperparameter a controls the degree of this modulation: lower value of a results in sharper FlowFeat boundaries. However, a very low value of may amplify the negative effect of inaccurate flow predictions, which can also exhibit flow discontinuities.
The total recond-order flow reconstruction loss is simply a weighted sum:
Liaal Lo ALLI.
where Co is the sum of 2 and CE, and À is trade-off hyperparameter kept fixed across all models.
Experiments
We probe FlowFeat three diverse tasks: video object segmentation (VOS), semantic segmentation and monocular depth prediction. Our goal to demonstrate that FlowFeat offers substantial and consistent benefits across these downstream tasks well across backbone models, regardless of their pre-training strategy. Overall, we train FlowFeat on top of five backbone models: Masked Autoencoder (MAE) (8] based ViT-B16. DINO 9 based ViT-B16 and ViT-S16, and DINO2 37 based on ViT-B 14 and ViT-S14 14. As the decoder architecture and the only trainable component FlowFeat, we use the DPT model 4 which is uutime--efficien (ef. Tab. supp. material). The
122 based on ResNet-34. However, our ablation experiments in Sec. 4.4 with the older RAFT model 47 and unsupervised flow 44 show that this choice of the flow estimator is not critical. Furthermore. Fig. 6 illustrates the resilience of the training to inaccurate flow targets. We report the results for two Flow variants. FlowFeat Y7' trains on 3471 video sequences from YouTube- VOS (CC BY 4.0, [54)). For larger backbones, we train FlowFeat- K on Kinetics-44e dataset (CC BY 4.0, 23 containing 147646 videos We compare our FlowFeat variants to the corresponding encoder model, as well as FeatUp [5] pre-trained on COCO-Stuff
flow distillation relies on NEA-RAFT
We exclude videos containing montage of multiple clips ensure temporal coherence.
5

---

2017 (val). FlowFeat sig- Table 1: Video object seg- mentation (VOS) with linear probing and label propaga- tion (local KNN) on DAVIS-
Linear Probing
Local KNN
Method
Train Data
FF
Jm
Fm
JF
Jm
Fm
V-JEAA 5
VideuMix2M
S
49.0
46.1
51.9
56.7
55.6
57.8
VideoMAE X8
Kinetics
43.3
40.9
45.8
55.1
54.6
55.6
MAE-B16 IN
ImageNet
40.8
43.1
44.3
42.8
45.8
nificantly improves the vOs
+ FlowFeat-K
Kinetics
53.8
50.1
57.5
89.1
57.3
60.8
accuracy of the baselines across all tested scenarios. It further outperforms previous and concurrent upsampling techniques (FeatUp [15] and Pre-training FlowFeat on the larger Kinet-
DINO-BI6 g
ImageNet
52.3
49.1
55.4
62.3
60.7
64.0
+ FlowFFea-YY
-YT-VOS
55.5
52,5
58.5
64.0
62.7
65.3
FlowFeet-KK
Kinetics
56.9
60.1
66.0
DI6 S
ImageNet
49.6
46.8
52.4
61.5
59.9
63.1
FennUp 1S
COCO-S
52.4
49.6
55.2
63.7
62.4
64.9
+ FlooFFeat-YT
+YT-VOS
54.1
51.1
57.0
63.7
62.0
65.5
LoftUp
20
+HowFeat-K
Kinetics
59.5
66.5
DINO2-B14 X
LVD"
61.6
58.5
64.7
66.4
64.4
ics datasets tends to produce a stronger representation. LVD* 71.2
68.5
-HooFeat-YT
-T-VOS
65.7
62.2
69.2
69.0
66.9
+ +FlowFeal-K K
Kinetics
66.1
62.3
69.9
69.9
67.7
72.1
refers to the distillation from a model pre- pre-trained on LVD 66.6
DINO2-S14 377
LVD"
57.5
54.2
60.7
65.1
63.7
+ FeaUUp 15
COCO-S
60.5
57.4
63.6
65.5
65.0
[37] LoftUp (20) uses SAM, 69.6
LoftUp 20
+SAIB 25
63.0
59.6
66.4
66.0
64.7
67.4
+ +lowFeel-Y
YT-VOS
65.8
62.0
69.7
67.6
65.6
trained with mask supervision
FlowFFeat-K K
Kinetics
64.6
61.0
68.2
68.5
66.1
70.9
on SA1B 25
(CC BY 4.0/ Flickr, ( Recall that FeatUp stacks multiple bilateral upsamplers and preserves the feature dimensionality. For instance, FeatUp yields representations with dimensionality 384 for VÝT-S, whereas FlowFeat is more compact and has a fixed dimensionality of 128 across all variants. This allows us to evaluate FlowFeat in complementary fashion to the backbone encoding by jointly fitting a high-resolution probe on FlowFeat and a low-resolution probe on the fixed encoder.
Implementation details (see also Appendix B). Training FlowFeat is computationally inexpensive. To train one model, we use single GPU with 46GB of memory. The training proceeds with mini- batches of 128 images, input resolution 224 x 224 and Adam optimiser [24] 31 with learning rate 10 and no weight decay. For the hyperparameters. we empirically set 0.1,
0.1 and 1.0 and did not observe sensitivity to moderate deviations from these values. We train FlowFeat for 500 epochs on YouTube-Ve VOS and for 100 epochs on Kinetics. In wall-clock time with one A40 GPU, the training takes only 24 hours and 3 days for YouTuue--VOT and Kinetics, respectively.
4.1 Video object segmentation
We evaluate FlowFeat on semi-supervised video object segmentation (VOS) using 30 validation sequences from DAVIS-2017 (CC BY-SA 4.0, [40).. The task is to propagate the ground-truth annotation defined in the first frame the rest of the video. Therefore, performing well on this task would indicate the capacity for temporal invariance as well as pixel-level semantic discrimination.
Previous evaluation protocols for VOS employ variant of a localised k-nearest neighbour classifier 2 28). referred to as local KNN in the following. This probing technique is known to be brittle, exhibiting high volatility w.r.t. its hyperparameters (34). For consistency with previous work, we stick to the implementation
local KNN provided by Caron et al. [9]. However, we additionally evaluate VOS with linear probing, the more established and interpretable technique in representation learning 16). Linear probing extends seamlessly to the VOS task. Specifically, for each video, we train linear classifier using the ground-truth segmentation provided for the first frame. We apply the linear classifier to the remaining frames to obtain the segmentation result. For both probing strategies linear probing and local KNN we compute the mean region similarity Im. the mean contour- based accuracy Fm and their mean J F. Tab. reports the results. Across all pre- pre-training methods and metrics, FlowFeat achieves a consistent and substantial improvement VOS accuracy. The benefit is especially significant for MAE-BE 16, where FlowFeat improves the baseline by staggering 13.0% / 14.8% JF with linear probing / local KNN. However, FlowFeat also surpasses stronger baselines. e.g. DINO2-B14 (+4.5% / +3.5% J.F) and FeatUp (+3.8% 6 / +2.8% JF for DINO-S16 and +5.3% +3.0% for DINO2-S14 J.). As illustrated in Fig. (right), the improvement is especially pronounced at the object boundaries. FeatUp enhances VOS accuracy for both baselines (DINO-SI 116 and DINO2-S14), but these improvements are more modest. FeatUp also struggles with inputs of higher resolution, introducing static feature artefacts, see the supplemental videos and further analysis

---

Table 2: Probing semantic segmentation and monocular depth. On COCO-Stuff 2017 (val), FlowFeat advances the segmentation quality across all baselines as well. A lightweight refinement using FlowFeat+++ (numbers in parentheses) further boosts the accuracy without any parameter training. On NYUv2 (val), FlowFeat significantly improves the depth accuracy across all pre-trained encoders contrast FeatUp, which struggles to improve upon baselines.
Semantic Segmeniation
Depth Estimation
Method
mloU
pAce
RMSE L
6,
82
6g
MAE-BI6 1 [ + FealWFea-K
46.0
71.5
0.4534
83,68
96.98
99.28
47.2
72,9
0.4400
84.43
97.18
99.35
DINO-B16 X
46.1
72.0
0.4287
86.15
97.61
99, 47
FlowFaaa-K
73.7
0.4176
86.87
97.71
99.50
FeatUp DINO-S16 (++)
(42. 1)
69.5
(69.9)
0.4624
83.54
96.90
99.32
DINO-S16 Ø
39.6
67.5
0.4634
83,60
96.94
99.32
+ FFlowFeat-Ye -e (++)
44.7
(45.9)
71.4
(72.5)
0.4410
85,26
97. 17
99. 30
FFlowFeat-K (++)
(45.4)
71.3
(72.3)
0.4422
84.81
97.19
99.37
DINO2-B14 [3
78.0
94.14
+ FlowFeat-K
60.4
79.8
0.2791
95.55
99.52
99.93
FeatUp DINO2-S1445
58.3
(58.5)
79.1
(79.2)
0.3207
93.29
99.15 18
99.86
DINO2-S14 [3]
56.2
77.3
0.3294
92.97
99. 11
99.85
+HowFeat-Y
(59.4)
78.7
(79.7)
93.91
+ FFlwFFeat K (++)
58.1
(59.6)
78.9
(79.9)
0.3061
94.12
99.31
99.88
Tab. 6. Similarly, the contemporaneous work, LoftUp (20) achieved inferior accuracy despite the
implicit leverage of vast mask supervision via SAM 255 Video-based models, such as V-JEPA 55) and VideoMAE
are also remarkably ineffective.
Overall, the improvements on VOS metrics provide compelling evidence that FlowFeat encapsulates high degree of temporal invariance and feature detail, with complementary properties to the encoder representation. Furthermore, the larger Kinetics dataset tends to produce stronger variant of FlowFeat. This observation indicates that FlowFeat has the promising capacity to scale with the ever-increasing volume of real-world videos.
4.2 Semantic segmentation
We follow the setting of FeatUp [13] and use POCO-Stuff 2017 with C = 27 coarsely annotated categories [8] Since FlowFeat focuses on motion patterns rather than global semantic alignment, may lack consistent semantic structure across images; therefore, employ attention probing [5] to derive image-specific class prototypes. In more detail, we define = 27 learnable queries attending the FlowFeat representation with single layer of attentoonn We freeze the models and train the probes on 256 x 256 centre crops using the cross-entropy loss. Additionally, we demonstrate that FlowFeat can further boost the segmentation accuracy with simple adaptation of lightweight post-processing technique. Concretely, we adapt the local mask refinement strategy (PAMR) []. but leverage FlowFeat instead of the image intensity refine the segmentation result. Note that such refinement is not possible by the use the probes alone due to their feed-for forward nature. We refer to this straightforward extension as FlowFeatt+. Appendix B.2 provides further details.
Tab. reports the mean pixel accuracy and the mean IoU. The results align with our observations in VOS experiments: FlowFeat boosts the accuracy across all baseline models. Particularly notable are the improvements w.r.t smaller models. For example, FlowFeat surpasses DINO-S16 by 5.1% and 4.6% with FlowFeat-YT and FlowFeat-K, respectively. Without the refinement, FlowFeat performs competitively with FeatUp [S]] based on DINO2-S14 and outperforms for DINO-S16. Furthermore, the FlowFeat- example, FlowFeat- K++ improves over FlowFeat- K by a notable margin of 1.5% mloU. By contrast, FeatUp does not profit from the refinement much.
refinement significantly enhances the segmentation quality. For
Fig. 5 visualises the segmentation results for the DINO2-S14 backbone, with and without refinement. Initial predictions of the probes are coarse and lack detail, especially around object boundaries. Leveraging the high-resolution FlowFeat representation (visualised with PCA), the refinement leads to sharper mask alignment with image boundaries.
7

---

Inpot
DIN02-S14
FlowFeat-K
FlowFeat-K (PCA)
FlowFeat-KK+
Ground
Figure 5: Semantic segmentation and post-hoc refinement (++) with FlowFeat, The segmentation masks from FlowFeat exhibit high level of boundary accuracy. The FlowFeat representation, visualised with PCA, identifies prominent scene elements with a fine-grained detail. lightweight post-hoc refinement FllowFeat-K+H based on PAMR [1]. leverages the pairwise pixel similarity embedded by FlowFeat (instead of image intensities) to improve the results further.
In summary, FlowFeat provides significant boost also for downstream semantic tasks. The feature representation offers high degree of spatial detail and also lends itself well to lightweight post- processing without the need for additional training.
4.3 Monocular depth estimation
We evaluate FlowFeat on geometric task, monocular depth estimation, using NYUv2 [36], following the evalu- ation protocol of Banani al. Similar to the VOS set- ting, we compare FlowFeat against self self-supervised back- bones: MAE [88] DINO [ and DINO2 37 As in
Inpul
semantic segmentation, we use attention probing [5] to ex- tract the depth-speeific prototypes from FlowFeat. Specifi- cally, we utilise the AdaBins (6] formulation that quantises the depth into 256 bins. The depth value is weighted sum of the predicted distribution across the bins and the corresponding depth value the bin. Following Banani et al. (4). we optimise the model using weighted combi- nation of the scale-invariant depth loss [6] and a gradient loss. Appendix B provides further details probe imple- mentation and training.
DINO2-S1A S14
Flat (PCA)
Adhering to the setting of Banani et al. 44, we train the probes on the NYUv2's s training set (24231 images) and evaluate the models on 480 x 480 centre crops of the 1 1449 validation images [3). As the standard depth met- rics, we report the root-mean-squared error (RMSE) and the inlier rates at three thresholds. Specifically, corre- sponds to the fraction of depth predictions d satisfying max(d/ (d/d", .d' /d) < 1.25' w.r.t. the ground-trute trutt d".
FlowFeat
Ground Truth
Tab. summarises the quantitative results. In line with our observations on VOS and semantic segmentation, FlowFeat achieves notable boost across all baseline mod- els. For example, FlowFeat-K reduces RMSE w.r.t. the DINO-S16 model by 0.051 and increases the d1 by 3. 3.16%. By contrast, we did not observe benefit from the high- resolution FeatUp, which appears to be biased towards the pre-training resolution of 224 224. Notably, we did not observe such detrimental bias in FlowFeat.
Figure 4: Depth probing. FlowFeat significantly improves depth estimates for challenging elements, such non- Lambertian surfaces (e.g. left, the piano), intricate structures (e.g. middle, the bicy- cle), and under- and oversaturated image areas (e.g. right, bathroom).

---

Table Ablation study on DAVIS-2017 (val),
Inpul
Target Flow
Our Estimate
FlowFFesl (PCA)
Following Sec. 4.1 we perform linear probing on VOS in set of ablation experiments. The A reports the absolute difference in the correspond- ing metric w.r.t. the baseline. The ridge regular- isation in FlowFeat is crucial, but the choice of the flow estimator is not instrumental.
Baseline: DINO2-S14
JF /A
Jm /
Fm /A
+Random DPT
58.7
55.2
62.2
FloowFentYY
69.7
maive
56.7 -9.1
52.8 -922
60.5 -9.2
Figure 6: Resilience to inaccurate flow targets.
(bJ 0.001, En.
58.2 -7.6
54.9 -7.1
61.5-8.1
[e] w/o Lv. of Eq, 6
64.3 -1.5
-1.0
62.7-2.0
Despite inaccurate and even artefact- artefact-prone prone predic-
(d)
67.0
tions from optical flow networks, FlowFeat learns
(e) w/o CLI of. Eq.
65.3 -0.5
60.6-0.4
69.0-0.7
reasonable flow approximation without compro-
(f) RAFT
65.2 -0.6
61.6
-0.
68.8 -099
SMURF (unsupervised)
64.1 -1.7
60.7
-1.
67.5 -22
mising the feature representation.
(b) temp. window x
65.5
-0.3
62.1
+0.1
68.9 -08
( frame only
65.8 0.0
62.2
+0.2
69.4 -0.
We visually inspect the results in Fig. 4 for the DINO2-S14 14 backbone. In contrast to the low-quality depth estimates extracted from the frozen encoder. FlowFeat representation exhibits an impressive degree of fine-grained detail. This is indeed noteworthy. considering that FlowFeat originates from video data and was not trained for such static scenes. FlowFeat also robust to under- and oversaturated image areas (ef. Fig. 4 the rightmost column) and infers highly plausible depth where the ground truth is not available due to surface specularity (ef. Fig. 4 the piano).
In summary, the results suggest that the motion profiles embedded by FlowFeat provide strong geometric cues. FlowFeat not only enhances the depth awareness across all baselines, but also scales compellingly with the increased amount of video data for pre-training: FlowFeat-K outperforms the less data-intensive FlowFeat-Y YT across virtually all settings and metrics.
4.4 Ablation study
We conduct an ablation study of FlowFeat on the DAVIS-200 2017 (val) benchmark. The study follows the evaluation protocol with linear probing from Sec. 4.1 Using ViT-S14 backbone pre-trained with DINO2 [37], we train number of FlowFeat configurations YouTube- [54]. Tab. B reports results. As a sanity check, we verify that a randomly initialised DPT decoder has virtually no effect the VOS accuracy (cf. Tab. 3 in grey). Similarly, (a) naïvely fitting optical flow with single linear layer (instead of a distribution) is futile. Next, (b) we verify the benefit of estimating the optimal operator A in Eq. (5) with ridge regression. To compare with the baseline setting of y 1 (ef. Eq. (4) ). we trained the model with 7 = 10-3 We found the training numerically unstable with O. Thus, setting r to 10 provides reasonable approximation to removing the effect of L2- regularisation on the linear operator A. In this case, the VOS accuracy drastically deteriorates across all metrics, which justifies the crucial need for ridge regularization. (c) We train FlowFeat without the second-order term, Lo in Eq. (6). A drop in the downstream accuracy (e.g. -2.0% Fm) suggests that FlowFeat exploits motion boundaries in its representation, in line with the established view that motion boundaries are strong semantic cues. (d) Replacing the L1 reconstruction loss by L2 distance in Eq. (5) reduces J&F from 65.8% 63.4%, supporting the robustness of L, in comparison to bg- Next, (e) we explore a configuration without the 1.1 reconstruction term by setting X = 0 in Eq. 7 Surprisingly, the drop in accuracy not substantial. This suggests that the training process can succeed with the gradient loss alone. However, we observed that including the LI reconstruction term tends to improve the convergence speed consistently across all models. In the next experiments (f.g), we replace SEA-RAFT [52] with the RAFT model [7] and the unsupervised SMURF [44]. respectively. The drop in the VOS accuracy not significant, which demonstrates that obtaining FlowFeat is not sensitive to a specific choice of the flow model. Fig. further examines the training resilience to inaccurate target flow. both examples, the optical flow from the pre-trained network is inaccurate (even artefact-pronee, yet it has no visible effect on the quality of FlowFeat. Curiously, the second example in Fig. also reveals one limitation of FlowFeat: the apparent motion of the tail and the head of the snake have opposite directions, which decouples their feature representation.

---

Finally, (h, we test two strategies for sampling frame pairs from a video in Tab. Our base configuration uses a temporal window of 5 frames and can select frame t' either from the past or the future. We increase this temporal window to 9 (h), which leads to slight deterioration of VOS accuracy - presumabby due to the more challenging estimation of optical flow. The setting (i) selects the immediate next future frame as t'. Here, the VOS accuracy barely changes. This implies that (i) FlowFeat does not simply overfit apparent motion (compare to (a)), and (ii) the motion samples across the dataset, not just temporal window, play a more critical role in embedding motion profiles.
Further study. Our further analysis, provided in the supplemental material, shows that: (1) FlowFeat scales well with the input resolution, further improving the VOS accuracy when the input resolution is doubled (cf. Tab. 6): (2) the accuracy gains from FlowFeat do not arise merely from higher resolution of the feature map per se, but from its complementary motion-derived properties (cf. Tab. Sa); (3) Feaw also scales effectively to larger transformer models (e.g. ViT-I.) (ef. Tab. 5b).
5 Limitations
Application scope. FlowFeat relies on pre-trained optical flow network and video data for training. assumes either brightness constancy in the video stream or availability of synthetic data for pre- training the optical flow model. While these assumptions generally hold for standard RGB videos, they may not apply in other domains, such as medical imaging (e.g. MRI, CT), thermal imaging or low-light scenarios.
Frozen backbone. Recall that train- ing FlowFeat involves updating only the decoder parameters, while keep-
Model
Person
Wall
Landscape
Vegetation
Ground
DINO-S16
65.5
33.3
FlowFeat-YY -YT
75.6
50.0
50.8
69.9
37.0
ing the encoder parameters fixed. Consequently, the encoder represen- tation imposes an upper bound on FlowFeat's 's downstream accuracy, es- pecially in terms of high-frequency content. Although we have shown that FlowFeat generalises across widely used self-supervised encoders, such as MAE 18) DINO (9) DINOv2 [37 and across different model capacities, FlowFeat may be less effective with backbones that underrepresent high- frequency details in their intermediate feature maps.
DINO-B16
72.9
51.4
51.0
70.3
38.9
FlowFeet-K
77.8
52.9
53.1
71.7
39.6
DINOv2-S14
76.9
57.6
59.2
71.0
44.6
FlowFeat-YY
81.7
59.3
60.5
73.0
45.7
DINOv2-B14
77.0
59.2
59.6
20.3
44.9
FlowFeat-K
72.3
45.1
MAE-B16
52.7
66.9
FlowFeat-K
78.6
51.3
53.7
69.9
38.8
Table 4: Semantic segmentation accuracy COCO-Stuff (IoU, %). As expected from motion parallax, the gains (potentially) dynamic classes (e.g. "person")) are larger compared to that of typical background categories (e.g. "veg- etation"). Nevertheless, FlowFeat leads a consistent seg- mentation improvement across all categories.
Owing to its training approach, FlowFeat tends to empha-
Motion bias.
sise image regions with larger magnitudes of expected motion, typically corresponding foreground dynamic objects, relative to the static background areas. To quantitatively assess this behaviour, we report per-category IoU scores COCO-Stuff in Tab. 4. following the probing protocol described in Sec. 4 We observe that the improvement on the "person" category is indeed more pronounced than for static classes. Nevertheless, FlowFeat yields consistent accuracy gains across all categories, regardless of whether they are static or dynamic in nature.
Conclusion
We presented FlowFeat, a pixel-dense and versatile representation embedding motion profiles. Our experiments provide compelling evidence that FlowFeat enhances the representation power of pre- trained encoders across all downstream tasks considered our study. Specifically, FlowFeat possesses temporal consistency and exhibits a remarkable level of spatial detail, encompassing semantic and geometric cues without explicit supervision. More broadly, our work addresses motion stochasticity in a principled fashion, revealing a powerful synergy between optical flow networks and large video datasets. FlowFeat takes a significant step towards label-efficient and versatile models for high-precisin tasks, such image-based based 3D reconstruction, object-level segmentation and tracking.
10

---

Acknowledgements.
This work was supported by the ERC Advanced Grant SIMULACRON and DFG project 250/26-1 YouTbe,b. thanks Junhwa Hur and Jochen Gast their valuable feedback.
References
[1] N. Araslanov and S. Roth. Stagle-stage semantic segmentation from image labels. CVPR, 2020.
[2] N. Araslanov, S. Schaub-Meyer, and S. Roth. Dense unsupervised learning for video segmentation. In NeurIPS, 2021.
[3] M. Assran, Q Duval. I. Misra, P. Bojanowski, P. Vincent, M. G. Rabbat, Y. LeCun, and N. Ballas. Self-supervised learning from images with joint-embedding predictive architecture. In CVPR, 2023.
[4] M. E. Banani, A. Raj, K. Maninis, A. Kar, Y. Li, M. Rubinstein, D. Sun, I. J. Guibas, J. Johnson, and
V. Jampani. Probing the 3D awareness of visual foundation models. In CVPR, 2024.
[5] A. Bardes, Q Garrido, J. Ponce, X. Chen, M. G. Rabbat, LeCun, M. Assran, and N. Ballas, Revisiting feature prediction for learning visual representations from video. TMLR, 2024.
[6] S. Bhat, Alhashim, and P. Wonka. AdaBins: Depth estimation using adaptive bins. CVPR, 2020.
[7] T. Brox and J. Malik. Object segmentation long term analysis of point trajectories. In ECCV, 2010.
[8] H. Caesar, Uijlings, and V. Ferrari. COCO-Stuff: Thing and stuff classes context. In CVPR, 2018.
[9] Caron, Touvron, Misra, Jégou, Mairal, Bojanowski, and A. Joulin. Emerging properties self-supervised vision transformers. ICCV, 2021.
[10] M. Caron, N. Houlsby, and C. Schmid. Location-aware self-supervised transformers for semantic segmen- tation. WACV, 2024.
[11] T. Chen, S. Korbblith, M. Norouzi, and G. E. Hinton. A simple framework for contrastive learning of visual representations. In ICML. 2020.
[12] X. Chen and K. He. Exploring simple siamese representation learning. In CVPR, 2021.
[13] C. Feichtenhofer, H. Fan, Y. Li, and K. He. Masked autnencoders as spatiotemporal leamers. In NeurlPS, 2022.
[14] K. Fragkiadaki. P. Arbeláez, P. Felsen, and J. Malik. Learning to segment moving objects videos. In CVPR, 2015.
[15] S. Fu, M. Hamilton, L. E. Brandt, A. Feldman, Z. Zhang, and W. T. Freeman. FeatUp: model-agnostic framework for features at any resolution. ICLR, 2024.
[16] Grill, Struh, F. Altché, C. Tallec, H. Richemond, E. Buchatskaya, Doersch, B. Á. Pires, Z. Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. Munos, and M. Valko. Bootstrap your own latent A new approach to self-supervised learning. In NeurIPS, 2020.
[17] T. Han, Xie, and A. Zisserman. Self-supervised co-training for video representation learning. In NeurIPS, 2020.
[18] He. X. Chen. S. Xie, Li. Dollár, and R. Girshick. Masked autoencoders are scalable vision leamners. In CVPR, 2022.
[19] O. J. Hénaff, S. Koppula, F. Shelhamer, D. Zoran, A. Jacgle, A. Zisserman, J. Carreira, and Arandjelovic. Object discovery and representation networks. In ECCV, 2022.
[20] H. Huang, A. Chen, V. Havrylov, A. Geiger, and D. Zhang. LoftUp: Leaming coordinate based feature upsampler for vision foundation models. arXiv:2504. 140.2 [cs. CVJ, 2025.
[21] A. Jabri, A. Owens, and A. A. Efros. Space-time correspondence as contrastive random walk. In NeurIPS, 2020.
[22] S. Jcon, D. Min, S. Kim, and K. Sohn. Mining better samples for contrastive learning of temporal correspondence. In CVPR, 2021.
[23] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vjjyanarasimman, E. Viola, T. Green, T. Back, P. Natsev, M. Suleyman, and A. Zisserman. The Kinetics human action video dataset. arXiv:1705.06955 (es. CVJ. 2017.

---

[24] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[25] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, and W.-Y. Lo et al. Segment anything. In CVPR, 2023.
[26] J. Kopf, M. F. Cohen, D. Lischinski, and M. Uyttendacle. Joint bilateral upsampling. ACM Trans. Graph., 26(3):96, 2007.
[27] Z. Lai and W. Xie. Self- supervieee video representation learning for correspondence flow. In BMVC, 2019.
[28] Z. Lai, E. Lu, and W. Xie. MAST: A memory- self-supervised tracker. In CVPR, 2020.
[29] C. Liu, A. Torralba, W. T. Freeman, F. Durand, and E. H. Adelson. Motion magnification. ACM Trans. Graph., 24(3):519-526, 2005.
[30] P. Liu, M. Lyu, King, and Xu. Learning by distillation: A self-supervised learning framework for optical flow estimation. IEEE Trans. Puttern Anal. Mach. Intell., 44(6): 5041, 2022.
[31] Loshchilov and F. Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[32] A. Mahendran, J. Thewlis, and A. Vedaldi. Cross pixel optical-flow similarity for self-supervised learning. In ACCV, 2018.
[33] D. Marr and Vaina. Representation recognition of movements of shapes. Proceedings of the Royal Society of London. Series B. Biological Sciences, 214(1 197):501-5244 1982.
[34] D. McKee, Z. Zhan, B. Shuai, D. Modolo, J. Tighe, and S. Lazebnik. Transfer of representations to vidco label propagation: Implementation factors matter. 2:03::220300555. [cs.CV] 2022.
[35] S. Meister, J. Hur, and S. Roth. UnFlow: Unsupervised learning of optical flow with a bidirectional census loss. In AAAI, 2018.
[36] P. K. Nathan Silberman, Derek Hoiem and R. Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012.
[37] M. Oquab, T. Darcet, T. Moutakanni, and H. V. et al. DINOv2: Learning robust visual features without supervision. arXiv:2304.07199 [es. 2023.
[38] D. Pathak, R. B. Girshick, P. Dollár, T. Darrell, and B. Hariharan. Learning features by watching objects move. In CVPR, 2017.
[39] P. O. Pinheiro, A. Almahairi, Y. Benmalek, F. Golemo, and A. C. Courville, Unsupervised learning of dense visual representations. NeurlPS, 2020.
[40] Pont-Tuset, Perazzi, S. Caelles, Arbeláez, Horkine and L. Van Gool. The 2017 DAVIS challenge on video object segmentation. arXiv:1704.0000 00675 (cs.CV), 2017.
[41] R. Ranftl, A. Bochkovskiy, and V. Koltun. Vision transformers for dense prediction. ICCV, 2021.
[42]
and J. Malik. Motion segmentation and tracking using normalized cuts. In ICCV, 1998.
[43] Son. Contrastive learning for space-time correspondence via self-cycle consistency. CVPR, 2022.
[44] A. Stone, D. Maurer, Ayvaci, Angelova, and Jonschkowski. SMURF: self-tcaching multi-frame unsupervised RAFT with full-image warping. In CVPR, 2021.
[45] D. Sun, X. Yang, M. Liu, and Kautz, PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. In CVPR, 2018.
[46] A. Tarvainen and H. Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi cemi-supervised deep lcarning results. In NIPS, 2017.
[47] 7. Teed and J. Deng. RAFT: Recurrent all-pairs field transforms for optical flow. In ECCV, 2020.
[48] Z. Tong, Y. Song, J. Wang, and L. Wang. VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training. In NeurlPS, 2022.
[49] C. Vondrick, A. Shrivastava, A. Fathi, S. Guadarrama, and K. Murphy, Tracking emerges by colorizing videos. In ECCV, 2018
12

---

[50] N. Wang. W. Zhou, and H. Li. Contrastive transformation for self-supervised correspondence learning. In AAAI, 2021.
[51] X. Wang. A. Jabri, and A. A. Efros. Learning correspondence from the cycle-consistency of time, In CVPR, 2019.
[52] Y. Wang, L. Lipson, and J. Deng. SEA-RAFT: Simple, efficient, accurate RAFT for optical flow. In ECCV, 2024.
[53] J. Xu and X. Wang. Rethinking self-supervised correspondence learning: A video frame-level similarity perspective. In ICCV, 2021.
[54] N. Xu, L. Yang, Y. Fan, D. Yue, Y. Liang, J. Yang. and T. S. Huang. YouTube-VOS: A large-scale video object segmentation benchmark. arXiv: arXiv:/809.000 [cs.CV], 2018.
[55] J. Zbontar, L. Jing, I. Misra, LeCun, and S. Deny. Barlow Twins: Self-supervised learning via redundancy reduction. In ICML, 2021.
[56] I.. Zelnik-Manor and M. Trani. Degeneracies, dependencies and their implications in multi-body and multi-sequence factorizations. In CVPR, 2003.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim that FlowFeat encompasses semantic and geometric cues, while being spatially precise and temporally consistent. We extensively support these claims empirically with five baselines on three corresponding probing tasks (semantic segmentation, depth estimation and video object segmentation), as well as qualitatively analyse the feature maps (ef. Fig.
T and the supplemental material).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The snake example in Fig. 6 the corresponding discussion in Sec. 4.4 and Sec. 5 highlight multiple limitations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
14

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our work is predominantly empirical, although it includes some theoretical aspects (e.g. ridge regression), which are widely well-understood.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Sec. 4 and Appendix B provide details of the training and probing settings.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
15

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: The training data (YouTube- VOS and Kinetics-400) as well the evaluation datasets (e.g. DAVIS-2011 are publicly available. We release the code and the pre-trained models, and provide the corresponding URLs.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to rum to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.co wwppulllgguiiiieeeeeeeeeepeeebll tttpg/dde//eeeeubbmmbiecese wwtp///oddesissiiiooae wommisssioopicyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Sec. elabbraates on the data splits, hyperparameters, and the optimiser used for training. Appendix B further details the evaluation settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
The errors bars are not included due to the limited computational budget.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All experiments have been performed on one A40 GPU (cf. Sec. 4 "Imple- mentation details"). Appendix C also performs a computational analysis.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
This research has adhered to all aspects of the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This research has foundational nature, since it does not aim at deployment- level accuracy on any particular downstream task.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Answer: [No]
Justification:
We do not perceive such risks in this research due to its foundational nature.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Asset owners/creators credited with the corresponding citation. Most assets used in this work (the datasets DAVIS-2011 2017, YouTube- -oS and Kinetics-400) are publicly available and include the license (CC BY-SA 4.0, CC BY 4.0, CC BY 4.0, respectively). However, we have been unable to identify the license of the NYUv2 dataset, which is widely used in our community.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
18

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pa se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We release the code and pre-trained models with detailed instructions upon acceptance under Apache License 2.0.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The work did not use crowdsourcing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The work did not use crowdsourcing experiments or research with human subjects.
Guidelines:
19

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The core method development in this research does involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
20


Reviews:
[{"review_id": "h9FliszCmZ", "reviewer": "Reviewer_meu9", "summary": "This paper proposes a new method for training an upsampling decoder for image features in a self-supervised way by taking advantage of video data. The goal is to take image features from a frozen, large-scale pre-trained backbone (encoder) and upsample them to dense features for better performance on dense tasks such as semantic segmentation and depth estimation. The work is most similar to LoftUp and FeatUp which are other feature upsampling methods with similar use cases. FlowFeat claims to be different by not requiring runtime optimization like FeatUp and also not relying on pre-trained models such as SAM to generate full resolution pseudo-GT like LoftUp. By contrast, FlowFeat claims to be fully unsupervised, fast to inference, efficient to train, and also comes with temporal consistency advantages compared to other methods.\n\nThe primary method for generating these upsampled features is by using a DPT style decoder head which is trained to predict dense \"motion profiles\" from a single image. This isn't quite optical flow because predicting flow from a single image is an under constrained task, thus the actual goal is to train a feature vector such that a simple linear transformation will produce the correct flow for any temporally neighboring frame. The supervision for these flow outputs comes from a pre-trained optical-flow network such as RAFT or SMURF.", "questions": "1. How much feature map upsampling are you able to achieve with good results using FlowFeat?\n2. Do the random crops used as input to the flow model have to be a specific resolution?\n3. Does the resolution of the flow network's outputs affect the DPT decoder's ability to upsample?", "limitations": "Given the reliance on flow, I'd like to see the authors dive a bit more into the performance on \"static background\" elements in images (for example, segmentation performance on a tree in the background). This seems like it may be a limitation, but is hard to see from the qualitative examples provided. Additionally pointing out that the method only really applies to RGB web images, and not other domains is a good idea (as mentioned in the weaknesses section).", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "## Strengths\nThe main strength of FlowFeat is it's unique formulation which makes it generalizable (no per-frame optimization) and inexpensive to train. The core idea of using motion cues and optical flow as a supervisory signal for dense features is brilliant and it clearly has positive impacts of tasks requiring temporal consistency such as video object segmentation. The linear transformation & motion profile section is also well explained, and in combination with a public code release makes a compelling case for wider adoption and testing in a wide variety of tasks. One example I can think of is that I suspect the features produced by FlowFeat might make for a good drop-in replacement for those used by point tracking methods.\n\n## Weaknesses\nThe main weaknesses of the paper are twofold. One is that the flow based features appear to lose background features when visualized the same way as those from FeatUp (for example). This is best seen in Figure 1 in the skateboarding example where FlowFeat seems to smush the background (sky tree etc.) into an amorphous blob. It's also visible in the videos provided in the supplemental. The second is that it's positioned as an unsupervised method, which I guess technically it is (especially if you use SMURF) but needing a good flow estimation network does limit it's applicability somewhat (for example, to other domains such as medical images where pre-trained flow methods might not work as well). Though really I consider this second weakness to be more of a nitpick.", "was_revised": true, "final_justification": "The authors adequately answered my questions, and clarified the slight limitations of their proposed method. I do not see this as a fundamental issue since the method is still a strict improvement over the baseline. I maintain my rating to accept.", "rebuttal": "Thank you for your considerate, very encouraging feedback. \n\n> **\"[...] flow based features appear to lose background features when visualized [...]\" / \"[...] dive a bit more into the performance on \"static background\" elements [...]\"**\n\nThis is an accurate observation. Due to motion parallax, FlowFeat will prioritise foreground elements, where the apparent motion is expected to have a larger magnitude than that of the background pixels. However, FlowFeat also retains useful pixel-level information about the background structure. We show this empirically in Tab. 2 and elaborate further next.\n\nRecall that NYUv2 is composed of static elements, which are semantically akin to background. On this benchmark, the improvements of FlowFeat over the baselines are consistent across all scenarios and notably surpass FeatUp. This suggests that FlowFeat not only preserves the static details (also visualised in Fig. 3, third row), but also leverages motion parallax as a useful depth cue.\n\nOur segmentation results on COCO-Stuff are similarly consistent. As suggested, we further inspected the segmentation accuracy of non-dynamic categories. Specifically, we analysed the per-category IoU for semantic classes attributable to background areas (e.g. “trees”). As we report below, FlowFeat improves on those categories across all models, though somewhat less significantly than on (potentially) dynamic objects. For reference, we also provide the accuracy for ‘person’, as a typical foreground instance.\n\n| Model | Person | Wall | Landscape | Vegetation | Ground | \n|-----------|-----------|-----------|-----------|-----------|-----------|\n| DINO S16 | 69.3 | 46.5 | 43.9 | 65.5 | 33.3 |\n| +FlowFeat-YT | **75.6** | **50.0** | **50.8** | **69.9** | **37.0** |\n\n| Model | Person | Wall | Landscape | Vegetation | Ground | \n|-----------|-----------|-----------|-----------|-----------|-----------|\n| DINO B16 | 72.9 | 51.4 | 51.0 | 70.3 | 38.9 |\n| +FlowFeat-KT | **77.8** | **52.9** | **53.1** | **71.7** | **39.6** |\n\n| Model | Person | Wall | Landscape | Vegetation | Ground | \n|-----------|-----------|-----------|-----------|-----------|-----------|\n| DINOv2 S14 | 76.9 | 57.6 | 59.2 | 71.0 | 44.6 |\n| +FlowFeat-YT | **81.7** | **59.3** | **60.5** | **73.0** | **45.7** |\n\n| Model | Person | Wall | Landscape | Vegetation | Ground | \n|-----------|-----------|-----------|-----------|-----------|-----------|\n| DINOv2 B14 | 77.0 | 59.2 | 59.6 | 70.3 | 44.9 |\n| +FlowFeat-KT | **83.0** | **61.7** | **61.3** | **72.3** | **45.1** |\n\n| Model | Person | Wall | Landscape | Vegetation | Ground | \n|-----------|-----------|-----------|-----------|-----------|-----------|\n| MAE B16 | 72.2 | 50.8 | 52.7 | 66.9 | 36.1 |\n| +FlowFeat-KT | **78.6** | **51.3** | **53.7** | **69.9** | **38.8** |\n\n\nWe will add analysis of this limitation to the final revision (e.g. in the additional 10th page). Thank you for the suggestion.\n\n\n> **\"[...] needing a good flow estimation network does limit it's applicability somewhat [...]\"**\n\nIndeed, our approach requires a pre-trained optical flow network and video data. However, if the video sequences in another domain satisfy the assumption of brightness constancy, optical flow networks will likely still operate reliably. Consequently, a fine-tuned FlowFeat on the domain would also provide reasonable representations. However, we acknowledge that the success of FlowFeat ultimately depends on the training data available in the domain.\n\n> **Q.1 How much feature map upsampling are you able to achieve with good results using FlowFeat?**\n\nAs we show in the supplemental videos and Tab. 4 (supp.), scaling up the input to FlowFeat to resolution 448x448 improves the downstream accuracy on VOS further – in contrast to FeatUp. This resolution is already close to the upper bound of DAVIS (480p), which we used in the experiments. Following your suggestion, we have now inspected FlowFeat for the input resolution of 1080p. Since we cannot share any visual data, we report our observations. The PCA visualisation exhibits sharp boundaries, aligned with the image gradient and (potential) motion discontinuities. We observed no trace of artefacts akin to those emerging in FeatUp (c.f. supp. videos). Please note that evaluating transformers at resolution 1080p significantly increases their inference time and the memory footprint, thus we could not timely evaluate the models on benchmarks.\n\n\n> **Q.2 Do the random crops used as input to the flow model have to be a specific resolution?**\n\nWe feed frame pairs at resolution 224x224 to the flow network. Then, we sample the random crops from the produced flow map to align them with the input and output of the FlowFeat network. Therefore, the target flow maps used to train FlowFeat are bilinearly upsampled by a factor from the range (1, 2.5]. We did not find the training to be sensitive to moderate deviations from this range. \n\n> **Q.3 Does the resolution of the flow network's outputs affect the DPT decoder's ability to upsample?**\n\nA low-resolution optical flow would conceivably result in smooth boundaries in the feature map of FlowFeat. Conversely, a high-resolution optical flow would increase the boundary sharpness, while also increasing the computational costs. We found a good compromise by running optical flow at the training resolution of FlowFeat (224x224) and upsampling the flow map to fit the input crops with bilinear interpolation, as described previously.\n\n> **[...] the method only really applies to RGB web images, and not other domains [...]**\n\nWe will point out the current application scope of FlowFeat to RGB images from the Internet – thank you for suggesting it."}, {"review_id": "zLIeAPCx4B", "reviewer": "Reviewer_7P8o", "summary": "This paper introduces FlowFeat, a novel, high-resolution feature representation for images that is learned in a self-supervised manner from video data. The core idea is to distill \"motion profiles\"—distributions of plausible motion—from optical flow estimates into a dense, pixel-level feature map. This is achieved by training a lightweight decoder on top of a frozen, pre-trained image encoder. The training objective forces the learned features to be able to reconstruct the optical flow between pairs of video frames via a dynamically computed linear transformation. The authors demonstrate that FlowFeat significantly enhances the performance of several state-of-the-art visual encoders on a variety of dense prediction tasks, including video object segmentation, monocular depth estimation, and semantic segmentation. The method is shown to be label-efficient, computationally inexpensive to train, and robust to inaccuracies in the underlying optical flow estimation, offering a practical step toward creating more versatile and detailed image representations.", "questions": "1. In Section 4.1, you state that for linear probing on DAVIS-2017, a linear classifier is trained on the ground-truth segmentation of the first frame. Given that the first frame contains only a single annotated instance for each object, how do you construct a balanced training set for the linear classifier, especially for distinguishing the object from a diverse background?\n\n2. The ablation study in Table 3 shows that removing the L1 reconstruction loss (experiment d) leads to only a minor drop in accuracy, suggesting the gradient loss L_{∇} is doing most of the work. This is surprising. Can you explain the reason?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1. The core idea of embedding \"motion profiles\" by learning to approximate optical flow with a distribution of linear transformations is both novel and well-motivated. It leverages the strong geometric and semantic cues inherent in motion without requiring expensive manual annotations.\n\n2. The paper presents a comprehensive set of experiments across three distinct and challenging dense prediction tasks. FlowFeat consistently and significantly improves upon strong baseline encoders (MAE, DINO, DINOv2) and outperforms or performs competitively with alternative upsampling techniques like FeatUp. The improvements are particularly notable for weaker encoders like MAE.\n\n3. The ablation studies and qualitative results demonstrate that the method is robust to the choice of the optical flow network and resilient to inaccurate or artifact-prone flow estimations, which is a critical property for real-world application.\n\n\n\nWeakness\n\n1. The training relies on the assumption that two different, overlapping crops from the same initial frame should be able to predict the flow in their corresponding regions of the target frame using the same linear operator A *. While this works well empirically, the justification for this core assumption could be stronger.\n\n2. While the paper commendably points out a limitation with non-rigid objects (the snake example), a broader discussion of potential failure modes would be beneficial. For example, how does it handle scenes with very little or no motion, or scenes with extremely complex, non-linear motion patterns?\n\n3. The authors state that state-of-the-art networks, such as transformers, generate low-resolution feature grids that are suboptimal for dense prediction tasks. How is this suboptimality quantified or evaluated in practice?", "was_revised": true, "final_justification": "Thank you to the authors for their responses. I find the work interesting and valuable to the community. My main concern was the lack of theoretical analysis in the proposed learning-based model. However, as the authors clarified during the rebuttal phase, providing such an analysis may be non-trivial. Therefore, I am willing to raise my score. That said, I strongly recommend including a self-consistent theoretical analysis in the camera-ready version.", "rebuttal": "Thank you for your time and your valuable feedback. \n\nWe agree with both of your main points: the justification of the assumptions and the discussion of the limitations could be expanded upon, for which we had limited space in the submission. NeurIPS will provide an additional page for the camera-ready version, and we will be happy to use the allocated space to address your concerns, as we do below:\n\n>  **[...] the justification for this core assumption could be stronger.**\n\nThe main idea behind the assumption derives from the following observation:\nthe apparent motion (optical flow) is structured and spatially correlated. Therefore, the motion of two overlapping crops will be correlated as well. We model this correlation by a shared linear mapping A*, which predicts optical flow in both crops. As a result, the feature representation in both random crops will have a (partially) shared and spatially correlated structure.\n\nNote that the intuition behind the cropping is not dissimilar to previous work (e.g. DINO), where two random crops are required to share a (global) representation. Our novelty is a dynamic computation of the global representation (A*), which additionally allows for efficient learning of per pixel motion-aware representations.\n\n> **[...] how does it handle scenes with very little or no motion, or scenes with extremely complex, non-linear motion patterns?**\n\nWe assume that the question relates to the training process, since FlowFeat is a *monocular* model (it takes only one image as the input).\n\nWhen there is no motion, there is a trivial linear mapping (zero vector), so the learning gradient will be zero, effectively discarding the training sample.\n\nState-of-the-art optical flow networks can handle relatively complex non-rigid motion, hence provide useful cues for training FlowFeat also in challenging non-rigid scenarios. Moreover, training FlowFeat is not very sensitive to errors in optical flow – thanks to the ridge regularisation in the least-squares solver (c.f. Eq. (4)) and the (robust) L1 training loss. However, we excluded compilation videos (i.e. multiple short clips stacked together) from the Kinetics dataset before training, since they break motion continuity. \n\n> **The authors state that state-of-the-art networks, such as transformers, generate low-resolution feature grids that are suboptimal for dense prediction tasks. How is this suboptimality quantified or evaluated in practice?**\n\nAll our experiments with the baselines are dense tasks, which exemplify one way of quantifying and evaluating the low-resolution feature grids. This setting corresponds to our baselines in Tabs. 1 and 2 (e.g. DINO, MAE).\n\nQualitatively, the limitations become particularly apparent in Figs. 1, 4 and 5.\n\n> **[...] how do you construct a balanced training set for the linear classifier, especially for distinguishing the object from a diverse background?**\n\nWe do not finetune or re-balance the training set for the linear probe. Indeed, the classification problem for fitting the linear probe in VOS can be highly imbalanced. However, this is not an issue, because the imbalance observed in the first frame may reflect the imbalance in the rest of the video and can therefore serve as a useful bias. A heuristic-free probing strategy is more transparent and makes the results more interpretable.\n\n> **The ablation study in Table 3 shows that removing the L1 reconstruction loss (experiment d) leads to only a minor drop in accuracy [...]**\n\nThis is because motion discontinuities are known to correlate well with semantic boundaries.\nRecall that we compute the gradient loss based on our lower-bound flow approximation with $A^\\ast$ (c.f. Eq. (6)). Therefore, minimising the gradient loss with the solution $A^\\ast$ will also lead to a lower reconstruction loss (c.f. Eq. (5)). This implies that the gradient loss jointly performs two functions: it focuses on motion discontinuities while minimising the reconstruction loss.\n\nIn practice, we observed that training FlowFeat with the gradient loss alone may be unstable for some backbone models and datasets -- presumably due to the spatial sparsity of the learning gradient (the loss vanishes for pixel areas with constant motion). Therefore, we keep both loss terms for consistency across all our experiments.\n\n\n**We hope that our response resolves your concerns and look forward to a more favourable recommendation.**"}, {"review_id": "mb6DLn6khU", "reviewer": "Reviewer_2G2T", "summary": "The paper proposes FlowFeat, a method that, based on an off-the-shelf generic visual representation (which are typically low-resolution), predicts high-resolution generic features using a decoder. Supervision for the decoder is primarily obtained via optical flow between frames from open-set videos, resulting in a high-resolution, dense supervision signal. The learned representation, starting from multiple different backbones, is evaluated across multiple different downstream tasks and demonstrates broad improvements over the respective base representation's performance.", "questions": "1. Regarding W1: Is this the case? If so, why not pass the image to the model in addition to the features? If not, this should be made clear throughout the paper (Fig. 2, Sec. 3, ...).\n2. What happens if the representations obtained from FlowFeat are downsampled to the same resolution as the base model's representations? Evaluating in this setting as well would show whether the quantitative gains stem from increased resolution or from improved data representation (i.e., making the data that was already present in the base embedding more accessible for linear probing on the given tasks).\n3. Regarding W2: Could the authors please elaborate on the motivation behind these choices, and provide ablations compared to intuitive baselines (e.g., L2 loss, no weighting, ...)?\n4. Are the representations directly related to the base encoder's representations in any way?\n\nAddressing these questions (and, ideally, also the other weaknesses mentioned) would help clarify what the method actually achieves and demonstrate its significance. I'm willing to raise the score to accept if the authors convincingly address these points and other reviewers do not uncover major unaddressed weaknesses.", "limitations": "Limitations are touched upon only very briefly and spread throughout the paper rather than being consolidated in a single place. Consolidating them into a limitations paragraph in the conclusion and extending upon them would improve the paper.\nThe PCA visualizations also appear to place a significant emphasis on moving objects, thereby obscuring the visibility of background elements, unlike other approaches. It would be worth discussing whether this constitutes a limitation of the approach and specific downstream capabilities, or whether it might simply be an artifact of the PCA visualization.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nThe proposed method is relatively simple yet appears to be effective. While I'm not sure that describing the representation as embedding \"statistical motion patterns\" is a very fitting description, the idea of predicting a representation that captures which parts of the scene could move together and then solving for the actual motion to obtain supervision is interesting.\nThe experiments cover different tasks (video object/semantic segmentation and monocular depth) and show strong performance across them, outperforming the base representation across multiple standard off-the-shelf backbones.\n\n**Weaknesses**\n1. As far as I understood the paper, the feature decoder does not receive a copy of the image. This would mean that its success in providing high-resolution features is wholly dependent on the source features containing high-resolution image information, unlike FeatUp, which accounts for the full-resolution image. This appears to be a significant potential weakness of the method to me.\n2. The authors make some non-obvious design choices, such as using an L1 loss instead of an L2 loss in Eq. 5 and the formulation of the weighting term in Eq. 6, without explaining them in detail or providing ablation studies.\n3. Evaluations are only performed on small S- and B-scale models. It would be very valuable to demonstrate that the presented findings also generalize to larger models commonly used for practical applications.\n4. Comparisons with FeatUp in Tab. 2 lack evaluation with the refinement stage for FeatUp and some base models, which should be included for a fair comparison.\n5. Sec. 2 and 3 lack any structuring into subsections and/or paragraphs with headings, making them unnecessarily hard to read. Other sections have similar problems, but less severe. Introducing some additional structure could make the paper significantly more accessible.", "was_revised": true, "final_justification": "As detailed in the strengths section, I consider the work interesting and valuable to the community. My initial rating of 4 stemmed from some open questions & weaknesses, which have mostly been resolved during the rebuttal, hence my raised score.\n\nOverview of weaknesses & questions raised in original review\nI consider mostly/fully resolved (Y), partially resolved (~), and unresolved (X):\n\n| W1   | W2   | W3   | W4   | W5   | Q1     | Q2   | Q3     | Q4   |\n| ---- | ---- | ---- | ---- | ---- | ------ | ---- | ------ | ---- |\n| ~    | Y    | Y    | Y    | Y    | Y [W1] | N    | Y [W2] | Y    |\n\nAdditional details:\n\n- W1: primarily about presentation, not a major weakness\n- Q2: authors promised suggested eval for later, did not (yet) deliver during rebuttal, so unclear what result will be", "rebuttal": "Thank you for your thoughtful and encouraging feedback.\n\n> **[...] the feature decoder does not receive a copy of the image.**\n\nWe agree that this could be a valid concern. However, the DPT architecture, which FlowFeat uses, has skip connections to intermediate layers of the encoder, from which high-frequency information can be restored (despite the constant resolution in transformers). We kindly refer to Ranftl et al., [41] for architecture details and analysis. As can be seen from our qualitative examples (c.f. Figs. 1,3,4,5), FlowFeat exhibits fine-grained boundaries, suggesting no difficulty in recovering high-frequency information.\n\n> **The authors make some non-obvious design choices, such as using an L1 loss instead of an L2 loss in Eq. 5 and the formulation of the weighting term in Eq. 6**\n\nWe actually experiment with the two terms in our ablation study, please refer to Tab. 6, (c,d).\n\nSince our training signal is an (potentially inaccurate) optical flow estimate, L1 distance is a natural choice due to its robustness in comparison to the L2 loss.\nWe have verified this intuition by replacing the L1 training loss with the L2 loss. The results below, obtained on the VOS benchmark with DINO2-S14, confirm that L1 loss is a more appropriate choice.\n\n| Loss | J&F-Mean | J-Mean | F-Mean | \n|-----------|-----------|-----------|-----------|\n| L1 (baseline) | **65.8** | **62.0** | **69.7** |\n| L2 | 63.4 | 59.8 | 67.0 |\n\n> **[...]  It would be very valuable to demonstrate that the presented findings also generalize to larger models commonly used for practical applications.**\n\nPlease note that demonstrating consistent benefits across 5 commonly used backbone architectures already provides strong empirical support for our claims. Nevertheless, we have now experimented with larger models, DINOv2 ViT-L and MAE ViT-L, and report the results on VOS with linear probing. As expected, FlowFeat (pre-trained on YouTube-VOS) consistently and significantly improves the baselines, and surpasses the VOS accuracy of smaller models. These results suggest that FlowFeat also generalises to larger models.\n\n| Model | J&F-Mean | J-Mean | F-Mean | \n|-----------|-----------|-----------|-----------|\n| DINOv2 ViT-L | 59.4 | 55.8 | 63.0 |\n| +FlowFeat-YT | **66.9** | **63.4** | **79.4** |\n\n| Model | J&F-Mean | J-Mean | F-Mean | \n|-----------|-----------|-----------|-----------|\n| MAE ViT-L | 46.7 | 44.4 | 49.0 |\n| +FlowFeat-YT | **55.4** | **52.0** | **58.9** |\n\n\n> **Comparisons with FeatUp in Tab. 2 lack evaluation with the refinement stage for FeatUp and some base models, which should be included for a fair comparison.**\n\nPlease note that FeatUp employs a stack of bilateral solvers; PAMR, at best, only approximates the bilateral solver with multiple local operations. Moreover, as can be seen in Fig. 1, FeatUp tends to produce feature maps with inaccurate or “smudgy” boundaries. The refinement with PAMR will correspondingly \"smudge\" the segmentation masks, if used with the FeatUp’s feature maps, degrading the segmentation accuracy.\n\nAs a remark, this is related to the earlier concern on high-frequency information. If FlowFeat lacked the high-frequency detail, the refinement would have no benefit. The results in Tab. 2 demonstrate the opposite.\n\n> **Sec. 2 and 3 lack any structuring into subsections and/or paragraphs with headings [...]**\n\nThank you for the great suggestion. We will improve readability of Sec. 2 and 3 by structuring the text into run-in headings.\n\n> **Q.1 Regarding W1: Is this the case? [...]**\n\nAs we discussed above – not exactly. The DPT decoder has skip connections, which allows FlowFeat to recover high-frequency detail, confirmed experimentally and qualitatively.\n\n> **Q.2 What happens if the representations obtained from FlowFeat are downsampled to the same resolution as the base model's representations? [...]**\n\nThis is an interesting idea.\nHowever, our experimental setting already accounts for the possibility of ‘mere’ upsampling of the feature representation in two ways.\n\nFirst, we compare to FeatUp, which performs upscaling with a bilateral solver without computing a new representation. Intuitively, the FeatUp features remain in the ‘convex hull’ of the original representation available in the backbone. FlowFeat provides improved downstream accuracy on virtually all tasks (without refinement), which shows that the gains do not stem from the increased resolution alone.\n\nSecond, in the local KNN setting of VOS evaluation (see Tab. 1), we keep the same hyperparameters of label propagation (e.g. the size of the local window) across all models (c.f. ll. 50-54, supp. material). This requires us to *downsample* the feature maps of FlowFeat and FeatUp, which effectively implements the proposed setting. Here, FlowFeat also provides a tangibly higher accuracy across all baselines.\n\nOverall, these results demonstrate that the gains come not merely from the increased resolution, but from the improved data representation as well.\n\n> **Q.3 Regarding W2: Could the authors please elaborate on the motivation behind these choices?**\n\nAs discussed above, the setting (d) in Tab. 3 already trains a model with the weight set to zero (i.e. no flow reconstruction loss). This leads to a slight decrease in the downstream accuracy on VOS.\n\n> **Q.4 Are the representations directly related to the base encoder's representations in any way?**\n\nWe are not sure about the question, but the FlowFeat representation comes from the DPT decoder, which has skip connections to the encoder model – this defines one relation. Moreover, FlowFeat is motion-driven, as it focuses on the foreground elements with large expected motion. By contrast, the base encoder’s representation is appearance-driven. Therefore, the two representations are different and complementary.\n\n> **Q.5 Limitations are touched upon only very briefly and spread throughout the paper rather than being consolidated in a single place. [...]**\n\nThank you for the great suggestion. We will consolidate the limitations in one place. Note that NeurIPS will provide an additional page for the camera-ready version, and we will be happy to use the allocated space to address this concern.\n\n> **Q.5 [...] The PCA visualizations also appear to place a significant emphasis on moving objects [...]**\n\nThis is an accurate observation and an expected outcome of motion parallax. However, our experimental protocol with semantic segmentation and depth estimation empirically validate that FlowFeat still preserves useful background information. We further elaborate on this point above in our response to reviewer **meu9**, to which we kindly refer.\n\n\n**Thank you for considering our response. We hope that it addresses your concerns and provides grounds for a more endorsing recommendation.**"}, {"review_id": "uqb7d4BEzi", "reviewer": "Reviewer_zphZ", "summary": "The paper introduces a new feature representation named FlowFeat, a multi-task feature representation learned through a novel distillation technique that embeds distributions of plausible apparent motion (motion profiles). FlowFeat leverages optical flow networks and diverse video data in a self-supervised framework to produce pixel-level features with high spatial precision and temporal consistency. The authors evaluate FlowFeat on three dense prediction tasks—video object segmentation (VOS), monocular depth estimation, and semantic segmentation—demonstrating consistent improvements over state-of-the-art encoders and upsampling strategies like FeatUp. Key contributions include:\n1.A self-supervised training framework combining optical flow networks and video data to embed motion profiles.\n2.Empirical validation showing FlowFeat enhances representation quality across multiple tasks and backbone architectures.", "questions": "Please see the weaknesses", "limitations": "Limitations are discussed, but more are needed to further validate the performances of FlowFeat with respect to different aspects, e.g. scene parallax, occlusions, textureless areas etc.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Pros.\nThe idea of distilling motion profiles via optical flow networks is innovative and addresses a clear gap in dense feature learning.  FlowFeat is computationally efficient, trainable on a single GPU, and scales well with input resolution without fine-tuning. The framework is shown to be resilient to inaccurate flow estimates and works with unsupervised flow networks.\n\nCons. \nThree tasks are selected for evaluation, e.g., VOS, MDE, and semantic segmentation. However, one of the most important task, optical flow estimation, are omitted. Given that the flowfeat is computed from video frames, optical flow is the natural task that should be evaluated.   \n\nWhile the empirical results are strong, the paper lacks a deeper theoretical analysis of why motion profiles lead to better feature representations. For instance, how does the distribution of linear transformations generalize across tasks?\n\nThe comparison to FeatUp is limited; why does FlowFeat outperform it? Is it purely due to motion cues, or are there architectural differences?\n\nThe discussion of limitations (e.g., non-rigid motion handling in Fig. 6) is brief. How does FlowFeat perform on highly dynamic or occluded scenes? Are there failure modes not shown?\n\nCan you provide more intuition or analysis for why motion profiles lead to better features? For example, is there a connection to equivariance or invariance in representation learning?", "was_revised": true, "final_justification": "Thanks for the rebuttal. Some of my concerns are addressed. Therefore, I would like to raise my score. Please include intuition analysis and discussions of limitations in the revision. Thanks", "rebuttal": "Thank you for your time and your valuable feedback.\nPlease note that NeurIPS will provide an additional page for the camera-ready version. We plan to use the allocated space to expand our analysis of limitations and provide more intuition behind the approach. Thank you for these suggestions.\n\nWe clarify and respond to your comments below.\n\n> **[...] optical flow is the natural task that should be evaluated.**\n\nObserve that FlowFeat is a *monocular* model – much like DINO and MAE. \nSince FlowFeat learns motion patterns from a pre-trained optical flow network, we do not expect -- and do not claim -- that FlowFeat will outperform the teacher network on flow estimation. \n\nInstead, we focus on cross-task generalisation, as the common practice in representation learning. Therefore, we use benchmarks that are very different from optical flow estimation.\nSpecifically, we evaluate FlowFeat on **three benchmarks**, train **9 model variants** of FlowFeat and test **2** datasets for pre-training (YouTube-VOS and Kinetics). We believe that this evaluation scope and the consistency of the results across all experiments provides sufficient empirical evidence for our technical contribution.\n\n> **[...] the paper lacks a deeper theoretical analysis of why motion profiles lead to better feature representations. For instance, how does the distribution of linear transformations generalize across tasks?**\n\nA theoretical analysis of our work is currently infeasible due to the fact that we are training a deep model on weakly curated video data with stochastic optimisation. However, the intuition behind generalisation of FlowFeat is the following.\n\nThe least-squares solution A* will be specific to the image pair, on which it was computed, and it will not generalise to fit the optical flow of another frame pair. \nHowever, by training the model on many frame pairs -- implying many different linear projections A* -- our embeddings learn to satisfy the induced distribution of those linear transformations w.r.t. the apparent motion. As a result, the network must encode non-trivial feature maps embedding plausible apparent motion.\n\n> **[...] why does FlowFeat outperform [FeatUp]? Is it purely due to motion cues, or are there architectural differences?**\n\nThis is a valid concern. However, we compare to FeatUp while ensuring the same backbone architecture. Therefore, the observed improvements come directly from the motion cues.\n\n> **[...] How does FlowFeat perform on highly dynamic or occluded scenes? Are there failure modes not shown?**\n\nThe VOS benchmark contains some highly dynamic and occluded scenes, and is broadly adopted in our community for these reasons. As shown quantitatively (c.f. Tab. 1), FlowFeat consistently improves the baseline accuracy. \n\nAs shown in some qualitative results, FlowFeat allocates less capacity to background elements. Quantitatively, the improvement on background areas is more modest in comparison to the improvement on the foreground regions. Nevertheless, FlowFeat demonstrates consistent benefits in terms of segmentation and depth metrics.\n\n\n> **Can you provide more intuition or analysis for why motion profiles lead to better features? For example, is there a connection to equivariance or invariance in representation learning?**\n\nIntuitively, the field of apparent motion is highly structured -- recall the Gestalt principle “what moves together belongs together”. Now, let us consider constant motion as an example. In this case, approximating constant flow (no motion boundaries) is trivial both in terms of A* and the feature map. However, to approximate complex motion patterns with a linear projection A*, the network must learn a non-trivial feature map, which groups pixels that tend to have similar motion. As we train the network on a distribution of A*’s (induced by randomly sampling frame pairs), the network will learn a highly structured and spatially meaningful feature representation.\n\nIndeed, FlowFeat is based on invariance of A* for overlapping image crops: the observed motion in one image crop must be sufficient to explain the motion in its adjacent crop.\n\n\n**Thank you for reading our response. As we did not recognise any critical concerns in the initial review, we hope for a more supportive recommendation.**"}]

Author rebuttal:

Meta-review: The manuscript addresses the problem of increasing the resolution of feature representations. Two contributions are claimed: the method, an effective self-supervised training framework that exploits flow networks and large video datasets, and an evaluation of the obtained feature representation (FlowFeat). Both parts are considered as strength by the reviewers and the AC. Main points of criticism are on evaluation details and discussion of short comings and edge cases. The weaknesses are of minor nature, while the strengths are dominating, thus the recommendation to accept the paper. As the topic of increased feature representation resolution is of broad relevance, it is suggested to present the work as spotlight, but even an oral would work.
All reviewers were rather positive from the beginning and after rebuttal and discussion, two suggest acceptance, and two BA. The main weaknesses addressed in the rebuttal were about edge cases and additional reflections. Several reviewers raised their assessment, but also pointed out to include the additional theoretical (7P8o) and intuitive (zphZ) reflections in the final version.
Decision: Accept (spotlight)