Title: FlowFeat: Pixel-Dense Embedding of Motion Profiles
Venue: NeurIPS 2025
Accepted: yes
Keywords: representation learning

optical flow
Abstract: Dense and versatile image representations underpin the success of virtually all computer vision applications. However, state-of-the-art networks, such as transformers, produce low-resolution feature grids, which are suboptimal for dense prediction tasks. To address this limitation, we present *FlowFeat*, a high-resolution and multi-task feature representation. The key ingredient behind FlowFeat is a novel distillation technique that embeds a distribution of plausible apparent motions, or *motion profiles*. By leveraging optical flow networks and diverse video data, we develop an effective self-supervised training framework that statistically approximates the apparent motion. With its remarkable level of spatial detail, FlowFeat encodes a compelling degree of geometric and semantic cues while exhibiting high temporal consistency. Empirically, FlowFeat significantly enhances the representational power of five state-of-the-art encoders and alternative upsampling strategies across three dense tasks: video object segmentation, monocular depth estimation and semantic segmentation. Training FlowFeat is computationally inexpensive and robust to inaccurate flow estimation, remaining highly effective even when using unsupervised flow networks. Our work takes a step forward towards reliable and versatile dense image representations.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new feature representation named FlowFeat, a multi-task feature representation learned through a novel distillation technique that embeds distributions of plausible apparent motion (motion profiles). FlowFeat leverages optical flow networks and diverse video data in a self-supervised framework to produce pixel-level features with high spatial precision and temporal consistency. The authors evaluate FlowFeat on three dense prediction tasks—video object segmentation (VOS), monocular depth estimation, and semantic segmentation—demonstrating consistent improvements over state-of-the-art encoders and upsampling strategies like FeatUp. Key contributions include:\n1.A self-supervised training framework combining optical flow networks and video data to embed motion profiles.\n2.Empirical validation showing FlowFeat enhances representation quality across multiple tasks and backbone architectures.\n\nSTRENGTHS AND WEAKNESSES:\nPros.\nThe idea of distilling motion profiles via optical flow networks is innovative and addresses a clear gap in dense feature learning.  FlowFeat is computationally efficient, trainable on a single GPU, and scales well with input resolution without fine-tuning. The framework is shown to be resilient to inaccurate flow estimates and works with unsupervised flow networks.\n\nCons. \nThree tasks are selected for evaluation, e.g., VOS, MDE, and semantic segmentation. However, one of the most important task, optical flow estimation, are omitted. Given that the flowfeat is computed from video frames, optical flow is the natural task that should be evaluated.   \n\nWhile the empirical results are strong, the paper lacks a deeper theoretical analysis of why motion profiles lead to better feature representations. For instance, how does the distribution of linear transformations generalize across tasks?\n\nThe comparison to FeatUp is limited; why does FlowFeat outperform it? Is it purely due to motion cues, or are there architectural differences?\n\nThe discussion of limitations (e.g., non-rigid motion handling in Fig. 6) is brief. How does FlowFeat perform on highly dynamic or occluded scenes? Are there failure modes not shown?\n\nCan you provide more intuition or analysis for why motion profiles lead to better features? For example, is there a connection to equivariance or invariance in representation learning?\n\nQUESTIONS:\nPlease see the weaknesses\n\nLIMITATIONS:\nLimitations are discussed, but more are needed to further validate the performances of FlowFeat with respect to different aspects, e.g. scene parallax, occlusions, textureless areas etc.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes FlowFeat, a method that, based on an off-the-shelf generic visual representation (which are typically low-resolution), predicts high-resolution generic features using a decoder. Supervision for the decoder is primarily obtained via optical flow between frames from open-set videos, resulting in a high-resolution, dense supervision signal. The learned representation, starting from multiple different backbones, is evaluated across multiple different downstream tasks and demonstrates broad improvements over the respective base representation's performance.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThe proposed method is relatively simple yet appears to be effective. While I'm not sure that describing the representation as embedding \"statistical motion patterns\" is a very fitting description, the idea of predicting a representation that captures which parts of the scene could move together and then solving for the actual motion to obtain supervision is interesting.\nThe experiments cover different tasks (video object/semantic segmentation and monocular depth) and show strong performance across them, outperforming the base representation across multiple standard off-the-shelf backbones.\n\n**Weaknesses**\n1. As far as I understood the paper, the feature decoder does not receive a copy of the image. This would mean that its success in providing high-resolution features is wholly dependent on the source features containing high-resolution image information, unlike FeatUp, which accounts for the full-resolution image. This appears to be a significant potential weakness of the method to me.\n2. The authors make some non-obvious design choices, such as using an L1 loss instead of an L2 loss in Eq. 5 and the formulation of the weighting term in Eq. 6, without explaining them in detail or providing ablation studies.\n3. Evaluations are only performed on small S- and B-scale models. It would be very valuable to demonstrate that the presented findings also generalize to larger models commonly used for practical applications.\n4. Comparisons with FeatUp in Tab. 2 lack evaluation with the refinement stage for FeatUp and some base models, which should be included for a fair comparison.\n5. Sec. 2 and 3 lack any structuring into subsections and/or paragraphs with headings, making them unnecessarily hard to read. Other sections have similar problems, but less severe. Introducing some additional structure could make the paper significantly more accessible.\n\nQUESTIONS:\n1. Regarding W1: Is this the case? If so, why not pass the image to the model in addition to the features? If not, this should be made clear throughout the paper (Fig. 2, Sec. 3, ...).\n2. What happens if the representations obtained from FlowFeat are downsampled to the same resolution as the base model's representations? Evaluating in this setting as well would show whether the quantitative gains stem from increased resolution or from improved data representation (i.e., making the data that was already present in the base embedding more accessible for linear probing on the given tasks).\n3. Regarding W2: Could the authors please elaborate on the motivation behind these choices, and provide ablations compared to intuitive baselines (e.g., L2 loss, no weighting, ...)?\n4. Are the representations directly related to the base encoder's representations in any way?\n\nAddressing these questions (and, ideally, also the other weaknesses mentioned) would help clarify what the method actually achieves and demonstrate its significance. I'm willing to raise the score to accept if the authors convincingly address these points and other reviewers do not uncover major unaddressed weaknesses.\n\nLIMITATIONS:\nLimitations are touched upon only very briefly and spread throughout the paper rather than being consolidated in a single place. Consolidating them into a limitations paragraph in the conclusion and extending upon them would improve the paper.\nThe PCA visualizations also appear to place a significant emphasis on moving objects, thereby obscuring the visibility of background elements, unlike other approaches. It would be worth discussing whether this constitutes a limitation of the approach and specific downstream capabilities, or whether it might simply be an artifact of the PCA visualization.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces FlowFeat, a novel, high-resolution feature representation for images that is learned in a self-supervised manner from video data. The core idea is to distill \"motion profiles\"—distributions of plausible motion—from optical flow estimates into a dense, pixel-level feature map. This is achieved by training a lightweight decoder on top of a frozen, pre-trained image encoder. The training objective forces the learned features to be able to reconstruct the optical flow between pairs of video frames via a dynamically computed linear transformation. The authors demonstrate that FlowFeat significantly enhances the performance of several state-of-the-art visual encoders on a variety of dense prediction tasks, including video object segmentation, monocular depth estimation, and semantic segmentation. The method is shown to be label-efficient, computationally inexpensive to train, and robust to inaccuracies in the underlying optical flow estimation, offering a practical step toward creating more versatile and detailed image representations.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The core idea of embedding \"motion profiles\" by learning to approximate optical flow with a distribution of linear transformations is both novel and well-motivated. It leverages the strong geometric and semantic cues inherent in motion without requiring expensive manual annotations.\n\n2. The paper presents a comprehensive set of experiments across three distinct and challenging dense prediction tasks. FlowFeat consistently and significantly improves upon strong baseline encoders (MAE, DINO, DINOv2) and outperforms or performs competitively with alternative upsampling techniques like FeatUp. The improvements are particularly notable for weaker encoders like MAE.\n\n3. The ablation studies and qualitative results demonstrate that the method is robust to the choice of the optical flow network and resilient to inaccurate or artifact-prone flow estimations, which is a critical property for real-world application.\n\n\n\nWeakness\n\n1. The training relies on the assumption that two different, overlapping crops from the same initial frame should be able to predict the flow in their corresponding regions of the target frame using the same linear operator A *. While this works well empirically, the justification for this core assumption could be stronger.\n\n2. While the paper commendably points out a limitation with non-rigid objects (the snake example), a broader discussion of potential failure modes would be beneficial. For example, how does it handle scenes with very little or no motion, or scenes with extremely complex, non-linear motion patterns?\n\n3. The authors state that state-of-the-art networks, such as transformers, generate low-resolution feature grids that are suboptimal for dense prediction tasks. How is this suboptimality quantified or evaluated in practice?\n\nQUESTIONS:\n1. In Section 4.1, you state that for linear probing on DAVIS-2017, a linear classifier is trained on the ground-truth segmentation of the first frame. Given that the first frame contains only a single annotated instance for each object, how do you construct a balanced training set for the linear classifier, especially for distinguishing the object from a diverse background?\n\n2. The ablation study in Table 3 shows that removing the L1 reconstruction loss (experiment d) leads to only a minor drop in accuracy, suggesting the gradient loss L_{∇} is doing most of the work. This is surprising. Can you explain the reason?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new method for training an upsampling decoder for image features in a self-supervised way by taking advantage of video data. The goal is to take image features from a frozen, large-scale pre-trained backbone (encoder) and upsample them to dense features for better performance on dense tasks such as semantic segmentation and depth estimation. The work is most similar to LoftUp and FeatUp which are other feature upsampling methods with similar use cases. FlowFeat claims to be different by not requiring runtime optimization like FeatUp and also not relying on pre-trained models such as SAM to generate full resolution pseudo-GT like LoftUp. By contrast, FlowFeat claims to be fully unsupervised, fast to inference, efficient to train, and also comes with temporal consistency advantages compared to other methods.\n\nThe primary method for generating these upsampled features is by using a DPT style decoder head which is trained to predict dense \"motion profiles\" from a single image. This isn't quite optical flow because predicting flow from a single image is an under constrained task, thus the actual goal is to train a feature vector such that a simple linear transformation will produce the correct flow for any temporally neighboring frame. The supervision for these flow outputs comes from a pre-trained optical-flow network such as RAFT or SMURF.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\nThe main strength of FlowFeat is it's unique formulation which makes it generalizable (no per-frame optimization) and inexpensive to train. The core idea of using motion cues and optical flow as a supervisory signal for dense features is brilliant and it clearly has positive impacts of tasks requiring temporal consistency such as video object segmentation. The linear transformation & motion profile section is also well explained, and in combination with a public code release makes a compelling case for wider adoption and testing in a wide variety of tasks. One example I can think of is that I suspect the features produced by FlowFeat might make for a good drop-in replacement for those used by point tracking methods.\n\n## Weaknesses\nThe main weaknesses of the paper are twofold. One is that the flow based features appear to lose background features when visualized the same way as those from FeatUp (for example). This is best seen in Figure 1 in the skateboarding example where FlowFeat seems to smush the background (sky tree etc.) into an amorphous blob. It's also visible in the videos provided in the supplemental. The second is that it's positioned as an unsupervised method, which I guess technically it is (especially if you use SMURF) but needing a good flow estimation network does limit it's applicability somewhat (for example, to other domains such as medical images where pre-trained flow methods might not work as well). Though really I consider this second weakness to be more of a nitpick.\n\nQUESTIONS:\n1. How much feature map upsampling are you able to achieve with good results using FlowFeat?\n2. Do the random crops used as input to the flow model have to be a specific resolution?\n3. Does the resolution of the flow network's outputs affect the DPT decoder's ability to upsample?\n\nLIMITATIONS:\nGiven the reliance on flow, I'd like to see the authors dive a bit more into the performance on \"static background\" elements in images (for example, segmentation performance on a tree in the background). This seems like it may be a limitation, but is hard to see from the qualitative examples provided. Additionally pointing out that the method only really applies to RGB web images, and not other domains is a good idea (as mentioned in the weaknesses section).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-05"}