Title: DINO-Foresight: Looking into the Future with DINO
Venue: NeurIPS 2025
Accepted: yes
Keywords: Future Feature Prediction

Foundation Models

Masked Feature Transformers

Self-supervised learning

Multi-task Learning
Abstract: Predicting future dynamics is crucial for applications like autonomous driving and robotics, where understanding the environment is key. Existing pixel-level methods are computationally expensive and often focus on irrelevant details. 
To address these challenges, we introduce DINO-Foresight, a novel framework that operates in the semantic feature space of pretrained Vision Foundation Models (VFMs). Our approach trains a masked feature transformer in a self-supervised manner to predict the evolution of VFM features over time. By forecasting these features, we can apply off-the-shelf, task-specific heads for various scene understanding tasks. In this framework, VFM features are treated as a latent space, to which different heads attach to perform specific tasks for future-frame analysis. Extensive experiments show the very strong performance, robustness and scalability of our framework.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel framework for dynamic future prediction in autonomous driving. The authors propose operating in the feature space of pretrained Vision Foundation Models (VFMs) rather than at the pixel level like existing methods.\nThe approach casts dynamic future prediction as a self-supervised task with causal masking, where input tokens from previous frames are extracted using powerful VFMs and fed to a transformer. The model learns to predict latent tokens for future frames, which can then be decoded using off-the-shelf, task-specific prediction heads.\nResults demonstrate competitive or superior performance compared to previous works while being significantly more efficient due to operating in compressed feature space rather than raw pixels.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is well-written and thoroughly ablated.\n2. The addressed task is very interesting.\n3. The idea of using only features from a single powerful VFM is smart and elegant, as it enables the possibility of quickly scaling to new modalities by simply training a new head on top of the original VFM features.\n\nWeaknesses:\n1. The authors claim that \"VAE latents often lack semantic alignment, making them hard to interpret or use in downstream scene understanding.\" I think this is a strong claim, and in general, this is not true. There are many works that are actually successful in using VAE features. Although it is for images, works such \"Massively Multimodal Masked Modeling\" by Mizrahi et al., showed how VQ-VAE can provide useful features for solving many downstream tasks. I think it should be straightforward to apply discrete video tokenizers such as MAGVIT-v2 or MaskViT in the context of 4M to obtain a strong model that solves downstream tasks such as segmentation or depth on videos.\n2. This work is very similar to V-JEPA, but viewed from the perspective of autonomous driving and downstream applications. Although I still believe this approach is interesting and looking at direct applications is useful, the paper should at least cite V-JEPA and highlight the main differences from an algorithmic perspective.\n3. Futurist is cited as a competitor work and is listed in Table 1 with better results, but it is not discussed in Section 4.2.\n4. Overall, I have the feeling that the paper is a composition of many different components from other papers, such as 2D-3D decomposition of attention, two-phase training, etc., and it lacks fundamental novel components.\n\nQUESTIONS:\n1) I wonder if a better approach would be to use a discrete pretrained tokenizer rather than a pretrained VFM, as this would make the training process much simpler: i.e., predict next tokens within a vocabulary size instead of continuous features after PCA. Why did the authors decide to use a continuous representation rather than a discrete one? What is the advantage?\n\n2) Did the authors consider alterative masking strategies and different simple loss functions like simple L2/L1? It would be interesting to see how these affect the results.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces DINO-Foresight, a self-supervised framework through forecasting the temporal evolution of VFM features (such as DINOv2), and it can be further adapted to various scene understanding tasks. Unlike prior video prediction methods that rely on generative models in pixel or VAE latent space, DINO-Foresight directly operates in the VFM feature space. The core idea is to predict the temporal evolution of rich, high-level VFM features using a masked transformer model, allowing seamless plug-and-play with downstream task-specific heads for applications like semantic segmentation, depth prediction, and instance segmentation. Besides, experimental results on several future prediction tasks demonstrate a unique advantage of DINO-Foresight.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. This paper leverages the strengths of VFMs and further improves them by forecasting the temporal evolution of VFM features, finally providing a powerful vision backbone.\n\n2. The backbone can be seamlessly integrated with various task heads, and bring consistent improvements across these tasks.\n\n\n**Weaknesses**\n1. This paper builds upon existing models and designs, such as DINO and temporal MAE. Although the authors argue that DINO-Foresight is trained by directly forecasting VFM features, differing from standard MAE, it cannot adequately uphold the contribution's novelty.\n\n2. The paragraph is not well-structured. For example, the paragraph in line 48 feels somewhat abrupt, and the paragraph in line 81 overemphasizes the differences.\n\n3. Except for the feature prediction component, the model builds on prior designs, like FUTURIST, but the experiment results fail to demonstrate a significant improvement over these existing approaches.\n\nQUESTIONS:\nplease see weaknesses\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces  DINO-Foresight, a self-supervised framework for semantic future prediction that leverages pretrained Vision Foundation Models (VFMs). The core idea of the framework is to forecast VFM features over time using a masked feature transformer. Unlike traditional approaches that rely on low-level pixel or latent space representations, DINO-Foresight directly predicts semantic features, which are then applied to various downstream tasks such as semantic segmentation, depth prediction, and surface normal estimation. The paper demonstrates that the proposed method outperforms state-of-the-art methods in future-frame prediction across various tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- This paper is well-motivated, and the writing is easy to follow.\n- Extensive experiments on multi-task performance. In Table1, the authors compare their method with multiple counterparts on several forecasting tasks, including depth estimation and instance segmentation. Such comprehensive comparisons greatly demonstrate the superiority of the proposed method.\n- The idea of predicting semantic features (from vision foundation models) of future frames enables more efficient learning and inference by omitting the generation of low-level pixel details. This solution might be practically useful and insightful for research areas that are sensitive to computational efficiency, such as real-time autonomous driving.\n\nWeaknesses:\n- Lack of discussion with some prior works. Besides the related work listed in Semantic Future Prediction (in lines 86-99), some prior works are missing from the discussion and comparison, such as V-JEPA and DINO-WM. \n\n- Lack of investigation on the scalability of the model size and data scale. Experiments are conducted on a fixed model size and data scale without any indication of how task performance would be with more/less parameters and data.\n\n\nReferences:\nV-JEPA: Revisiting Feature Prediction for Learning Visual Representations from Video\n\nDINO-WM: DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning\n\nQUESTIONS:\nPlease see the Weaknesses above.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDINO-Foresight introduces a self-supervised framework for predicting future scene dynamics by forecasting vision foundation model (DINOv2) features of future frames of videos. A key contribution is the use of high resolution features from multiple layers of the DINO model and the use of PCA for dimensionality reduction. This allows the model to predict future depth maps, semantic segmentation maps and surface normals of future frames of videos from steet scene datasets: CityScapes and nuScenes.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n•\tThe paper demonstrates state of the art results on a wide range of tasks ranging from the prediction of future semantic segmentation maps to depth maps.\n\n•\tThe proposed method is simple and computationally efficient.\n\n•\tThe paper includes adequate ablations highlighting the effectiveness of the proposed approach.\n\n•\tThe paper is well written and easy to understand.\n\n**Weakness:**\n\n•\tNovelty: The proposed method is relatively simple – it predicts the future in the feature space instead of RGB/semantic/depth pixel space. The key contribution is the integration of high level features using a PCA dimensionality reduction step. A key downside of predictions in the DINO feature space is the inability to predict future RGB frames as DINO features still discard a lot of high-frequency details in the original RGB input sequence. \n\n•\tGeneralization: The proposed model is trained and test on the same datasets: Cityscapes and nuScenes. However, the source DINO model is inherently generalizable across datasets. It would be interesting to see zero-shot applications to datasets such as Waymo Open in the street scene domain.\n\n•\tEvaluation metrics: In case of street scenes, nearby objects have a larger apparent motion, while faraway objects can appear static. The paper uses evaluation metrics that do not take such motion into account. While this is also an issue with prior work, it would be interesting to use a motion aware evaluation metric, to highlight that the proposed model can predict motion correctly. \n\n•\t Multi-modality: The distribution of future, e.g., in case of street scenes, is highly multi-modal – there are many distinct likely futures, i.e., a vehicle can turn to the left or to the right. The proposed model cannot capture this multi-modal distribution.\n\n•\tDiverse scenarios: The proposed model is limited to street scenes, it is not clear if the model can be applied to arbitrary videos for future frame prediction. It would be interesting to include evaluation on diverse domains.\n\nQUESTIONS:\n•\tThe paper should highlight its novel technical contributions in more detail.\n•\tDoes the proposed approach generalize to novel scenarios?\n•\tCan the proposed approach handle scenarios where the distribution of future states is multi-modal?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}