DINO-Foresight: Looking into the Future with DINO
Authors: Efstathios Karypidis
Ioannis Kakogeorgiou
Spyros Gidaris
Nikos Komodakis
Venue: NeurIPS 2025 poster ()
Keywords: Future Feature Prediction
Foundation Models
Masked Feature Transformers
Self-supervised learning
Multi-task Learning
Primary area: deep_learning
TL;DR: DINO-Foresight forecasts future VFM features to enable robust and scalable semantic scene understanding across multiple tasks.

Abstract: Predicting future dynamics is crucial for applications like autonomous driving and robotics, where understanding the environment is key. Existing pixel-level methods are computationally expensive and often focus on irrelevant details. 
To address these challenges, we introduce DINO-Foresight, a novel framework that operates in the semantic feature space of pretrained Vision Foundation Models (VFMs). Our approach trains a masked feature transformer in a self-supervised manner to predict the evolution of VFM features over time. By forecasting these features, we can apply off-the-shelf, task-specific heads for various scene understanding tasks. In this framework, VFM features are treated as a latent space, to which different heads attach to perform specific tasks for future-frame analysis. Extensive experiments show the very strong performance, robustness and scalability of our framework.

DINO-Foresight: Looking into the Future with DINO
Efstathios Karypidis Ioannis Kakogeorgiou*
Spyros Gidaris
Nikos Komodakis
Archimedes, Athena Research Center, Greece
vvaleo.ai
National Technical University of Athens
University of Crete 9IACM-Forth
Abstract
Predicting future dynamics is crucial for applications like autonomous driving and robotics, where understanding the environment is key. Existing pixel-level methods are computationally expensive and often focus on irrelevant details. To address these challenges, we introduce in the semantic feature space of pretrained Vision Foundation Models (VFMs). Our approach trains masked feature transformer in self-supervised manner to predict the evolution of VFM features over time. By forecasting these features, we can apply off-the-shell task-specific heads for various scene understanding tasks. In this framework, VFM features are treated as a latent space, to which different heads attach to perform specific tasks for future-frame analysis. Extensive experiments show the very strong performance, robustness and scalability of our framework. Project page and code
novel framework that operates
Introduction
Predicting future states in video sequences is key challenge in computer vision and machine learning. with important applications autonomous systems like self-driving cars and robotics (Finn et al., 2016 Dosovitskiy and Koltun 2017). These systems must navigate dynamic environments safely, yet predicting future states remains eisficult-eseccilll in complex scenarios involving multi-object interactions over long time horizons.
Recent approaches focus on generating realistic RGB future frames using latent generative modeling (Rombach et al., (2022). These methods first compress RGB data into latent space, such as continuous (Rombach et al., (0222) or discrete (Esser al. 202T) Variational Autoencoder (VAE) representations. Then, they train generative
diffusion models (Zheng et al.) 2024 Gao al. 2024 Ho al. 2022a b. Harvey et al., 2022, Brooks et al. 2024). autoregressive models (Hu al.) [2233] Hong et al. 2023 Kondratyuk et al. [2244] Wang et al. (2024), or masked video generation (Yu et al. 2023 2024 to predict future states in this compressed space. While this reduces dimensionality and improves training stability (Rombach et al. 2022). VAE latents often lack semantic alignment, making them hard to interpret or use in downstream scene understanding tasks (see Table 2). Moreover, these methods must model both low-level appearance and high-level semantics, even though decision-ni decision-making systems (e.g., self-driving cars) primarily need semantic scene understanding- -what objects exist and where they are. Latent generative approaches may waste capacity on irrelevant low-level details, compromising temporal semantic accuracy.
In contrast, vision foundation models (VFMs) have revolutionized scene understanding with robust, transferable features (Oquab et al. 2024 Radford et al. 2021] Venkaaaramanan et al. 20255] Bardes et al. 2024). This raises key question: can VFM features serve semantically rich, high- dimensional latent space for precise future prediction?
In this work, we propose precisely this idea. Instead of predicting future low-level VAE latents, we forecast the temporal evolution of VFM features directly. This shift brings several important advantages: (a) Beyond low-level latents. It leverages semantically meaningful representations,
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Context Frumes
Segmenlation
VFM Encoder
Head
ofs
Masked Feature Transformer
Depth Prediction
VFM Encoder
Predicted Features
Head
Surface Normals
Future Frame
Head
VFM
Smooth
Encoder
Library of Prediction Heads At Test-Time
Train-Time
Figure Forecasting VFM Features for Future Frames. At the core of our approach is the prediction of future VFM feature evolution. To this end, we train masked transformer model in self-supervised manner to forecast these features from context frames, minimizing SmoothL1 loss between predicted and actual future features. By forecasting these rich and versatile features, task-specific prediction heads such as semantic segmentation, depth, and surface normals can be effortlessly employed time, enabling modular and efficient multi-task understanding.
inheriting strong scene understanding. (b) Dynamic semantics over raw frames. It avoids full-frame synthesis, letting the model focus meaningful dynamics, reducing complexity and sidestepping challenges like multimodal pixel distributions. (c) Scalable multi-task support and modular integration. Forecasted features align with downstream tasks, enabling plug-and-plas integration with pretrained classifiers, segmenters or task-specific heahs-without need retrain the core feature predictor (see Figure This represents significant departure from prior semantic feature prediction methods (Luc et al. 2017), which face significant challenges with multi-tase salliittskk scalability. Such approaches either require training a separate model for each task (Nabavi et al. 2018 Chiu et al. 2020 Terwilliger et al. 2019) or involve predicting features from multiple task-specific models simultaneously (Hu al. 2020 Karypidis et al. 2025). resulting in more complex and less scalable architectures.
In this work we show that forecasting high-dimensional VFM features is not only feasible but also achieves strong performance, offering a new path toward semantically grounded future prediction.
In summary, our contributions are: (1) We introduce DINO-Foresight, self-supervised method for semantic future prediction that builds on the key idea of forecasting VFM features- our core contribution. Unlike latent generative methods that rely on low-level VAE features, our approach delivers precise future scene understanding without modeling unnecessary appearance details, en- abling unified model for diverse scene understanding tasks. (2) To realize this idea, we design an efficient masked feature transformer (see that propagates multi-layer, high-resolution features critical for achieving strong task performance. (3) Experimental results demonstrate unique advantage of our approach our single model successfully handles multiple future-frama understanding tasks (semantic segmentation, instance segmentation, depth prediction, and surface normal prediction) where previous approaches required multiple specialized models. Furthermore, show in Appenddx Subsection A.2) intermediate features within our masked transformer further improve downstream task performance, highlighting its potential as a self-supervised learning strategy that enhances the already strong VFM features.
Related Work
Video Prediction is an extensively studied problem. Early approaches based Convolutional LSTMs
(Nabavi al. [2018] | Xu al. 2018, Wang et al.) [2018] Castrejon et al., 2019 [ee et al., 2022 Wu
et al. 2021 Gao et al. (022) struggled to maintain both visual quality and temporal consistency. Subsequent developments introduced generative adversarial networks and variational autoencoders
(Yan et al. 202 Vondrick et al. 2016b Castrejon et al. 2019 Lee et al. 2018 Babaeizadeh et al.)
2218), alongside diffusion models (Ho et al.] [202aa] Gao al. 2024 Harvey et al.] (022). to
enhance spatial-temporal coherence and improve the quality of predictions. Furthermore, transformer-

---

based models have also been adapted to videos, utilizing auto-regressive and masked modeling
objectives to capture video dynamics (Yu al., 2024 2023 Gupta et al. 2023 Wang al. 2024).
Latent Generative Models Current state-of-the-ard video prediction methods (Gupta et al. 2023 Yu et al. 2023 2024; Gao et al. (024) build upon latent generative approaches (Rombach et al.
2022). These frameworks employ an autoencoder to compress RGB frames into a latent space, then train generative models to predict future states in this compressed representation. While sharing superficial similarities with our seyrroach-including the use of latent spaces and masked transformers in some cases (Gupta al.) 2023] Yu et al. 2003) -these methods differ fundamentally. Their latent spaces primarily encode low-level visual information, requiring reconstruction back to RGB space and forcing the model to simultaneously handle both appearance details and semantic changes.
Our key innovation lies in forecasting VFM features that natively encode high-level semantic infor- mation, enabling direct application of task-specific prediction heads. As demonstrated in Table 2 conventional VAE latent spaces cannot match this capability. Our experimental results in Table 1 further show these methods' limitations in predicting semantic scene evolution, highlighting the distinct advantages of VFM feature forecasting for scene understanding tasks. Concurrently to our work, DINO-WM (Zhou et al., 2025), also leverages DINOv2 features for world modeling with action-condittoned planning in simulated environments, while our work targets multi-task dense semantic forecasting in real world scenarios.
Semantic Future Prediction An emerging approach for future-framme prediction focuses on fore-
casting intermediate features from an encoder rather than raw RGB values (Nabavi et al. 2018 Lin
et al. 2021 Sarie et al. 2020 Sun et al. 2019 Chen and Han 2019 Jin et al. 2017
Luc al.
2018 Sarić et al. 2019 ; Hu et al. 2021 Vondrick et al. 20 Zhong et al. 2023) ). This strategy
models abstract encoder representations, which task-specifie heads use for downstream tasks. Early methods in this paradigm include F2F (Luc et al.] 2018), which regresses Mask-RCNN's feature pyramid, and F2MF (Saric et al. 2020), which improves feature prediction using flow-based warping. APANet (Hu et al. (021)) aggregates multi-level features via an auto-path mechanism for instance segmentation, while PFA (Lin et al. (20T)) enhances global structures and suppresses local details for more predictable features. Recently, Futurist (Karypidis et al., (025)) introduced a multi-modal semantic forecasting approach for semantic segmentation and depth maps. While effective, these methods often rely on task- or dataset-specife encoders, limiting practicality and scalability. To address this, we use VFM encoders, which, due to large-scale pre-training, perform well across diverse tasks and generalize effectively to new scenes without retraining.
Vision Foundation Models (VFMs) VFMs have transformed computer vision, achieving strong performance across a range of visual tasks. Trained on large-scale datasets, these models learn rich, transferable visual representations. Notable examples include DINOv2 (Caron et al. |021 Oquab
ctt al. 2024), a self-supervised model based on self-distillation; Franca (Venkataramanan et al. 20255) ) fully open VFM for scalable self-supervised representation learning; CLIP and its variants (Radford) ctt al. 2021 Fang ct al. 2023] Sun et al.) 2023 Fang et al.] 2024). which align visual representations with natural language; SAM (Segment Anything Model) (Kirillov et al. 2023). a foundation model for image segmentation and @JEPA(Barder et al.) 2024), a video representation learning method via feature prediction in latent space. In this work, we explore the potential of VFM features for semantic future prediction tasks, connecting static visual understanding with dynamic prediction.
Multi-Task Learning Multi-Task Learning (MTL) is a learning paradigm that enables simultaneous
training of models multiple related tasks (Maninis et al.) 2019; Misra al. 2016 Neseem et al..
2023
Vandenhende et al. (022). promoting shared representations and improving performance across
tasks. Traditional MTL frameworks often use parameter sharing (Kendall et al. 2018 Sener and
Koltun 2018 Bekoulis et al. 2018) or task interaction allowing exchange of information (Bragman
2019 Chen al. 2223aa bj Misra et al., 2016, Ruder et al., 2019). Other approaches employ
strategy that incrementally increases the model's s depth during training, enabling the network to
learn task-specific representations in a more sesource-efficieen way Aich et al.) (2023); Choi and
Im (2023): Guo et al. (2020): Lu et al. (2017); Zhang et al.) (2022). Recently. the emergence of large-scale pretrained models has led to the introduction of adapter-based multi-task fine-tuning
approaches (Liang et al.) 2022 Liu et al. 2022). In our work, we leverage VFM features to provide
a unified, scalable and modular framework for future prediction. Our approach enables seamless integration of multiple tasks without retraining or complex adaptations.
3

---

3 Methodology
Our semantic future prediction approach builds on forecasting VFM features powerful represen- lations that excel at scene understanding tasks and generalize well to unseen environments. We realize this through a masked transformer model trained via self-supervision to predict the temporal evolution of VFM features. This forecasted feature space serves as a latent representation that can flexibly connect to various off-the-shelf spec-iic heads for future-frame analysis.
Figure provides an overview of our approach, with the key components detailed in the following
sections: Section 3.1 describes how target features are generated from multi-layer VFM features.
Section 3.2 presents our formulation of VFM feature forecasting as a masked feature modeling problem and details the model architecture. [Section [3.3] covers efficient training techniques for high-resolution VFM feature prediction. Finally. SeallyySSettonn 3.4 introduces our modular framework for multi-task future-frame analysis and details how prediction heads are trained and integrated.
3.1 Hierarchical Target Feature Construction
In Figure 2, provide an overview of how the target feature space for the feature prediction model is constructed. Below, we outline the main steps involved.
Multi-Layer VFM Feature Extraction Scene understanding models often benefit from pro- cessing features across multiple layers of an im- age encoder (Ranftl et al., 2021, Long et al.) 2015: Lin et 2017 et al. 2022 et al. 2017] Chen et 2017)) especially when the encoder is frozen, in our approach. To fully leverage the pretrained VFMs, we propa- gate features extracted from multiple layers of the VFM. We ablate the impact of multi-layer features in Appendix Subsection A.4
Visien Transfomerr
Cencat
Targst Fartrce
Traneformer Block
Tokenowise
Dom, Reduction
Transfunnc
Conceseraied
Hiaranhizzl
Transfommer
Features
Block
Transformer
Our framework uses VFMs based on the Vision Transformer (ViT) architecture, though the ap- proach can extended to other architectures. Given sequence of N image frames X E N x3 let F(!) E X Done rep-
Figure 2: Hierarchical Target Feature Construc- tion for the Feature Prediction Model. Our framework constructs feature space by extract- ing and concatenating multi-layer features from frozen ViT encoder, capturing semantic informa- tion at varying abstraction levels. PCA is applied to reduce dimensionality, creating compact features.
resent the features extracted from layer of the VIT model. Here, Dene is the feature dimension, and x W is the spatial resolution, which are consistent across layers in ViT-based models. To form the target feature space on which the feature prediction model operates, we concatenate the features from L layers along the channel dimen- sion, resulting in F Fooncat E IR X XW xL-Deme These concatenated features capture rich semantic information from the input images at multiple levels of abstraction.
Dimensionality Reduction The concatenated features F Fooncat have high dimensionality, so we apply dimensionality reduction to simplify the feature prediction task while retaining essential information. In this work, we use Principal Component Analysis (PCA) to reduce the dimensionality. transforming where D < L Dene- These
Fooncat into lower-dimennsooal representation FPCA E R x PCA-reduced features, F FPCA. serve as the target features on which the future feature prediction operates, i.c., FTRG = FPCA. In Appenddi Subsection A.I we show that this PCA-based compression retains essential information and enhances downstream performance.
3.2 Self-Supervised Future Feature Prediction with Masked Transformers
Masked Feature Transformer Architecture Inspired by previous video generation models (Chang et al. 2022 Gupta et al. 2023 Karypidis et al. 2025). we implement the future feature prediction task using self-supervised masked transformer architecture. The task involves predicting future frames in a video sequence consisting of N frames, where N₂ are context frames and N, are future

---

frames to be predicted, such that N N: + Np. Given the target features F FIRG E RNXHHWWX x for these N frames, the future-fraae tokens are masked, and the transformer must predict these missing tokens by processing all the tokens from the entire sequence, i.e., all the N H W tokens.
The architecture begins with a token embedding stage, where each token is projected from D dimensions into the transformer's s hidden dimension Daec through a linear layer. During training, the tokens corresponding to the future frames are replaced with a learnable Dae-dimmnsional [MASK] vector. During inference, these [MASK] tokens are appended after the context frames. Each token also receives position embedding to retain both temporal and spatial information across the sequence.
The tokens are then passed through a series of transformer layers. Standard self-attention layers in transformers have quadratic time complexity with respect to the number of tokens, making them computationally expensive for high-resolution, multi--rame sequences. To address this, we follow the approach from recent video transformers (Amab et al. 2021 Gupta al. [023 Karypidis et al. (025) and decompose the attention mechanism into temporal and spatial components. Temporal attention applied across tokens with the same spatial position in different frames, capturing the dynamics over time. Spatial attention, on the other hand, operates within individual frames, focusing on spatial interactions. Thus, each transformer layer consists of temporal Multi-Head Self-Attention (MSA) layer, a spatial MSA layer, and a feedforward MLP layer. After passing through the transformer layers, linear prediction layer maps the output token embeddings from the hidden dimension Decc back the feature dimension D, producing the predicted feature map F FTRG E N
Training Objective We frame the future-framm frame prediction as a continuous regression problem and optimize self-supervised training objective based on the S loss between the predicted features FIR and the ground truth features FTRG at the masked locations. The loss is defined
Σ L (FTRO(D),ETRO()) ETRO(P)) PEP
CMPM =
E TEX
(I)
where X' denotes the training dataset, [N positions across the H W spatial dimensions of the Np future frames, and FTRG(p) and FIRG (p) are the ground truth and predicted feature vectors at position p, respectively. L(-;-) computes the 1moothL.1 loss between two feature vectors:
H] [1 W] represents the set of
D
0.5 (ara 1a)a
if aa yal < B,
L(x, v) =
8
ra Val 0.53, otherwise.
d=1
In our experiments, we set 3 0.1.
3.3 Compute-Eree Effficiet Training Strategies for High- Rego-ueesottion Feature Forecasting
Using high-resolution features is crucial for pixel-wiss wiie scene understanding tasks, such as segmen- tation depth prediction, where low-resolution features struggle to capture small objects or fine spatial structures (Ranitl et al. 2021; Cheng et al., 2022 Chen et al. 2017 Touvron et al. 2019). To achieve good performance with a spatial resolution of H' x W = 448 x 896. For a ViT with a patch size of 14 x 14, as used in DINOv2 (Oquab et al.)
these tasks, aim to forecast VFM features extracted from frames
2024) and EVA-CLID (Fang et al. 2024), this results in feature maps with resolution H x W
32 x 64, corresponding to 2048 tokens per frame.
However, training ViTs on such high-resolution inputs is computationally expensive in terms of both time and memory (Dosovitskiy et al., 2020). To address this challenge while maintaining efficient training. we explore the following strategies:
Low- Rew-Reeslttion Training with High- Resh-utsoottion Inference In this approach, we train on frames with a lower resolution of 224 x 448, resulting in features with a resolution of 16 x 32. During testing, we use high-resolution frames (448 896) and adapt the position embeddings through interpolation. However, this strategy leads to suboptimal performance due to a distribution shift between the training and test data, which causes inaccurate feature forecasting.
Sliding- Window Approach for High- Resolution Inference Inspired by sliding- window techniques used in segmentation tasks (Strudel et al. 2021), this strategy trains the model with cropped feature
5

---

maps. The VIT encoder extracts features from high-resolution frames (448 x 896), producing high- resolution tokens (e.g., 32 64 for patch size of 14 x 14). During training, we sample local crops of size 16 32, taken from the same spatial locations across frames. The model is trained on these cropped features. At inference time, the model processes the high-resolution features in a sliding-window window manner using the same crop dimensions. This approach efficiently handles large inputs while avoiding the computational cost of full-resolution training.
Two-Phase Training with Resolution Adaptation This strategy employs two-phase training 2019; Kolesnikoy et al. 2020). First, the model is trained on low-resolution frames (224 x 448) for several epochs, focusing learning broad feature forecasting. Then, the model is fine-tuned on high-resolution (448 896) for a small number of epochs. This adaptation phase improves the model's ability to handle high-resolution features without incurring the computational cost of training from scratch at the higher resolution. As shown in our experiments, both strategies are effective, but the two-phase approach yields better feature forecasting performance. This is likely because the masked transformer has access to a larger spatial context when propagating VFM features in future frames.
process (Oquab et al. 2024 Touvron et al.
3.4 Modular Framework for Future-Frame Multi-Tast Predictions
Our framework supports a library of interchangeable task-specific prediction heads that operate on the preficted future freames, enabling flexible multi-task future understanding. The design modular: each head operates independently, allowing tasks to be added or removed without retraining the core feature prediction model.
We focus on four pixel-wise prediction tasks: semantic segmentation, instance segmentation, depth prediction, and surface normals prediction. However, our framework is general and can support other scene understanding tasks, such as object detection and panoptic segmentation. For seman- tie segmentation, depth prediction, and surface normals prediction, Transformer (Ranftl et al., (022)) (DPT) architecture, which is well-suited to our setup. DPT lever- ages multi-layer features from ViT-based encoders incosees--coniiseeet with the features predicted by model-and progressively refines them into high-resolution predictions using convolutional fusion. For the instance segmentation task, use a Mask2Former (Cheng et al. 2022) head.
the Dense Prediction
Prediction heads can be trained directly on frozen VFM features and then applied "off-the-shelf" to unture-framee features predicted by the model. Additionally, they can be trained to account for the PCA stage by applying PCA compression and decompression to the multi-layer fcatures. This approach is useful for cases where prediction heads trained on annotated 21) images, without requiring video data, and then added to the library for future-frame predictions.
Experiments
4.1
Experimental setup
Data. We assess our approach using the Cityscapes (Cordts et al., 2016) and nuScenes (Caesar et al. 2020) datasets, both offering video sequences from urban driving environments. The Cityscapes dataset includes 2,975 training sequences, 500 for validation, each with 30 frames captured 16 fps and resolution of 1024 2048 pixels. The 20th frame in each sequence is annotated for semantic segmentation with 19 classes. The nuScenes dataset comprises of 700 training scenes and 150 validation scenes, captured at frame rate of 12 Hz, with each sequence extending for 20 seconds.
Implementation details. By default, we use DINOv2-Reg with ViT-B/14 as the default VFM visual encoder for our method. For the masked feature transformer we built upon (Besnier and Chen 2023 implementation. We use 12 layers with hidden dimension of d = 1152 and sequence length N (with N. = 4 context frames and N₂ future frame). For end-to-end training, we use the Adam optimizer |Kingma and Ba 2015) with momentum parameters 31 0.9, 3g = 0.99, and a learning rate of 6.4 x 10 with cosine annealing. Training is conducted on 100 40Gb GPUs with an effective batch size of 64. We train DPT (Ranftl et al.) 2021 heads for the semantic segmentation, depth prediction, and surface normals estimation tasks, and Former (Cheng et al. 2022) head for the instance segmentation task. More implementation details in Appendix Subsection D

---

Table 1: Comparison with state-of-the-art on multiple forecasting tasks. Methods that do not handle a task are marked with '-' For approaches requiring separate prediction models per task (e.g., PFA), we show multiple entries (semantic/instance), ALL:: mloU of all classes. MO: mloU of movable objects. or ft is the VISTA model fine-tuned on Cityscapes. Compared approaches
include: 3Dconv-F2F-
(Chiu et 2020), Di110-S2S (Lue et al. 2017), F2F (Luc et al.
2018), ConvLSTM (Chiu et al., 2020), FeatRepro. (Vora et al. 2018 Bayesian S2S (Bhat
tacharyya et al. 2019). 3Dconv S-tan-FFF-SSSS F2F-SFFFF (Chiu et al. 2020), ooformF2F (Sarić et al. 2019 LSTM
AM $2S (Chen and Han 2019). APANet (Hu et al., 2021), LSTM M2M
(Terwilliger et al. 2019
IndRNN-Stack (Graber et al. 2021 DiffAttn-Fuse (Graber et al.
2022). F2MF (Saric et al.
2020 and PFA (Lin al. 2021 Futurist ist (Karypidis et 2025 and VISTA (Go a 2024).
SEMANTIC SEOMENTATION
INSTANCE SEOMENTATION
DEPTH
SURFACE NORMALS
METHOD
SHORT
MID
SHORT
MID
SHORT
MID
SHORT
MID
ALL
MO
ALL
MO
AP50
AP
AP50
AP
61
61
11.25
11.25
3DcoEV-F2F-RGT
57.0
40.8
D1 11110-828
59.4
55.3
47.8
40.8
F2F
61.2
41.2
39.9
19.4
7.7
ConvLSTI
60.1
PeatRepro
Bayesian 825
65.1
51.2
3DooxV-F2F-SBD
65.5
50.5
DeformF2F
65.5
63.8
53.6
49.9
LSTN AM $25
65.8
$1.3
CPConVLSTM
44.3
25.6
11.2
APAMet
64.9
51.4
46.1
23.2
29.2
12.9
65.1
IndRWB-Stack
67.6
60.8
58.1
52.1
DiffAtta-Fuee
67.9
61.2
58.1
51.7
P2N
69.6
67.7
57.9
54.6
PFA (semantic)
71.1
69.2
60.3
56.7
PFA finstance)
48.7
14,8
Puturist
73.9
74.9
62.7
61.2
96.0
91.9
Oracle
77.0
77.4
77.0
77.4
66.2
40.4
66.2
40.4
89.1
89.1
95.3
95.3
VISTA
64.9
62.1
53.9
51.0
33.1
17.7
19.8
9.0
86.4
82.8
93.0
90.0
P.N0-Poreeight (ours)
21.8
21.7
37.6
50.5
27.3
12.6
94.4
Evaluation Metrics. To evaluate our method's performance we use the following metrics: For mean Intersection over Union (mloU) in two ways: (1) mloU (ALL), which includes all semantic classes, and (2) MO-mloU (MO), which considers only movable object classes like person, rider, car, truck, bus, train, motorcycle, and bicycle. For instance segmentation, we measure performance using average precision at 0.50 IoU threshold (AP50) and also the mean average precision over IoU thresholds from 0.50 0.95. For depth prediction, we use mean Absolute Relative Error (AbsRel) and depth accuracy (6)). For surface normals, we compute the mean angular error (m.) and the percentage of pixels with angular errors below 11.25 (11.25%1). Definitions of depth and surface normals metrics are provided in Appendix Subsection
semantic segmentation,
Evaluation Scenarios Following prior work (Luc et al. 2017 Nabavi al. 2018), we evaluate our model on Cityscapes for short-term prediction (3 frames, 0.188) and mid-term prediction (9 frames, 0.54s). On nuScenes, which has more static scenes (i.c., with less movement) than Cityscapes, we use mid-term prediction (9 frames, 0.75s) and long-term prediction (18 frames, 1.5s). Further details in Appendix Subsection
Baselines We evaluate our method against three baselines. The first, the Oracle baseline, directly accesses the target future frame, establishing an upper performance bound. The second, Copy-Last, copies the most recent context frame to predict the target frame, providing a lower performance bound. Both baselines use DINOv2-Reg ViT-B encoder with DPT heads for semantic segmentation, depth prediction, and surface normals estimation, and a Mask2Former head for instance segmentation. The third baseline leverages VISTA (Gao al., 2024). a sfate-of-thh-an world model that uses video latent diffusion to generate future RGB frames from three context frames. This is large-scale model, comprising 2.5 billion parameters and trained on 1,740 hours driving videos, which we fine-tune on Cityscapes and nuScenes using the same frame-rate our model. VISTA generates future RGB frames (with action-free conditioning). which are processed by DINOv2-Reg encoder with DPT heads (identical to other baselines) for semantic segmentation, depth and surface normals prediction.
7

---

Table 2: Comparison of VFM encoders across tasks. For each encoder (DINOv2, CLIP, SAM), we show performance on segmentation (ALL, MO), depth estimation (6) accuracy, AbsRel error), and surface normal prediction (m, percentage within 11.25%).
SEGMENTATION
DEPTH
SURFACE NORMALS
ENCODER
MUTHOD
SHORT
MID
SHORT
MID
SHORT
MID
ALL.
MO
ALL1
MO1
61
AbsR
81
AbsR..
m4
11.25"
11,25
Oracle
77.0
77.4
89.1
.108
3.24
DINOV2
Copy Last
54.7
52.0
40.4
32.3
84,1
1154
77.8
.2112
4.41
89.2
5.39
84.0
Oquab et al.
2024
Prediction
71.8
71.7
59.8
57.6
88.6
.114
85.4
.136
3.39
94.4
4.00
91.3
EVA2-CLIP
Oracle
71.0
69.5
71.0
69.5
85.2
.123
85.2
.123
3.37
94.5
3.37
94.5
Copy
51.9
47.7
29.5
75.6
.216
4.52
Fang et al
2024
Prediction
66.3
64.2
54.5
49.6
85.1
.122
82.5
.145
3,56
93.4
4.18
90.1
Oracle
69.8
63.9
69.8
63.9
84,8
.143
84.8
.143
3.01
96.0
3.01
96.0
Copy Last
49.4
41.8
36.8
26.0
78.3
.211
73.4
.267
4.84
87
5.77
82.4
Kirillov
2023
Prediction
65.3
43.9
77.6
.209
3.80
Oracle
47.3
61.5
.251
5.3
86.1
86.1
VAE
Copy Last
37.1
25.6
28.5
16.5
60.7
.252
59.2
.286
5.8
83.2
6.3
80.1
Rombach et al
2022
Prodiction
33.4
17.9
24.7
9.8
64.1
.281
61.4
.394
6.5
80.5
8.0
73.2
4.2 VFM feature forecasting results
Comparison with State-of-the-Art Table 1 compares method with state-of-the-ee approaches for semantic/instance segmentation, depth estimation, and surface normal forecasting. The results highlight our key advantage: a single feature prediction model that achieves competitive superior performance across multiple scene understanding tasks. In contrast, prior works either require separate prediction models per task (Luc et al. 2018) or handle at most two tasks simultaneously (Karypidis et al.) (2025). This demonstrates the flexibility and practicality of our VFM feature forecasting approach.
Unified Representations for Multiple Tasks: VFM Features vs. RGB Pixels An alternative to forecasting VFM features is predicting future frames directly in RGB space, which also supports performing multiple downstream tasks through standard scene understanding models. For comparison, we fine-tune VISTA (Gao et al. 2024) to generate five future frames from three context frames (8 frames in total). We process these synthesized frames (both short-term and mid-term) with DINOV2- Reg ViT-B and DPT heads (similar to method's setup) for fair evaluation. Despite VISTA's large-scale training and model size (2.5B parameters), it achieves lower performance on semantic segmentation, depth estimation, and surface normal prediction. Extended evaluations nuScenes (provided AAppendix Table 7] show similar trends for depth and surface normal estimation. Our approach is also far more computationally efficient: mid-term forecasting on Cityscapes' 500 validation scenes takes approximately 5 minutes versus VISTA's 8.3 hours (both on a single A100 GPU). This highlights accurate semantic prediction while being significantly more resource-efficient.
method's advantages of operating in VFM feature space achieving
we evaluate our method using three VFM encoders to extract features for our feature prediction model: DINOv2 with registers (Darcet et al. Oquab al.) 2024) (self-superve supervsdce.,... EVA2-CLIP (Fang et al., (024) (vision-language con- trastive), and SAM (Kriillvv et al. 2023) (supervised instance segmentation). For each, we use the ViT-B variant. We also include Copy-Last and Oracle baselines for comparison. The results show that: (1) DINOv2 consistently outperforms other encoders across all tasks, achieving the best results for both short- and mid-term predictions.
Comparison of VFM Visual Encoders In Table 2|
(2) This aligns with expectations, the DINOv2- --seed Oracle also performs best in all cases. (3) Our model effectively predicts iiture-frame features for all VFMs, significantly improving over the Copy-Last baseline. Based on these findings, select DINOv2 as our default VFM encoder.
we evaluate using VAE latents (Rombach et al. 2022) (used latent generative models) instead of VFM features, training DPT prediction heads on these latents. Results are significantly worse, as expected, since these latents lack high-level information, and even DPT oracles perform poorly. This highlights the advantage of forecasting VFM features over VAE latents.
Future Prediction: VFM features vs VAE-based latents Additionally, in Table 2
8

---

Table 3: Continuous vs. Discrete VFM Representations. Comparison of continuous DINOv2 features (our approach) against 4M's DINOv2 tokenizer with discrete codes. Unlike other tables and for fair comparison with the 4M tokenized variant, we use the DINOv2 ViT-B w/o Reg model and extract features from the last layer only. Results semantic segmentation forecasting.
CONTINUOUS
DISCRETE (4M TOKENIZED)
METHOD
SHORT
MID
SHORT
MID
ALL
MO
ALL
MO
ALL
MO
ALL1
MO
Oracle
72.9
74.0
72.9
74.0
70.2
71,4
70.2
71.4
Copy Last
54,7
51.9
40.5
32.2
53.7
51.0
40.0
31.6
Prediction
68.9
69.3
57.3
55.0
61.7
60.9
53.7
51.0
Table 4: Strategies for Training-Efffcient High-Resolution Feature Forecasting.
RESOLUTIONS
ADAPTATION
SHORT-TERM
MID-TERM ALL
(Train-Tesst)
APPROACH
ALL
MO
MO
Oracle
(a) 224-+224
N/A
68.24
66,41
68.24
66.41
(b) 4484
N/A
76.97
77.40
76.97
77.40
Forecasting
(c) 224 +224
N/A
64.50
62.63
55.49
52.62
(d) 2212244444 +448
Pos. interp.
64.34
64.29
48.31
44.60
(e) +44822
Sliding win.
71.26
71.11
58.75
56.78
(f) (2248:448)
Two-phase
71.81
71.71
59.78
57.65
Discrete vs. Continuous VFM Representations Recent work in generative modeling has explored
both discrete and continuous representations for image and video generation (Chang et al. 2022 Yu
al. 2023 Yan et al. 202 Razavi al. 2019 Li et al. 2024 Tschannen al. 2024 Recent
findings favor continuous representations Li et al. 2024 Tschannen et al.) (024). ), showing that
removing vector quantization can improve generation quality while retaining the benefits of sequence modeling. To further investigate this in the context of VFM feature forecasting, we employ 4M's pretrained DINOv2 tokenizer (Bachmann et al. 2024), which encodes DINOv2 features (without register tokens) into discrete codes from vocabulary of size 8192. We train a quantized variant of DINO-Foresight (ViT-B14, single-layer features) using cross-entropy loss to predict these discrete codes, which are decoded back to DINOv2 features at inference time. Results are reported in Tabll While the discretized variant achieves comparable oracle performance the continuous VFM feature case, our continuous VFM feature forecasting approach yields superior future semantic prediction results. These findings suggest that preserving the rich, continuous representations from VFtsswithout quantization-ofiee offers clear advantages for dense semantic forecasting tasks.
we compare the resolution- adaptation strategies from Comminecioo 3.3 reporting results for future semantic segmenta- tion. High-resolutie faatuii features are essential: comparing the low-resolution Oracle baseline (model (a)) with the high- reshuutioot Oracle baseline (model (b)) highlights the importance of high-resolution features for strong segmentation performance. Consequently. forecasting low-resolution features (model (c)) results in significantly poorer segmentation than models predicting high-resolution fea- tures (models (e) and (f)). Adapting a model trained on low-resolution features for high-resolution inputs by simply adjusting position embeddings during inference (model (d)) leads suboptimal results, even underperforming compared to low-resolution forecasting. The other two adaptation strategies-Sliding Window (model (e)) and two-phase training with resolution increase (model (f) achieve considerably better results, demonstrating their effectiveness. The two-phase approach simpler and yields the best performance, so we adopt it as our default for high resh-uesolution feature forecasting.
Training-Efiice Efficient Strategies for Hesol-tto Feature Forecasting Table 4
Additional Ablations and Analysis We conduct comprehensive ablation studies in the appendix to further validate our approach. First, in[Appendix Subsection A.3] we demonstrate strong zero-shot generalization by training on Cityscapes and evaluating on nuScenes without fine-tuning. The result- ing performance is only slightly worse than models trained directly nuScenes, while surpassing

---

Scene Frunidurt (1)
Scene Munster (15)
Sccne Frankfurt (1)
Scene Munster (15)
Last Context Frame
Future Frame
Last Context Frame
Future Frame
Last Context Frame
Feture Frame
Last Context Frame
Peture Frame
Ours
Oracle
Ours
Ocicle
Ourr
Oracle
Ours
Oracle
Segnt
Segrn.
Depih
Normas Depth
Normals
[a] Short: Term
(b) Mid-Term
Figure 3: Future predictions for semantic segmentation, depth, and surface normals. Noisy segmentations the bottom of the image (in both predicted and Oracle results) are due to unannotated regions in Cityscapes that are ignored during DPT training. This artifact affects only segmentation, the predicted features, as evident the clear depth and surface normal predictions.
all baselines. Second, in Appendix Subsection A.5 we investigate the impact of different mask- ing strategies (random full masking) and loss functions (I.1, MSE, Smoothl.1, and SmoothL. with cosine similarity). Third, in [pppeeddix Subsection A.6, we analyze scalability across model sizes (Small: 115M, Base: 258M, Large: 460M parameters) and data scale (Cityscapes alone vs. Cityscapes+nuScenes combined), demonstrating consistent performance improvements with increased capacity and data diversity. These results highlight the promising robustness and scalability of our VFM feature forecasting approach.
4.3 Qualitative results.
In Figure 3
present qualitative results from method applied semantic segmentation, depth estimation, and surface normal prediction tasks, with both short-term and mid-term future predictions. Our single VFM feature prediction model produces meaningful outputs across all tasks, demonstrating the benefits of leveraging the feature space of large-scale pre-trained VFMs for future prediction.
5 Conclusion
In this work, we introduced DINO- DINO-Foresight, a self-supervised framework for semantic future prediction that shifts the paradigm from forecasting low-level latent representations to predicting high-dimensional, semantically rich VFM features. This shift offers several key advantages: enhances scene understanding by leveraging structured semantic information, avoids complexity of full-frame synthesis, and enables scalable, modular integration with downstream tasks.
To realize this approach, we designed masked feature transformer that efficiently propagates high- resolution, multi-layer VFM features over time. Our experiments show that forecasting such features not only feasible but also highly effective-ddeoonstaaiin strong performance across diverse frame- understanding tasks including semantic segmentation, instance segmentation, depth estimation, and surface normal prediction. Unlike prior methods that rely on multiple task specific models, our single model handles all tasks seamlessly, validating the scalability and versatility of our framework.
Overall, this work lays the foundation for a new class of unified and modular future prediction systems, grounded in semantic reasoning rather than low-level reconstruction.
Acknowledgements This work has been partially supported by project MIS 5154714 of the National Recovery and Resilience Plan Greece 2.0 funded by the European Union under the NextGen- erationEU Program. Hardware resources were granted with the support of GRNET. Also, this work was performed using HPC resources from GENCI-IDRIS (Grants 2023-A0141014182. 2023- ADOL 10011012884 and 204--A0010))288R33)
10

---

References
Abhishek Aich, Samuel Schulter, Amit K. Roy-Chowdhury, Manmohan Chandraker. and Yumin Suh. Efficient controllable multi-task architectures. ICCV, 2023.
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, Cordelia Schmid. Vivit: video vision transformer. In ICCV, 2021.
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H. Campbell, and Sergey Levine. Stochastic variational video prediction. In ICLR, 2018.
Roman Bachmann, Oguzhan Fatih Kar, David Mizrahi, Ali Garjani, Mingfei Gao, David Griffiths, Jiaming Hu, Afshin Dehghan, and Amir Zamir. 4m-21: An any-to-ane any vision model for tens of tasks and modalities. In NewrIPS, 2024.
Adrien Bardes, Quentin Garrido. Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun. Mido Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. Transactions on Machine Learning Research, 2024.
Giannis Bekoulis, Johannes Deleu, Thomas Demeester, and Chris Develder. Adversarial training for multi- context joint entity and relation extraction. In NeurlPS, 2018.
Victor Besnier and Mickael Chen. A pytorch reproduction of masked generative image transformer. arXiv preprint arXiv:23 14400, 2023.
Apratim Bhattacharyya, Mario Fritz, and Bernt Schicle. Bayesian prediction of future street scenes using synthetic likelihoods. ICLR, 2019.
Felix IS Bragman, Ryutaro Tanno, Schastien Ourselin, Daniel C. Alexander, and Jorge Cardoso. Stochastic filter groups for multi-task CNNs: Learning specialist and generalist convolution kernels. In ICCV, 2019.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufci Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, al. Video generation models as world simulators. OpenAl Blog, 1:8, 2024.
Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijhom. nuscenes: multimodal dataset for autonomous driving. In CVPR, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jegou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Lluis Castrejon, Nicolas Ballas, and Aaron Courville. Improved conditional vmns for video prediction. In CVPR, 2019.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In CVPR, 2022.
Liang-Chieh Chen, George Papandreou, Lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
Ting Chen, Simon Korblithh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In /CML, 2020.
Tianlong Chen, Xuxi Chen, Xianzhi Du Abdullah Rashwan, Fan Yang. Huizhong Chen, Zhangyang Wang, and Yeqing Li. Adamv Adaptive multi-tase task vision mixture-of-experts. In ICCV, 2023a.
Xin Chen and Yahong Han. International Conference on Multimedia and Expo (ICME), 2019.
context encoding for scene parsing prediction. In 2019 IEEE
Zitian Chen, Yikang Shen, Mingyu Ding. Zhenfang Chen, Hengshuang Zhao, Erik G. Learned- Miller, and Chuang Gan. Mod-squad: Designing mixtures of experts as modular multi-task learners. In CVPR, 2023b.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Maskod-attention mask transformer for universal image segmentation. In CVPR, 2022.
Hsu- kuang Chiu, Ehsan Adeli, and Juan Carlos Niebles. Segmenting the future. IEEE Robotics and Automation Letters. 2020.

---

Wonhyeok Choi and Sunghoon Im. Dynamic neural network for multi-task learning searching across diverse network topologies. In CVPR, 2023.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson. Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urhan scene understanding. In CVPR, 2016.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In ICLR, 2024.
Alexey Dosovitskiy and Vladlen Koltun. Learning to act by predicting the future. In ICLR, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Hcigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2020.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In CVPR, 2023.
Yuxin Fang, Quan Sun, Xinggang Wang. Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. Image and Vision Computing. 2024.
Chelsea Finn, Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. NewrlPS, 2016.
Shenyuan Jiazi Yang. Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. In NeurIPS, 2024.
Zhangyang Gao, Cheng Tan, Lirong Wu, and Stan Z Li. Simvp: Simpler yet better vidco prediction. In CVPR, 2022.
Spyros Gidaris, Andrei Bursuc, Oriane Siméoni, Antonín Vohecky, Nikos Komodakis, Matthieu Cord, and Patrick Perez. MOCA: Self-supervised representation learning by predicting masked online codebook assignments. Transactions on Machine Learning Research, 2024.
Rohit Girdhar, Alaaeldin El-Nouby, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. OmniMAE: Single Model Masked Pretraining on Images and Videos In CVPR, 2023.
Colin Graber, Grace Tsai, Michael Firman, Gabricl Brostow, and Alexander Schwing. Panoptic Segmentation Forecasting.. In CVPR, 2021.
Colin Graber, Cyril Jazra, Wenjie Luo, Liangyan Gui, and Alexander G Schwing. Joint forecasting of panoptic segmentations with difference attention. CVPR, 2022.
Sean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bemardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent a new approach to self-supervised learning. In NeurlPS, 2020.
Pengsheng Guo, Chen- Yu Lee, Daniel Ulbricht. Learning to branch for multi-task learning. In JCML, 2020.
Agrim Gupta, Stephen Tian, Yunzhi Zhang, Jigjun Wu, Roberto Martin-Martín, and Li Fei-Fei. Maskvit: Masked
visual pre-training for video prediction. In ICLR, 2023.
William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Dietrich Weilbach, and Frank Wood. Flexible diffusion modeling of long videos. In NeurIPS, 2022.
Jing He, Haodong LI, Wei Yin. Yixun Liang, Leheng Li, Kaiqiang Zhou, Hongbo Zhang, Bingbing Liu, and Ying-Cong Chen. Lotus: iiffussio-bbsed visual foundation model for high-quality dense prediction. In ICLR, 2025.
Kaiming He, Xinlei Chen, Saining Xie. Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
Greg Heinrich, Mike Ranzinger, Hongxu Yin. Yao Lu, Jan Kautz, Andrew Tao, Bryan Catanzaro, and Pavlo Molchanov. Radiov2.5: Improved baselines for agglomerative vision foundation models. In CVPR, 2025.
12

---

Jonathan Ho, William Chan, Chitwan Saharia. Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma. Ben Poole, Mohammad Norouzi, David Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint 1rX:0:2210.02303, 2022a.
Jonathan Ho. Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. In NeurlPS, 2022b.
Wenyi Hong. Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. In ICLR, 2023.
Anthony Hu, Fergal Cotter, Nikhil Mohan, Corina Gurau, and Alex Kendall. Probabilistic future prediction for video scene understanding. ECCV, 2020.
Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gian- luca Corrado. Gaia-1: A generative world model autonomous driving. arXiv preprint arXiv:2309. 17800, 2023.
Jian-Fang Hu, Jiangxin Sun, Zihang Lin, Jian-Huang Lai, Wenjun Zeng, and Wei-Shi Zheng. Apanet: Auto-path aggregation for future instance segmentation prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Xiaojie Jin. Xin Li, Huaxin Xiao, Xiaohui Shen, Zhe Lin, Jimei Yang. Yunpeng Chen, Jian Dong, Luoqi Liu, Zequn Jie, et al. Video scene parsing with predictive feature learning. In ICCV, 2017.
Ioannis Kakogeorgiou, Spyros Gidaris, Bill Psomas, Yannis Avrithis, Andrei Bursuc, Konstantinos Karantzalos, and Nikos Komodakis. What to hide from your students: Attention-guided masked image modeling. In ECCV, 2022.
Ioannis Kakogeorgiou. Spyros Gidaris. Konstantinos Karantzalos, and Nikos Komodakis. Spot: Self-training with patch-order permutation for object-centric learning with autoregressive transformers. In CVPR, 2024.
Efstathios Karypidis, Toannis Kakogeorgiou, Spyros Gidaris, and Nikos Komodakis. Advancing semantic future prediction through multimodal visual sequence transformers. In CVPR, 2025.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty weigh losses for scene geometry and semantics. In CVPR, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In CVPR, 2023.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In ECCV, 2020.
Dan Kondratyuk, Lijun Yu. Xiuye Gu. Jose Lezama. Jonathan Huang, Grant Schindler, Rachel Hornung. Vighnesh Birodkar, Jimmy Yan. ShangChang Chiu, Krishna Somandepalli, Hassan Akbari, Yair Alon, Yong Cheng, Joshua V. Dillon, Agrim Gupta, Meera Hahn, Anja Hauth, David Hendon, Alonso Martinez, David Minnen, Mikhail Sirotenko, Kihyuk Sohn, Xuan Yang, Hartwig Adam, Ming-Hsuan Yang. Irfan Essa, Huisheng Wang, David A Ross, Bryan Scybold, and Lu Jiang. Vidcopoct: A large language model for zero-shot video generation. In ICML, 2024.
Alex X Lee, Richard Zhang, Frederik Ehert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXiv:/8044 804.01523, 2018.
Sangmin Lec, Hak Gu Kim, Dae Hwi Choi, Hyung-ll Kim, and Yong Man Ro. Vidco prediction recalling long-term motion context via memory alignment learning. In CVPR, 2021.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. NeurlPS, 2024.
Xiwen Liang, Yangxin Wu, Jianhua Han, Hang Xu, Chunjing Xu, and Xiaodan Liang. Effective adaptation in multi-task co-training for unified autonomous driving. In NewrIPS, 2022.
Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming Hc, Bharath Hariharan, and Serge Belongic. Feature pyramid networks for object detection. In CVPR, 2017.
Zihang Lin, Jiangxin Sun, Jian-Fang Hu, Qizhi Yu, Jian Jian-Huang Lai, and Whi-Shi Zheng. Predictive feature learning for future segmentation prediction. In CVPR, 2021.
13

---

Yen-Cheng Liu. CHIH--AN MA, Junjiao Tian. Zijian He, and Zsolt Kira. Polyhistor: Parameter-efficient multi-task adaptation for dense vision tasks. In NeurlPS, 2022.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
Yongxi Lu, Abhishek Kumar. Shuangfei Zhai, Yu Cheng, Tara Javidi, and Rogerio Feris. Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification. In CVPR, 2017.
Pauline Luc, Natalia Neverova, Camille Couprie, Jakob Verbeek, and Yann LeCun. Predicting deeper into the
future of semantic segmentation. In ICCV, 2017.
Pauline Luc, Camille Couprie, Yann Lecun, and Jakob Verbeek. Predicting future instance segmentation by forecasting convolutional features. In ECCV, 2018.
Kevis-Kokee Kokissi Maninis, Iijaa Radosavovic, and lasonas Kokkinos. Attentive single-tasking of multiple tasks. In CVPR, 2019.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In CVPR, 2016.
Seyed Shahabeddin Nabavi, Mrigank Rochan, and Yang Wang. Future semantic segmentation with convolutional Istm. In BMVC, 2018.
Marina Neseem, Ahmed Agiza, and Sherief Reda. AdaMTL: Adaptive Input-dependent Inference for Efficient Multi-Task Learning. In CVPRW, 2023.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniee, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaacldin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes. Po- Yao Huang, Shang- Wen Li, Ishan Misra. Michael Rabbat, Vasu Sharma, Gabriel Synnaeve. Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions Machine Learning Research, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In JCML, 2021.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In CVPR, 2021.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vac-2. In NewrlPS, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. Latent multi-task architecture learning. In AAAI, 2019.
Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po- Yao Huang. Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaced, Judy Hoffmman Jitendra Malik, Yanghao Feichtenhofer. Hiera: a hierarchical vision transformer without the bells-and whistles..es.. In ICML, 2023.
Christoph
Josip Šaric, Marin Oršic, Tonái Antunović, Sacha Vražić, and Siniša Segvić. Single level -eatutreto-feature forecasting with deformable convolutions. In 41 DAGM German Conference on Puttern Recognition, 2019.
Josip Saric, Marin Orsic. Tonci Antunovic, Sacha Vrazic, and Sinisa Segvic. Warp the future: Joint forecasting of features and feature motion. CVPR, 2020.
Ozan Sener and Vladlen Koltun. Mask learning as mblti-objecctvv optimization. In NewrlPS, 2018.
Sophia Galko-Galouchenko Spyros Gidaris, Antonin Vobecky, Andrei Bursuc, and Nicolas Thome. Dip: Unsupervised dense in-context post-training of visual representations. In ICCV, 2025.
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In CVPR, 2021.
14

---

Jiangxin Sun, Jiafeng Xie, Jian-Fang, Hu, Zihang Lin, Jianhuang Lai. Wenjun Zeng, and Wei-Shi Zheng. Predicting future instance segmentation with contextual pyramid convistms, In Proceedings the 27th ACM International Conference on Multimedia, 2019.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303. /53399 2023.
Adam Terwilliger, Garrick Brazil, and Xiaoming Liu. Recurrent flow-guided semantic forecasting. In WACV, 2019.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. VideoMAE: Masked autocncoders are data- learners for self-supervised video pre-training, In NeurlPS, 2022.
Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Herve Jegou. Fixing the train-test resolution discrepancy. In NewrlPS, 2019.
Michael Tschannen, Cian Eastwood, and Fabian Mentzer. Givt: Generative infinite- vocaaulary transformers. In ECCV, 2024.
Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Mare Proesmans, Dengxin Dai, and Luc Van Gool. Multi-task learning for dense prediction tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Shashanka Venkataramanan, Valentinos Pariza. Mohammadreza Salehi, Lukas Knobel, Spyros Gidaris, Elias Ramzi, Andrei Bursuc, and Yuki M Asano. Franca: Nested matryoshka clustering for scalable visual representation learning. arXiv preprint arXiv:2507. 11.0007..4177 14137, 2025.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Anticipating visual representations from unlaheled video. In CVPR, 201
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In NeurlPS, 2016b.
Suhani Vora, Reza Mahjourian, Soeren Pirk, and Anelia Angelova. Future semantic segmentation using 3d structure. arXiv preprint 1.1i:181111135555 2, 2018.
Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang. and Qiao. Videomac v2: Scaling video masked autoencoders with dual masking. In CVPR, 2023.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li. Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint crXiv:2409./8886) 2024.
Yunbo Wang, Lu Jiang, Ming- Hiing- Yang. Li-Jia Li, Mingsheng Long, and Fei-Fei. Eidetic 3d Istm: model for video prediction and beyond. In JCLR, 2018.
Chen Wei, Haoqi Fan, Saining Xie. Chao- Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. In CVPR, 2022.
Haixu Wu, Zhiyu Yao, Jianmin Wang, and Mingsheng Long. Motionmnn flexible model for video prediction with varying motions. In CVPR, 2021.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yes Yen Lo, and Ross Girshick. Detectron2. 2019.
https
//githu.. com/ sttp://fcccooorreserrrc/eeettrrt
Jingwei Xu, Binghing Ni, Zefan Li, Shuo Cheng, and Xiaokang Yang. Structure preserving video prediction. In CVPR, 2018.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104.1 201557 2021.
Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024a.
Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. In NewrIPS, 2024b.
Lijun Yu. Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In CVPR, 2023.
15

---

Lijun Yu, Jose Lezama, Nitesh Bharadwaj Gundavarapu. Luca Versari, Kihyuk Sohn, David Minnen. Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander Hauptmann, Boqing Gong, Ming-Hsuan Yang. Irfan Essa, David Ross, and Lu Jiang. Language model beats diffusion tokenizer key to visual generation. In ICLR. 2024.
Lijun Zhang, Xiao Liu, and Hui Guan. Automtl: A programming framework for automating efficient multi-task learning. In NeurlPS, 2022.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, 2017.
Wenzhao Zheng, Ruiqi Song. Xianda Guo, Chenming Zhang, and Long Chen. Genad: Generative end-to-end autonomous driving. In ECCV, 2024.
Zeyun Zhong, David Schneider, Michael Voit, Rainer Stiefelhagen, Jürgen Beyerer. Anticipative feature fusion transformer for multi-modal action anticipation. In WACV, 2023.
Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. DINO-W World models on pre-trained visual features enable zero-shot planning. In /CML, 2025.
16

---

Appendix
A Additional Results
A.1 Impact of Dimensionality Reduction
In our work, we examine PCA-based dimensionality reduction method and find that compressing features in this way does not compromise performance on semantic segmentation and depth prediction downstream tasks (Table 5). In fact, reducing the dimensionality simplifies the modeling process and leads to improved performance. Specifically, for semantic segmentation forecasting, PCA enhances short-term
for moving objects-while its effect on mid-term predictions is negligible. Similarly, for depth forecasting, dimensionality reduction consistently boosts performance across all metrics for both short- and mid-term predictions.
Table 5: Impact of Dimensionality Reduction. Reduction performed using PCA. Results on semantic segmentation and depth forecasting.
SEGMENTATION
DEPTH
DIM. REDUCTION
SHORT
MID
SHORT
MID
ALL1
MO
ALL1
MO
81
AbsRl
811
AbsR
71.3
70.4
59.9
57.6
87.9
.122
84.8
.147
71.8
71.7
59.8
57.6
88.6
.114
85.4
.136
A.2 Emerging Visual Representations in the Future-Frame Masked Feature Transformer
Self-supervised representation learning has achieved remarkable progress, with numerous studies focusing on extracting robust visual features from unlabeled images and videos (Bardes et al.
2024
Tong et al. 2022 Girdhar et al. 2023] Ryali et al. 2023 Wang et al. 2023 Chen et al.)
2020 Caron
al. 2021 He et al.] 2022; Grill et al., 2020 Wei al. [2022] Kakogeorgiou et al. 2024
2022
Gidaris et al. 2024; Venkataramanan et al. 2025 Sirko Brrko-Galouchenko et al. 2025). Inspired by
investigate the potential of future-frame masked feature transformer as a self-supervised method for enhancing VFM visual features. Specifically, we train DPT heads for
these advancements,
semantic segmentation and depth prediction, using only the features predicted by the masked feature transformer but also additional features extracted from intermediate transformer layers. We examine features from the 6th, 9th, 10th, 11th, and 12th (last) layers of the transformer to assess whether these intermediate representations can further improve the strong VFM features predicted by our masked transformer.
Predicted + Intermediate Transformer Features
Predicted VPM features only
AbsRel Reduction
72.0
.004
.004
mloU
60.0
.002
.002
71.5
59.5
.000
.000
101112 Transformer Layer
6 101112 Transformer Layer
12 Transformer Layer
10
101110
6 9 101112 Transformer Layer
(d) Mid-term, Depth Figure 4: Impact of Intermediate Transformer Features Future Segmentation and Depth Prediction. Results are shown for semantic segmentation and depth prediction heads using two feature sets: only the VFM features predicted by the masked feature transformer (dashed line) and combined features from both predicted and intermediate transformer layers (blue bars). We evaluate DPT heads trained on features from the 6th, 9th, 10th, 11th and I 12th layers. For segmentation (barplots (a) and (b)), we report mloU across all classes. For depth (barplots (c) and (d)), we show the reduction in AbsRel metric (higher is better) when adding intermediate layer features.
(a) Short-term, Segm.
(b) Mid-term, Segm
(c) Short-term, Depth
17

---

Results, shown in Figure 4 indicate that incorporating intermediate transformer features from the masked transformer enhances segmentation performance, with one exception (6th-layer features for short-term segmentation). Notably, the best segmentation results are achieved using features from the 9th layer. Although the improvements are modest, this aligns with expectations given the strength of the predicted VFM features alone. Similar results are observed for future depth prediction, where intermediate features also led to performance gains, with the best depth results achieved using 6th-layer features (see Figure 4b. While exploring self-supervised learning was not the primary aim of our work, we find these results intriguing, as they suggest that future prediction methods hold promise self- self-suuereieed visual representation learners. We hope this work spark further research into this direction.
A.3 Additional Comparisons with VISTA
In Table 6] we present comprehensive evaluation comparing our method against VISTA (Gao et al. 2024) for semantic segmentation, instance segmentation, depth and surface normal estimation on the Cityscapes dataset. We extend this evaluation to the nuScenes (Caesar al., 2020) dataset in Table 7 for depth and surface normals. Notably, DINO-Foresight demonstrates strong zero-shot gencralization capabilities. When trained only on Cityscapes and directly evaluated on nuScenes without fine-tuning, performance degradation is minimal, while still significantly outperforming all baselines. The results show that DINO-Fores igght consistently surpasses the performance of VISTA on both datasets.
Table 6: Comparison across tasks on Cityscapes. We use DINOv2 encoder and show performance segmentation (ALL,, MO), instance segmentation (AP50, AP), depth estimation (o, accuracy, AbsRel error), and surface normal prediction (m, percentage within 11.25%). VISTA is the VISTA model fine-tuned on Cityscapes.
SUMANTIC SEOM.
SURFACE NORMALS
METION
StOIkT
MID
ALLL1
MD]
API
6.1
5:1
AboR.
m..
11:25*
Drecle
77.4
89.1
Copy Last
54.7
52.0
32.3
24.7
9.5
2.3
84.1
.144
77.8
.12
99.2
VISTA
62.1
53.6
$1.0
13.1
17.7
19.8
9.0
86.4
.14
82.8
.53
3.75
93.0
4.30
DIRO Foresight
71.8
71.7
59.8
$7.6
50.5
26.6
27.3
12.6
88.6
.114
85.4
.136
3.39
94.4
4,00
91.3
Table 7: Comparison across tasks at nuScenes. We use DINOv2 encoder and show performance on depth estimation (&1 accuracy, AbsRel error) and surface normal prediction (m, percentage within 1 1..25).. VISTA is the VISTA model fine-tuned on nuScenes. The last row evaluates zero-shot generalization: FiNo-Foresight trained on Cityscapes and directly evaluated on nuScenes.
DEPTH
SURFACE NORMALS
МETHOD
MID
LONG
MID
LONG
81 1 AbsR
6 1 AbsR
11.25
f
m 11.25
Oracle
82.6
.206
82.6
.206
3.09
97.1
3.09
97.1
Copy Last
73.4
.353
68.4
.468
4.66
88.6
5.31
85.3
VISTA
74.6
.337
70.8
.421
4.46
90.8
4.96
88.2
DINO-Foresight
80.7
.218
76.3
.299
3.59
93.9
4.22
90.6
DINO-Foresight (zero-shot) 78.4
.269
72.1
.377
4.03
92.3
4.77
88.4
A.4 Impact of Multi-Layer Features
Scene understanding models benefit from utilizing features from multiple layers of frozen image encoder. To fully exploit the pretrained DINO features, we integrate representations from several layers into the DPT head. "Talle & presents an ablation comparing multi-layer DINO features to using only the final layer (layer 12) for semantic segmentation on Cityscapes. The results demonstrate that aggregating features from layers 3, 6, 9, and enhances performance, with ALL (all semantic classes) and MO (movable objects classes) scores rising from 1/73/ to 77.0/77.4.
18

---

Table 8: DINO+DPT Features Ablation. Ablation of multi-layer DINO features as imput to the DPT head versus using only the last layer features, evaluated on semantic segmentation on Cityscapes.
SEGMENTATION
LAYERS
ALL
MO
12
72.1
73.4
3,6,9112
77.0
77.4
A.5 Masking strategy and loss function ablations
We evaluate the impact of masking strategies and loss functions on forecasting performance. As shown in Table 9| full masking (masking all future features) consistently outperforms random masking across all metrics, for semantic segmentation and depth forecasting. This demonstrates that masking all future features forces the model learn more robust temporal dynamics. Regarding loss functions (Table IO), we find that our framework is robust to loss function choice, with LI, MSE, SmoothL1, and I+Cooih achieving comparable performance.
Table Impact of Masking Strategies on Cityscapes. We compare random masking versus full masking for semantic segmentation, depth estimation, and surface normal prediction. Full masking demonstrates consistently superior performance across all metrics.
SEMANTIC SEGMENTATION
DEPTH
SURFACE NORMALS
STRATEGY
SHORT
MID
SHORT
MID
SHORT
MID
ALL1
MOT
ALL1
MOT
AbsR
6.1
AbsRt
mn]
11.25"
ml.
11,25
Random Masking
88.2
84.7
3.48
Full Masking 71.8
71.7
59.8
57.6
88.6
.114
85.4
.136
3,39
94.4
4.00
91.3
Table 10: Loss Function Comparison on Cityscapes. We evaluate different loss functions (L1, MSE, SmoothL1, EmoothhL+Cosine for semantic segmentation, depth estimation, and surface normal prediction. Results demonstrate that our framework is robust to loss function choice, with all variants achieving comparable performance.
SEMANTIC SEGMEN LATION
DEPTH
SURFACE NORMALS
Loss FUNCTION
SHORT
MID
SHORT
MID
SHORT
MID
ALL
MO1
ALL1
MO1
6, 1
AbsR
6.1
AbsR.
m,
11.25%
11,25
LI
71.7
71.7
59.7
57.6
88.6
.118
85.6
.138
3.41
94.3
4.00
91.3
MSE
57.8
.117
85.4
.138
3.40
Smoothl.1+Cos
71.7
71.4
59.8
57.5
88.7
.116
85.5
.137
3.40
94.4
3.98
91.4
SmoothL1
71.8
71.7
59.8
57.6
88.6
.114
85.4
.136
3.39
94.4
4.00
91.3
A.6 Model Size and Data Scale Scalability
We investigate how performance scales with model size and training data diversity. As shown in Table I1, evaluate three model variants-Small (115M), Base (258M), and Large (460M) parameters-by by modifying the hidden dimension and number of attention heads while keeping dataset size and training duration fixed. Results demonstrate consistent performance improvements with increased model capacity, particularly for mid-term predictions, indicating that larger models better capture complex temporal dynamics VFM features. Regarding data scale (Table 12), combine Cityscapes and nuScenes datasets with probabbioy sampling during training and evaluate on Cityscapes. The model trained on combined datasets achieves consistent improvements across all tasks compared to training on Cityscapes alone, with gains particularly pronounced for mid-term semantic segmentation. These findings demonstrate that our framework effectively scales with both increased model capacity and diverse training data, motivating further exploration with larger models and datasets to fully realize the potential of forecasting VFM features for multi-task scene understanding.
19

---

Table 11: Model Size Scalability Cityscapes. We evaluate three model sizes Small (115M), Base (258M), and Large (460M) parameters- across semantic segmentation, depth estimation, and surface normal prediction. Results demonstrate consistent performance improvements with increased model capacity.
acrooss
SEMANTIC SEGM.
DEPTH
SURFACE NORMALS
MODEL VARIANT
HIDDEN DMM
ATT HEADS
SHORT
MID
SHORT
MTD
SHORT
Mm
ALLT
MU1
ALL
MU'
6,
Absk]
4,7
Abs8)
11,25"
m]
11:25"
Small (115M)
768
87.7
84.8
3.59
(258M)
854
Large (460M)
Table 12: Data Scale Scalability. We compare training on Cityscapes alone versus training on combined Cityscape++uSccene data. Results demonstrate that increasing training data diversity leads to improved performance across all tasks and temporal horizons.
SEMANTIC SEGMENTATION
DEPTH
SURFACE NORMALS
SHORT
MID
SHORT
MID
SHORT
MID
ALL1
MO
ALL1
MO
6,
AbsR
6.1
AbsRj
ml
11.25"
ml.
11.25
Cityscapes
71.7
.114
3.39
4.00
Sityscapes-nueeeees
72.3
72.2
61.0
59.4
88.6
.117
85.7
.136
3.41
94.4
3.95
91.6
A.7 More Visualizations
In Figure and 6 we present additional qualitative results illustrating the prediction of se- mantic segmentation, depth maps, and surface normals. Specifically, we compare our method, DINO-Fores eights,tithhe against the Oracle, which involves using future RGB frames as inputs for differ- ent prediction heads, well as VISTA Gao al. (2024). As illustrated in in/Figure DINO- Foressgtt maintains superior integrity of motion dynamics and geometric consistency across frames, resulting in more accurate predictions
In Figure 7 and [ we offer additional qualitative results derived from utilizing DINO-Foresight for the prediction of semantic segmentation and depth maps and surface normals over extended time intervals. These outcomes are achieved through the use of autoregressive rollouts. Beginning with a series of four context frames (Xr-9 9 Xt), the model is capable of predicting up 48 subsequent frames, equivalent to 2.88 seconds, with predictions occurring at an interval of every third frame. Our model consistently delivers accurate predictions over the entire forecasted duration, effectively capturing motion dynamics and maintaining consistency across different modalities. This performance underscores its robustness and versatility, which are related its capability of predicting the features of foundation model. As final remark, it is important to note that the noisy segmentation predictions observed at the bottom of the images, present in both the predicted and Oracle results, are attributed to unannotated regions in the Cityscapes dataset that are disregarded during DPT training. This artifact impacts only the segmentation outcomes of DPT head and does not affect the predicted future features, evidenced by the clear and accurate depth and surface normal predictions.
B Limitations and Future Work
In this work, introduced DINO-Foresiiht, a simple yet effective method for semantic future prediction based on forecasting VFM features. Our approach delivers strong results while opening several exciting directions for future research.
First, our method uses a straightforward masked transformer with SmoothL1 loss. While forecasting VFM features avoids the challenges of modeling complex pixel distributions, current implementa- tion deterministic. However, our framework can easily be extended to capture uncertainty- for example, by adding diffusion loss (as in MAR (Li al. 2024) modeling tokens with Gaussian mixture model (as in GIVT (Tschannen et al.) 2024)). These extensions would better handle future ambiguity while maintaining the simplicity of approach.
Although we explored strategies to reduce training compute for high-resolution feature prediction, inference-tee time compute demands remain unchanged. Future work could address this by adopting
20

---

hierarchical transformer architectures (Ryali et al. 2023), which would not only improve efficiency but also enable the model handle even higher feature resolutions.
Another promising direction is scaling DINO-Foresight to larger datasets and models. Our ex- periments on model size (1 115MM 460M parameters) and data diversity (CitsscapestnuSeenes) demonstrate consistent performance improvements, particularly for mid-term predictions, suggesting that further scaling to even larger models and more diverse datasets could yield substantial gains in forecasting performance.. Furthermore, the flexibility of our framework allows seamless integration of newer VFM encoders, such as RADIOv2.5 (Heinrich al. 2025), which combines multiple VFMs into a single, more powerful model, enhancing its multi-task task future scene understanding capabilities.
Overall, these research directions highlight the flexibility and growth potential of our approach, paving the way for further advancements in semantic future prediction.
C Broader Impact
Our work enables efficient and scalable semantic future prediction by forecasting semantically rich VFM features. This allows flexible integration with different scene understanding tasks without retraining, making it useful for applications like autonomous driving and robotics. While we do not foresee risks in our approach, we must remain mindful that the pretrained Vision Foundation Models we build upon may carry biases, which could potentially influence our semantic future predictions.
D Implementation Details
We provide implementation details for the heads trained different downstream tasks. The DPT head is used for semantic segmentation, depth estimation, and surface normal estimation. We adopt the Ranttl et al. (202T) implementation from Depth Anything Yang al. (2024a b)), setting the feature dimensionality to 256 and configuring dptoutchannels = [128, 256, 512, 512]. For all tasks, models are trained for 100 epochs with batch size of 128 (16 GPUs). The learning rate is set to 0.0016, using the AdamW optimizer with linear warmup for the first 10 epochs, and weight decay is 0.0001. For semantic segmentation, we use a polynomial scheduler and cross-entropy loss with 19 classes. For depth estimation, we use cosine annealing scheduler and cross-entrops loss, with 256 classes. For surface normal estimation, we employ a polynomial scheduler and a loss function combining cosine similarity and L2 loss with weighted averaging, using 3 classes.
For the Mask2Former head used in instance segmentation, implement our approach using the official Fask2Former Cheng et al. (2022) and Detectron Wu et al. (2019) codebases. The main dif- ference compared to the official Fask2Former configuration for Cityscapes instance segmentation is the input feature maps. In our approach, the four multi-scale feature maps expected by Mask2Former are derived from the PCA features. These PCA features are first projected to 128, 256, 512, and 1024 dimensions and then resized so their spatial resolutions are x4, x2, x1, and x0.5 relative to the original resolution of the DINOv2 ViT-B outputs. We train using the Adam W optimizer, with a batch size of 64 (8 x 8 GPUs), learning rate of 0.00032, weight decay of 0.05, and 67. 07,500 iterations, with polynomial scheduler.
Regarding Vista, we fine-tuned the model with 8 frames in total (3 as context and 5 future frames) for cityscapes, while for Nuscenes we used 9 frames in total (3 as context and 6 future frames) to support long-term forecasting
E Definitions of Evaluation Metries
For depth prediction, we use two metrics: the mean Absolute Relative Error (AbsRel), defined b,
as M Σ la.-b.l where a; and by are the predicted and ground truth disparities at pixel i, and M is the number of pixels. We also evaluate depth accuracy using 61, the percentage of pixels < 1.25. For surface normal evaluation, we compute the mean angular error
where max
a.s bz 6. a.
mL as 1 N Σi= cos-1
n.n;
where n, and ñ, are the predicted and ground truth normals,
[n: [:
21

---

respectively. Furthermore, we measure precision through the percentage of pixels with angular errors below 11,25,, calculated as Σ> I(0 < 11.25%)
F Details of Evaluation Scenarios
Regarding Cityscapes, the target frame for both short and mid term prediction is 20. We subsample sequences by factor before inputting to the model. For short-term prediction. the model uses frames 8, 11, 14, and 17 as context to predict frame 20 (with context length N. = 4 and NN = 1). context and predicts frame 20
For mid-term prediction, the model uses frames 2, 5, 8, and auto-regressively through frames 14 and 17. We calculate segmentation metrics on the 20th frame using Cityscapes ground truth. For depth and surface normals, we rely on pseudo-annotations from D Yang al. (2024b) and Lotus He et al. (2025). respectively, due to the lack of true annotations Cityscapes. Regarding nuScenes, the target frame for both mid and long term prediction is 29. Again, we subsample sequences by factor of 3 before input to the model. For mid-term prediction, the model uses frames 11, 14, 17, and as context and predicts frame auto-regressively through frames 23 and 26. For long-tern prediction, the model uses frames 2, 5, 8, and as context and predicts frame 29 auto-regressively through frames 14,17,20,23 and 26. Again for depth and surface normals, rely (2024b) and Lotus He et al. (2025), respectively. due to the lack true annotations in nuScenes.
-auuddoanntttiions from DeythAnything V2 Yang al.
22

---

Last Context Frame
Future Frame
VISTA (Output)
Ours
Oracle
VISTA
Segm.
Depth
Surf. Normals
(a) Short-Term
Last Context Frame
Future Frame
VISTA (Output)
Ours
Oracle
VISTA
Segm.
Depth
Surf. Normals
(b) Mid-Term
Figure 5: Visualization of future predictions for semantic segmentation, depth, and surface normals. The illustrated scene is Frankfurt (01 (017082-017111))
23

---

Last Context Frame
Future Frame
VISTA (Output)
Ours
Oracle
VISTA
Segm.
Depth
Surf. Normals
(a) Short-Tem
Last Context Frame
Future Frame
VISTA (Output)
Ours
Oracle
VISTA
Segm.
Depth
Surf. Normals
(b) Mid-Term
Figure 6: Visualization of future predictions for semantic segmentation, depth, and surface normals. The illustrated scene is Munster (15).
24

---

Context Frames
X
X
X
Predicted Frames
X0.a(0.18s)
(0.36s)
X (0.54s)
X (0.
16 (0.99s)
(1.18
X1421 (1.26s)
Xe+24 (1.4ss)
X1+27( (1.62s)
Xo (1.8s)
X143a (1.98s)
X xa 16s)
Ne (2.34s)
X au (2.52s)
X 40 (2.7s)
X 48 (2.88s)
Figure 7: Long-term semantic segmentation, depth and surface normal predictions. The illustrated scene Frankfurt (01 (011791-011820)) DIND-Foresight consistently preserves motion dynamics and intricate details in complex scenes over extended time horizons.
25

---

Context Frames
X
X
X t-3
X
Predicted Frames
X143(0.18s)
X1.6(0.36s)
(0.5g0
X (2s)
(0.1s(
(1.18(1.08s)
X1421 X142((1.265)
(1.25(1.445)
X (1.62s)
(1.30(1.8))
X149s(1.98s)
X1e36 (2. 16s)
X (2.34s)
Xe (2.52s)
X (2.7s)
X (s.88888
Figure Long-term semantic segmentation, depth and surface normal predictions. The illustrated scene is Frankfurt (01 (006570-006599).. DINO-Foresighe excels in predicting the motion of the nearby car but faces challenges with distant. low-motion objects, highlighting areas for future improvement.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately describes the proposed method and align with the results presented in Table Table 2 and Table
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations of our work in Appendix Subsection B
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
27

---

Justification:
We do not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide all the implementation details for our experiments section and in the Appendix Subsection D
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
28

---

reproducible, they Answer: [No]
Justification:
We will publicly release our code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
public/guided www.eeoeaaiiiiiooooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/ //nips httpcc/cpplllgggiuueee
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: To our knowledge, we have included all required dataset splits, hyperparame- ters and experimental specifications needed to reproduce the resuls.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Running large generative models multiple times with differentt random seeds demands computational resources beyond our capacity. Statistical significance testing remains uncommon the field of large image/video generative models.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We specify the hardware used (8xA 100 40G GPUs) for our experiments in section 4 and in the Appendix Subsection
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't It make it into the paper).
paper (e.g., preliminary or failed experiments that http:rsss../pllll/cccccccoooeeeo
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have read the code of ethics and made sure that this paper conforms it in every aspect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the societal impact of our work in Appendix Subsection
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
30

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The proposed method for semantic future prediction do not present significant misuse concerns that would require specialized release safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We appropriately cite all code implementations, datasets and models utilized in conducting our experiments and evaluating our models.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
31

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The source code will be open sourced upon acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We not do involve crowdsourcing experiments research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We not involve human subjects our work, therefore IRB approval is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
32

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLM usage doe not impacts the core methodology, scientific rigorousness and originality of this work.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non sta-standard components.
Please refer to LLM policy for what should or should not be described.
33


Reviews:
[{"review_id": "VTU1NGJdcp", "reviewer": "Reviewer_BMEg", "summary": "DINO-Foresight introduces a self-supervised framework for predicting future scene dynamics by forecasting vision foundation model (DINOv2) features of future frames of videos. A key contribution is the use of high resolution features from multiple layers of the DINO model and the use of PCA for dimensionality reduction. This allows the model to predict future depth maps, semantic segmentation maps and surface normals of future frames of videos from steet scene datasets: CityScapes and nuScenes.", "questions": "•\tThe paper should highlight its novel technical contributions in more detail.\n•\tDoes the proposed approach generalize to novel scenarios?\n•\tCan the proposed approach handle scenarios where the distribution of future states is multi-modal?", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n•\tThe paper demonstrates state of the art results on a wide range of tasks ranging from the prediction of future semantic segmentation maps to depth maps.\n\n•\tThe proposed method is simple and computationally efficient.\n\n•\tThe paper includes adequate ablations highlighting the effectiveness of the proposed approach.\n\n•\tThe paper is well written and easy to understand.\n\n**Weakness:**\n\n•\tNovelty: The proposed method is relatively simple – it predicts the future in the feature space instead of RGB/semantic/depth pixel space. The key contribution is the integration of high level features using a PCA dimensionality reduction step. A key downside of predictions in the DINO feature space is the inability to predict future RGB frames as DINO features still discard a lot of high-frequency details in the original RGB input sequence. \n\n•\tGeneralization: The proposed model is trained and test on the same datasets: Cityscapes and nuScenes. However, the source DINO model is inherently generalizable across datasets. It would be interesting to see zero-shot applications to datasets such as Waymo Open in the street scene domain.\n\n•\tEvaluation metrics: In case of street scenes, nearby objects have a larger apparent motion, while faraway objects can appear static. The paper uses evaluation metrics that do not take such motion into account. While this is also an issue with prior work, it would be interesting to use a motion aware evaluation metric, to highlight that the proposed model can predict motion correctly. \n\n•\t Multi-modality: The distribution of future, e.g., in case of street scenes, is highly multi-modal – there are many distinct likely futures, i.e., a vehicle can turn to the left or to the right. The proposed model cannot capture this multi-modal distribution.\n\n•\tDiverse scenarios: The proposed model is limited to street scenes, it is not clear if the model can be applied to arbitrary videos for future frame prediction. It would be interesting to include evaluation on diverse domains.", "was_revised": true, "final_justification": "The rebuttal addressed all my concerns.", "rebuttal": "Thank you for your comprehensive review and insightful feedback. We appreciate the assessment of our work and the constructive suggestions for improvement. Below, we provide our responses to each of your points:\n\n**Clarifying the technical contribution of our work**\n\nWe appreciate your assessment of the method's simplicity as a strength while noting concerns about novelty. Our core contribution lies in demonstrating that modeling the temporal evolution of VFM features enables effective multi-task semantic future prediction. While the individual technical components are not novel, our design reveals that VFM features maintain semantic richness when predicted temporally, enabling a unified framework that operates across multiple scene understanding tasks without task-specific training. \n\n**Predicting future RGB frames from DINO**\n\nWhile out of scope for this work, we explored the ability of our method to predict future RGB from predicted DINO features. To this end, we trained offline a DPT head with simple L2 loss to \"reconstruct\" RGB from DINOv2 features using just ~3000 RGB images and then applied it to our prediction transformer following our pipeline. Preliminary results show that in the oracle setting, DINOv2+DPT can sufficiently reconstruct RGB frames with very low MSE, and in the prediction setting provides sufficient RGB predictions with clear structure but blurry regions in high motion/ambiguous areas. This preliminary experiment demonstrates that while our focus remains on semantic prediction for autonomous driving applications, our framework shows promising potential for extension to RGB prediction - potentially through a generative decoder (e.g., diffusion models) that could convert predicted DINO features to RGB frames.\n\n**Zero-shot Evaluation**\n\nYou raise a valuable point regarding the importance of cross-dataset evaluation. While our primary analysis has focused on Cityscapes and nuScenes, the model is built on top of DINOv2 features, which are inherently designed for strong generalizability across diverse visual domains. Due to time constraints and the lack of standardized benchmarks and data loading modules for Waymo (Panoramic Video Panoptic Segmentation Subset), we provide preliminary zero-shot results on nuScenes validation set using a model trained solely on Cityscapes. Notably, the zero-shot performance on nuScenes is only slightly worse than that of a model trained directly on nuScenes and surpasses all baselines. These initial findings demonstrate encouraging potential for zero-shot transfer beyond the original training dataset, supporting the view that our approach can generalize effectively to new domains. We see comprehensive cross-dataset and zero-shot evaluations as an important direction for future work.\n\n*Short Term Results on Nuscenes*\n\n| Method              | Depth (δ1↑) | Depth (AbsRel↓) | Normals (m↓) | Normals (11.25°↑) |\n| :---               | :---:       | :---:           | :---:         | :---:              |\n| Oracle              | 82.6        | 0.206           | 3.09          | 97.1               |\n| Copy Last Baseline  | 73.4        | 0.353           | 4.66          | 88.6               |\n| Vista               | 74.6        | 0.337           | 4.46          | 90.8               |\n| Nuscenes Trained    | 80.7        | 0.218           | 3.59          | 93.9               |\n| Cityscapes Trained  | 78.4        | 0.269           | 4.03          | 92.3               |\n\n\n*Mid Term Results on Nuscenes*\n\n| Method              | Depth (δ1↑) | Depth (AbsRel↓) | Normals (m↓) | Normals (11.25°↑) |\n| :---               | :---:       | :---:           | :---:         | :---:              |\n| Oracle              | 82.6        | 0.206           | 3.09          | 97.1               |\n| Copy Last Baseline  | 69.5        | 0.442           | 5.15          | 86.3               |\n| Vista               | 70.8        | 0.421           | 4.96          | 88.2               |\n| Nuscenes Trained    | 76.0        | 0.299           | 4.27          | 90.5               |\n| Cityscapes Trained  | 73.7        | 0.346           | 4.54          | 89.8               |\n\n**Motion-aware evaluation**\n\nWe appreciate the suggestion to incorporate motion-aware evaluation metrics. In this work, we have followed standard benchmarks and evaluation protocols established by prior works, including MO-mIoU, which measures segmentation performance specifically on movable objects and thus partially addresses the dynamic aspects of street scenes. Incorporating motion-weighted evaluation techniques would be an interesting addition to assess the model’s ability to predict dynamic scene elements. To the best of our knowledge, such a comparison or implementation is not available in the standard benchmarks we adopted. However, due to the limited available time for the rebuttal, we were unable to implement such metric at this point. We are open to adopting such metrics in the final version of the paper or as part of future work. \n\n**Multi-modality**\n\nYou correctly identify that our current approach produces deterministic predictions rather than capturing the multi-modal nature of future possibilities. We acknowledge this limitation in our paper's limitations section, where we discuss that our deterministic architecture does not account for inherent uncertainty in future predictions and propose incorporating stochastic elements such as diffusion losses on tokens to better capture this ambiguity. Despite predicting the most plausible future based on observed conditions, it does not capture the full distribution of possible outcomes. This represents an important future direction for handling scenarios where multiple futures are equally plausible (e.g., vehicles at intersections).\n\n**Domain diversity**\n\nWe acknowledge that our evaluation is currently limited to street scenes, which constrains broader claims about general applicability. However, our approach operates on frozen DINOv2 VFM features, which are inherently generalizable and not domain-specific, having been trained on diverse visual data from multiple domains. By conducting temporal modeling directly in the VFM feature space, we further reduce the domain gap that would typically exist in raw RGB space, increasing the likelihood that our method can transfer to varied input domains beyond autonomous driving. Thus, extending our evaluation to more diverse video datasets is an important direction for future work. ."}, {"review_id": "H3FAY60oar", "reviewer": "Reviewer_Bct7", "summary": "This paper introduces  DINO-Foresight, a self-supervised framework for semantic future prediction that leverages pretrained Vision Foundation Models (VFMs). The core idea of the framework is to forecast VFM features over time using a masked feature transformer. Unlike traditional approaches that rely on low-level pixel or latent space representations, DINO-Foresight directly predicts semantic features, which are then applied to various downstream tasks such as semantic segmentation, depth prediction, and surface normal estimation. The paper demonstrates that the proposed method outperforms state-of-the-art methods in future-frame prediction across various tasks.", "questions": "Please see the Weaknesses above.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- This paper is well-motivated, and the writing is easy to follow.\n- Extensive experiments on multi-task performance. In Table1, the authors compare their method with multiple counterparts on several forecasting tasks, including depth estimation and instance segmentation. Such comprehensive comparisons greatly demonstrate the superiority of the proposed method.\n- The idea of predicting semantic features (from vision foundation models) of future frames enables more efficient learning and inference by omitting the generation of low-level pixel details. This solution might be practically useful and insightful for research areas that are sensitive to computational efficiency, such as real-time autonomous driving.\n\nWeaknesses:\n- Lack of discussion with some prior works. Besides the related work listed in Semantic Future Prediction (in lines 86-99), some prior works are missing from the discussion and comparison, such as V-JEPA and DINO-WM. \n\n- Lack of investigation on the scalability of the model size and data scale. Experiments are conducted on a fixed model size and data scale without any indication of how task performance would be with more/less parameters and data.\n\n\nReferences:\nV-JEPA: Revisiting Feature Prediction for Learning Visual Representations from Video\n\nDINO-WM: DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning", "was_revised": true, "final_justification": "Accept: Technically solid paper with a good rebuttal that addresses my previous concerns.", "rebuttal": "Thank you for reviewing our work and providing thoughtful and constructive feedback. Below, we provide our responses to each of your points:\n\n**Discussion with some prior works**\n\nThank you for raising this important point. We acknowledge our manuscript could benefit from a more thorough discussion of additional related methods such as V-JEPA and DINO-WM.\n\n- V-JEPA is a self-supervised pretraining method for learning general video representations (i.e., video encoders) via masked feature prediction, without explicit focus on future feature forecasting. In contrast, our work specifically targets the prediction of semantically rich features from frozen Vision Foundation Models (VFMs) to enable downstream future scene understanding. This reflects a fundamental difference in objectives: V-JEPA aims to learn transferable video representations, while our approach emphasizes multi-task semantic forecasting.\n- DINO-WM utilizes DINO features for world modeling, primarily applied to action-conditioned planning in simulated environments with synthetic data. Our work, instead, focuses on real-world autonomous driving scenarios and multi-task future prediction, with an emphasis on accurate dense (pixel-wise) semantic forecasting rather than planning. We would like also to mention that these two works were developed concurrently.\n\nWe will revise the manuscript to incorporate these discussions.\n\n**Model size and Data Scale scalability**\n\nYou raise an excellent point about scalability analysis. While our initial experiments focus on demonstrating the core concept with a fixed architecture, investigating how performance scales with model size and data would provide valuable insights into our method's potential. To this end, we provide preliminary results of model size and data scale experiments conducted to assess the impact on downstream task performance:\n\nThe following two tables demonstrate how varying the model size—while keeping dataset size and training duration fixed—affects downstream task performance for both short-term and mid-term future prediction. We evaluate a small and a large variant alongside our default (base) architecture by modifying the hidden dimension and number of attention heads. Results indicate that our framework benefits from increased model capacity, with improvements particularly notable for mid-term predictions. These findings motivate future work to explore even larger variants and leverage more data to fully realize the potential of forecasting VFM features for multi-task scene understanding.\n\n*Short Term Results:*\n\n| Model Variant      | Hidden Dim | Att Heads | Segm (mIoU↑) | Segm (MO_mIoU↑) | Depth (δ1↑) | Depth (AbsRel↓) | Normals (m↓) | Normals (11.25°↑) |\n|:------------------|:----------:|:---------:|:------------:|:---------------:|:-----------:|:--------------:|:------------:|:----------------:|\n| Oracle             | -          | -         | 77.0         | 77.4            | 89.1        | 0.108          | 3.24         | 95.3             |\n| Copy Last Baseline | -          | -         | 54.7         | 52.0            | 84.1        | 0.154          | 4.41         | 89.2             |\n| Small (115 M)      | 768        | 6         | 71.1         | 70.8            | 87.7        | 0.125          | 3.58         | 93.7             |\n| Base (258 M)       | 1152       | 8         | 71.8         | **71.7**            | **88.6**        | **0.114**          | 3.39         | **94.4**             |\n| Large (460 M)      | 1536       | 12        | **71.9**         | **71.7**            | **88.6**        | 0.116          | **3.40**        | **94.4**             |\n\n*Mid Term Results:*\n| Model Variant      | Hidden Dim | Att Heads | Segm (mIoU↑) | Segm (MO_mIoU↑) | Depth (δ1↑) | Depth (AbsRel↓) | Normals (m↓) | Normals (11.25°↑) |\n|:------------------|:----------:|:---------:|:------------:|:---------------:|:-----------:|:--------------:|:------------:|:----------------:|\n| Oracle             | -          | -         | 77.0         | 77.4            | 89.1        | 0.108          | 3.24         | 95.3             |\n| Copy Last Baseline | -          | -         | 40.4         | 32.3            | 77.8        | 0.212          | 5.39         | 84.0             |\n| Small (115 M)      | 768        | 6         | 59.3         | 57.3            | 84.8        | 0.142          | 4.13         | 90.8             |\n| Base (258 M)       | 1152       | 8         | 59.8         | 57.6            | 85.4        | **0.136**          | **4.00**         | 91.3             |\n| Large (460 M)      | 1536       | 12        | **60.2**         | **58.3**           | **85.5**        | 0.137          | **4.00**         | **91.5**            |\n\n\nThe following two tables illustrate the impact of varying training data size—while keeping model size fixed—on downstream task performance for both short-term and mid-term future prediction. Specifically, we combine the Cityscapes (CS) and nuScenes (NU) datasets, ensuring equal-probability sampling from each during training. Preliminary results on the Cityscapes test set show that the model trained on the combined datasets achieves consistent improvements in semantic future prediction across all tasks. These findings highlight the benefit of increased and diverse training data and motivate further exploration of scalability with larger datasets and extended training to better realize the potential of forecasting VFM features for multi-task scene understanding.\n\n*Short Term Results:*\n\n| Training data               | Segm (mIoU↑) | Segm (MO_mIoU↑) | Depth (δ1×100↑) | Depth (AbsRel↓) | Normals (Mean Ang Err↓) | Normals (11.25°↑) |\n|:----------------------------|:-------------:|:---------------:|:---------------:|:---------------:|:-----------------------:|:-----------------:|\n| Oracle           | 77.0         | 77.4            | 89.1        | 0.108          | 3.24         | 95.3 |\n| Copy Last Baseline | 54.7         | 52.0            | 84.1        | 0.154          | 4.41         | 89.2             |\n| Cityscapes    | 71.8          | 71.7            | **88.6**          | **0.114**           | 3.39                    | **94.4**             |\n| Cityscapes+Nuscenes | **72.3**          | **72.2**            | **88.6**           | 0.117           | 3.41                    | **94.4**           |\n\n*Mid Term Results:*\n| Training data                | Segm (mIoU↑) | Segm (MO_mIoU↑) | Depth (δ1×100↑) | Depth (AbsRel↓) | Normals (Mean Ang Err↓) | Normals (11.25°↑) |\n|:----------------------------|:-------------:|:---------------:|:---------------:|:---------------:|:-----------------------:|:-----------------:|\n| Oracle           | 77.0         | 77.4            | 89.1        | 0.108          | 3.24         | 95.3    |\n| Copy Last Baseline |  40.4         | 32.3            | 77.8        | 0.212          | 5.39         | 84.0             |\n| Cityscapes    | 59.8          | 57.6            | 85.4            | **0.136**           | 4.00                    | 91.3             |\n| Cityscapes+Nuscenes | **61.0**          | **59.4**            | **85.7**            | **0.136**          | **3.95**                    | **91.6**            |"}, {"review_id": "mYwyzDvrFk", "reviewer": "Reviewer_Vuqi", "summary": "The paper introduces DINO-Foresight, a self-supervised framework through forecasting the temporal evolution of VFM features (such as DINOv2), and it can be further adapted to various scene understanding tasks. Unlike prior video prediction methods that rely on generative models in pixel or VAE latent space, DINO-Foresight directly operates in the VFM feature space. The core idea is to predict the temporal evolution of rich, high-level VFM features using a masked transformer model, allowing seamless plug-and-play with downstream task-specific heads for applications like semantic segmentation, depth prediction, and instance segmentation. Besides, experimental results on several future prediction tasks demonstrate a unique advantage of DINO-Foresight.", "questions": "please see weaknesses", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. This paper leverages the strengths of VFMs and further improves them by forecasting the temporal evolution of VFM features, finally providing a powerful vision backbone.\n\n2. The backbone can be seamlessly integrated with various task heads, and bring consistent improvements across these tasks.\n\n\n**Weaknesses**\n1. This paper builds upon existing models and designs, such as DINO and temporal MAE. Although the authors argue that DINO-Foresight is trained by directly forecasting VFM features, differing from standard MAE, it cannot adequately uphold the contribution's novelty.\n\n2. The paragraph is not well-structured. For example, the paragraph in line 48 feels somewhat abrupt, and the paragraph in line 81 overemphasizes the differences.\n\n3. Except for the feature prediction component, the model builds on prior designs, like FUTURIST, but the experiment results fail to demonstrate a significant improvement over these existing approaches.", "was_revised": true, "final_justification": "Thanks for the authors' detailed responses, which has essentially addressed my concerns. I am willing to raise my rating.", "rebuttal": "Thank you for reviewing our work and providing thoughtful and constructive feedback. Below, we provide our responses to each of your points:\n\n**Difference with DINOv2 and temporal MAE**\n\nWhile our method leverages DINOv2 features and employs masked transformer architectures, our core contribution centers on **modeling the temporal dynamics of VFM features to achieve accurate, multi-task, dense semantic future prediction**. In contrast, methods like Temporal MAE (or VideoMAE) focus on self-supervised pretraining of video representations (i.e., learning generic video encoders) through masked video modeling objectives. This represents a fundamental difference in goals: while their aim is to learn transferable video representations, ours is to enable precise future semantic forecasting.\n\n**Paragraph structure**\n\nWe appreciate your feedback on the manuscript structure. Indeed, in both cases mentioned, we would like to emphasize our main contribution and highlight differences with latent generative approaches respectively. We will restructure the paragraphs to provide smoother transitions and clear flow.\n\n**Advantages over prior works**\n\nWhile Table 1 shows competitive numerical results with FUTURIST, there are important practical advantages that our method provides:\n1) Unified framework: Our single model handles multiple tasks (semantic segmentation, depth, surface normals, instance segmentation) simultaneously using expressive and general frozen VFM representations, unlike prior works that would require task-specific training for each modality and dataset. For instance, FUTURIST would need to precompute pseudo-labels separately for each modality and dataset, significantly increasing implementation complexity.\n2) Scalability: New tasks can be added by simply plugging in new heads that can be trained with limited labels, without fine-tuning the feature extractor or the prediction architecture. Furthermore, since Dino-Foresight operates directly on RGB inputs rather than precomputed modality-specific features, it can readily leverage additional raw video data for further improvements. This avoids potential misalignment issues that can arise when mixing precomputed features or pseudo-labels across datasets with differing label spaces or class definitions. Also, once FUTURIST is trained on specific modalities, it must be retrained from scratch to accommodate additional modalities.\n3) DINO-Foresight is the only method to deliver future predictions for all four fundamental scene understanding tasks simultaneously, while being readily extensible to many more new tasks through its modular design. In contrast, prior works provide future predictions for to two tasks simultaneously at most."}, {"review_id": "3qec72bx7j", "reviewer": "Reviewer_S83A", "summary": "This paper introduces a novel framework for dynamic future prediction in autonomous driving. The authors propose operating in the feature space of pretrained Vision Foundation Models (VFMs) rather than at the pixel level like existing methods.\nThe approach casts dynamic future prediction as a self-supervised task with causal masking, where input tokens from previous frames are extracted using powerful VFMs and fed to a transformer. The model learns to predict latent tokens for future frames, which can then be decoded using off-the-shelf, task-specific prediction heads.\nResults demonstrate competitive or superior performance compared to previous works while being significantly more efficient due to operating in compressed feature space rather than raw pixels.", "questions": "1) I wonder if a better approach would be to use a discrete pretrained tokenizer rather than a pretrained VFM, as this would make the training process much simpler: i.e., predict next tokens within a vocabulary size instead of continuous features after PCA. Why did the authors decide to use a continuous representation rather than a discrete one? What is the advantage?\n\n2) Did the authors consider alterative masking strategies and different simple loss functions like simple L2/L1? It would be interesting to see how these affect the results.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-written and thoroughly ablated.\n2. The addressed task is very interesting.\n3. The idea of using only features from a single powerful VFM is smart and elegant, as it enables the possibility of quickly scaling to new modalities by simply training a new head on top of the original VFM features.\n\nWeaknesses:\n1. The authors claim that \"VAE latents often lack semantic alignment, making them hard to interpret or use in downstream scene understanding.\" I think this is a strong claim, and in general, this is not true. There are many works that are actually successful in using VAE features. Although it is for images, works such \"Massively Multimodal Masked Modeling\" by Mizrahi et al., showed how VQ-VAE can provide useful features for solving many downstream tasks. I think it should be straightforward to apply discrete video tokenizers such as MAGVIT-v2 or MaskViT in the context of 4M to obtain a strong model that solves downstream tasks such as segmentation or depth on videos.\n2. This work is very similar to V-JEPA, but viewed from the perspective of autonomous driving and downstream applications. Although I still believe this approach is interesting and looking at direct applications is useful, the paper should at least cite V-JEPA and highlight the main differences from an algorithmic perspective.\n3. Futurist is cited as a competitor work and is listed in Table 1 with better results, but it is not discussed in Section 4.2.\n4. Overall, I have the feeling that the paper is a composition of many different components from other papers, such as 2D-3D decomposition of attention, two-phase training, etc., and it lacks fundamental novel components.", "was_revised": true, "final_justification": "After carefully reading the other reviews and the response to my concern, I've decided to raise my score to accept.\nI strongly suggest the authors to better discuss the similarities with V-jepa however.", "rebuttal": "Thank you for reviewing our work and providing thoughtful and constructive feedback. Below, we provide our responses to each of your points:\n\n**Claims about VAEs**\n\nWe appreciate the  point regarding VAE features and acknowledge the success of VAE-based approaches like 4M [2,3]. However, we highlight several important distinctions:\n\nFirst, our claim is that standard VAEs trained with image reconstruction objectives **achieve inferior semantic alignment in their latent spaces compared to powerful encoders like DINOv2**. To substantiate this claim, we conducted experiments using a publicly available VQ-VAE from the LDM framework [1]. As shown in Table 2 of our main paper, DPT heads trained on this VAE's latent embeddings significantly underperform compared to those using DINO and other VFMs in the oracle scenario. We further validated this by training a model to predict future discrete tokens within this VAE's vocabulary using cross-entropy loss, where predicted codes were converted to continuous features for downstream tasks. We will incorporate these findings in the relevant section (lines 322-326) of the main paper.\n\nRegarding 4M, we clarify that it employs separate VQ-VAE tokenizers for each modality, including one specifically trained for DINOv2 features (trained to reconstruct these features). This DINOv2 feature tokenizer differs fundamentally from standard image tokenizers. Our claim specifically concerns the limitations of image-based VAE tokenizers for scene understanding tasks. (As an aside, we note that our PCA reduction of DINOv2 features can be viewed as a simple continuous tokenizer alternative.)\n\n**Question 1: Discrete vs. Continuous features**\n\nIn continuation to the previous discussion, and in response to the reviewer’s request, we compared forecasting performance using 4M's publicly available DINOv2 tokenizer (with cross-entropy loss on the predictor) against our proposed approach. For fair comparison, we retrained a simplified DINO-Foresight (ViT-B14 without registers, using only last-layer features). While the tokenized variant shows slightly worse oracle performance, DINO-Foresight without the VQ-VAE tokenization demonstrates significantly superior results for semantic future prediction.\n\n|  **Method**           |  **Short Term (mIoU↑)** |  **Short Term (MO_mIoU↑)** | **Mid Term (mIoU↑)** | **Mid Term (MO_mIoU↑)**  |\n|:--------------------------------|:----------------------:|:------------------------:|:--------------------:|:-----------------------:|\n| Oracle                          | 72.9                   | 74.0                     | 72.9                 | 74.0                    |\n| Oracle (4M tokenized)            | 70.2                   | 71.4                     | 70.2                 | 71.4                    |\n| Copy Last Baseline              | 54.7                   | 51.9                     | 40.5                 | 32.2                    |\n| Copy Last Baseline (4M tokenized)  | 53.7                   | 51.0                     | 40.0                 | 31.6                    |\n| Prediction                      | **68.9**                   | **69.3**                     | **57.3**                 | **55.0**                    |\n| Prediction (4M tokenized)          | 61.7                   | 60.9                     | 53.7                 | 51.0                    | 53.7                 | 51.0                    |\n\n\n\n**Comparison with V-JEPA**\n\nThank you for highlighting this. We have cited V-JEPA in supplementary material and will be happy to add it to the main manuscript. However, there are key algorithmic differences: V-JEPA is a self-supervised pretraining method for learning general video representations (i.e., video encoders) via masked feature prediction, without any explicit focus on future feature forecasting (e.g., its evaluation is focused on video understanding tasks such as action classification). In contrast, our work specifically targets the future prediction of semantically rich features from frozen Vision Foundation Models (VFMs) to enable downstream future scene understanding. This reflects a fundamental difference in objectives: V-JEPA aims to learn transferable video representations, while our approach emphasizes multi-task semantic forecasting.\n\nAdditionally, since V-JEPA's goal is to learn the video encoder, it jointly trains both encoder and predictor in an end-to-end manner, ultimately discarding the predictor after training. In contrast, Dino-Foresight keeps the VFM encoder frozen, training only the temporal predictor for forecasting. This design choice simplifies optimization and leverages the strong semantic consistency of DINOv2 features, making Dino-Foresight particularly well-suited for future prediction tasks, while V-JEPA remains more suitable for video understanding\n\n**Futurist discussion**\n\nYou're right that we should better discuss Futurist. While Table 1 shows our competitive results, Futurist requires training separate models for each task (semantic segmentation and depth), whereas our single model handles all tasks simultaneously. This scalability advantage represents a key practical contribution (see provided discussion about advantages over futurist in the response to Reviewer Vuqi). We will expand this discussion in Section 4.2.\n\n**Technical novelty**\n\nWhile we build on established components, our core contribution is demonstrating that **forecasting VFM features** creates a unified semantic representation space for multi-task future prediction. This shifts the paradigm from task-specific or pixel-level forecasting to semantic feature forecasting. The 2D-3D attention decomposition and two-phase training are practical implementation choices rather than our primary novelties. Although these components are not novel themselves, we believe our work makes an important contribution by providing an accurate, open-source framework (code to be released) that will significantly advance research in feature-based future prediction.\n\n\n**Question 2: Alternative masking strategies and loss functions**\n\nWe conducted extensive ablations on different loss functions and masking strategies, as shown in the tables below. \n\nResults demonstrate that different loss functions do not provide significant differences in performance:\n\n*Short Term Results:*\n| **Loss Function** | **Segm (mIoU↑)** | **Segm (MO_mIoU↑)** | **Depth (δ1↑)** | **Depth (AbsRel↓)** | **Normals (m↓)** | **Normals (11.25°↑)** |\n|:-----------------|:----------------:|:-------------------:|:----------------:|:--------------------:|:-------------------:|:------------------------:|\n| L1 | 71.7 | 71.7 | 88.6 | .118 | 3.41 | 94.3 |\n| MSE | 71.7 | **71.9** | 88.6 | .117 | 3.40 | **94.4** |\n| SmoothL1+cos | 71.7 | 71.4 | **88.7** | .116 | 3.40 | **94.4**|\n| SmoothL1 | **71.8** | 71.7 | 88.6 | **.114** | **3.39** | **94.4** |\n\n*Mid Term Results*\n| **Loss Function** | **Segm (mIoU↑)** | **Segm (MO_mIoU↑)** | **Depth (δ1↑)** | **Depth (AbsRel↓)** | **Normals (m↓)** | **Normals (11.25°↑)** |\n|:-----------------|:----------------:|:-------------------:|:----------------:|:--------------------:|:-------------------:|:------------------------:|\n| L1 | 59.7 | 57.6 | **85.6** | .138 | 4.00 | 91.3|\n| MSE | **60.0** | **57.8** | 85.4 | .138 | 3.99 | **91.4** |\n| SmoothL1+cos | 59.8 | 57.5 | 85.5 | .137 | **3.98** | **91.4** |\n| SmoothL1 | 59.8 | 57.6 | 85.4 | **.136** | 4.00 | 91.3 |\n\nFor masking strategies, we implemented random masking for the future frame and compared it with our full masking approach. Our experiments demonstrate the superiority of the full masking strategy on future frame prediction over random masking.\n\n*Short Term Results*\n| **Masking Strategy**                | **Segm (mIoU↑)** | **Segm (MO_mIoU↑)** | **Depth (δ1↑)** | **Depth (AbsRel↓)** | **Normals (m↓)** | **Normals (11.25°↑)** |\n|:------------------------------|:---------------:|:------------------:|:---------------:|:-------------------:|:------------------------:|:-----------------------:|\n| Random Masking | 70.5            | 70.2               | 88.2            | .121                | 3.48                     | 94.0                    |\n| Full Masking     | **71.8**        | **71.7**           | **88.6**        | **.114**            | **3.39**                 | **94.4**                |\n\n*Mid Term Results*\n| **Masking Strategy**                | **Segm (mIoU↑)** | **Segm (MO_mIoU↑)** | **Depth (δ1↑)** | **Depth (AbsRel↓)** | **Normals (m↓)** | **Normals (11.25°↑)** |\n|:------------------------------|:---------------:|:------------------:|:---------------:|:-------------------:|:------------------------:|:-----------------------:|\n| Random Masking | 58.0            | 55.7               | 84.7            | .148                | 4.11                     | 90.8                    |\n| Full Masking     | **59.8**        | **57.6**           | **85.4**        | **.136**            | **4.00**                 | **91.3**                |\n\n\n[1]. High-Resolution Image Synthesis with Latent Diffusion Models\n\n[2]. 4M: Massively Multimodal Masked Modeling\n\n[3]. 4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities"}]

Author rebuttal:

Meta-review: This submission received all four positive scores from the reviewers (1xBA, 2xA, 1xSA). The reviewers especially appreciated simplicity and effectiveness of the approach, together with thorough evaluations. The remaining questions were addressed in the rebuttal. The final recommendation is therefore to accept.
Decision: Accept (poster)