Midway Network: Learning Representations for Recognition and Motion from Latent Dynamics
Authors: Christopher Hoang
Mengye Ren
Venue: ICLR 2026 Poster (2026)
Keywords: self-supervision learning
representation learning
latent dynamics
natural videos
object recognition
motion understanding
computer vision
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Midway Network is a new SSL architecture that extends latent dynamics modeling to natural videos to learn rich representations for object recognition and motion understanding.

Abstract: Object recognition and motion understanding are key components of perception that complement each other.
    While self-supervised learning methods have shown promise in their ability to learn from unlabeled data, they have primarily focused on obtaining rich representations for either recognition or motion rather than both in tandem.
    On the other hand, latent dynamics modeling has been used in decision making to learn latent representations of observations and their transformations over time for control and planning tasks.
    In this work, we present Midway Network, a new self-supervised learning architecture that is the first to learn strong visual representations for both object recognition and motion understanding solely from natural videos, by extending latent dynamics modeling to this domain.
    Midway Network leverages a _midway_ top-down path to infer motion latents between video frames, as well as a dense forward prediction objective and hierarchical structure to tackle the complex, multi-object scenes of natural videos.
    We demonstrate that after pretraining on two large-scale natural video datasets, Midway Network achieves strong performance on both semantic segmentation and optical flow tasks relative to prior self-supervised learning methods.
    We also show that Midway Network's learned dynamics can capture high-level correspondence via a novel analysis method based on forward feature perturbation.
    Code is provided at https://github.com/agentic-learning-ai-lab/midway-network.

Published as a conference paper at ICLR 2026
MIDWAY NETWORK: LEARNING REPRESENTATIONS FOR RECOGNITION AND MOTION FROM LATENT DYNAMICS
Chris Hoang, Mengye Ren New York University (ch3451, mengye] ) @nyu. edu
ABSTRACT
Object recognition and motion understanding are key components of perception that complement each other. While self self- supervisdd learning methods have shown promise in their ability to learn from unlabeled data, they have primarily focused on obtaining rich representations for either recognition or motion rather than both in tandem. On the other hand, latent dynamics modeling has been used in decision making to learn latent representations of observations and their transformations over time for control and planning tasks. In this work, we present Midway Network, a new self-supervised learning architecture that is the first to learn strong visual representations for both object recognition and motion understanding solely from natural videos, by extending latent dynamics modeling to this domain. Midway Network leverages midway top-down path infer motion latents between video frames, as well as dense forward prediction objective and hierarchical structure to tackle the complex, multi-object scenes of natural videos. We demonstrate that after pretraining on two large-scale natural video datasets, Midway Network achieves strong performance on both semantic segmentation and optical flow tasks relative prior self-supervised learning methods. We also show that Midway Network's s Iearned dynamics can capture high-level correspondence via a novel analysis method based on forward feature perturbation. Code is provided at this link
INTRODUCTION
Animals and humans are able to recognize objects and predict their motion by observing the dynamic world with little no supervision. Inspired by this capability. research in deep learning has made significant progress in emulating "learning by observing." Prior work has shown that observing objects through time via video streams can serve as powerful learning signal (Földiák 1991, Wiskott & Sejnowski] 2002 Wang & Gupta 2015 Srivastava et al. 2015). Others have shown that self-supervised learning (SSL) methods can learn strong visual representations from vast amounts of unlabeled data (Goyal et al. 2022 Oquab et al. 2023 Fan et al. 2025).
Among a number of perception abilitics attained via observation, object recognition and motion understanding are two intertwined core components. Recognition allows one identify the same object across views to establish correspondence; conversely, motion links the same object across spacetime enable learning of its invariant properties (Simonyan & Zisserman 2014 Xu & Wang) 2021). However, most prior work on visual SSL has focused on learning representations for either object recognition motion understanding, but not both in tandem. Image SSL methods (Chen et al.) 20205] He et al., 2020; Grill et al. 2020 Caron et al., 202 Assran et al., 2023) have demonstrated strong capabilities in learning semantic representations, but most operate on iconic, i.c. single-subject, image datasets which are human-curated and additionally lack temporal information to learn motion. More recently, some have proposed performing SSL on natural videos, which depict real-world scenes and can approximate the observational perspective of animals. Nonetheless, these methods either do not utilize motion transformations for training (Gordon et al. 2020 Venkataramanan et al. 2024) rely on external optical flow networks to incorporate motion a learning signal (Xiong et al. 2021 Wang et al. 2025). On the other hand, self-supervised methods that focus on learning motion pixel-correspondence (Liu et al.) 2019 Jonschkowski et al.) 22200] Luo al. 2021 Stone jet al. 202T) or cross-viie view reconstruction task (Weinzaepfel et al. 2023) result in poor semantic representations. Only few works aim to learn both semantic and motion features (Bardes et al.

---

Published as a conference paper at ICLR 2026
(a) Traditional SSL static, iconic images using augmentations
Dense SSL natural videos with external motion predictor
[e] Our Midwayy Network Infers motion
latents for SSL on natural videos
&
[|P
aulli
114e
Reas lE
1| de
3,11
latents
Figure 1: (a) Traditional SSL methods focus on learning representations for object recognition and lean on curated, iconic image data for training. (b) Dense SSL methods extend training to natural videos, but either do not utilize motion transformations (Gordon et al. [020) Venkataramanan et
2024) for training or rely external networks to incorporate motion (Xiong et al. [021] Wang et al. 2025). (c) Our proposed Midway Network jointly learns representations semantics and motion from solely natural videos via latent dynamics modeling. The learned image-level representations
can be used towards downstream object recognition and motion understanding tasks.
2023), but these methods still depend curated, iconic image data for training. How can we jointly learn rich representations for object recogmition and motion understanding solely from natural videos?
Theories in neuroscience have proposed that animals use internal inverse and forward dynamics models and future sensory prediction. i.e., predictive coding, to perform motor control, planning, and perception (Shidara et al. 1993 Miall Wolpert 1996 Wolpert al.) 1998, Rao & Ballard 1999. Friston 2005). Works in decision making have also suggested using latent dynamics modeling for representation learning, but focus on control and planning tasks in simulated and lab environ- ments (Brandfonbrener et al. 20233 Schmidt & Jiang 2024 Cui et al. (0044). Together, these studies point to latent dynamics modeling as natural mechanism for learning useful representations of visual observations and their transformations over time, e.g.. motion.
Building on this observation, we propose Midway Network, a new SSL architecture that is the first to learn both recognition and motion understanding solely from natural videos, by extending latent dynamics modeling to this domain. Midway Network is centered around a midway top-down path, which infers motion latents between video frames via inverse dynamics that subsequently used to condition the forward predictions. We rely on two design choices in order to better model the complex, multi-object scenes in natural videos. First, we formulate the forward prediction objective over dense features, rather than global features like in previous works (Cui et al. 2024). Second, Midway Net- work introduces a hierarchical architecture with backward and lateral layers refine the motion latents and representations over multiple feature levels, inspired by optical flow networks (Sun et al. 2018).
Midway Network shows strong capability of learning image-level representations for object recognition and motion understanding after pretraining on large-scale natural video datasets. In particular, Midway Network outperforms prior SSL. methods on optical flow tasks while also achieving competitive performance on semantic segmentation tasks for both BDD 1001 (Yu et al.) 2020) and Walking Tours (Venkataramanan et al.] (024) pretraining. We also show that Midway Network's 's dynamics models can capture high-level correspondence, supported by evidence from our novel analysis method based on forwarded feature perturbation. Finally, our ablation studies demonstrate that our hierarchical design components are important for downstream performance.
To summarize, our contributions are:
We present Midway Network, novel SSL architecture that is the first to learn rich image-level representations for object recognition and motion understanding solely from natural videos. It lever- ages dense forward prediction objective and hierarchical design to better capture the complexity of natural videos.
We show that Midway Network achieves strong performance both optical flow (FlyingThing MPI-Sintel) and semantic segmentation (BDD100K, CityScapes, WT-Sem, ADE20K) when pre- trained natural video datasets, compared to prior SSL baselines which only attain substantial performance one of the two tasks.
We demonstrate Midway Network's 's ability to capture high-level correspondences between video frames with evidence from our novel analysis method based on forwarded feature perturbation.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Predictive modeling. This work builds upon research in predictive modeling from neuroscience and deep learning. Many works in neuroscience have explored predictive coding, a theory positing how the brain continuously predicts future sensory inputs with hierarchical networks perform
perception (Rao & Ballard 1999 |ao & Sejnowski] 1999] Lee & Mumford 2003] Friston 2005
Summerfield al. 2006). In particular, Friston'ss 's theory (Friston 2005) describes how perception may be split into recognition, inferring causes of sensory inputs, which is reminiscent of representation learning and inverse dynamics, and generation, predicting (future) sensory inputs from causes, which is akin to forward dynamics. Biological evidence of predictive coding has also been found, such as in monkey neural cells after receptive field excitation (Livingstone] [998) and in functional magnetic resonance imaging data of human subjects following visual stimuli under varying expectation levels (Egner et al. 2010). In deep learning, prior works such as PredNet (Chalasani & Principe 2013; Lotter et al.]
2017) have proposed architectures inspired by predictive coding to perform video prediction. More generally, there has been a line of research in leveraging prediction of future frames in videos as a learning objective (Softky 1995 Finn et al., 2016; Villegas et al. 2018 8 Feichtenhofer itt al. 2022). Others have developed predictive modeling methods that perform video prediction in latent feature space (Vondrick et al. 2016 Bardes al.) 2024). Midway Network is inspired by these predictive modeling to natural videos with a new hierarchical architecture in order to learn rich representations for object recognition and motion understanding.
ideas, extending
Dynamics modeling. Prior works have suggested that animals use internal inverse and forward dynamics models for motor control and planning (Wolpert et al.] [995]] MMall & Wolpert 1996]
Flanagan & Wing 1997; Wolpert et al. 1998. Kitazawa et al. 1998| Jordan & Rumelhart 2013). Inverse and forward dynamics have subsequently been used in works like DynaMo (Cui et al.
2024) to learn latent visual and action representations for robotic manipulation and control tasks (Brandfonbrener et al. 2023] Chen et al. 2024 Ye et al.] 2025). but they have only focused on simulated or controlled environments. World models are concurrent line of work which learn a latent dynamics model of the environment to enable efficient policy learning and
planning, but
prior works such as Dreamer and V-JEPA 2 have relied ground truth reward signals or actions (Hal
& Schmidhuber] 2018 Hafner et al. 2019 2020 Schwarzer et al. 2021 Hu et al.) 2023 Assran al. 2025). In particular, DINO-WM (Zhou al. 2024) proposed training a forward dynamics
predictor over DINOv2 (Oquab et al. 2023) patch features, but this method also required access to
ground truth actions. More recently, generative models, such as the Genie series, have emerged as promising approach for learning world models and interactive environments (Menapace et al. 2022 Yang et al., 2024, PParker-Hlddrr et al. [024] [Sun et al., 2024)). Midway Network utilizes inverse and forward dynamies tackle new problem: learning rich image-level representations for recognition and motion understanding solely from natural videos. It leverages dense forward prediction and a new hierarchical refinement architecture to capture the complex, multi-object scenes in this data domain.
Visual self-supervised learning. SSL on visual data has enjoyed a long history, from denoising autoencoders /Vincent et al.
et al. 2016 Chen et al. 2020a) He et al. 2022) to joint embedding (Grill et al. [0200] Chen et al. |002bb; He ct al. 22200 Caron ct al.) 2021 | Bardes et al. 2022) and joint-embedding predictive (Assran al. 2023 Garrido et al. 2024) models. These works primarily aim to learn semantic representations from iconic, single-subject images. Following their
success, others have proposed methods learn from dense, multi-subject images by leveraging losses
on local features (Wang et al. 2021 Xie et al. 202T] Bardes et al. 2022 Zhang et al. (023). While prior work uses motion from natural videos to learn visual representations (Xiong et al.) [022] Wang
et al.] al.]|[222 (0225), these approaches either rely on external supervised flow networks or use motion only to construct training views (Jabri et al.) 2020) Gordon et al.) [200;] Venkataramanan et al. 2024).
In contrast, our work also jointly learns representations of the motion transformations themselver lves. separate line of work focuses on learning motion as a cross-view pixel correspondence (Liu cet al. 2019 Jonschkowski et al. 2020 Luo et al.] 2021 Stone et al. 2021) or reconstruction task Weinzaepfel et 2022 2023); however, the resulting features have poor recognition capability,
Video SSL methods (Tong et al. 2022 Wei et al. 2022, Bardes al. 2024) tackle learning clip-level representations for action recognition tasks, whereas Midway Network and our baselines target image-
level representations. While few video SSL works (Qing et al., 2022) also explore hierarchical designs for learning, these hierarchies are only related to the temporal structure of videos for sampling training pairs. Finally, MC-JEPA seeks learn both semantic and motion features (Bardes et al. 2023), but unlike Midway Network, it still relies on curated, iconic image data for training.
3

---

Published as a conference paper at ICLR 2026
source
Midway motion latents
target
2
4+1
backward
prodiction loss
1
2
m
x
Cayn
Zees
Xe+1
Figure 2: Midway Network employs hierarchical design in which the midway path infers motion latents m between source and target features in a top-down manner. Within each level of this hierarchy, backward layers with top-down and lateral connections refine the souree features 2f- Forward prediction blocks, conditioned on the refined features of and motion latents ml+1, predict the dense target features 2 4 and the prediction loss Ldyn jointly trains all components at each level.
MIDWAY NETWORK
We present Midway Network, new SSL architecture that uses latent dynamics modeling to Iearn rep- resentations for object recognition and motion understanding solely from natural vidcos. At the heart of Midway Network is a midway path that infers motion latents to describe the transformation between source and target video frame. The visual encoder extracts features from the raw video frames, and the backward layers refine these features with lower-level information in a top-down manner. The for- ward dynamics model, conditioned on the source frame backward features and motion latents, predicts the dense target frame features, and the resulting prediction error is used to jointly train all components of the model. Midway Network employs a hierarchical design, where the forward prediction objective is placed at multiple feature levels, and the forward predictions from higher feature levels are used as the input to refine the motion latents at lower levels. The architecture illustrated in Figure 2 and the computations for the dense forward prediction objective are summarized in Algorithm]
Preliminaries. The model inputs are pairs of source and target video frames, 2t and Tt-1- Following the SSL knowledge distilla- tion paradigm (Grill et al., 2020; Caron et al.) 2021), we encode the video frames into features using source and target networks, 2 fo(x:) and žt+1 = (2(4411)) where is updated using an exponential moving average of the student parameters 6. Midway Network operates at multiple feature levels, so we use the notation that 2' are the features of the 1-th level, ordered from lowest level 1 to highest level T.
Algorithm 1 Dense forward prediction objective.
1: m L+1 + 0, of + 2.
2: + fo( R6+1 f- fo(0et1), #t+1 +- et
for 1 I. 1 do:
m 141 + midway(m' +2 gl+1 2t11 + m42 of + backward(2/ vft1).
6:
E+ < predictor(v/ m (1).
7:
# + || #l41
#/41
1/41
1144,112
8: Clag 11b41 111.
end for
10: Lidgn
Cayne
Motion latents via midway path. The midway path aims to learn motion latents that capture the transformation between observations over time via inverse dynamics. Specifically, the midway inverse dynamics model is transformer that takes in previous motion latents m and the source and target features 2 and 2 as input, and outputs the motion latents m' for the next level. The motion latents accumulate over levels, i.e. m = midway hondwaym'+ 2/, =f41) + m/+1, The initial motion latents are learnable tokens. For every level besides the top level, we use the output of the higher level's forward prediction, z/. instead of the features 2 as input. Thus, the model learns to refine the motion latents in top-down manner, conditioned on the higher-level predictions. This design is motivated by how prior optical flow methods (Sun et al.] (2018] Jonschkowski et al. [2020) would use intermediate flow estimates to warp features before computing cost volumes, which would subsequently be used to refine flow predictions at lower levels.
4

---

Published as a conference paper at ICLR 2026
Backward features. Prior works, from Ladder Networks Valpola 2015) to PooDLe (Wang et al. 2025), have proposed backward layers with top-down and lateral connections to relieve higher-level features of the burden of encoding low-level details. In this work, backward layers used to refine features in a top-down manner by using lateral connections to incorporate lower-level information. Specifically, the backward layers transformer blocks that use cross-attention (Lin al. (022)) where laterally-comnected features z) are used as queries that attend to higher-level backward features
of 1+1 which serve as keys and values.
Dense forward prediction. The forward dynamics model is also transformer that takes in backward features of and motion latents m as input, concatenated along the spatial dimension, and predicts the dense features of the target frame. The dense forward prediction objective is then to minimize the prediction error between the predicted features 2/411 and the realized target features 1-4-1 The prediction error the mean squared error between the normalized dense predictions and targets:
Cayg = 2+ 112.
Forward prediction gating. In a standard transformer block, the input token value is always propagated forward due to the residual connection this biases the computation towards the identity mapping. However, we would like the forward transformer model to learn whether the object captured by input token has moved, i.e., if its features can be computed from tokens at other spatial locations, rather than defaulting to the identity location. Thus, we introduce learnable gating units for the residual connection in the transformer layers of the forward dynamics model. The gating unit is a multi-layer perceptron that learns vector-wise gating weight between 0 and for the residual connection of each input token of Ut. Specifically, the transformer block modified with gating unit g such that the input to the feedforward network, h, is computed as:
Add & &Norm
Gating
Multi-Head Attention
1/2
Figure 3: Attention layer with gating unit on tt.
= g(z) x + Attention(z).
(2)
We do not use gating units in the first transformer block to provide sufficient information for initial estimates of attention, nor do we them for the motion latents to fully propagate the motion information. In our experiments, we find that the gating units improve semantic feature quality and interpretability of the learned dynamics models, as shown in Section 4.4
Invariance objective. We utilize an additional joint-embedding invariance objective over smaller crops to encourage the visual encoder to learn semantic features, following PooDLe (Wang et al., 2025). In our experiments, we use the DINO (Caron et al.) (2021) objective with projection heads on top of the source and target networks. This can be viewed as a form of regularization for the features that are subsequently used in the latent dynamics modeling.
4 EXPERIMENTS
We evaluate Midway Network by pretraining on large-scale natural video datasets, BDD100K (Yu) et al. 2020) and Walkings Tours (WT) /Venkataramanan et al.) 2024), and evaluating the learned image and motion latent representations on downstream semantic segmentation and optical flow tasks. In our experiments, we study whether Midway Network learns good visual features for both object recognition and motion understanding. We further analyze how each component of Midway Network contributes to downstream performance and what information does its dynamics models capture.
4.1 SETUP
Pretraining, We pretrain Midway Network on two large-scale video datasets from different domains. BDD100K (Yu et al. 2020> is dataset of 100,000 dashcam driving videos collected in varying weather, lighting, and of conditions from New York and the San Francisco Bay Area. Each video is 40 seconds long 720p and 30 fps. We pretrain all 70,000 videos in the train split. Walk- ing Tours (WT) Venkataramanan et al. (2024) is a dataset of 10 first-person YouTube walking videos collected in various cities of Europe and Asia, with outdoor and indoor scenes, and natural transitions in lighting and location. The videos range from 59 minutes to 2 hours 55 minutes, at 720p and 30 fps. We pretrain on the Venice video following DoRA (Venkataramanan et al. 2024)'s 's original setup."
Due computational constraints, we did not pretrain all 10 videos.
5

---

Published as a conference paper at ICLR 2026
Downstream evaluations. We evaluate Midway Network's pretrained representations semantic segmentation tasks to gauge object recognition capability. For BDD pretraining, we perform linear and UperNet readout on the BDD and CityScapes (Cordts et al., 2016) benchmarks following FlowE (Xiong et al.,
2021). For WT pretraining, we perform UperNet finetuning
on the WT-Sem (Wang al.; 2025) and ADE20K (Zhou et al.] (017)) benchmarks following 2024) and PooDLe (Wang et al., (025). For linear readout only, we use the backward layer features following PooDLe. We also evaluate Midway Network on optical flow tasks to assess motion understanding. We follow CroCo v2 (Weinzaepfel et al., 2023)'s finetuning evaluation protocol, replacing their binocular decoder with our midway inverse dynamics and forward dynamics models - baselines without binocular components also use the dynamies models, but with randomly initialized weights. We finetune models pretrained on BDD on TirtanAir (Wang et al.) (020), MPI-Sintel (Butler et al.) 2012), FlyingThinge' (Mayer et al., 2016). and FlyingChairs (Dosovitskiy et al., 2015) datasets, and evaluate on the corresponding validation splits of FlyingThings and MPI-Sintel. We report mean inlersecttio-over-nniiii (mloU) and pixel-level accuracy (Acc) for semantic segmentation, and endpoint error (EPE) for optical flow. More details on evaluation settings are provided in Appendix [
DoRA (Venkataramanan et al.
Baselines. We compare Midway Network to iconic image SSL methods (DINO, iBOT (Caron et al. 2021 Zhou et al., 2) multi-object SSL methods (DoRA, PooDLe (Venkataramanan et al. 2024 2025) ), and masked reconstruction methods (CroCo v2, VideoMAF, MAE (Weinzaepfel al. 2023 Tong et al. 2022 He al.. 2022)). DoRA uses 8-frame clips for training. VideoMAE uses 16-frame clips, and iBOT and MAE use single frames. Midway Network and all other baselines learn from pairs of frames. We also implement a modified version of DynaMo (Cui et al. 2024) that uses ViT-S as the encoder and includes the DINO invariance objective. We use official implementations to pretrain the baselines on BDD and WT. All baselines are trained on 224 x 224 resolution, except for PooDLe in Table 2 which uses 512 x 1024.
Wang et al.
Implementation. We use ViT-S and ViT-B sized vision transformers for our visual encoders. For the midway inverse dynamics, forward dynamics, and backward models, we use decoder-only transformers (Vaswani et al. 2017), with the backward layers using cross-attention (Lin et al., 2022) blocks. We largely follow the guidelines provided by PooDLe (Wagg et al. (025) on data sampling from natural videos. Specifically, we sample pairs of frames 0.5 ~ 1 seconds apart, one per video per epoch for BDD, and 0.5 seconds apart, for all possible pairs per epoch for WT Veniee. For the dense forward prediction objective, we sample larger crops of area range [0.2, 0.4] at the same location for both frames. We take smaller initial crops of area range [0.05, 0.2) at the same location for both frames, from which global and local crops are sampled for the DINO joint-embedding objective. All crops are resized to 224 x 224 resolution. Appendix provides more details on implementation. compute resources, and comparisons of training cost across the different methods.
4.2 SEMANTIC SEGMENTATION AND OPTICAL FLOW RESULTS
Table 1: Semantic segmentation and optical flow evaluations for BDD100K 224 x 224 resolution pretraining, Sem. Seg. is conducted with frozen backbone and optical flow is conducted with finetuning. DynaMo is modified to use a VIT-S encoder and DINO objective.
BDDBBOK
Seg. UperNet
Cityscapes Sem, Seg
Optical Flow
Methedl
Arch
Fp-
1/tc
1Auz
Nper
LEFE
#lyingThings
JEPE
LEPE
PooDLe Wang
R50
310)
35.1
87.8
47.4
91.0)
44.5
$3.0
55.4
1807 2 Amrent 2003
VIT:S
800
27.2
85.4
35.5
66.7
32.0
36.2
44.5
90.3
18.5
18.0
13.0
13.7
DINO
VIT:S
300
36.7
89.3
49.3
92.0
41.5
90.4
$7.9
93.3
15.8
13.8
11.5
10.8
Vatee
WITS
304)
7%
$0.5
10:9
58.0
6.4
46.9
11:7
62.99
15.2
16.1
7.3
7.6
v2
Neleonggl frut 2003
VIT:5
300
21.2
80.0
31.9
$7.0
24.0
31.5
37.5
89.0
9.7
9.4
5.1
3.8
DoRA
N0 10 min
2004
VIT.5
300
30.4
87.2
40.5
90.0
36.2
35.2
51,3
91.9
15.5
15.1
11.5
11.9
DymaM0
Laa 1004
VIT-S
300
36.8
89.4
47.4
91.7
41.2
90.3
$7.2
03.1
Midsog (ds. onl)
VITS
304)
16.6
13.5
11.7
109
Midwoy
VIT.5
300
39.7
90.3
50.4
92.4
90.9
499
Caporetal ai (200)
300
90.9
92.7
12.1
OnCa V2 wwioooprre af
300
BDD100K pretraining. Table shows results on BDD100K and CityScapes semantic segmentation, and Things and MPI-Sintel optical flow benchmarks after BDD100K pretraining. Notably, Midway Network is the only model to perform well on both semantic segmentation and optical flow tasks overall. For semantic segmentation, Midway Network outperforms all baselines on BDD100K, and its learned visual features also transfer well to CityScapes, where they are competitive with the best pest--efformng baseline, PooDLe, which relies on an external supervised optical flow network. Note that even without the backward network, our model achieves 39.2 mloU and 90.1 Acc 1DD100K

---

Published as a conference paper at ICLR 2026
Image
Ground Truth
PooDLe
CroCo v2
DoRA
DynaMo
Figure 4: Visualization of BDD semantic segmentation UperNet readout. Midway Network is able to produce cleaner object boundaries, particularly for the cyclist on the right.
Table 2: Semantic segmentation and optical flow evaluations for WT- Venice 224 x 224 resolution pretraining. Sem. Seg. and optical flow conducted with finetuning. *PooDLe 512 x 1024 resolution pretraining from their original table (Wang et al. 2025). 'iBOT results taken from DoRA Venkataramanan et al. 2024).
WT-Sem Sem, Seg.
ADEZOK Sem. Seg.
Optical Flow
Method
Arch
Ep.
TmlooU
UperNet
TAcc
UperNet TmnloU
TAsc
FlyingThingg JEPE
JEPE
JEPE (c)
MPI-Sintel
JEPE
PooDLe Wong al 2025
R50
20
13.7
85.4
36.6
77.9
BBT Zhou
2011
VIT.S
100
33.9
Video
10t dral
VITSS
100
67.9
7,8
55.6
15.9
15.8
7.0
7.4
DINO
Kinceer mi AM1
VTTS
100
11.0
83.0
29.2
74.7
15.5
14.0
12.4
13.8
CroCo
VTTS
11.3
84.4
9.6
9.1
DoRA
Keikalaram 0E
VIT-
13.6
85.7
35.2
77.7
17.9
13.3
12.4
12,4
Midway
VIT:S
100
13.1
85.4
33.4
76.9
7.7
7.4
5.2
6.6
linear readout, continuing to outperform the baselines. Midway Network also surpasses all baselines' performance FlyingThings and MPI-Sintel optical flow. As shown by Midway Network (enc. only), performance on optical flow drops drastically if we do not initialize the midway inverse and forward dynamics models with the pretrained weights, indicating that the dynamics models have learned features that are useful towards motion estimation. We also demonstrate that Midway Network's 's downstream performance also scales with larger model sizes, from ViT-S to ViT-B. While CroCo v2 edges out Midway Network optical flow for ViT-B, Midway Network does not suffer the same tradcoff on semantic segmentation performance as CroCo v2. Figure 4 and Figure compare predicted segmentation masks for BDDI0OK, and optical flow for FlyingThing and MPI-Sintel, respectively, across different methods.
Walking Tours pretraining. Table results WT-Sem and ADE20K semantic segmentation, and FlyingThings and MPI-Sintel optical flow benchmarks after WT We-Venice pretraining. Again, Midway Network is the only method achieve strong, competitive performance both semantic segmentation and optical flow tasks. Note that PooDLe was pretrained at high resolution (512 x 1024) and utilized external supervised optical flow networks. We include additional visualizations of predicted segmentation masks and optical flow for WT-Venice pretraining in Appendix
Source
Targht
Ground Touth
Michwny Network
CroCave
DINO
Figure 5: Visualization MlyingTTingg and MPI-Sintel optical flow evaluations after finetuning. Midway Network is able generate accurate optical flow predictions compared to CroCo v2.
7

---

Published as a conference paper at ICLR 2026
Table 3: Ablation studies on Midway Network components evaluated on BDDI00K semantic segmentation linear readout and MPI-Sintel optical flow finetuning.
Variant
Latent Dynamics
Backward
Multi-Level
Refinement
Gating
1mloU
JPPE
Base model
28.3
6.2
30.4
4.4
30.0
5.0
30.4
5.2
31.1
3.9
Full model
/
31.5
4.1
No backward
/
30.4
3.7
8 No multi-level
30.3
5.2
No refinement
30.8
5.1
Table 4: Ablation studies Midway Network model capacity and time delta sampling hyperpa- rameter evaluated on BDDI00K semantic segmentation linear readout and MPI-Sintel optical flow finetuning.
(a) Capacity of midway path and forward dynamics
(b) Time delta for sampling frames
Ablation
fmloU
JEPE
Time Delta
tmloU
JEPE
Full model
31,5
4.1
0.16 sec
31.0
4.1
0.5x midway dim
31.3
6.7
0.5 sec
32,0
4.3
2x midway dim
31.0
3.3
sec
31.2
4.6
1-layer midway
31.9
6.9
sec
31.0
4.9
2-layer midway
31.2
6.4
1 l-aer forward
29.6
5.0
2-layer forward
30.2
4.8
4.3 ABLATION STUDIES
perform a series of ablation studies, shown in Table where we cumulatively add components of Midway Network until we reach the full model. For the ablations, we pretrain variants of Midway Network on BDDI00K for 100 epochs and evaluate on BDD semantic segmentation with lincar readout and on MPI-Sintel optical flow (clean renderings) after finetuning FlyingChairs and FlyingThings following CroCo V2 and prior optical flow methods. For reference, we run seeds for the full model (row 6), and obtain standard deviation of 0.06 mloU and 0.08 EPE. More technical details are found in Appendix B
First, we find that adding latent dynamics modeling immediately adds large boost to performance (row 2). Next, we observe that the hierarchical structure of the backward network and multi-level learning work together with motion latent refinement provide further gains both recognition and motion understanding (row 5). Finally, using gating units improves recognition (row 6) as well as visual interpretability of the learned dynamics, as shown in Figure We also that removing any of the introduced design components from Midway Network harms performance by decent margin (rows? 7 -)).
We also investigate how varying capacity of the midway path and forward dynamics model affects performance in Table Lal Reducing midway path capacity primarily harms EPE, whereas adding capacity improves EPE and hurts mloU, likely because motion latents can capture more information from paired frames, but the forward prediction objective is made easier. Performance drops with fewer forward model layers. We further show in Table 4b that Midway Network is relatively robust to different time deltas for sampling frames in pretraining.
4.4 ANALYSIS OF DYNAMICS
To probe the extent to which Midway Network has learned dynamics after pretraining on natural videos, we introduce new analysis method based forwarded feature perturbation. First, we encode a pair of frames to get features 2, and 2141 and compute motion latents m between them, as usual. Then, sample random vector r as a tangent vector to the selected feature the source frame. We perform forward prediction to propagate the perturbation to the predicted target features' tangent vectors the propagation is done
(O, 1) and "perturb" selected spatial feature by associating
8

---

Published as a conference paper at ICLR 2026
Source
Target
Heatmap
Source
Target
Heatmap
Figure 6: Heatmaps from forwarded feature perturbation. Features are perturbed at green squares in Source, which also depicted in Target the location highlight the motion between frames. Midway Network without gating units exhibits identity bias (bottom right, red border).
Midway forward perturaation
DING-INTK feature similaitty
Midway
perturbation
simblaity
Figure 7: High-level tracking using forwarded feature perturbation and/or feature similarity, Midway Network is able to track high-level regions such as the cyclist's foot (top row, pink square).
via forward mode automatic differentiation. The cosine similarity between the random vector and the langent vectors of the predicted features then represents the sensitivity of each spatial feature in the target frame to the initial perturbation. This process is repeated times, and the similarity scores averaged to obtain a final heatmap over the target frame spatial locations. In Figure 6 we observe that the highest similarity regions in Target correctly correspond with the initial perturbation locations in Source (green square), indicating that the dynamics models can capture high-level correspondences. We also see that Midway Network without gating units (bottom right, red border) learns an incorrect identity mapping where the highest similarity region is the same location the initial perturbation.
We may also use forwarded feature perturbation as a form high-level tracking. First, for consecutive pairs of frames, we compute perturbation heatmaps over the target spatial features by individually perturbing each spatial feature in the source frame. Then, for the first frame of the video, select an initial location and take the top-5 locations in the next frame with the highest perturbation heatmap scores; from these locations, we select the one with the highest feature similarity. This process repeats with the newly selected location until we have track across all

---

Published as conference paper at ICLR 2026
frames. Figure shows these tracking results in comparison to selecting the next location based on highest feature similarity with DINO (Caron et al. (202]) pretrained on ImageNet (INIK). Despite being trained in latent space, Midway Network is able to roughly track high-level regions over time, whereas the DINO-INIK feature similarity baseline tracks quickly diverge.
In Figure 8 further compare the perturbation heatmaps optical flow estimated from RAFT Teed Deng 2020) as well as heatmaps produced by cosine similarity of last-layer dense features from different models. We convert heatmaps optical flow by computing the (x. y) delta from each source token to the target token with highest heatmap value. We show the highest-valued target token, "Pred (K=1)," and token with highest feature similarity out of the top-2 highest-waas To get optical flow at the token level, "GT (Token)," we sample optical flow at the center of each source token and retrieve the mapped target token. Forwarded feature perturbation less noisy and more well-aligned with RAFT's optical flow compared to feature similarity baselines.
tokens, "Pred (K=2)."
Midway forward perturbation
Midway forward perturbation feature similarity
forward
Midway feature similarity
DINO-BDD feature similerity
DINO-IN1K
feature similarity
Source
Target
GT (Piwal)
GT (Token)
Pred (K=1)
Pred (K-2)
Figure Optical flow estimates (Pred) derived from forwarded feature perturbation and feature similarity heatmaps compared to RAFT-predicted optical flow maps (GT) BDD.
CONCLUSION
Object recognition and motion understanding complementary aspects of perception, yet most self-supervised methods have focused on learning representations for only one facet. We aim to bridge this gap by extending latent dynamies modeling to the natural video domain. In this work, we propose Midway Network, the first self-supervised learning architecture to learn representations for both recognition and motion solely from natural videos, leveraging an inverse dynamics midway path, a dense forward prediction objective. and a hierarchical structure to capture the complex, multi-object scenes. Midway Network learns strong image-level representations for both recognition and motion, and in many cases, outperforms prior approaches on semantic segmentation and optical flow estimation. We have demonstrated that Midway Network can used across different video datasets and scales well with larger models - training more diverse data and continuing to scale model capacity could further improve performance. An exciting avenue for future work is to leverage the motion and dynamics captured by Midway Network for real-world planning tasks. Possible next steps towards this direction include incorporating forward dynamics predictor within world modeling framework.
data and using Midway Network's
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
We thank members of the NYU Agentic Learning AI Lab for their helpful discussions. CH is supported by the DoD NDSEG Fellowship. The work is supported in part by the Institute of Information & Communications Technology Planning Evaluation (IITP) under grant RS-2024- 00469482, funded by the Ministry Science and ICT (MSIT) of the Republic of Korea connection with the Global AI Frontier Lab International Collaborative Research. The compute is supported by the NYU High Performance Computing resources, services, and staff expertise.
REFERENCES
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with bonnt-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2023.
Mahmoud Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes. Mojtaba Komeili, Matthew Muckley, Ammar Rizvi, Claire Roberts. Koustuv Sinha, Artem Zholus, Sergio Arnaud, Abha Gejji, Ada Martin, Francois Robert Hogan, Daniel Dugas, Piotr Bojanowski, Vasil Khalidov, Patrick Labatut, Francisco Massa, Mare Szafraniec, Kapil Krishnakumar, Yong Li, Xiaodong Ma, Sarath Chandar, Franziska Meier, Yann LoCun, Michael Rabbat, and Nicolas Ballas. V-jepa 2: Self-supervised vidco models enable understanding, prediction and planning. arXiv preprint 0rXiv:2506.09985. 2025.
Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: wtaeeeiiaaaiinccovvviiiiii regularization for self-supervised learning. In International Conference on Learning Representations, 2022. URL net.
htttps://opennevill http:///wwweeeeeeeoooooooooo-o-e F lUb
Adrien Bardes, Jean Ponce, and Yann LeCun. Mc-jepa: A joint-embedding predictive architecture for self-supervised learning of motion and content features. arXiv preprint arXiv:2307.12699, 2023.
Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mido Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. Transactions on Machine Learning Research, 2024.
David Brandfonbrener, Ofir Nachum, and Joan Bruna. Inverse dynamics pretraining learns good representations for multitask imitation. In Advances in Neural Information Processing Systems, 2023. StRphtts:: http///ppppeeeeeeeeeeeeeeeeeeeee net.eewww.eeeeeeeeeeeeeeeeeeeeee wow.:ooommmniii-T----
Daniel Butler, Jonas Wulff, Garrett B Stanley, and Michael J Black. A naturalistic open source movie for optical flow evaluation. In European Conference on Computer Vision, 2012.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, and Armand Joulin. Emerging properties in self-supervised vision transformers. In IEEE International Conference on Computer Vision, 2021.
Rakesh Chalasani and Jose C Principe. Deep predictive coding networks. arXiv:1301.3541, 2013.
arXiv preprint
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David I Luan, and llya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pp. 1691 1703. PMLR, 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, 2020b.
Yi Chen, Yuying Ge, Yizhuo Li, Yixiao Ge, Mingyu Ding, Ying Shan, and Xihui Liu. Moto: Latent motion token as the bridging language for robot manipulation. arXiv preprint arXiv:2412.04444 2024.
MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. hhttps://qi thwp:///ggttbb.te httph/uwu..oooooooooooooeeee http::/ammmmammaaataaaaeaa ht/pmm/gmmmmtttti.ioo.. 2020.

---

Published as a conference paper at ICLR 2026
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bermt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE/CVF conference Computer Vision and Pattern Recognition, 2016.
Zichen Jeff Cui, Hengkai Pan, Aadhithya Iyer, Siddhant Haldar, and Lerrel Pinto. Dynamo: In-domain dynamics pretraining for visuo-motor control. Advances in Neural Information Processing Systems, 2024. URL
ew. net. /ttprr//wwweeeeeeeeorrr////////a
Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Häusser, Caner Hazirbas, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In IEEE International Conference on Computer Vision, 2015.
Tobias Egner, Jim M Monti, and Christopher Summerfield. Expectation and surprise determine neural population responses in the ventral visual stream. Journal of Neuroscience, 16649):16001111600, 2010.
David Fan, Shengbang Tong, Jiachen Zhu, Koustuv Sinha, Zhuang Liu, Xinlei Chen, Michael Rabbat, Nicolas Ballas, Yann LeCun, Amir Bar, et al. Scaling language-free visual representation learning. arXiv preprint 2rx1022504.01017, 2025.
Christoph Feichtenhofer, Haoqi Fan, Yanghao Li, and Kaiming He. Masked autoencoders as spatiotemporal learners. In Advances in Neural Information Processing Systems, 2022. URL
httppss//openrevvew net wwwp::eeee/ooooddeeeeeeee.m. www.eoorrruuaaa//.
3mdl
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. Advances in Neural Information Processing Systems, 2016.
J Randall Flanagan and Alan Wing. The role of internal models in motion planning and con- trol: evidence from grip force adjustments during movements of hand-held loads. Journal of Neuroscience, 17(4):1519-155 1528, 1997.
Karl Friston. A theory of cortical responses. Philosophical transactions of the Royal Society of London. Series B, Biological sciences, 360:815-36, 04 2005, doi: 1.1109/stb.2005..166
Peter Földiák. Learning invariance from transformation sequences. Neural Computation, 1991.
Quentin Garrido, Mahmoud Assran, Nicolas Ballas, Adrien Bardes, Laurent Najman, and Yann LeCun. Learning and leveraging world models in visual representation learning, 2024. URL
htttps://axixy: https://axxiv.orr//aaa.. http:/bs/224000 00504
Daniel Gordon, Kiana Ehsani, Dieter Fox, and Ali Farhadi. Watching the world go by: Representation learning from unlabeled vidcos, 2020.
Priya Goyal, Quentin Duval, Isaac Seessel, Mathilde Caron, Ishan Misra, Levent Sagun, Armand Joulin, and Piotr Bojanowski. Vision models are more robust and fair when pretrained on uncurated images without supervision, 2022. URL htt.:tttts//aaae org. abs/////// /2002.220.22 08360
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. In Advances in Neural Information Processing Systems, 2020.
David Ha and Jürgen Schmidhuber. World models. In Advances in Neural Information Processing Systems, 2018
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pp. 2555-2565, 2019.
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. International Conference on Learning Representations, 2020.
URL htl.https:22 http:/ttts///peeee httpt:::///pperreiieeee www.:eeet/ooommmeeeetomoo 1TCAADD
12

---

Published as a conference paper at ICLR 2026
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16000-16009, June 2022.
Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-I: A generative world model for autonomous driving. arXiv preprint arXiv:2309. 10809.770800 17080, 2023.
Zhaoyang Huang, Xiaoyu Shi, Chao Zhang, Qiang Wang, Ka Chun Cheung, Hongwei Jifeng Dai, and Hongsheng Li. FlowFormer: transformer architecture for optical flow. European Conference Computer Vision, 2022.
Allan Jabri, Andrew Owens, and Alexei A Efros. Space- correspondence as a contrastive random walk. Advances in Neurul Information Processing Systems, 2020.
Rico Jonschkowski, Austin Stone, Jonathan Barron, Ariel Gordon, Kurt Konolige, and Anelia Angelova. What matters in unsupervised optical flow. In European Conference on Computer Vision, 2020.
Michael I Jordan and David Rumelhart. Forward models: Supervised learning with a distal teacher. In Backpropagation, pp. 189-236. Psychology Press, 2013.
Shigeru Kitazawa, Tatsuya Kimura, and Ping-Bo Yin. Cerebellar complex spikes encode both destinations and errors in movements. Nature, 392(6675):4944497 1998.
Tai Sing Lee and David Mumford. Hierarchical bayesian inference in the visual cortex. Journal of the Optical Society of America A, (4(7))143441444 2003.
Hezheng Lin, Xing Cheng, Xiangyu Wu, and Dong Shen. Cat: Cross attention in vision transformer. In IEEE International Conference on Multimedia and Expo, 2022.
Pengpeng Liu, Michael R. Lyu, Irwin King, and Jia Xu. Selflow: Self-supervised learning of optical flow. CVPR, 2019.
Margaret Livingstone, Mechanisms of direction selectivity in macaque vl. Neuron, 20(3):509-526, 1998.
William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsupervised learning. In International Conference on Learning Representations. 2017.
Kunming Luo, Chuan Wang, Shuaicheng Liu, Haoqiang Fan, Jue Wang, and Jian Sun. Upflow: Upsampling pyramid for unsupervised optical flow learning. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 1045-1054, 2021.
Nikolaus Mayer, Eddy Ilg, Philip Häusser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, 2016.
Willi Menapace, Stéphane Lathuilière, Aliaksandr Siarohin, Christian Theobalt, Sergey Tulyakov, Vladislav Golyanik, and Elisa Ricci. Playable environments: Video manipulation in space and time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1188-3593, 2022.
Chris Miall and Daniel M Wolpert. Forward models for physiological motor control. Neural networks, 1 1996.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arX 2023.
13

---

Published as a conference paper at ICLR 2026
Jack Parker-Holder, Philipp Ball, Jake Bruce, Vibhavari Dasagi, Kristian Holsheimer, Chris- tos Kaplanis, Alexandre Moufarek. Guy Scully, Jeremy Shar, Jimmy Shi, Stephen Spencer, Jessica Yung, Michael Dennis, Sultan Kenjeyev, Shangbang Long, Vlad Mnih, Harris Chan, Maxime Gazeau, Bonnie Li, Fabio Pardo, Luyu Wang. Lei Zhang, Frederic Besse, Tim Harley, Anna Mitenkova, Jane Wang. Jeff Clune, Demis Hassabis, Raia Hadsell, Adrian Bolton, Satinder Singh, and Tim Rocktäschel. Genie 2: A large-scale foun- dation world model. 2024. URL https://deepmindd wow.eplliiinggoglle.....o google e /wd//issooooo scover/blog/ gerle-2-a-Large--ee woweeeooooatattooooooo-----
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings the Vision and Pattern Recognition, pp. 2536-2544 2016.
the IEEE/CVF conference on Computer
Zhiwu Qing, Shiwei Zhang, Ziyuan Huang, Yi Xu, Xiang Wang, Mingqian Tang, Changxin Gao, Rong Jin, and Nong Sang. Learning from untrimmed videos: Self-supervised video representation learning with hierarchical consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In IEEE International Conference on Computer Vision, 2021.
Rajesh Rao and Dana Ballard. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nature neuroscience, 2.79-87, 02 1999. doi: 10.1038/4580.
Rajesh Rao and Terrence J Sejnowski. Predictive sequence learning in recurrent neocortical circuits. In S. Solla, Leen, and K. Müller (eds.), Advances Neural Information Processing Systems, vol- ume 12. MIT Press, 1999. URL httptttspppooccciiiggtsts////ees newrips. cc/ paper ccppaper _iles/ile//leeelll
Dominik Schmidt and Minqi Jiang. Learning to act without actions. In International Conference on Learning Representations, 2024.
Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin,
R Devon Hjelm, Philip Bachman, and Aaron Courville. Pretraining representations for data- efficient reinforcement learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https //openreview pet/foruue id-idd--ss XpSAviinMa
M Shidara, K Kawano, H Gomi, and Kawato. Inverse-dynamics model eye movement control by purkinje cells in the cerebellum. Nature, 365(6441)::0-533 1993.
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. Advances in Neural Information Processing Systems, 2014.
William Softky. Unsupervised pixel-prediction. Advances in Neural Information Processing Systems, 1995.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using Istms. In International Conference Machine Learning, pp. 843-852. PMLR, 2015.
Austin Stone, Daniel Maurer, Alper Ayvaci, Anelia Angelova, and Rico Jonschkowski. Smurf: Self- teaching multi-frame unsupervised raft with full-image warping. In Proceedings of the IEFF/CVF conference on Computer Vision and Pattern Recognition, pp. 3887-3896, 2021.
Christopher Summerfield, Tobias Egner, Matthew Greene, Etienne Koechlin, Jennifer Mangels, and Joy Hirsch. Predictive codes for forthcoming perception the frontal cortex. Science, 31 4(5003): 1311-1314, 2006.
Deqing Sun, Xiaodong Yang, Ming- Yu Liu, and Jan Kautz. PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018
14

---

Published as a conference paper at ICLR 2026
Yihong Sun, Hao Zhou, Liangzhe Yuan, Jennifer J. Sun, Yandong Li, Xuhui Jia, Hartwig Adam, Bharath Hariharan, Long Zhao, and Ting Liu. Video creation by demonstration, 2024. URL
aittps://ariis org/ /tt.////1.. 09551
Zachary Teed and Jia Deng. Raft: Recurrent all- pll-pirrs field transforms for optical flow. In European Conference Computer Vision, 2020.
Zhan Tong, Yibing Song. Jue Wang, and Limin Wang. VideoMAE: Masked autoencoders are data- efficient learners for self-supervised video pre-training. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Harri Valpola. From neural pea to deep unsupervised learning. Analysis and Learning Machines. 2015.
leamiigg In Advances in Independent Component
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Shashanka Venkataramanan, Mamshad Nayeem Rizve, João Carreira, Yuki M Asano, and Yannis Avrithis. Is imagenet worth 1 video? learning strong image encoders from 1 long unlabelled video. In International Conference on Learning Representations, 2024.
Ruben Villegas, Dumitru Erhan, Honglak Lee, al. Hierarchical long-term video prediction without supervision. In International Conference on Machine Learning, 2018.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie. Yoshua Bengio, Pitrre-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in deep network with a local denoising criterion. Journal of machine learning research, 11(12), 2010.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Anticipating visual representations from unlabeled video. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2016.
Alex N. Wang, Chris Hoang, Yuwen Xiong, Yann LeCun, and Mengye Ren. Poodle: Pooled and dense self-supervised learning from naturalistic videos. In International Conference on Learning Representations, 2025.
Wenshan Wang, Delong Zhu, Xiangwei Wang. Yaoyu Hu, Yuheng Qiu, Chen Wang, Yafei Hu, Ashish Kapoor, and Sebastian Scherer. Tartanair: dataset to push the limits of visual slam. JEEE/RSJ International Conference Intelligent Robots and Systems, 2020.
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In IEEE International Conference on Computer Vision, 2015.
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training, In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2021.
Chen Wei, Haoqi Fan, Saining Xie, Chao- Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2022.
Philippe Weinzaepfel, Vincent Leroy, Thomas Lucas, Romain Brégier, Yohann Cabon, Vaibhav Arora, Leonid Antsfeld, Boris Chidlovskii, Gabriela Csurka, and Revaud Jérôme. Croco: Self-supervised pre-training for 3d vision tasks by cross-view completion. In NeurlPS, 2022.
Philippe Weinzaepfel, Thomas Lucas, Vincent Leroy, Yohann Cabon, Vaibhav Arora, Romain Brégier, Gabriela Csurka, Leonid Antsfeld, Boris Chidlovskii, and Jérôme Revaud. CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flow. In ICCV, 2023.
Laurenz Wiskott and Terrence Sejnowski. Slow feature analysis: Unsupervised learning of
invariances. Neural computation, 14(4):715-770, 2002.
15

---

Published as a conference paper at ICLR 2026
Daniel M Wolpert, Zoubin Ghahramani, and Michael I Jordan. An internal model for sensorimotor integration. Science, 269(5232):1880-115 1995.
Daniel M Wolpert, R Chris Miall, and Mitsuo Kawato. Internal models in the cerebellum. Trends in cognitive sciences, 2(9):338-344 3478 1998.
Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Yuwen Xiong, Mengye Ren, Wenyuan Zeng, and Raquel Urtasun. Self-supervised representation learning from flow equivariance. In IEEE International Conference on Computer Vision, 2021.
Jiarui Xu and Xiaolong Wang. Rethinking self-supervised correspondence learning: A video frame-level similarity perspective. In IEEE International Conference on Computer Vision, pp. 10075-10085, 2021.
Sherry Yang, Yilun Du, Seyed Kamyar Seyed Ghasemipour, Jonathan Tompson, Leslie Pack Kael- bling, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. In International Conference on Learning Representations, 2024.
Seonghyeon Ye, Joel Jang, Byeongguk Jeon, Se June Joo, Jianwei Yang, Baolin Peng, Ajay Man- dlekar, Reuben Tan, Yu- Wei Chao, Bill Yuchen Lin, Lars Liden, Kimin Lee, Jianfeng Gao, Luke Zettlemoyer, Dieter Fox, and Minjoon Seo. Latent action pretraining from videos. In International Conference Learning Representations, 2025. URL htl.htts:: /ttp:///ppeevviieeeee www.net.ee... nwt//frrtrm forum? id Bd-VYOe2eB0ehi
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Shaofeng Zhang. Feng Zhu, Rui Zhao, and Junchi Yan. Patch-level contrasting without patch correspondence for accurate and dense contrastive representation learning. International Conference Learning Representations, 2023.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE/CVF conference Computer Vision and Pattern Recognition, 2017.
Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. Dino-wm: World models on pre-trained visual features enable zero-shot planning, 2024. URL http::://arx..voorr//a////
04983
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. In International Conference Learning Representations, 2021.
16

---

Published as a conference paper at ICLR 2026
APPENDIX
A ADDITIONAL RESULTS
A.I LONGER PRETRAINING
We provide additional experiments WT-Venice pretraining below. Table shows that Midway Network's downstream performance continues to improve with longer pretraining.
Table 5: Semantic segmentation and optical flow evaluations for additional experiments on WT- Venice 224 x 224 resolution pretraining. Sem. Seg. and optical flow are conducted with finetuning.
WTT-Sem Sem. Seg. UperNet
ADE20K Sem. Seg.
Optical Flow
UperNet
FlyingThingg
MPI-Sintel
Method
Arch
Ep.
tmloU
1Acc
tmloU
fAcc
LEPE (c)
JEPE (n)
JEPE (c)
JEPE (0)
Midway
VIT-$
100
13.1
85.4
33.4
76.9
7.7
7.4
5.2
6.6
Midway
VIT-S
300
14.8
86.5
36.9
78.2
7.3
6.9
A.2 ADE20K LINEAR READOUT
Table 6 shows evaluation results for ADE20K semantic segmentation linear readout. Performance trends follow the UperNet finetuning results in Table Again, Midway Network is competitive with baselines, PooDLe and DoRA, and furthermore, does not rely on an external supervised optical flow network and can jointly learn representations for motion understanding.
Table 6: ADE20K semantic segmentation linear readout evaluations for WT Venice 224 224 resolution pretraining. PooDDee 512 1024 resolution pretraining from their original table (Wang
et al. 2025
Method
Arch
Ep.
Tmlol
TAcc
PooDLee
Wang el
2025
R50
14.6
59.0
MAE He
VIT-S
100
7.4
55.1
er
DINO
Carin et di 202
VIT-S
100
6.9
48.2
2022
DoRA Nenkalarummaann el 2024
VIT-S
100
14.1
63.5
Midway
VIT-S
100
12.1
61.3
A.3 OPTICAL FLOW FROZEN READOUT
Table Prooides evaluation results for optical flow linear readout. Here, the backbone parameters of each method frozen and only the DPT (Ranftl al. (021)) head is trained using the same data as the optical flow finetuning experiments. Midway Network's learned representations again achieve strong performance relative to the baselines.
Table 7: Optical flow frozen readout evaluations for BDD100K 224 x 224 resolution pretraining.
FlyingThings
MPI-Sintel
Method
Arch
JEPE (e)
JEPE
JEPE (c)
JEPE (f)
iBOT
VIT.S
DINO Lrron d 202
VIT-S
300
19.0
17.5
14.0
13.5
VideoMAE Tony
VIT.S
CroCo v2 Meinzapple delle e 2023
VIT-S
300
39.2
39.2
24.0
23.9
V
Midway (enc. only)
VIT-S
300
18.8
17.0
12.5
11.7
Midway
VIT.S
DINO Caron 2021
ViT-B
300
19.0
17.4
14.2
13.2
CroCo V2 Weinze pte of al 2023
VIT-B
300
39.2
39.2
24.0
24.1
Midway
VIT-B
21.7
20.2
13.7
12.9
B IMPLEMENTATION DETAILS
In this section, provide additional details the implementation of Midway Network, the pretraining and evaluation setups, and compute resources used for our experiments. The experiments were implemented using the FyTorch framework.
17

---

Published as a conference paper at ICLR 2026
B.1 ARCHITECTURE
The ViT encoders have 12 feature levels, and we perform the dense forward prediction objective levels 3, 6, and The midway path infers motion latents with feature inputs level 12 for the level 9 objective and refines them as described in Section f for levels 6 and 3. The midway inverse dynamies model at each level a 4-block transformer with feature dimension 192, with linear projectors to map from and to the original feature dimension. We use 10 learnable tokens for the motion latents. The backward layers are 1-block cross-attention transformers with feature dimension equal to the dimension of the underlying ViT encoder, i.e. 384 for ViT-S and 768 for ViT-B. The forward dynamics model at each level is a 4-block transformer with feature dimension equal to the underlying encoder dimension well. The learnable gating units are placed at all but the first block. Each gating unit is a multi-layer perceptron with hidden layer of same dimension as the encoder, GELU activation, and a final sigmoid activation. To bias the initial gating weights towards i.c. the original felgh-weighted residual connection, we add bias of 4 to the input of the sigmoid.
We follow DINO (Caron al. 202T) for implementation of the joint-embedding invariance objective, using the same projection heads, centering and sharpening operations, and temperature schedules as described in their paper. Given that we have 2 paired video frames as input, we can sample global crops and 8 local crops from each frame and compute the loss between crops across frames to leverage the natural temporal motion augmentation. The loss also symmetrical, where we compute the loss for the original frame ordering as well as the reversed ordering. We utilize this setup for the DINO baseline as well for fair comparison. The final loss is an equal-weighted sum of the dense forward prediction loss, averaged over the feature levels, and the joint-embedding invariance loss:
C
Clyg + Line-
(3)
L 1=1
B.2 PRETRAINING
We outline the hyperparameters used for pretraining in Table 8 The hyperparameters largely follow the DINO (Caron et al. 202T) training recipe. We use the same hyperparameters for BDD100K and Walking Tours pretraining. For BDD100K we utilize repeat sampling following MAE-st (Feichten hofer et al. 2022), which samples = 5 frames each time a video is seen for faster data loading. Therefore, treat each pass through the dataset R epochs.
Table 8: Hyperparameters used for full Midway Network experiments.
Hyperparameter
Value
Learning rate
10 4
Learning rate warmup
10 epochs
Learning rate schedule
cosine
Batch size
200
Weight decay
0.04
Weight decay end
0.4
Optimizer
Adam
Betas
(0.9, 0.999)
Gradient clip norm
3.0
Drop path rate
0.1
Use FP16
True
B.3 BASELINES
We use the official implementations to pretrain the basclines BDD100K and Walking Tours. We use the released checkpoints for DINO, DoRA, and PooDLe on Walking Tours; semantic segmentation finetuning results for MAE, DINO, DoRA, and PooDLe also from the original table in PooDLe (Wang et al. 2202)).
B.4 EVALUATION
For the semantic segmentation tasks, we follow the ViT-based setup described in PooDLe Waan et al. 2025), based on the mnseqmentation (Contributors) 2020) codebase. The linear and U UperNet readout setups for BDD100K and CityScapes were originally from FlowE (Xiong et al. 202 ; the
UperNet finetuning setup for ADE20K was originally from iBOT (Zhou al. 2021
18

---

Published as a conference paper at ICLR 2026
For the optical flow tasks, we follow the finetuning evaluation setup described in CroCo v2 (Weinza eefel et al. 2023) and use their official implementation. Our main results follow CroCo v2 s setup for Table from their paper; ablation studies follow their setup for their Table 11 ("smaller training data") to match the settings of other optical flow methods. The primary difference that replace CroCo v2's decoder with Midway Network's midway inverse dynamics and forward dynamics models. We use the following as input to the DPT (Ranftl et al. (022)) that outputs the optical flow predictions: dense tokens of encoder feature level 12, dense spatial tokens corresponding to the target frame processed by the midway model at the highest level of the dense objective, dense token prediction of the forward model at the highest objective level, and dense token prediction of the forward model at the lowest objective level. For reference, the midway model processes the dense spatial tokens from the source and target frames alongside the motion latents. We use this architecture for all other baselines besides CroCo v2 with randomly initialized weights, they do not have binocular components.
B.5 COMPUTE AND TRAINING COSTS
Table provides a comparison on training cost in FLOPSs per single training example and model size in parameters for Midway Network and the baseline methods. Midway Network uses less than half of the FLOPs of prior video data-based learning methods, PooDLe and DoRA. The dynamics networks of Midway Network iterative refinement operations used by prior flow methods such as RAFT (Teed & Deng, (020) and FlowFormer (Huang et al. 2022 Table Io shows the compute resources used for the experiments.
parameters to capture motion information, but avoid costly
Table 9: Training cost (GLOPs per example) and model size (millions of parameters) of Midway Network and baseline methods.
Method
Training cost (GFL.Ps)
Parameters (millions)
Midway Network
21.7 (encoder), 36.6 (dynamics networks)
PooDLe
202.3
23.5 (encoder), 12.1 (spatial decoder)
DoRA
202.1
21.7 (encoder)
CroCo v2
6.9
21.7 (encoder), 7.2 (decoder)
DynaMo
68.9
21.7 (encoder), 13.0 (dynamics networks)
VideoMAE
11.6
22.0 (encoder). 2.0 (decoder)
iBOT
35.3
21.7 (encoder)
DINO
50.4
21.7 (encoder)
Table 10: Compute resources and time used for Midway Network experiments.
Experiment
Epochs
Resources
Time
BDD 10D1 VIT-S pretraining
300
2 A100 GPUs
66 hours
BDD100K VIT-B pretraining
300
8 RTX A6000 GPUs
27 hours
BDD100K VIT-S ablations
100
2 A10 GPUs
24 hours
Walking Tours ViT-S pretraining
100
4 RTX A6000 GPUs
29 hours
19

---

Published as a conference paper at ICLR 2026
C MORE VISUALIZATIONS
We show additional visualizations of predictions from the semantic segmentation evaluations in Figure 9 for CityScapes, Figure 9 for WT-Sem, and Figure I1 for ADE20K, and optical flow evaluations for models pretrained on Walking Tours in Figure T2 We also provide visualizations of optical flow evaluations comparing Midway Network and CroCo v2 for different model sizes in Figure
Image
Midway
DINO
CroDov2
DoRA
DynaMo
Figure 9: Visualization of CityScape semantic segmentation UperNet readout. Midway Network generates cleaner boundaries, particularly for the crossing pedestrians.
Imago
Ground Truth
Midway Network
PooDLe
DINO
CreCo v2
DoRA
MAE
Figure 10: Visualization of WT-Sem semantic segmentation UperNet finetuning. Midway Network is able to produce reasonable segmentation masks, even in cluttered scenes.
Image
Ground Truth
Midway Network
CroCo v2
Image
Ground Truth
Midway Network
CroCo v2
Figure 11: Visualization of ADE20K semantic segmentation UperNet finetuning, Midway Network generates more accurate segmentation masks compared to CroCo v2.
We also include more examples of the forwarded feature perturbation analysis of Midway Network's learned dynamics, with heatmaps in Figure 14] and high-level tracking Figure
20

---

Published as a conference paper at ICLR 2026
Source
Ta
Ground Truth
Midway Network
CreCo v2
DING
Figure 12: Visualization of FlyingThings and MPI-Sintel optical flow evaluations after finetuning for models pretrained on WT Venice, Midway Network is able to generate more accurate optical flow predictions compared CroCo v2 and DINO.
Target
Figure 13: Visualization of FhyingTTings and MPI-Sintel optical flow evaluations after finetuning for Midway Network and CroCo v2 pretrained on BDD for varying model sizes. Moving from ViT-Small ViT-Base primarily provides fine-grained improvements in optical flow estimation.
Source
Terget
Heetmap
Source
Terget
Heatmap
Heatmaps for forwarded feature perturbation in Source (green squares); shown in Target the same location to highlight motion. The learned dynamics can capture high-level correspondence, such the right taillight of the black car (bottom left).
Figure 14:
21

---

Published as a conference paper at ICLR 2026
Midway forward perturbation
DINO-IN1K similarity
Midway forwerd perturbation
DINO-IN1K feature similarity
Figure 15: High-level tracking using forwarded feature perturbation and/or feature similarity. Midway Network able to track high-level regions through motion transformations. such as the back of the toddler (top row, pink square).
22


Reviews:
[{"review_id": "vRu0VyhKFc", "reviewer": "Reviewer_RAZm", "summary": "This paper introduces a new self-supervised learning framework that jointly learns object recognition and motion understanding from natural videos. It extends latent dynamics modeling to the video representation learning domain. The model combines an inverse-dynamics midway path, a dense forward prediction objective, and a hierarchical refinement structure to learn both semantic and motion features from unlabeled data.", "questions": "1. In Table 1, ViT-B only modestly improves flow over ViT-S and CroCo v2 still wins. What’s the failure mode for Midway at that scale?\n2.The forwarded feature perturbation analysis is cool, could it be extended into a quantitative metric?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper tackles an important and underexplored problem, learning both recognition and motion representations from unlabeled videos\n2. It’s conceptually well-grounded, drawing inspiration from neuroscience to motivate the overall framework.\n3. The architecture itself is Innovative. The midway path for motion latent inference feels like a fresh and thoughtful design choice.", "weaknesses": "1. It’s hard to disentangle architecture gains from data scale and model capacity\n2. While the combination is novel, many components are incremental extensions of ideas from CroCo, DynaMo, or PooDLe.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xpJfmTLZdD", "reviewer": "Reviewer_BVMm", "summary": "This paper presents a novel self-supervised learning architecture for learning visual representations that jointly capture object recognition and motion understanding from video inputs. The proposed Midway Network achieves this by approximating a hierarchical latent embedding to model motion signals, thereby removing the need for an external motion predictor. Extensive experiments on object segmentation and optical flow tasks demonstrate the effectiveness of the approach.", "questions": "1. Figure 1 is referenced in the text but not included in the main manuscript.\n\n2. Were all the additional components used during inference, or only in training?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. By integrating latent optical flow estimation within a self-supervised framework, the proposed method enables simultaneous learning of motion information and content features through a single encoder. This design allows motion cues to be naturally incorporated into semantic representations.\n\n2. The joint learning of motion and semantic features leads to consistent improvements in both semantic segmentation and optical flow performance.", "weaknesses": "1. While the paper contains several strong ideas, the overall framework is somewhat difficult to follow at first. It took multiple readings to fully grasp how the individual components contribute to the overall system. This could be improved by adding a concise, high-level overview of the architecture—perhaps at the beginning of Section 3—along with a summarizing figure that highlights the key components and their interactions.\n\n2. Since one of the main contributions is the joint learning of motion and semantic representations from video, it would strengthen the paper to include a comparison with MC-JEPA to better contextualize performance gains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "egg7UE8fxN", "reviewer": "Reviewer_9dmU", "summary": "This paper introduces Midway Network, a self-supervised learning framework that jointly learns object recognition and motion understanding from natural videos through latent dynamics modeling. The key idea is to integrate inverse and forward dynamics modules within a hierarchical architecture, where motion latents describe transformations between consecutive frames and are refined top-down across feature levels. Experiments show that Midway Network achieves strong performances in both semantic segmentation and optical flow tasks.", "questions": "Q1. Why are optical flow results for PooDLe not reported in Tables 1–2?\nWas there a fundamental limitation in adapting its architecture for flow prediction, or were these experiments omitted for other reasons?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "S1. Unified SSL framework for both object recognition and motion understanding: The paper presents a coherent approach that bridges predictive coding theory with modern self-supervised Transformers, achieving representation learning for both semantics and motion within a single framework.\n\nS2. In-depth analysis: The analysis of learned motion latents in Section 4.4 (forwarded feature perturbation) is technically sound and creative. It provides interpretable evidence that the model captures spatial correspondences.\n\nS3. Diverse Evaluation Domains: The proposed method is evaluated on several tasks and benchmarks.", "weaknesses": "W1. Incremental novelty: the proposed inverse + forward dynamics framework for learning latent motions follows a well-established approach used in latent world models and video prediction literature [a, b, c]. Moreover, the hierarchical refinement design is conceptually similar to the Spatial Dynamics Module (SDM) used in PooDLe..\n\nW2. No guarantee of motion: The paper does not theoretically ensure that the learned latent $m_t$ encodes motion rather than directly leaking target feature information ($𝑧_{𝑡+1}$). Although empirical analyses (optical flow results, Sec. 4.4 perturbation study) suggest motion-like behavior, the model could still “hack” the objective by embedding target information in $𝑚$ making the representation less interpretable.\n\nW3. Limited quantitative analysis of motion learning:\nSection 4.4 provides compelling qualitative examples, but the conclusions would be stronger with quantitative validation. For instance, comparing perturbation-based correspondence against pseudo ground-truth from an off-the-shelf tracker could more rigorously substantiate the claimed motion alignment.\n\nW4. Marginal effect of design components: As seen in Table 3 (e.g., 2→7, 3→8, 4→9 comparisons), adding design components (e.g., backward layers, gating) only yields small incremental gains, which questions their relative contribution to the overall improvement.\n\nW5. (Minor) Lack of planning or world-model evaluation: Despite the discussion of potential applications to planning and world modeling, no experiments demonstrate this capability, slightly limiting the perceived impact of the proposed approach. \n\n[a] Bruce et al., “Genie: Generative Interactive Environments,” ICML, 2024.\\\n[b] Ye et al., “Latent Action Pretraining from Videos,” ICLR, 2025.\\\n[c] Gao et al., “AdaWorld: Learning Adaptable World Models with Latent Actions,” ICML, 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J5um38HQkK", "reviewer": "Reviewer_m9iU", "summary": "The authors, in this work, introduce a new self-supervised learning (SSL) architecture called Midway Network.\n\n- They extend the concept of latent dynamics modeling, used in control and decision-making, to the visual perception domain.\n\n- Midway network is a self-supervised video model designed to learn both: Object recognition (semantic understanding) and Motion understanding (how things move over time) purely from natural, unlabeled videos — without relying on curated image datasets or external motion labels like optical flow.\n\n- Midway Network outperforms or matches the best SSL baselines (like DINO, DoRA, CroCo v2) on both tasks: Semantic segmentation and Optical flow estimation.", "questions": "Please see the discussion in weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "### 1. Conceptual Soundness\n- The authors ground their approach in predictive coding and latent dynamics modeling, both well-established ideas in neuroscience and machine learning.\n\n- The theoretical motivation, that perception arises from predicting sensory changes is sound.\n\n- The authors identify a clear gap: self-supervised models typically learn either recognition (DINO, iBOT) or motion (CroCo, FlowE), not both.\n\n### 2. Architectural Novelty\n- The architecture is a careful combination of: 1. Inverse dynamics (midway path) to infer motion, 2. Forward prediction (dense feature-level) to learn temporal coherence, and 3. Hierarchical refinement (inspired by optical flow networks e.g., PWC-Net, RAFT).\n\n- The design is justified by analogies to biological systems and prior SSL hierarchies (e.g., Ladder Networks, DINO).\n\n- The inclusion of gating units in transformer residuals is thoughtful as it prevents trivial identity mappings, a known issue in predictive models.\n\n### 3. Experimental Soundness\n- Comprehensive evaluations across recognition (semantic segmentation) and motion (optical flow).\n\n- Extensive comparison against relevant baselines: DINO, DoRA, PooDLe, CroCo v2, DynaMo, etc.\n\n- Consistent use of natural video datasets (BDD100K and Walking Tours) supports the claim of \"learning from natural data only.\"\n\n- Sufficient ablation studies test each architectural component's impact on both semantic and motion metrics.\n\n### 4. Interpretation Soundness\n- The \"forwarded feature perturbation\" method is a novel and interpretable way to visualize what the dynamics model learns.\n\n- It qualitatively demonstrates non-trivial motion correspondence — a rare strength in SSL papers.", "weaknesses": "### 1. Conceptual Limitations\n- While conceptually coherent, \"latent dynamics\" is borrowed from control/planning literature and adapted here somewhat heuristically — the paper lacks a strong theoretical derivation connecting latent dynamics to semantic learning in videos.\n\n- The link between motion prediction and semantic invariance is intuitive but not formally analyzed.\n\n### 2. Architectural Limitations\n- The midway and backward paths add significant complexity; it’s unclear if all components are essential (though the ablation studies help).\n\n- The architecture might overfit to short temporal correlations (1-second gaps between frames) and may not generalize to longer-horizon motion.\n\n### 3. Experimental Limitations\n- Only two pretraining sources (BDD and WT-Venice). That’s small compared to large-scale pretraining regimes (e.g., Kinetics-700, Ego4D). Therefore, it is hard to tell if results scale to diverse or indoor/outdoor mixed environments.\n\n- The authors do not measure performance over multiple time steps (e.g., predicting motion 10 frames ahead).\n\n- Some of the baselines (e.g., PooDLe) use higher resolution or external flow networks, making cross-comparison imperfect.\n\n- Few metrics report standard deviation or multiple seeds.\n\n### 4. Interpretation Limitations:\n- The forward feature perturbation analysis is qualitative. There’s no quantitative measure of how well it aligns with ground-truth motion.\n\n- The theoretical justification for why this analysis reflects \"high-level correspondence\" is intuitive but not formalized.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)