DynaMind: Reasoning over Abstract Video Dynamics for Embodied Decision-Making
Authors: Ziru Wang
Mengmeng Wang
Jade Dai
Teli Ma
Guo-Jun Qi
Yong Liu
Guang Dai
Jingdong Wang
Venue: ICML 2025 poster (2025)
Keywords: decision-making; language commands; manipulation;
Primary area: applications->robotics
TL;DR: We propose DynaMind, a framework that enhances decision-making through dynamic reasoning, bridging the gap between modalities by abstracting video content into dynamic representations.

Abstract: Integrating natural language instructions and visual perception with decision-making is a critical challenge for embodied agents. Existing methods often struggle to balance the conciseness of language commands with the richness of video content. To bridge the gap between modalities, we propose extracting key spatiotemporal patterns from video that capture visual saliency and temporal evolution, referred to as dynamic representation. Building on this, we introduce DynaMind, a framework that enhances decision-making through dynamic reasoning. Specifically, we design an adaptive FrameScorer to evaluate video frames based on semantic consistency and visual saliency, assigning each frame an importance score. These scores are used to filter redundant video content and synthesize compact dynamic representations. Leveraging these representations, we predict critical future dynamics and apply a dynamic-guided policy to generate coherent and context-aware actions. Extensive results demonstrate that DynaMind significantly outperforms the baselines across several simulation benchmarks and real-world scenarios.

DynaMind: Reasoning over Abstract Video Dynamics for Embodied Decision-Making
Ziru Wang
Mengmeng Wang? 21 Jade Dai Teli Ma Guo-Jun Qi Yong Liu 5
Guang Dai Jingdong W
Abstract
Language: the light on, pull the microwave door, and open the cabinet. Video
Video
Integrating natural language instructions and vi- sual perception with decision-making is crit- ical challenge for embodied agents. Existing methods often struggle to balance the concise- ness of language commands with the richness of video content. To bridge the gap between modal- ities, we propose extracting key spatiotemporal patterns from video that capture visual saliency and temporal evolution, referred as dynamic representation. Building this, we introduce DynaMind, framework that enhances decision- making through dynamic reasoning. Specifically, we design an adaptive FrameScorer to evaluate video frames based on semantic consistency and visual saliency, assigning each frame an impor- tance score. These scores used to filter dundant video content and synthesize compact dynamic representations. Leveraging these rep- resentations, we predict critical future dynamics and apply a dynamic-guided policy to generate coherent and context-aware actions. Extensive results demonstrate that DynaMind significantly outperforms the baselines across several simula- tion benchmarks and real-world scenarios.
(a) Different videos achieve the same language instruction
nurn light on, pull the microwave door, and open the cabinet
2
h
h
Previous image sequences
Proposed dynamic sequences
(b) Previous vs. proposed method
Figure I. The mismatch between the simplicity and singularity of language and the diversity and complexity of vidcos. (b) In- stead of directly mapping language to video content, DynaMind bridges this gap by abstracting video into dynamic representations, enabling high-level dynamic reasoning miliiinnmmaingg.
visual perception, and action planning, along with the flexi- bility to adapt to complex environments.
1. Introduction
The Vision-Language-AAcio approach to this challenge by combining visual, linguistic, and action modalities. It allows agents to connect language instructions with visual contexts and generate task-specific actions (I.i et al., 2023; Wu et al., 2024; Chen et al., 2024). Furthermore, advancements in language-to-videe have opened new development opportunities. These models generate image sequences visualizing detailed occurrences, using generated frames as intermediate states to guide low- level action control (Du et al., 2024; Liang et al., 2024).
framework offers a promising
Natural language instructions provide efficient interface for human-computer interaction, enabling embodied agents to make sequential decisions based on brief language de- scriptions (Zhou et al., 2024; Liang et al., 2024). This process requires the integration of language understanding,
models
'SGIT AI Lab, State Grid Corporration of China, "Zhejiang University of Technology. 'The Hong Kong University of Science and Technology, Guangzhou, *Westlake University, 'Zhejiang Uni- versity, Baidu. Correspondence to: Mengmeng Wang <mengme- wang mong@gmmailccom>
However, these methods often rely on rigid mappings be- tween language instructions and video content, failing to bridge the gap between the abstract simplicity of language and the detailed specificity of video. Generally, single
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

DynaMind: Reasoning over Abstract Video Dynamics for Embodied Decision-Making
language instruction can correspond multiple videos (see Figure la), revealing limitations in handling vague seman- ties and hindering generalization for real-world tasks. To address these, prior research has focused on enhancing language information to reduce underspecification in com- mands. One approach extends semantic representations to create a more flexible semantic space (Wang et al., 2024), while another decomposes underspecified instructions into finer-grained semantic skills, using these details to predict future frames or actions (Garg et al., 2022; Liang et al., 2024). Although these methods enhance the precision of instruction interpretation, they remain limited by the in- herent constraints of semantic expression. Furthermore, approaches relying on predefined semantic skill libraries constrain adaptability and generalization to novel tasks or complex scenarios.
In summary, our study makes three key contributions: 1) We introduce the DynaMind framework, which abstracts video content into dynamic representations and aids decision- making through dynamic reasoning, thus reducing the mis- match between language and video. 2) We design a dynamic abstraction module with an adaptive FrameScorer to con- vert video into compact, expressive dynamic sequences, followed by a generation module to generate future dynam- ies and decision module predicts appropriate actions. 3) We empirically demonstrate DynaMind's effectiveness and generalization capabilities across various simulation exper- iments, provide visualizations of abstract video dynamics, and confirm its effectiveness in real-world tasks.
2. Related Work
Embodied Control under Language Instruction. Pre- dicting robotic actions from language instructions is key focus in embodied control (Li et al., 2023; Zhou et al., 2024; Liang et al., 2024; Wu et al., 2024; Chen et al., 2024). One prominent class of approaches is the Vision-Language- Action framework, which aligns visual, linguistic, and ac- tion modalities (Brohan al., 2022: Li et al., 2023; Wu et al., 2024; Chen et al., 2024). They typically use a pretraining- finetuning strategy, where vision-language models are first pretrained on large video datasets and then fine-tuned with task-specific video-action data. Another class of approaches aims to improve cross- cross-modal understanding by explicitly aligning visual and linguistic information (Yao et al., 2022: Chen et al., 2024; Mazzaglia et al., 2024; Kou et al., 2024; Ma et al., 2024; Wu al., 2024), typically using the multi- modal contrastive learning objective. Recently, advances in generative models have introduced new opportunities in this field. These methods generate video based on language instructions, using the generation as intermediate goals to guide low-level action decision (Du et al., 2024; Ko et al., 2024; Zhou al., 2024: Luo & Du, 2024; Tian et al., 2024). Specifically, some studies employ video diffusion models to predict future visual frames and infer actions through inverse dynamics models. Unlike previous methods, we bridge the gap between different modalities by abstracting dynamic representations from video, generating future dy- namics, and using these as conditions to predict actions. In contrast to the getraining-finetunin paradigm, we employs end-to-end training, reducing training complexity.
In response, we introduce a novel perspective: rather than refining language information, the attention can be di- rected toward abstracting video content (see Figure lb right). Building on this perspective, we propose the Dyna- Mind framework. Guided by language instructions, Dyna- Mind abstracts videos into high-level spatiotemporal fea- tures that capture visual saliency and temporal evolution, which we called dynamic representations. Unlike previous frame-by-fe by-frame methods that generate continuous image sequences, DynaMind focuses on dynamic reasoning. This shift allows it to better capture essential content of videos based on language instructions, enhancing cross-modal un- derstanding and decision-makee making. DynaMind is structured around three tightly integrated functional modules:
Video Dynamic Abstraction. This module converts the video into a sequence of high-level dynamic representa- tions using an adaptive FrameScorer. The FrameScorer evaluates the significance of each frame based on se- mantic consistency and visual saliency. By prioritizing critical frames and minimizing redundancy, this module emphasizes essential spatiotemporal features, producing a compact dynamic representation sequence that serves as a reliable foundation for subsequent processing.
Video Dynamic Reasoning. Using the abstract dynamic representations, this module models temporal evolution to predict future dynamics. captures global temporal dependencies, generating forward-looking predictions of future developments. It is particularly beneficial for long- horizon tasks that require complex reasoning.
Language and Video Grounding for Embodied Control. Language instructions are typically concise and abstract, with single instruction often corresponding to multiple videos that convey detailed and diverse information (Gabeur et al., 2020; Fang et al., 2023). This disparity in both the quantity and nature poses significant challenges for current research. To address this, recent studies have focused on enriching language representations to better integrate the
Dynamie-Guided Action Decision. Leveraging pre- dicted future dynamics and historical context, this module enables agents to execute context-aware actions through dynamic-guided policy. In contrast to methods that focus on udjacent-frame motion (Liang et al., 2024; Zhou et al., 2024), it models comprehensive relationships, ensuring more coherent Decision-making
2

---

DynaMind: Reasoning over Abstract Video Dynamics for Embodied Decision-Making
Muitimodal Encoder
Video Dynamic Abstraction (Sec. 3.1)
Video Dynamic Reasoning (Sec. 3.2)
2201
dynanic feature
image frame feature
Temporal Transformer
G
imarge onoodar
FrameScorer
video frama
hg
L reasonnng
Dynamic-Guided Action Decision (Sec. 3.3)
light and Then sid the drower
cc-c+1
langusge
languago
Q2
+
consistency
adtion
hryoa
Action Transformer
II
satient feoture
dynamic feature
Leaienny
Laction
actions
Figure 2. Overview framework of DynaMind. DynaMind begins with Video Dynamic Abstraction, which converts video into dynamic representations. Then, Video Dynamic Reasoning predicts future dynamics. Finally, Dynanic-Guided Action Decision uses the predicted dynamics infer the corresponding action sequence. These modules are integrated for shared feature encoding, enabling end-to-end training. During training, future frames and actions serve supervision, while inference relies solely on historical information.
strengths of both video and language modalities, providing more precise guidance to improve decision-making ability (Wang et al., 2024; Garg et al., 2022; Liang et al., 2024). class of methods involves extending semantic represen- tations to encompass a broader and more flexible semantic space (Croitoru et al., 2021; Wang al., 2024). While these enhancements enable agents interpret language in a more concrete and nuanced manner, they remain constrained by the inherent limitations of language representation. An- other widely used class of methods decomposes instructions into finer graingg semantic skills and generates correspond- ing image sequences for each sub-skill (Garg et al., 2022; Ju al., 2024; Liang et al., 2024). However, these high- level skills often lack precision when handling long-horizon dependencies, which leads to error accumulation the complexity of skill combinations increases. Furthermore, they typically rely on predefined skill libraries, limiting the agent's s adaptability to novel tasks or complex scenarios. In contrast, we adopt a video-centric perspective. By ab- stracting video into dynamic representations, we balance the contributions of both video and language modalities, enhancing decision-making ability.
ments for guiding the decision- makiin process. Unlike pre- vious methods that focus frame-by-frand video genera- tion (Zhou et al., 2024; Liang et al., 2024), DynaMind shifts its focus toward reasoning over dynamic representations to address challenges arising from the mismatch between concise language instructions and detailed video content. Specifically, we extract dynamic representations from the image sequence o' and combine them with language in- structions 1' using generative model G to predict future dynamics. To capture temporal dependencies and adapt to environmental changes, predicted dynamic representa- tions are periodically updated. Guided by these updated dynamics, actions are inferred through a policy model II.
To achieve this, DynaMind is structured around three mod- ules, shown in Figure 2. The first module, Video Dy- namic Abstraction ($3.1), converts video inputs into high- level dynamic representations. This module captures es- sential spatiotemporal features and patterns, providing a robust foundation for comprehending video content. The second module, Video Dynamic Reasoning ($3.2), predicts future video dynamics based on historical dynamics. This component models the temporal evolution of the video, en- abling the prediction of key transitions and patterns. Finally, Synamic-Guidedd Action Decision ($3.3) leverages the pre- dicted future dynamics to determine the action sequence.
3. Method
Overview. In decision decision-making tasks, the objective is to predict actions sequentially based on language instructions in order achieve a desired goal. A dataset consisting of multiple sequences is provided, where each sequence
Input Representations. The language input consists of language instruction 1', which is encoded into language embeddings q' using pre-trained DistilBERT model & (Sanh, 2019), which is frozen during training. The video input is represented as an image sequence, where each im- age frame is encoded into features el using a CNN-based encoder Pin, trained from scratch. This encoding reduces
(I', {of af), (oh, ob)..... ... (or, () includes in- structions 1", images of. and corresponding actions at. We introduce the DynaMind framework, which abstracts high- level dynamic representations to forecast future develop-
3

---

DynaMind: Reasoning over Abstract Video Dynamics for Embodied Decision-Making
high-dimensional visual data into a lower-dimensional fea- ture space, making it more manageable for the subsequent processing stage.
Training. We train F(-) by optimizing the entire dynamic sequence, minimizing two components of the loss: the se- mantic consistency loss Lconsisencc and the visual salieney loss Lsaliency The overall objective is:
3.1. Video Dynamic Abstraction
LB Ceonsistency + >Csaliency,
(3)
To abstract a video into high-level dynamic representations, we propose adaptive FrameScorer to evaluate frame im- portance. The importance scores guide the merging of rele- vant frames into compact dynamic representations.
where À is a hyperparameter balancing the two losses, In our experiments, À is set to for simplicity.
The semantic consistency loss ensures that the dynamic sequence is closely matched with the language instruction, maintaining task relevance. It defined as:
Previous studies primarily utilize pre-trained models (Nair et al., 2023; Ma et al., 2023) combined with contrastive learning to extract visual embeddings for each frame. These embeddings then compared to language embeddings from a predefined library of subtasks (Kou et al., 2024), and the resulting similarity scores are used to identify the bound- aries between subtasks in long-horizon videos. However, we focus on identifying key frames that capture significant spatiotemporal patterns within the video, rather than just detecting transitions between subtasks. Additionally, while prior methods often depend on predefined language annota- tions for subtasks or supervision from ground-truth rewards (Liu et al., 2023), these are not available in our study. In- stead, our FrameScorer (-) assigns an importance score wt based on each frame's 's semantic consistency and visual saliency. It uses a two-layer fully connected network with sigmoid activation to capture each frame's 's contribution to the overall video content.
N
Lconsistency
Σ D (H'
(4)
N
where N is the batch size, H represents the dynamic repre- sentation sequence for the i-th trajectory, q' the language embedding, and D(.--) measures cosine similarity.
To avoid representation collapse, we also focus on high- saliency (high-variance) frame features, which provide dis- tinetive visual information for differentiating video content. The visual saliency loss Lsalsnnyy ensures that the correlation between the dynamic representation sequence and the lan- guage embedding surpasses the correlation involving the high- varianar frame sequence. It is defined as:
N Lualieny = N Σ max (0, (V q') - D (H'
Using the FrameScorer, the abstrac- tion process consists of the following steps:
Abstraction Process.
where V represents the high-variance frame sequence, constructed as: V = [vi v /ir/cll. with each v being the feature of the frame with the highest variance in the n-th window. Notably, during training, the subsequent modules receive the dynamic representations as fixed inputs, without influencing the optimization of F(.).
Importance scoring with the FrameScorer. For a video with frames, where each frame o, E XX it is processed by the image encoder Pim to generate frame embeddings: e1:T- = [er, ..,..., er), e, = (m)),, where e₂ E R The FrameScorer F()) then assigns importance score w; to each frame:
By minimizing Ls. FF.) is trained to assign importance scores, enabling prioritization of key information while sup- pressing redundant details. During inference, historical image frames are fed to F(-), which computes importance scores to abstract them into dynamic representations, serv- ing as input for subsequent modules. Importantly, the above objectives are not enforced as strict constraints. Instead, they are introduced as soft loss terms in broader end-to-end supervised training framework, which also includes direct supervision from executed actions. These components act as flexible, task-driven guidance, encouraging the model to attend to relevant features while maintaining adaptability and avoiding over-reliance on auxiliary signals.
Wi+ = F(e:), Wy E (0, 1].
(1)
Here, F() evaluates both the frame's visual saliency and semantic consistency, as described in Equation 3.
Transformation with Sliding Window Fusion. A video is divided into non-overlapping windows of C frames. Within each window, features e, are fused using impor- tance scores Wi to form dynamic representation:
Dne Ei=(n-1)C+1 W
h
n 1 [T/C), (2)
DCC 21=(1-1)0+ Wt
where hn E D represents the dynamic feature of the n-th window. These window-level dynamics are ag- gregated into a global sequence of dynamics: H =
3.2. Video Dynamic Reasoning
To predict future development trends based on historical dynamic information, dynamic reasoning is achieved using autoregressive transformer, which is well-know for its
[h, hg....... hhg....,hrych hT/ccl which captures the global dynam- ics sequence of the video.

---

DynaMind: Reasoning over Abstract Video Dynamics for Embodied Decision-Making
capability in temporal modeling (Janner al., 2021; Han et al., 2021; Micheli et al., 2022). For training, after ab- stracting each video in the dataset into a dynamic sequence, represented as Mi.tr/c1 torical dynamic sequence - where 1 S n - [T/C], combined with the language embedding q' and temporal positional encoding fpus. and fed into the temporal trans- former model (Tforeer) to predict the future dynamic se- quence H2... Formally, the prediction process is expressed as: H22... = Tformer (iq', Himm-1] 1| + fpos). By employing an autoregressive mechanism, this approach incrementally predicts the future dynamic evolution.
sequences, and predicted future dynamic representations, enabling the capture of long-horizon dependencies.
[bj h2.... h l. The his-
Training. To align with dynamic reasoning, we extract multiple temporal windows of size C' from the image-action sequence as training data. Following feature extraction, the input comprises the historical frame features et-++1t-1b corresponding actions, and the goal condition 9t- These inputs are augmented using positional encoding fpos and subsequently processed by the action transformer. The train- ing objective Laction is designed minimize the discrep- ancy between the predicted action sequence C--++16-1 and the ground-truth action sequence At-C+1a-11 For discrete action predictions, binary cross- cross-entropy loss is employed, while MSE is used for continuous action predictions.
Training. By minimizing the mean squared error (MSE) loss function, we reduce the discrepancy between the gener- ated dynamic sequence H/... and the true dynamic sequence H2g.. where H2.,. = big.... hi and H2... is derived from the abstraction process described earlier. This opti- mization enables the model to progressively learn temporal dependencies and predict future dynamic evolution. During inference, the predicted dynamic representation há guides the downstream action decision module.
Hybrid Assignment. During early training, instability in the dynamic reasoning module can hinder the action prediction's 's performance. To mitigate this issue, a hybrid assignment strategy is introduced. Specifically, the goal ge is stochastically selected from two distinet sources: 1) the future dynamics forecasted by the dynamic reasoning mod- ule, or 2) the ground-truth frame features e, derived from the training data, which correspond to the frame immedi- ately succeeding the historical frame sequence Cc-C+it-1- During the early stages of training, the incorporation of e; serves stabilize the learning process. As training pro- gresses, there is progressive shift towards relying more on the predicted dynamics for end-to-end optimization.
Network. Our transformer model comprises two key com- ponents: the transformer block and the prediction head. The transformer block captures temporal dependencies among input embeddings, while the prediction head pro- duces module-specific outputs based on these embeddings. The transformer block integrates two complementary atten- tion mechanisms: self-attention and cross-attention. Self- attention, with causal masking, encodes temporal order and causal relationships. Cross-attention enables multimodal integration to produce task-relevant predictive representa- tions. These mechanisms alternate across transformer layers, resulting in enriched temporal representations.
During inference, this module processes historical frame fea- tures, updated action sequence, and predicted future dynam- ics. It autoregressively predicts the current action, which is then appended to the action sequence for subsequent predic- tions, ensuring both coherence and accuracy.
4. Experiments
3.3. Dynamic-Guided Action Decision
Previous action decision methods based on generative mod- els typically map adjacent frames to corresponding actions (Liang al., 2024; Zhou al., 2024). These approaches are not compatible with framework, they predict single- step actions based on generated adjacent image frames, whereas our focus is on using generated high-level dynamic representations for action decision-making.
Environments. We validate our method on simulation benchmarks and real-world scenarios, with simulation benchmarks including robotic manipulation tasks: LOReL Sawyer (Nair et al., 2022) and Franka Kitchen (Gupta et al., 2020), and navigation task, BabyAl Bedevaller-Boiveed et al., 2018), A summary can be found in SA.
Baselines.
We use the following basclines, detailed in $B. Vanilla Imitation Learning Methods: Vanilla BC (Step- puttis et al., 2020) and DT (Chen et al., 2021).
Different from them, we propose a solution that utilizes an action transformer to predict action sequences that tran- sition from current state to future dynamics. Our action transformer follows the same structure described in $3.2, but with a different input formulation. Unlike those methods that solely rely on image frames for action prediction (Liang et al., 2024; Zhou et al., 2024), we integrate historical in- formation integration and multi-source information fusion. This includes historical frame sequences, historical action
Multimodal Alignment Methods:
GR-/ (Wu et al., 2024): A transformer model designed for predicting videos under language conditions, fine-tuned align actions with both videos and language.
MT-R3M (Wu et al., 2024): An advanced model of GR-1
5

---

DynaMind: Reasoning over Abstract Video Dynamics for Embodicd Decision-Making
Table Task-wise success rates on LOReL Sawyer, DynaMind outperforms all other methods terms of average performance. The results calculated over 3 seeds, Best methods and those within 10% of the best are highlighted in bold.
Task
Random
Vanilla BC
RL
DT
LISA
SkillDiffuser
DynaMind (ours)
closer drawer
52%
50%
58%
10%
100 %
95%
100%
open drawer
14%
0%
8%
60%
20%
55%
80 %
turn faucet
24%
12%
13%
0%
%
57%
turn faucet right
15%
31 %
0%
30%
25%
26%
move black mug right
12%
73%
0%
20%
60%
18%
39%
move while mug down
5%
0%
30%
10%
20% %
Average over tasks
20%
29%
13%
1.%
40%
43%
53.67 %
Method
Success Rate
0.58
DT LOSA
DT
28.63%
0.05
Success Rale
0.9
MT-33M
LISA
28.69%
DynuMind
GR-1
32.94%
0.05
0.03
MT-R3M
30.50%
DynaMind 39.81%
Figure 3. Success rates on Franka Kitchen. The four plots on the right illustrate the success rates of completing to subtasks within single episode, while the left plot shows the average success rate all tasks. The evaluation is repeated 100 times.
that explicitly aligns video and language using pre- trained vision encoder, R3M (Nair et al., 2023).
Language-Decomposed Methods:
LISA (Garg et al., 2022): method that decomposes language instructions into fine-grained semantic skills and executes them via behavior transformer.
LISA
SkillDiffuser (Liang et al., 2024): A method for predicting skills from language, integrating skill-conditioned video generation and an inverse dynamics model.
DynaMind (ours)
Figure 4. Qualitative Results in LOReL Sawyer. We visualize the performance of different methods composite task, where the agent is required to open the drawer and turn the faucet to the right. Due space limitations, only subset of video frames shown.
4.1. Performance Comparison
Performance on LOReL Sawyer. We evaluate the per- formance various methods on the LOReL Sawyer dataset, which consists of 50,000 trajectories, all generated in a robot manipulation environment built on the MetaWorld (Yu et al., 2020). To ensure fair comparison, is designed to maintain similar parameter count to the baseline models and adopts the same visual and language encoder architecture as in SkillDiffuser (Liang et al., 2024). Table summarizes the quantitative results across tasks, demonstrating that our method outperforms others, includ- ing language-deccompssdd approaches such as LISA (Garg al., 2022) and SkillDiffuser. These methods tackle the mismatch between brief language instructions and complex video content by decomposing and refining the instructions. In contrast to these methods, our DynaMind abstracts key content from the video, reduces redundancy, and enhances udaptability to variations in scene dynamics and task com- plexity, leading to superior task performance.
Qualitative results are presented in Figure 4. DynaMind suc- cessfully completes the task. Although LISA successfully opens the drawer, it fails to turn faucet right, likely due to errors in language instruction decomposition that prevent it from providing the correct guidance for the subsequent task. We also evaluate DynaMind's 's ability to follow new language instructions that were not seen during training, but convey the same meaning. As shown in Table 2, the results demonstrate its strong language understanding and general- ization by narrowing the gap between video and language. Detailed results for each instruction type are in S..1.
method
Performance on Franka Kitchen. We evaluate the per- formance of DynaMind Franka Kitchen, which presents significant challenges due to its complex interactions and
6

---

DynaMind: Reasoning over Abstract Video Dynamics for Embodicd Decision-Making
Open the microwave door
Move to the knob and activate bottom hurner Activate the top burner
Open the cabinet door
Epoch 500
Epoch 1000
Prant Sgore
19.6
0.0
25
50
75
100 Time Steps
125
150
175
200
Figure Visualization our method adaptive scoring image frames. The top row displays critical frames within an episode. The bottom row shows the importance score of the frame at each time step. This allows DynaMind extract relevant information from the video while filtering out redundant content, effectively bridging the gap between complex video and concise language instructions.
Table 2. Performance on instruction generalization.
Table 3. Performance on BabyAL.
Method
DT
LISA
SkillDiffuser DynaMind
Task Go
Vanilla BC
DT
LISA
DynaMind
Success Rate
18.07%
30.14%
39.71%
53.73%
33.3%
49.3%
59.4%
72.7
SynthSeq BossLevel
12.9%
42.3%
46.3%
50.7
20.7%
44.5%
49.1%
52.3%
long-horizon tasks. consists of seven interactive objects. During evaluation, the agent sequentially completes four subtasks according to the language instructions, with each subtask involving interaction with different object. We compare DynaMind with several baselines, including LISA method), GR-1 (Wu al., 2024),
4.2. Ablation Study
A Ablation on dynamic abstraction. We conduct ablation studies to assess the role of dynamic abstraction using the adaptive FrameScorer. We explore how different frame weighting and selection strategies affect dynamic abstrac- tion and model performance. We replace the FrameScorer with: Equal weighting (all frames weighted equally); ii) Random weighting (random weights for each frame); iii) Random frame selection (one randomly selected frame per window). The results in Figure 6 show significant per- formance drop across all configurations. Both Equal and Random weighting hinder the model's ability to identify key frames, while Random frame selection demonstrates that single frame cannot capture the dynamics. Notably, the contribution of FrameScorer is pronounced in the Kitchen environment compared to BabyAl. This discrep-
(a minguage-decompose and MT-R3M (Wu et al., 2024) (both multi-modal alignment methods). As shown in Figure 3, DynaMind outperforms the baseline methods, showing particularly strong perfor- mance when handling more subtasks. These findings further reinforce the foundation of our method: direct alignment be- tween video and language is often ineffective, video data frequently contains redundant information, and language is inherently abstract. Additional qualitative results can be found in the $E.2. We also assess DynaMind's performance under varying amounts of training data. Results in $F. show our method generalizes better than others even with limited data and maintains superior scalability as the amount of training data increases.
Equal wiightngg Raadoo wiightng
Performance on BabyAL. Additionally, we evaluate the performance of our method on another long-horizon task, BabyAl navigation, which requires the sequential execution of multiple subtasks, as shown in Table 3. We specifically assess performance under low-data conditions, where only 1k randomly sampled trajectories from the dataset are used for training. The results demonstrate that our method is able extract more valuable information from the limited data.
Hasdon edaction
Steceen Kate
GoToSeq
SynthSqq
BeseLLevel
Kiteken
Figure 6. Ablation on dynamic abstraction.

---

DynaMind: Reasoning over Abstract Video Dynamics for Embodied Decision-Making
ancy can be attributed differences in environmental com- plexity: Kitchen features rich visual content and temporal redundancy, where adaptive frame selection offers greater benefits, whereas the relatively simple structure of BabyAl Al reduces the necessity for such abstraction. is worth noting that FrameScorer is only one part of the full method. The performance gains observed BabyAl, despite the weaker role of FrameScorer, underscore the effectiveness of the other modules in DynaMind.
suite (batch size 64) using identical hardware and settings (NVIDIA A800 GPU). The number of trainable parameters and GPU memory usage for each method is reported in Table 4. As shown in the table, our method achieves a favor- able balance between computational cost and task success rate. These results demonstrate that DynaMind provides an effective trade-off between training efficiency and perfor- mance, yielding improved outcomes without introducing substantial computational overhead.
Ablation on dynamic reasoning. We evaluate the impact of reasoning interval on performance in Video Dynamic Reasoning. In the BabyAl experiments, the default interval hyperparameter C' is set to 30. To evaluate its impact, we test values of 5, 10, and 100. The results in Figure 7 (a) demonstrate that moderate intervals lead to good perfor- mance, while both excessively small and large values result in performance degradation. Specifically, small interval results in frequent reasoning steps, leading to cumulative er- rors, while large interval causes to miss important dynamic representations, negatively impacting performance.
Table 4. Comparison of training efficiency.
Method
Params(M) GPU Memory(MiiB) Success Rate
LISA SkillDiffuser
7.52
690
40.0%
60.29
1136
43.0%
DynaMind
7.84
854
53.7%
4.4. Analysis Results
Abstracted dynamic representations convey key video in- formation. We demonstrate the adaptive scoring of image frames by the FrameScorer to visualize the information cap- tured in the abstracted dynamic representations, as shown in Figure 5. As training progresses, our method achieves two key outcomes: 1) it effectively captures the progression of events throughout the video, and 2) it better distinguishes between different image frames, assigning higher impor- tance those that are more relevant for task execution. This demonstrates that our method can indeed abstract key dynamic information from videos by focusing on frames that contain task-relevant content, effectively filtering out redundant or irrelevant information.
DELLG
Succos Raki
Succoss Rare Rate
Ours
0.5
GoToSeq
SynchSScq
BoesLevel
C-5 C-10
C-30 (4)
C-100
GoToSoq
SynthScq (b)
BossLevel
Figure 7. (a) Ablation on dynamic reasoning.
(b) Ablation
dynamic-guided action decision.
DynaMind capture the correlation between dynamics and language. To further narrow the gap between video and language, we combine DynaMind with the language- decomposed method LISA. However, the result does not yield the expected performance improvement (see Figure 8 (top)). To investigate the cause, plot the evolution of
Ablation action decision. We analyze the impact of Dynamic-Guided Action Decision by evaluating different configurations. We replace dynamic-guided component with language-guided (LG) and dynamic & language-guided (D&LG) configurations for comparison. We also evaluate the contribution of historical actions by removing this (de- noted as w/o Action). The results in Figure (b) show that our configuration performs best, especially in more com- plex tasks. This indicates that dynamic reasoning provides more effective dacision-making information than language instructions, as well as the rationality of the information we use for decision-making. Additionally, we provide addi-
Method
LISA
DynaMind+LISA
DynaMind
Success Rate
28.69%
33.19%
39.81%
0.08
LISA
DynaMind
0.06
0.14
tional studics in $E.3, including architecture configurations.
0.02
4.3. Comparison of Computational Cost
100
200
400
Epoch
600
800
1000
To quantitatively assess training efficiency, we compare our method with two representative baselines, SkillDiffuser and LISA. All models are trained on the LOReL. Sawyer task
Figure 8. Top: Results of the combined method. Bottom: Mutual information over training.
8

---

DynaMind: Reasoning over Abstract Video Dynamics for Embodied Decision-Making
mutual information during training in the Franka Kitchen environment. As shown in Figure 8 (bottom), the mutual information between dynamics and language in our method steadily increases throughout training. However, the mu- tual information between the decomposed language and the video in LISA does not exhibit a significant improve- ment. This indicates that the introduction of LISA does not further enhance mutual information, and may even lead to information loss during the decomposition process, nega- tively impacting overall performance. This phenomenon is consistent with our experimental results.
decomposed method LISA, highlighting the effectiveness of our approach in abstracting dynamic representations from video, which helps reduce the modality gap.
"pick up snack and place it into basket"
Sanco Roc
"fold towel"
Lika
The learned dynamic representations can be used to per- form new tasks. We evaluate DynaMind's ability to lever- age the abstracted dynamics learned from simpler tasks to tackle more complex ones. In the BabyAI Navigation envi- ronment, DynaMind is trained on the simpler GoToSeq task and tested more challenging tasks such SynthSeq and BossLevel. As shown in Table 5, our method outperforms the baselines, demonstrating its ability to transfer abstracted dynamic representations and adapt to more complex ones.
Figure 9. Left: Success rate averaged over 5 tasks. Right: Quali- tative results of DynaMind for 2 tasks in real-world experiments. More results and details can found in $F.
5. Limitation and Future Work
While our method transitions from frame-by- be-mee video generation to dynamic reasoning, substantially enhancing decision-making for embodied agents, also presents sev- eral limitations that suggest promising directions for future work. For instance, DynaMind, similar to many existing approaches, uses fixed dynamic reasoning intervals. While effective in most cases, such a rigid schedule can be subop- timal for tasks requiring more adaptive temporal reasoning. One potential solution is to learn task-specific reasoning trig- gers that activate dynamic reasoning at critical points during task progression, rather than depending on predetermined intervals. Another promising direction to adaptively in- crease the frequency of reasoning in response to significant changes in multimodal information, thereby ensuring rapid adaptation to evolving conditions. These directions high- light promising avenues for future work.
Table S. Performance on unseen tasks.
Unseen Task
DT
LISA
DynaMind
SynthSeg 31.0% 31.2%
33.1% 1%
40.0 %
BossLevel
32.4%
35.7% %
Additionally, we assess DynaMind's performance in execut- ing compositional tasks by leveraging the abstracted dynam- ies it has learned from simpler, shorter tasks. Specifically, we test its ability to handle novel combinations of language instructions that it has not encountered during training, As shown in Table 6, DynaMind significantly outperforms base- line algorithms, demonstrating its effectiveness in managing tasks with long-term dependencies and showcasing the gen- cralization ability of the abstracted dynamic representations on unseen compositional tasks.
6. Conclusion
Table 6. Performance on unseen compositional tasks on LOReL Sawyer.
In conclusion, we introduce DynaMind, a framework that shifts from video generation dynamic reasoning, effec- tively bridging the gap between multimodal information. By transforming videos into high-level dynamic sequences, DynaMind captures critical patterns, enabling agents to per- form dynamic reasoning for decision-making To achieve this, we design an adaptive FrameScorer that identifies key frames, abstracting dynamics, and a dynamic reasoning module to predict future dynamics. Additionally, dynamic- guided action decision module is incorporated to guide the decision process. Experimental results demonstrate that Dy- naMind outperforms existing methods in both performance and generalization. Our work showcases the feasibility of abstracting video into dynamics, thereby significantly en- hancing the decision-makiin abilities of embodied agents.
Method
DT
LISA
SkillDiffuser DynaMind
Success Rate 13.33% 20.89%
25.21%
36.67 %
4.5. Real-World Experiment
To validate the effectiveness of DynaMind in real-world scenarios, we train and test in a real-world setup using a Franka Research 3 arm. We design five tasks: pressing button, picking up milk box, pushing box to tar- get location, placing a snack into a basket, and folding a towel. As shown in Figure 9, results from 10 trials demon- strate that DynaMind can effectively complete tasks in real- world settings. Furthermore, it outperforms the language-

---

DynaMind: Reasoning over Abstract Video Dynamics for Embodicd Decision-Making
Acknowledgements
Vision-ECCV 2020: 16th European Conference, Glas- gow; UK, August 23-28, 2020, Proceedings, Part IV 16, pp. 214-229, Springer, 2020.
This work was supported by the National Natural Science Foundation of China (Grant No. 62403429) and the Nat- ural Science Foundation of Zhejiang Province (Grant No. LQN25F030008).
Garg, D., Vaidyanath, S., Kim, K., Song, J., and Ermon, S. Lisa: Learning interpretable skill abstractions from language. Advances in Neural Information Processing Systems, 15:21711-21722 2022.
Impact Statement
Gupta, A., Kumar, V., Lynch, C.,, Levine, S., and Hausman, K. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. In Conference Robot Learning, pp. 1025-1037. PMLR, 2020.
This paper presents work whose goal is to advance the field of Machine Learning in Robotics. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Han, K., Xiao, A., Wu, E., Guo, J., Xu, C., and Wang, Y. Transformer in transformer. Advances in neural informa- tion processing systems, 14:115988-5991 2021.
References
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J.,
Finn, C., Gopalakrishnam, K., Hausman, K., Herzog, A.,
Janner, M., Li, Q., and Levine, S. Offline reinforcement learning one big sequence modeling problem. Ad- vances in neural information processing systems, 34: 1273-1286, 2021.
Hsu, J., et al. Rt-1: Roboties transformer for real-world control at scale. arXiv preprint arXiv:2212.068) 2022.
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. De- cision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15099 2021.
Ju, Z., Yang. C., Sun, F., Wang. H., and Qiao, Y. Rethink- ing mutual information for language conditioned skill discovery on imitation learning. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 34, pp. 301-309, 2024.
Chen, Y., Ge, Y., Li, Y., Ge, Y., Ding, M., Shan, Y., and Liu, X. Moto: Latent motion token the bridging language for robot manipulation. arXiv preprint arXiv:2412.0444 2024.
Ko, P.-C., Mao, J., Du, Y., Sun, S.-.., and Tenenbaum, J. B. Learning to act from actionless videos through dense correspondences. In The Twelfth International Confer- ence on Learning Representutions, 2024. URL https: / //openreview net forum? Whb.hh-bbbp1p1100
Chevalier-Boiveet, M., Bahdanau, D., Lahlou, S., Willems, L., Saharia, C., Nguyen, T. H., and Bengio, Babyai: A platform to study the sample efficiency of grounded lan- guage learning. In International Conference on Learning Representations, 2018
Kou, L., Ni, F., ZHENG, Y., Liu, J., Yuan, Y., Dong, Z., and HAO, J. KISA: A unified keyframe identifier and skill annotator for long-horizon robotics demonstra- tions. In Forty-first International Conference on Machine Learning, 2024. URL https: /ppp/evveeeeeeeee//e/ee .nt// forum? 1d= Cac.oCC99HHHCC 9gowws.
Croitoru, I., Bogolin, S.-V., Leordeanu, M., Jin, H., Zisser- man, A., Albanie, S., and Liu, Y. Teachtext: Crossmodal generalized distillation for text-video retrieval. In Pro- ceedings of the IEEE/CVF International Conference Computer Vision. pp. 11593, 2021.
Li, X., Liu, M., Zhang, H., Yu, C.,, Xu, J., Wu, H., Cheang, C., Jing, Y., Zhang, W., Liu, H., et al. Vision-language foundation models effective robot imitators. arXiv preprint arXiv22311.01372 2023.
Du, Y., Yang, S., Dai, B., Dai, H., Nachum, O., Tenenbaum, J.. Schuurmans, D., and Abbeel, Learning universal policies via text-guided video generation. Advances in Neural Information P Processing Systems, 36, 2024.
Liang, Z., Mu, Y., Ma, H., Tomizuka, M., Ding, M., and Luo, P. Skilldiffuser: Interpretable hierarchical planning via skill abstractions in diffusion- task execution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16467-16476, 2024.
Fang. B., Wu, W., Liu, C., Zhou, Y., Song, Y., Wang, W., Shu, X., Ji, X., and Wang, Uatvr: Uncertainty-adape adaptive text-video retrieval. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 13723- 13733, 2023.
Liu, H., Zhuge, M., Li, B., Wang, Y., Faccio, F., Ghanem, B., and Schmidhuber, J. Learning to identify critical states for reinforcement learning from videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1955-1965, 2023.
Gabeur, V., Sun, C., Alahari, K., and Schmid, C. Multi- modal transformer for video retrieval. In Computer
10

---

DynaMind: Reasoning Abstract Video Dynamics for Embodicd Decision-Making
Luo, Y. and Du, Y. Grounding video models actions through goal conditioned exploration. arXiv preprint arXiv:4411007223, 2024.
IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 16551-16560, 2024.
Wu, H., Jing, Y., Cheang, C., Chen, G., Xu, Li, Liti, M., Li, H., and Kong. T. Unleashing large-scale video generative pre-training for visual robot manipulation. In The Twelfth International Conference Learning Representations, 2024. URL https: httpss////ppeeeieeeeeee net / forum 1d=NxodnGNNoC......
Lynch, C., Khansari, M., Xiao, T., Kumar, V., Tompson, J., Levine, S., and Sermanet, P. Learning latent plans from play. In Conference robot learning, pp. 1113-1132. PMLR, 2020.
Ma, Y., Chi, D., Wu, S., Liu, Y., Zhuang, Y., Hao, J., and King. I. Actra: Optimized transformer architecture for vision-language-action models in robot learning. arXiv preprint 4rXhv:2408.01147, 2024.
Yao. L., Huang, R., Hou, L., Lu, G., Niu, M., Xu, H., Liang, X., Li, Z., Jiang, X., and Xu, C. FILIP: Fine- grained interactive language-image pre-training. In In- ternational Conference on Learning Representations, 2022. URL https: / .wt.rieww.ee/formm forum? id=cpDhesEDC2.
Ma, J.J., Kumar, V., Zhang. A., Bastani, O., and Jayaraman, D. Liv: Language-image representations and rewards for robotic control. In International Conference Machine Learning, pp. 23301 23320. PMLR, 2023.
Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. Meta-world: A benchmark and evalua- tion for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1100. PMLR, 2020.
Mazzaglia, P., Verbelen, T., Dhoedt, B., Courville, A., and Rajeswar, S. GenRI: Multimodal-foundatee world mod- cls for generalization in embodied agents. In The Thirty- eighth Annual Conference cessing Systems, 2024. URL https: / ppenreviee...... net/forunts Eorum? i d= 9 Xx8UUUU
Neural Information Pro-
Zhou, S., Du, Y., Chen, J., Li, Y., Yeung, D.-Y., and Gan, C. Robodreamer: Learning compositional world models for robot imagination. arXiv preprint arXiv:2404. 12377, 2024.
Micheli, V., Alonso, E., and Fleuret, Transform- sample-efficient world models. arXiv preprint 2000002209.00588, 2022.
Nair, S., Mitchell, E., Chen, K., Savarese, S., Finn, C., et al. Learning language-condingg conditioned robot behavior from offline data and crowd-sourced annotation. In Conference Robot Learning, pp.
PMLR, 2022.
Nair, S., Rajeswaran, A., Kumar, V., Finn, C., and Gupta, A. R3m: universal visual representation for robot manipulation. In Conference on Robot Learning, pp. 892- 909. PMLR, 2023.
Sanh, Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. Proceedings of Thirty thir Conference on Neural Information Processing Systems (NIPS2019), 2019.
Stepputtis, S., Campbell, J., Phielipp, M., Lee, S., Baral, C., and Ben Amor, H. Language-conditione imitation learning for robot manipulation tasks. Advances Neural Information Processing Systems, 33:13139-13150 2020.
Tian, Y., Yang, S., Zeng, J., Wang, P., Lin, D., Dong, H., and Pang, J. Predictive inverse dynamics models are scalable learners for robotic manipulation. arXiv preprint arXiv:2412.15100 2024.
Wang, J.. Sun, G., Wang, P., Liu, D., Dianat, S., Rabbani, M., Rao, R., and Tao, Text is mass: Modeling as stochastic embedding for text-video retrieval. In Proceedings of the

---

DynaMind: Reasoning over Abstract Video Dynamics for Embodicd Decision-Making
A. Environment
LoReL Sawyer
Franka Kitchen
BabyAI
Figure 10. Environments.
LOReL Sawyer LOReL (Nair et al., 2022) is robot manipulation environment built on the MetaWorle (Yu et al., 2020), set in a tabletop scenario where a Sawyer robot interacts with objects such as faucet, a drawer, and two cups of different colors. The dataset consists of 50,000 trajectories generated from pseudo-expert data and plays data collected from the replay buffer of reinforcement learning policy and annotated with crowdsoureed, post-hoc language instructions. Although the trajectories fulfiill the language instructions, they are not necessarily optimal. This dataset is relatively inexpensive to collect in real-world settings (Lynch al., 2020), making it relevant for training algorithms that need be robust such noisy and random data. However, the inherent randomness in the trajectories makes them challenging for training. Despite these challenges, we evaluate our approach on the six tasks used in the original paper: closing the drawer, opening the drawer, turning the faucet right, turning the faucet left, moving the black mug to the right, and moving the white mug down. These tasks are evaluated using partially observed image data.
Franka Kitchen To highlight the complexity executing long-horizon sequential tasks, the Franka Kitchen (Gupta et al., 2020) environment is employed, where Franka robot operates within kitchen setting. Experiments are conducted using the Relay Policy Learning dataset, which contains demonstrations collected by human participants wearing VR headsets. Each demonstration consists of a sequence of four object interaction subtasks, selected from set of seven interactive objects: microwave, kettle, sliding cabinet, hinged cabinet, switch, and two stove burners. The experimental design involves selecting four-subtask combinations from these seven objects, with N demonstrations sampled for each combination during training. This setup explores how the number of demonstrations affects the method's effectiveness. During the evaluation phase, all tasks include randomized environmental variations. Performance measured by counting the number of subtasks successfully completed within fixed horizon of 280 time steps, averaged over 100 evaluation runs. The input to the model limited single-view RGB image, ensuring that the evaluation focuses on the method's 's effectiveness under constrained visual input conditions.
BabyAl AI Navigation The Al (Chevalier-Boisvell al., 2018) dataset includes various environment configurations, where the difficulty of levels and the complexity of navigation instructions gradually increase. Each level is set within grid world, where the agent observes partially visible 7x7 square region from an egocentric perspective. In this environment, synthetic natural language instructions guide the agent to perform navigation tasks under partial observability (e.g., unlocking doors) and move objects to specified locations. At simpler levels, the instructions are straightforward, while at higher difficulty levels, they become more complex, often involving multiple sequential subtasks. The dataset contains one million expert trajectories for each level, but only 0.1 % are used for training, allowing evaluation under limited data conditions. The method is tested with 100 distinct instruction sets from the Gym environment, covering variety of unseen layouts and language instructions, further assessing its generalization in data-constrained scenarios.
12

---

DynaMind: Reasoning over Abstract Video Dynamics for Embodied Decision-Making
B. Baselines
Vanilla Imitation Learning Methods:
These methods serve as baselines to assess how well our method improves upon vanilla methods.
Vanilla BC (Stepputtis et al., 2020): A classic supervised learning approach that replicates expert actions.
Decision Transformer (DT) (Chen al., 2021): A transformer-based behavior cloning method that capturing long-term dependencies in trajectory sequences.
Multimodal Alignment Methods:
They align information across modalities, either implicitly or explicitly. We use them as baselines to evaluate the rationale of extracting critical information from videos, rather than enforcing strict modality alignment.
GR-Z (Wu et al., 2024): A pre-trained transformer model designed for predicting videos under the language instruction, fine-tuned to align actions with videos and language.
MT-R3M (Wu et al., 2024): An advanced model of GR-1 that explicitly aligns video and language using a pre-trained vision encoder, R3M (Nair et al., 2023).
Language-Decompoee Methods: They break down concise language instructions into finer- fmer-grained semantic skills, We compare them with our method demonstrate the advantage of abstracting critical dynamic information directly from the video.
LISA (Garg et al., 2022): method that decomposes language instructions into fine-graine semantic skills and executes them via behavior transformer.
SkillDiffuser (Liang al., 2024): A method using a transformer to generate high-level semantic skills from concise instructions, combined with a diffusion model for next-frame prediction and a inverse dynamics model for single-step action prediction.
C. Additional Descriptions of Method
C.1. Additional Details of the Video Dynamic Abstraction Module
In the video dynamic abstraction module, we introduce adaptive FrameScorer to evaluate the importance of individual frames and abstract raw video sequences into high-level dynamic representations. At its core, FrameScorer is implemented as 2-layer fully connected network with sigmoid activation, designed to assign soft importance scores to each frame.
FrameScorer operates in lightweight manner and supervised through two global yet semantically grounded objectives: 1) Semantic consistency loss encourages the weighted dynamic representation, produced by FrameScorer, to align with the language embedding, guiding the model attend frames that reflect the task goal; 2) Visual saliency loss penalizes visually salient frames that lack semantic relevance, preventing the model from focusing on distractive yet functionally irrelevant content. These implicit supervision signals allow the model to discover semantically meaningful frames without relying on explicit annotations. In practice, both loss terms are assigned equal weights (1.0) and jointly optimized during training.
Moreover, the abstraction process is not solely guided by the aforementioned objectives. Since the model is trained in an end-to-end manner, FrameScorer also receives direct supervision from executed actions. Specifically, the importance scores generates affect downstream action predictions and are optimized through the action loss. This behavioral grounding ensures that the learned abstractions are not only semantically aligned but also functionally effective for task execution, particularly in complex or instruction-conditione scenarios. In practice, the action supervision loss is weighted by 1.0 in the overall training objective.
While video sequences are utilized during training to provide temporal and semantic context, the scoring mechanism remains frame-local. At inference time, FrameScorer evaluates each frame independently, enabling efficient online processing and ensuring scalability to long- horizoo sequences.
13

---

The dynamic reasoning module designed to predict future developments based on historical dynamics. It is implemented cycle of our DynaMind: Reasoning over Abstract Video Dynamics for Embodicd Decision-Making
C.2. Additional Details of the Video Dynamic Reasoning Module
I-layer, 4-head Transformer and operates within the broader framework.
During training, the video dynamic reasoning module receives as input a dynamic representation abstracted from ground- truth observation sequences. It is trained to predict the dynamic representation of future steps, which is also abstracted from future ground- frames. The predicted and target representations are compared using mean squared error loss, enabling the module learn temporally grounded reasoning patterns based on real trajectories. At inference time, the module reasons over the current dynamic representation to produce a predicted one, which guides the subsequent action decision. After the predicted action is executed in the environment, representation, which is then used for the next reasoning step.
observation is obtained and abstracted into an updated dynamic
Throughout both training and inference, the reasoning module operates on dynamic representations generated from real observed frames. This avoids recursive usage of model predictions, thereby preventing error accumulation and improving the stability and robustness the inference process.
C.3. Additional Details of the Dynamic-Guided Action Decision Module
We use a I-layer, 4-head Transformer for action decision in our implementation. The module takes three types of input: 1) Historical visual features: a sequence of image embeddings extracted from past frames. 2) Historical actions: a sequence of past actions, each projected into the same latent space as the visual features. 3) Goal token gt a special conditioning token encoding the target dynamics. This token is prepended the sequence and is either derived from ground-truth predicted future representations, depending on the training phase.
The fusion of these inputs proceeds in the following steps: 1) Each visual and action embedding is augmented with a learned timestep embedding to encode temporal order. 2) The state and action tokens are interleaved by timestep to form a temporally aligned sequence. 3) The goal token 9: is prepended to the full sequence, enabling the model to condition its attention on the future target. 4) The final sequence is normalized via LayerNorm and processed with an attention mask before being fed into the Transformer.
This design enables the model jointly reason over historical Mate-action trajectories while dynamically grounding its decision-making in predicted future dynamics. The unified representation captures both temporal dependencies and future intent, thereby improving the policy's 's ability to make coherent and goal- goal-directed decisions in long-horizon tasks.
During training, the model learns from ground-truth action sequences provided as supervision. During inference, the model generates actions step by step, conditioning each prediction on the previously generated actions and the accumulated observation history.
D. Additional Results on LOReL Sawyer
D.1. Instruction Generalization
We evaluate the generalization capability of DynaMind on language instructions, requiring it to understand and execute new instructions not encountered during training. We assess its performance in the LoReL using rephrased instructions outside the training data. As shown in Table 7, the results demonstrate that our method effectively adapts to unseen instructions, showeasing strong language understanding and generalization abilities.
D.2. Qualitative results in LOReL Sawyer
Qualitative results are presented in Figure 11. DynaMind successfully completes the entire task, whereas DT fails to open the drawer in test scenarios with random initializations, likely due to overfitting to the training data. Although LISA successfully opens the drawer, it fails to turn faucet right, likely due to errors in language instruction decomposition that prevent from providing the correct guidance for the subsequent task.
14

---

DynaMind: Reasoning over Abstract Video Dynamics for Embodicd Decision-Making
Table Performance instruction generalization.
Instruction
DT
LISA
SkillDiffuser DynaMind
seen
15%
40%
43.65%
57.78%
unseen noun
13.33%
33.33%
36.01%
54.45 %
unseen verb
28.33%
30%
36.70%
50.00 %
unseen verb+none
6.7%
20%
42.02%
53.34 %
human
26.98%
27.35%
40.16%
53.08 %
Average
18.07%
30.14%
39.71%
53.73%
"open drawer and turn faucet right"
DT
LISA
DynaMind (ours)
Figure 11. Qualitative Results in LOReL Sawyer: We visualize the performance of different methods on a composite task, where the agent is required open the drawer and turn the faucet the right. Due space limitations, only subset of video frames is shown.
E. Additional Results on Franka Kitchen
E.1. Performance on Different Date Scale.
DT
LISA
DynaMind(ours)
0.4
Success Rate
0.3
0.2
N-25
N-15
Figure 12. Performance on date scale.
We further assess DynaMind's performance under varying amounts of training data to test its robustness data-scarce conditions. We provide datasets with 5, 10, and 25 trajectories for each task. As shown in Figure 12, our method generalizes
15

---

DynaMind: Reasoning over Abstract Video Dynamics for Embodied Decision-Making
better than others even with limited data and maintains superior scalability the amount of training data increases.
E.2. Qualitative Results in Franka Kitchen
We visualize the performance of different methods on composite tasks. Figure 13 and Figure 14 show that our method successfully completes the task, while the language-decomposee method, LISA, and the multi-modal alignment method, GR-1, I, all fail due to their inability bridge the gap between video and language.
"open microwave door, activate the bottom burner, and activate the top burner, open the cabinet door"
LISA
GR-1
DynaMind (ours)
Figure 13. Qualitative results in the Franka Kitchen. The agent is required to perform four subtasks, including opening the microwave door, activating the top and bottom burners, and opening the cabinet door.
"move kettle to topleft burner, activate the bottom burner and turn on light switch, open slide cabinet"
LISA
GR-1
DynaMind (ours)
Figure 14. Qualitative results in the Franka Kitchen. The agent required to perform four subtasks, including moving the kettle to the top-left burner, activating the bottom burner and turning the light switch, and sliding open the cabinet door.
16

---

DynaMind: Reasoning Abstract Video Dynamics for Embodicd Decision-Making
E.3. Ablation of Architecture Configurations
We evaluate the following three configurations made in the architecture:
Shared-Tr: Shares Transformer parameters between the Dynamic Reasoning and Action Prediction components.
R3M: Uses a large-scale pretrained encoder (R3M) with frozen parameters, replacing our original language and image encoders.
Table 8. Ablation architecture configurations.
Method
Shared-Tr
R3M
DynaMind (ours)
Success Rate
33.72%
13.44%
39.81% %
The results in Table 8 show that while the Shared-Tr configuration performs slightly worse than non-shared configurations, it still significantly outperforms baseline methods. This suggests that the shared design has some limitations in adapting to the unique needs of individual tasks but strikes good balance between performance and computational efficiency, making suitable for vision ignores the inherent contradictions between these modalities, resulting in lower performance than our proposed configuration.
constraieed environments. In the R3M configuration, the static alignment between language and
F. Rorld Experiment
To demonstrate the effectiveness of DynaMind in real-world scenarios, we train and test the model in real-world setup equipped with Franka Research (FR3) robotic arm. The experimental assets and environment are shown in Figure 15. statically mounted RGB camera captures observations from a third- person perspective. We design five tasks: pressing button, picking up box of milk, pushing a box to a destination, placing a snack into basket, and folding a towel. These tasks involve various interactive objects and actions. For each task, we collect 20 demonstrations, all performed by human demonstrators. The trajectories conducted over 10 trials, and the average success rate is calculated. As shown in Table 9, the results demonstrate that DynaMind can make real-time predictions and effectively complete tasks in real-world settings, outperforming both DT, which does not address the language-vides gap, and LISA, which attempts to resolve it through language composition. Figure 17 shows the qualitative results of DynaMind real-world experiments, while Figure 16 illustrates that our method successfully completes the folding towel task, whereas LISA fails. This highlights the effectiveness our approach in abstracting dynamic representations from video and validates its capability in real-world environments.
third-person
recorded at 20 fps. To evaluate the performance of method, all experiments
Figure 15, Real-world scenario. The assets and environment configured the real-world experiments.
17

---

DynaMind: Reasoning over Abstract Video Dynamics for Embodicd Decision-Making
Table Results in the real world.
Success out of 10 trials
DT
LISA
DynaMind
press button pick up milk
100%
100%
100%
100%
100%
100%
push box to goal
100%
100%
100%
place into basket
60%
60%
90%
fold towel
40%
50%
80%
Average tasks
80%
82%
94%
LISA
DynaMind (ours)
Figure 16. Success and failure in folding towel.
press button
pick up milk
push box into goal
pick up snack and
place into basket
fold towel
Figure 17, Qualitative results DynaMind in real- world experiment.
18


Reviews:
[{"review_id": "bBq2184Eg8", "reviewer": "Reviewer_VsTD", "summary": "This paper aims to address the mismatch problems between abstract languages and the rich content of videos. It proposes dynamic abstraction to represent spatiotemporal latents as a substitute for videos. It generates dynamic abstraction by learning semantic consistency and visual saliency and learns the agent policy conditioned on dynamic abstraction. The empirical results show that its model can learn key video information, capture the correlation between languages and dynamic abstraction, and generalize to new tasks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer VsTD for the feedback. To ensure clarity, some responses are stated directly—we appreciate your understanding.\n>The fixed window size is less flexible\n- Fixed window sizes are standard practice, used in baselines like LISA and SkillDiffuser, and in some video understanding work.\n- Performance is stable under moderate window changes (Fig.7a), suggesting robustness. \n- We acknowledged the limitation(lines 886–888) and plan to explore adaptive horizons.\n\n>What is the image encoder—pretrained from SkillDiffuser or from scratch?\n\nThe image encoder is a CNN trained from scratch, not from SkillDiffuser or any pretrained source, ensuring that improvements reflect our own contributions. \n>Eq.4&5 compute similarity between abstraction of the short clip and instructions of a whole task—seems questionable.\n\nWe do not compute similarity between a short clip and language. Instead, as noted in lines 170–173 and 184–188, it is computed between the full dynamic representation of the entire trajectory and the task-level language instruction.\n>Inputs to Video Dynamic Reasoning differ between training and inference may impact performance.\n\nWe adopt a closed-loop process to reduce the train-test gap, a common strategy in decision-making. Both stages use dynamic abstracted from real observations, mitigating error accumulation. Specifically, during training, inputs come from actual frames; during inference, predicted dynamics guide actions, and real observations are appended iteratively.\n>The source of the goal token $g_t$\n\nAs noted in lines 271–274(left) and 220–229(right), during training, $g_t$ is gradually shifted from ground-truth future frame to predicted dynamics to stabilize learning. At inference, $g_t$ is taken from the predicted dynamics.\n>Confused why actions are $a_{t-C+1:t-1}$ but the input are $e_{t-C+1:t-1}$—shouldn't actions be autoregressively generated?\n\nThe model is auto-regressive, generating actions step by step during inference. Following common practice in sequence modeling, we apply parallel supervision over the action sequence during training to improve efficiency and stability.\n>A pseudocode might make training clear\n\nWe’ve prepared detailed pseudocode but couldn’t include it due to space limits. It will be added in the final version.\n>Fig.3 shows a sharp drop from 0.4(2 tasks) to 0.1(3 tasks)\n\nThe sharp drop in performance with more tasks reflects the challenge of long-horizon task, especially under constrained settings (lightweight model, limited data). As shown in Fig.3, all baselines struggle, while ours performs better.\n>Tab.2 uses paraphrased instructions for generalization. With T5-XXL, this is less of an issue due to embedding similarity.\n- We use the same lightweight pretrained language encoder (DistilBERT) as our baselines, yet achieve better instruction generalization(Tab.2). This indicates that **success relies not just on the language model, but on the ability to connect language with videos**.\n- Our method is resource-efficient and complementary to LLMs(11B for T5-XXL). While LLMs reduce linguistic variation, ours bridges the language–vision gap.\n\n>Fig.6:Why does FrameScorer help more in Kitchen than in BabyAI?\n- The difference stems from environment: FrameScorer helps in Kitchen with rich visuals and redundancy, while BabyAI’s simplicity reduces the need for abstraction.\n- FrameScorer is one part of our method;overall performance gains in BabyAI reflects the effectiveness of other modules. \n\n>Fig.7:Explain why D&LG performs poorly than dynamic-guided\n\nD&LG underperforms due to a mismatch: language encodes long-term goals, while dynamics reflect short-term cues. Their fusion introduces redundant or conflicting signals, hindering decisions—supported by low mutual information in Fig.8(bottom).\n>Fig.8:LISA’s mutual information is nearly zero—inconsistent with the original paper\n\nDue to environment differences: we compute MI in the more complex Franka Kitchen (lines 406–407), while LISA uses BabyAI. Higher diversity in Kitchen leads to lower MI.\n>Supplementary Material lacks details\n\nWe will include details to ensure reproducibility.\n>Not convinced that SkillDiffuser underperforms DynaMind on simple tasks.\n- SkillDiffuser results are directly taken from the original paper, without any modification.\n- SkillDiffuser underperforms on simple tasks, where skill reuse is limited. It suits long-horizon tasks but adds unnecessary complexity to simpler ones.\n- The two methods are complementary, not competing, and each has its strengths. Our goal is fair comparison under a unified protocol, not to claim superiority.\n\n>Text2video is expensive\n\nUnlike T2V methods that require full trajectory generation, our lightweight model predicts compact dynamics without heavy computation. On Lorel(A800 GPU, batch size 64), it achieves better SR with comparable or lower cost.\n-|Trainable Params(M)|GPU Memory(MiB)|SR\n-|-|-|-\nLISA|7.52|690|40%\nSkillDiffuser(T2V)|60.29|1136|43%\nOurs|7.84|854|53.6%", "claims_and_evidence": "1. Figure 1 shows the simplicity of language and the complexity of videos.\n2. Figure 5 shows the model can score high weight for key frames.\n3.", "methods_and_evaluation_criteria": "1. The fixed window size is less flexible, as also mentioned by the author. It needs to decide corresponding to the different tasks.\n2. What exactly is the image encoder? Is it pre-trained from SkillDiffuser or trained from scratch?\n3. Equation 4 and 5 compute the similarity between the abstraction of the short clip and language instructions of a whole task, which might include several subtasks. It seems not very reasonable.\n4. The inputs of Video Dynamic Reasoning are different in the training and inference stages. It might cause a performance drop in the inference.\n5. One input of action transformer is $g_t$. What is the $g_t$ from? Is it from the concatenation of $h$\n6. It's confused that the output actions are $a_{t-C+1:t-1}$ but the input frame features are $e_{t-C+1:t-1}$. Should it be autoregressively generated? \n7. There are three modules that need training. The whole training pipeline is unclear. A pseudocode might make this clear.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "1. Figure 3 shows the results on Franka Kichen. The success rate of completing 3 and 4 tasks is extremely low. I understand the difficulty of completing multiple tasks. However, the success rate drops from about 0.4 (2 tasks) to 0.1 (3 tasks).\n2. Table 2 shows the instruction generalization, where instructions are different while conveying the same meaning. It should not be a problem when the model is using a large language encoder like T5-XXL. The language embeddings would be similar.\n3. Figure 6 shows the ablation on dynamic abstraction. It would be better to discuss why FrameScorer improves performance much (about 0.1) on Kitchen but less (about 0.02) on BabyAI.\n4. Figure 7 shows the ablation on the action transformer. It would be better to discuss why D&LG performs poorly than dynamic-guided.\n5. Figure 8 shows the mutual information. The MI for LISA is extremely low (nearly zero), which does not align with the results (0.015) in the original paper.", "relation_to_broader_scientific_literature": "This paper proposes a solution (dynamic abstraction) for the mismatch between languages and videos.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n\n1. The empirical results show that the performance is improved by the method, and there are plenty of ablation studies to show the efficiency of the design.\n2. The dynamic abstraction is reasonable to solve the redundant information in the videos.", "other_comments_or_suggestions": "None", "questions_for_authors": "1. The method and training pipeline are very unclear to me; see the method part.\n2. I do not totally agree that text2video (SkillDiffuser) performs poorly than Dynamic Abstraction (DynaMind) in such simple tasks. I think text2video is expensive in such a setting.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wLET74MWna", "reviewer": "Reviewer_kRnT", "summary": "This paper proposes the DynaMind framework for video dynamic abstraction and reasoning, aiming to extract key dynamic information from long-horizon videos for future prediction and decision-making. First, a FrameScorer mechanism is designed to evaluate the importance of video frames based on visual saliency and semantic consistency, generating high-level dynamic representations through weighted fusion. Then, an autoregressive Transformer is employed for dynamic reasoning, leveraging temporal modeling to predict future evolution. Finally, an action Transformer integrates historical frames, past actions, and predicted future dynamics for long-term action decision-making. During training, multi-task loss optimization enhances video abstraction quality, while a hybrid assignment strategy stabilizes action prediction. Experimental results on the LOReL Sawyer robotic manipulation dataset demonstrate that DynaMind effectively reduces redundant information, improves adaptability to task complexity and scene variations, and outperforms existing language-decomposed task planning approaches.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer kRnT for recognizing the novelty and presentation of our work. We address each concern below and will revise the paper accordingly.\n>Comparison with more recent imitation learning methods from the past two years\n\nWe agree that including comparisons with more recent imitation learning methods strengthens the overall evaluation. In response, we add 3 recent methods:\n- Diffusion Policy(DP)[IJRR2023]: a diffusion-based imitation learning method.  \n- LCSD[ICAPS2024]: an extension of DP that introduces a language-conditioned skill learning module.  \n- PIDM[ICLR2025]: an end-to-end imitation learning approach that unifies vision and action by predicting actions from forecasted visual states.\n\nWe compare these methods with ours under 2 benchmarks:\n\nLorel Sawyer\n-|DP|LCSD|Ours\n-|-|-|-\nTask-wise SR|36.6%|45.5%|53.67%|\nRephrasal-wise SR|24.8%|35.8%|53.73%\n\nFranka Kitchen\n-|DP|PIDM|Ours\n-|-|-|-\nSR|33.42%|36.77%|39.81%\n- Lorel results are reported from LCSD.\n- Kitchen results are from our re-implementation using their public codebases.\n\n>Are there any directly comparable methods? \n\nTo the best of our knowledge, we are the first to address video-language modality imbalance from a video-centric perspective for language-conditioned decision-making. We compared with the most relevant works—LISA and SkillDiffuser—which approach the problem from the language side.\nNotably, a concurrent study [1] identifies similar modality imbalance in CLIP-style models, caused by information asymmetry between image and text. It further shows that smaller modality gaps lead to better performance, and that embedding dimensions contribute unequally. Though focused on a different task, it highlights the general importance of the problem and supports our motivation.\n[1] Two Effects,One Trigger.ICLR2025\n>In Table 1, some tasks (e.g.,move black mug right) show a large performance gap.\n\nThe short-horizon, low-complexity tasks are well-suited for vanilla imitation learning, which excels at fitting simple, deterministic behaviors. Interestingly, both our method and Text2Video approaches like SkillDiffuser underperform in these cases(line 282), likely due to indirect objectives introducing unnecessary complexity. This reflects a common but often overlooked limitation, which we plan to address for better adaptability across task complexities. In contrast, vanilla imitation methods struggle on more complex tasks with long-term dependencies or greater generalization demands (Table 1,2,6).\n>How does Hybrid Assignment handle early training and implement the transition?\n\nTo mitigate the impact of early-stage instability in the dynamic reasoning module on action prediction, we initially use ground-truth goal features (future frame representations) for stable supervision, then gradually shift to predicted dynamics to enable end-to-end learning while maintaining training stability. This transition is controlled by a linear annealing schedule with sampling probability $p_n=\\frac{n}{N}$, where $n$ is the current epoch and $N$ is the total epochs. \n>How are historical frames, actions, and predicted dynamics fed into the Action Transformer and fused?\n\nThe Action Transformer takes three inputs: 1) historical frame features, 2) historical actions (embedded into the same latent space), and 3) a goal token, as described in the last question.\nThe fusion process includes: adding timestep embeddings to encode temporal order, interleaving state and action tokens by timestep, prepending the goal token for global conditioning, and applying LayerNorm and an attention mask before feeding the sequence into the Transformer.\n>Results of integrating with language-centric methods\n\nThis is a thoughtful insight—it happens to align with a direction we’ve explored. Specifically, we integrated DynaMind with the language decomposition module from LISA (Figure 8), but it did not improve performance in the Franka Kitchen. To further investigate, we analyzed the mutual information during training(Fig.8, bottom). The results show that DynaMind progressively increases the mutual information between modalities, while LISA does not, suggesting that its language decomposition may lead to semantic information loss. Nevertheless, we consider our method orthogonal and potentially complementary to language-based approaches. In future work, we plan to explore integration with more advanced language decomposition methods.\n>Training cost of this method\n\nOur method is designed to be lightweight, using a small visual encoder and shallow Transformer blocks for dynamic reasoning and action prediction. To assess training cost, we compare with LISA (Transformer) and SkillDiffuser (Diffusion) under identical A800 GPU settings. We report parameter count and GPU memory usage (batch size 64, Lorel Sawyer). As shown in the table, Ours balances computational cost and success rate.\n-|Trainable Params(M)|GPU Memory(MiB)|SR\n-|-|-|-\nLISA|7.52|690|40%\nSkillDiffuser|60.29|1136|43%\nOurs|7.84|854|53.6%", "claims_and_evidence": "The thesis is supported by compelling evidence", "methods_and_evaluation_criteria": "The method proposed in this paper is suitable for this problem.", "theoretical_claims": "This paper has less theoretical arguments and more descriptive formulas.", "experimental_designs_or_analyses": "1. The imitation learning baselines used for comparison are somewhat outdated. Could the authors compare their method with more recent approaches from the past two years?  \n2. Are there any directly comparable methods? The Multimodal Alignment Methods and Language-Decomposed Methods serve as indirect baselines, which may not fully demonstrate the superiority of the proposed method. \n3. In Table 1, for some tasks (e.g., \"move black mug right\"), the performance gap is quite large compared to the best results. I suggest the authors analyze the reason for this discrepancy.", "relation_to_broader_scientific_literature": "Previous methods primarily addressed the gap between the simplicity of language abstraction and the complexity of video from a linguistic perspective. This paper, however, approaches the problem from the video perspective, offering a new viewpoint.", "essential_references_not_discussed": "The citations are fairly comprehensive, but there is a lack of comparison with the latest imitation learning methods.", "other_strengths_and_weaknesses": "The focus of this paper on the video component is quite innovative, and the writing is clear. The experiments were conducted in both virtual and real-world scenarios, and they are thorough. However, some methodological details and experimental results lack explanation. See the Question section for details.", "other_comments_or_suggestions": "See Question for details.", "questions_for_authors": "1. In Section 3.3, how does the Hybrid Assignment distinguish the early stages of training, and how is the transition implemented specifically?\n2. In Section 3.3, how are the historical frame sequences, historical action sequences, and predicted future dynamic representations input into the action transformer, and how are they fused?\n3. This paper focuses on abstracting information from videos. Have the authors attempted to integrate their approach with methods that primarily focus on language? If so, what were the results?.\n4. I am curious about the training cost of this method. Could the authors provide more details on this?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kir1Ttjgod", "reviewer": "Reviewer_fGaZ", "summary": "This paper proposes a novel method to leverage video data for decision making. To address the gap between abstract language and complex video, the paper proposes to learn abstract dynamic representations for video, rather than making language more detailed. The dynamic representation is learned by assigning higher score for key frames that capture significant spatiotemporal patterns. Based on learned dynamic representations, they predict the future dynamics, and then uses the predicted dynamics to infer the corresponding action sequence.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer fGaZ for the valuable feedback. Below, we respond to each comment and will revise the paper accordingly.\n>Works like LAPO, LAPA, and IGOR use video to learn latent actions for decision-making. PIDM relates to future prediction.\n\nThese works, like ours, target video-based decision-making. We briefly summarize the differences here and will provide a detailed comparison in the revised version.\nLAPO [ICLR2024] enables learning latent-action policies from raw video via consistency objectives, which can be quickly fine-tuned into expert-level policies. LAPA [ICLR2025] extends this by incorporating language. In contrast, our method avoids pretraining and latent action modeling, and instead learns video dynamics to address the modality gap through end-to-end training. IGOR [arXiv:2411.00785] learns a shared latent space for cross-embodiment transfer, while our method models dynamics within a single embodiment.\nPIDM [ICLR2025] predicts future states and feeds them into an inverse dynamics model to couple perception and control. Our method instead focuses on “what matters” rather than pixel-level predictions.\n\n>Comparation with MotoGPT,PIDM\n\nMotoGPT [arXiv:2412.04445] and PIDM are relevant as both leverage video and use Transformer-based architectures for policy learning. MotoGPT follows a three-stage pipeline: latent motion token modeling, generative model pretraining, and collaborative fine-tuning. PIDM adopts an end-to-end approach that predicts actions from forecasted states, coupling perception and control. \nGiven the structural similarity and its end-to-end design like ours, we compare with PIDM on the Franka Kitchen. For fairness, we use the same image and language encoders trained from scratch. A detailed comparison with MotoGPT will be included in the final version.\n\n-|Success Rate\n-|-\nPIDM|36.77%\nOurs|39.81%\n\n>Training details in appendix\n\nWe will include training details such as the image encoder, learning rate, epochs, batch size, and hyperparameters.\n>How FrameScorer learns to find keyframes from language with limited data? Can it extend with larger training datasets and pretrained visual representations?\n\n**Reasons for finding key frames with limited data**\n- Sub-task repetition: Many instructions share similar sub-tasks, providing multiple observations of the same semantic goal across different trajectories.\n- Multiple forms of weak supervision: FrameScorer is guided by two information-rich signals: focusing on semantically relevant frames and visually salient ones.\n\n**Regarding the scale of Franka Kitchen dataset**\n\nIn our main experiments, we trained the model with 25 trajectories per instruction (~300 video-action pairs each), constrained by computational resources. We anticipated the impact of dataset size and included an ablation study (Appendix, Fig.12) showing that our method scales well. Increased trajectory diversity helps the model better capture video-language patterns and improves performance.\n\n**Regarding pre-trained visual representations**\n\nWe had similar thoughts with replacing our lightweight visual encoder with the large-scale pretrained R3M(frozen during training; Appendix Table 7), but observed a performance drop. We attribute this to (1)modality misalignment—R3M’s CLIP-style modeling struggles to bridge language–vision gaps, and (2)domain gap—R3M is trained on human egocentric videos, which differ from our robotic tasks. Nonetheless, we see strong potential in pretrained models and are exploring adapter-based fine-tuning for better integration.\n>Common frameworks typically use LLMs to decompose tasks or letting the VLA predict the next sub-task instruction? Which is better compared to yours?\n\nThe methods you described often follow a language-centric paradigm, whereas our approach is video-centric, offering a complementary perspective. Language-centric methods provide modularity and interpretability but depend on accurate sub-task decomposition, which can cause cascading errors in open-ended or ambiguous tasks. Our method models task progression implicitly by capturing temporal patterns in video conditioned on the language instruction. We see combining both approaches as a promising direction for future work.\n>It is possible to split the language instruction into sub-instructions, and learn the Framescorer by matching them to dynamic features?\n\nInterestingly, this suggestion overlaps with a direction we explored by integrating our method with LISA, which decomposes language instructions into sub-instructions, corresponding to skills and matched to video dynamics (Fig.8). However, the integration did not improve performance. Mutual information analysis showed consistently low correlation between the decomposed language and video features by LISA, suggesting that the decomposition introduced noise or mismatches. Nonetheless, we believe this remains a promising direction and plan to explore stronger decomposition models or structured task planners in future.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No.", "experimental_designs_or_analyses": "For methods that predicts latent dynamic representations, the method could only be evaluated by the final performance. Because it is hard to say if the predicts latents are good or not. The most impressive results to me, is the learned key frames in Figure 5. Given that the authors didn't use pre-trained visual representations for image encoder, it is quite impressively that FrameScorer could find key frames corresponds to the language instructions, especially given the training dataset is not very large (can you describe the size of training data on Franka Kitchen?).  Is this because there are many repeated language instructions and repeated key frames so that the model learns to find the correspondence automatically? Do you think the success can be extended to more diverse language instructions with larger training dataset and even with pre-trained visual representations?", "relation_to_broader_scientific_literature": "The key contribution of this paper is a novel idea to make video representations more abstract, in order to address the mismatch between abstract language instruction and complex/detailed video data in methods that use video for learning decision making ability.", "essential_references_not_discussed": "There is a line of related works compressing visual changes into latent actions and then predicting latent actions, which also use video for pre-training decision making ability, should be discussed here. \n\nLearning to Act without Actions\nLAPA: Latent Action Pretraining from Videos\nIGOR: Image-GOal Representations Atomic Control Units for Foundation Models in Embodied AI\n\nRegarding predicting future frames, here is another work that is related.\n\nPredictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation", "other_strengths_and_weaknesses": "it could be better if the paper compare itself with motoGPT, which leverages video to pre-train decision making ability. PIDM also seems to be very related.", "other_comments_or_suggestions": "It would be better to present more training details in the appendix, for example, the image encoder used in the paper, or the learning rate, training epochs, etc.", "questions_for_authors": "I am curious how FrameScorer learns to find key frames corresponds to the language instructions, especially given the training dataset is not very large (can you describe the size of training data on Franka Kitchen?).  Is this because there are many repeated language instructions and repeated key frames so that the model learns to find the correspondence automatically? Do you think the success can be extended to more diverse language instructions with larger training dataset and even with pre-trained visual representations?\n\n\nThe setting seems a little bit strange to me, because the language instructions in the example (Fig1) and demo (Fig 5) both contain several sub tasks/instructions. And it seems the methods learns information about which sub-tasks the agent is current in, which sub-tasks are the  next to perform. However, in the most popular framework, we usually use LLM to decompose tasks into several sub-tasks, and feed the sub-tasks one by one into a VLA model. I am curious to learn what is the difference between the proposed method and feeding VLA with the current sub-task instructions, or taking one step further, ask VLA to predict the next sub-task instructions. Which one do the authors think would be better?\n\nMoreover, do you think it is possible to split the language instruction into several sub instructions, and learn the Framescorer by matching the correspondence between dynamic features and sub-task instructions?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "80CZb9gDzo", "reviewer": "Reviewer_mTuA", "summary": "This paper proposes to encode the manipulation video into `\"dynamic representation\" by assigning weight to frames. Leveraging this representation, future states are predicted, which are then used to output the action for robot control. The weight of each frame are determined by the variance and similarity between images and language. This approach is evaluated on two manipulation dataset and one navigation dataset.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate Reviewer mTuA’s recognition of the novelty. Please find our responses to each comment below.\n>Seen manipulation tasks during training;lacks evaluation on unseen tasks for generalization.\n\n- Randomized initializations within seen tasks are a standard evaluation protocol. In Table 1, tasks are fixed across train/test, but random initial states introduce diverse, unseen configurations, allowing evaluation of generalization to state and visual variations. Baselines follow the same setup.\n- Our paper included evaluations on unseen tasks.\n  - Instruction generalization(Table 2): Testing on paraphrased, unseen commands.\n  - Cross-task transfer(Table 4): Testing on more complex, unseen tasks.\n  - Compositional generalization(Table 6): Testing on novel task combinations.\n\n>2D navigation limitation.\n\nTo complement this, we extended experiments to iTHOR, which features realistic indoor navigation scenes. Results for other methods (AVDC, GVMA) are taken from [1];both belong to the line of learning to act from video.\n-|Kitchen|Living Room|Bedroom|Bathroom|Overall\n-|-|-|-|-|-\nAVDC|12.2|13.9|26.7|6.1|14.7\nGVMA[1]|48.3|42.7|51.0|52.7|48.7\nOurs|55.0|38.3|78.3|41.7|53.3\n\n[1] Grounding Video Models to Actions through Goal Conditioned Exploration.ICLR2025\n>Related work clarity.\n\nWe will clarify these connections in the revision. Unlike methods that model the environment or generate future frames, our approach predicts video dynamics, making it more suitable for long-horizon tasks. It also avoids large-scale pretraining, learning goal-conditioned representations for control.\n>The claim that \"a single language instruction can correspond to multiple videos\" motivates abstracted features from video, but video generation models can model the randomness of video given the text condition. The benefit of using abstracted features need further justification.\n\n- Our goal is to bridge the gap between abstract language and detailed video, which becomes more pronounced when one language instruction maps to multiple executions. We focus on mitigating the modality gap by learning compact dynamic representations that capture what matters most in a video, making ours fundamentally different from modeling trajectory diversity like video generation models.\n- As for the benefit of video abstraction, a concurrent vision-language study[2] shows that modality gaps—like the asymmetry between image and text in CLIP—negatively impact downstream performance. While their work is analytical, we build on similar insights and offer a practical solution in the embodied setting.\n[2]Two Effects,One Trigger.ICLR2025.\n\n>Abstraction vs. Transformer attention.\n- Explicit Inductive Bias vs. Data-Driven Attention. Unlike data-driven attention, our method introduces an explicit bias toward semantically relevant and visually salient frames.\n- Lightweight Structure. FrameScorer is a simple 2-layer MLP, lighter than Transformer-based alternatives and better suited for long-horizon tasks.\n- Empirical Validation.\n\n-|SR\n-|-\nReplace Abstraction with Attention|33.93%\nOurs|39.81%\n>Inference-time video input and use of history frames.\n\nOur method uses an online inference setup, where only current and past frames are available. During testing, frames are collected incrementally for real-time decision-making, consistent with standard practice in prior work.\n>The designed consistency loss for the abstraction directly use cosine similarity(CS) of the image features and text features. How to ensure they are aligned in the same feature space? \n\n- We clarify that the consistency loss is computed between abstracted video dynamics and language embeddings, rather than between raw image and text features. Instead of forcing alignment between inherently mismatched modalities, we introduce an abstraction module as a bridge.\n- While CS is a standard metric for cross-modal alignment, we understand the reviewer’s concern. Therefore, we estimate MI to capture global statistical dependency in the complex, long-horizon Franka Kitchen. The higher MI between abstracted dynamics and language, compared to raw pairs, further supports our approach.\n-|MI\n-|-\nDynamic↔Language|0.058\nVideo↔Language|0.011\n\n>Use of pretrained model.\n\nWe uses a frozen DistilBERT for language and no large-scale pre-trained visual features.\n>Clarification on higher similarity indicate higher importance.\n\nWe understand the reviewer’s concern, and we clarify that our method does not assume that higher similarity directly indicates frame importance. The semantic consistency loss is applied over the entire dynamic representation sequence and the language embedding, encouraging global task relevance rather than relying on per-frame similarity. Importantly, we avoid strong frame-level assumptions. Instead, frame importance is inferred contextually, guided by consistency and saliency losses, enabling the model to focus on what truly matters—not just what appears similar.\n>Missing period.\n\nWe’ve added the missing period.", "claims_and_evidence": "The main claim of this paper is the effectiveness of \"video dynamic abstraction\" in bridging the language instruction and video. This is supported by the comparison with baselines on three benchmarks and ablations.", "methods_and_evaluation_criteria": "The evaluation is thorough with different simulation benchmarks and real-world experiments. The method could fulfill the language-conditioned robot control problem.", "theoretical_claims": "Do not apply.", "experimental_designs_or_analyses": "This method is evaluated on two manipulation benchmarks and one navigation benchmark as well as 5 real-world tasks.\n\nMy major concern is about the setting in manipulation. It seems that all testing tasks are seen in the training set (line 570 and 579). Considering the language-conditioned manipulation setting, the author should evaluate the method on unseen tasks to validate the generalization ability. \n\nBesides, the navigation benchmark is simple 2D setting, which might fail to effectively reflect the performance of the proposed method.", "relation_to_broader_scientific_literature": "It could be related to text-conditioned manipulation. Specifically, it is related to model-based manipulation learning, robot control with video generation model, and visual representation learning for robotics.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "### Strengths\n\n1. The idea of abstracting dynamic information from video with key frame selection instead of generating video from text is novel.\n\n### Weaknesses\n\n1. The design of the proposed approach is not well-motivated. Firstly, the authors claim that \"a single\nlanguage instruction can correspond to multiple videos\", which lead to the design of abstracting dynamic representation from video. However, current video generation models like video diffusion could model the randomness of video given the text condition. The benefit of using abstracted features need further justification. Besides, the design of the \"Video Dynamic Abstraction\" module aims to assign different weight to frames. But transformer with attention mechanism could already perform weighted aggregation, which is more flexible than the hand-crafted losses for weight learning.\n\n2. The setting and design need further justification. I wonder the inference setting of this model since it requires a video input. What's the video input during test time? The history frames? Following the question, the designed consistency loss for the abstraction process directly use the cosine similarity of the image features and text features. How to ensure they are aligned in the same feature space? Is there a pre-trained model used? Furthermore, the assumption that higher similarity indicate higher importance need clarification.\n\n3. The evaluation seems to be problematic as written above.", "other_comments_or_suggestions": "### Minor\n\n1. More evaluation on unseen tasks and on real-world tasks could improve the quality of this paper. The navigation tasks should be in 3D robot navigation space. \n\n2. [Minor] There should be period after the bold text (e.g., line 184).", "questions_for_authors": "Please see the weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received overall positive reviews during the initial review period. Reviewers found the core idea of abstracting dynamic information from video, rather than making language more detailed, to be novel and well-motivated. The proposed framework, DynaMind, introduces a video-based abstraction pipeline for decision-making that bridges the gap between abstract language and complex video signals. Reviewers appreciated the strong experimental validation across both simulated and real-world tasks and the rich set of ablation studies. However, several concerns were raised, such as questions around the clarity of the training pipeline, limited evaluation on unseen tasks, and the simplicity of the navigation environment. Reviewers also suggested comparisons to more recent baselines and requested additional training and model details. The author's response effectively addressed many of these comments. 

The AC concurs with the reviewers that this paper presents a creative and empirically solid contribution and recommends acceptance. The authors are encouraged to incorporate the details from the rebuttal discussion into the camera-ready revision.
Decision: Accept (poster)