Video Generation with Learned Action Prior

## Abstract

Abstract Stochastic video generation is particularly challenging when the camera is mounted on a moving platform, as camera motion interacts with observed image pixels, creating complex spatio-temporal dynamics and making the problem partially observable. Existing methods typically address this by focusing on raw pixel-level image reconstruction without explicitly modelling camera motion dynamics. We propose a solution by considering camera motion or action as part of the observed image state, modelling both image and action within a multi-modal learning framework. We introduce three models: Video Generation with Learning Action Prior (VG-LeAP) treats the image-action pair as an augmented state generated from a single latent stochastic process and uses variational inference to learn the image-action latent prior; Causal-LeAP, which establishes a causal relationship between action and the observed image frame at time t 𝑡 t italic_t , learning an action prior conditioned on the observed image states; and RAFI, which integrates the augmented image-action state concept into flow matching with diffusion generative processes, demonstrating that this action-conditioned image generation concept can be extended to other diffusion-based models. We emphasize the importance of multi-modal training in partially observable video generation problems through detailed empirical studies on our new video action dataset, RoAM.

## 1 Introduction

Video prediction is a valuable tool for extracting essential information about the environment and can be utilized by other learning frameworks such as motion planning algorithms Hafner et al. (), and autonomous navigation and traffic management Claussmann et al. (); Bhattacharyya, Fritz, and Schiele (). However, the complex interactions among different moving objects in a scene present significant challenges for long-term video prediction Finn, Goodfellow, and Levine (); Finn and Levine (); Mathieu, Couprie, and LeCun (); Villegas et al. (); Gao et al. (); Villegas et al. (); Ebert et al. (); Sarkar, Ghose, and Bala (). Over the last decade, various works such as Srivastava, Mansimov, and Salakhudinov (); Oh et al. (); Vondrick, Pirsiavash, and Torralba (); Finn, Goodfellow, and Levine (); Mathieu, Couprie, and LeCun (); Villegas et al. (); Wichers et al. (); Oprea et al. (); Liang et al. (); Ebert et al. () have tried to address this problem employing recurrent deep architectures and using concepts such as optical flow decomposition to adversarial training for generating high-quality output.

Denton and Fergus (); Babaeizadeh et al. (); Lee et al. () have shown that video can be modelled as a latent stochastic process and variational inference can lead to reliable high-quality prediction on human action datasets such as KTH Schuldt, Laptev, and Caputo (), Human3.6M Ionescu et al. () and robotic datasets such as BAIR Robot Push Ebert et al. (). However, these datasets only consider a static camera and do not capture the complexities arising from a moving camera. Villegas et al. () showed that scaling deep stochastic variational architectures to a higher dimensional latent space can lead to better quality predictions in moving camera datasets such as KITTI. Gao et al. () disentangled motion-specific flow propagation from motion-agnostic generation of pixel data for higher fidelity. With the success of transformer models in natural language processing, there is growing interest in designing efficient visual transformers Dosovitskiy et al. () for video prediction tasks Ye and Bilodeau (); Gao et al. (). Recent works in diffusion-based video generation and prediction models Ho et al. (); Mei and Patel (); Davtyan, Sameni, and Favaro (); Harvey et al. (); Höppe et al. () have shown great promise in generating and predicting long-term high-fidelity videos.

In scenarios characterized by partial observability, especially when a camera is mounted on a moving platform, the captured image frames are influenced not only by the inherent scene dynamics but also by the actions of the mobile platform itself. This prevalent situation is particularly relevant in domains such as autonomous vehicles and mobile robotics. Numerous datasets, including KITTI Geiger et al. (), KITTI-360 Liao, Xie, and Geiger (), A2D2 et. al (), and the pedestrian video dataset by Caltech Dollar et al. (), have already illustrated the significance of this scenario in the context of autonomous cars. More recently, Sarkar et al. Sarkar et al. () have underscored the importance of modelling partially observable video data within the domain of indoor robotics, as demonstrated by their RoAM dataset. This further emphasizes the growing relevance of addressing partial observability in various robotic applications. However most of the past works like Sarkar, Ghose, and Bala (); Villegas et al. (); Gao et al. (, ) did not explicitly condition the generated future frames on the actions or movement of the recording camera even though they all highlighted the increased level of complexities arising from the interactions between the movement of the camera sensor and recorded image frames. This limitation can also be attributed, in part, to the absence of synchronized control action or actuator data in partially observable datasets like KITTI Geiger et al. () and A2D2 et. al (). These datasets do not provide the concurrent recording of steering actions taken by autonomous cars while capturing video data.

However, the recently introduced Robot Autonomous Motion or RoAM Sarkar et al. () dataset includes timestamped and synchronized control action data of the recording robotic agent along with stereo image pairs when the autonomous robot explored different indoor spaces capturing different human actions. This dataset presents the opportunity for vision scientists to design generative models that can be explicitly conditioned and model the on the robot action data in the context of video generation. Modelling this dynamic interaction is crucial for creating more accurate and realistic video predictions, especially in applications such as autonomous driving and robotic navigation.

In this paper, we propose a mathematical framework where the camera motion is considered as an extended part of the image state and we design generative models that can approximate the latent stochastic process with a learned image-action prior. We present 3 distinct models: (i) SVG-LeAP, (ii) Causal-LeAP and (iii) RAFI video diffusion model. SVG-LeAP and Causal-LeAP are both variational generative frameworks, with the key distinction being that Causal-LeAP considers a causal relationship between the action taken at time $t$, $a_{t}$, and the image observed, $x_{t}$. SVG-LeAP is based on the SVG-lp Denton and Fergus () model with the modification of considering the image-action pair as the joint observed state of the system instead of just the image. With RAFI we extend the concept of a joint image-action system state to flow matching in diffusion generative processes. RAFI is built on the sparsely conditioned flow matching model of RIVER Davtyan, Sameni, and Favaro ().

## 2 Prior works

Let us assume that $x_{t}$ is the current image frame from a sequence of frames $x_{1:T-1}$ from video data of dimension $d=[i_{h}\times i_{w}\times 3]$. Over the past decade, numerous mathematical frameworks have been proposed to model the generation process of $x_{t}$. In their seminal work, Denton and Fergus () introduced the stochastic learned prior model (SVG-lp), which has gained widespread acceptance and application within the computer vision community. This framework posits that a sequence of image frames from a video is generated from a latent Gaussian distribution. The latent distribution is learned through a variational training and inference paradigm using a set of observed image sequences. Mathematically here the current image frame is predicted as $\tilde{x}_{t}$ conditioned on the past observed frames $x_{1:t-1}$ and a latent variable $z_{t}$. Given that at the time of prediction $p(z_{t})$ is unknown, it is learnt with a posterior distribution $p_{\theta}(z_{t}|x_{1:t})={\mathcal{N}}(\mu_{\theta}(x_{1:t}),\sigma(x_{1:t}))$ approximated by a recurrent network parameterised by $\theta$. The sampled variable $z_{t}$ is then used to generate the current image frame $x_{t}$ conditioned on the past observed frames $x_{1:t-1}$. Denton proposed two methods for learning $p_{\theta}(z_{t}|x_{1:t})$: (i) with a fixed Gaussian prior and (ii) with a companion prior model $p_{\phi}(z_{t}|x_{1:t-1})$ and minimising the KL divergence loss between the two.

This learned prior model has subsequently been utilized in various video generation models, such as those by Villegas et al. (); Chatterjee, Ahuja, and Cherian () in recent years. While effective, this model does not explicitly address the integration of camera motion or other modalities of the video data generation process into the architecture. Camera motion plays a crucial role in the video generation process, especially when the camera is moving or mounted on a moving platform like a car or a robot. Villegas et al. () showed that with a significantly larger parametric space, SVG-lp can effectively generate and predict future image frames when tested on partially observable video datasets like KITTI, where the camera is mounted on a car. However, recent works, such as those by Sarkar et al. (), have demonstrated that long-term video prediction processes can be enhanced by explicitly conditioning the predicted frames on the motion of the camera.

Recently, diffusion models Ho et al. (); Davtyan, Sameni, and Favaro (); Voleti, Jolicoeur-Martineau, and Pal (); Song et al. (); Xu et al. (); Höppe et al. () have garnered attention from the computer vision community due to their capacity to generate and forecast high-fidelity video sequences. Rooted in the concepts of diffusion processes Ho et al. (), these models iteratively refine noisy data to produce high-quality image frames.

## 3 Action conditioned video generation

We introduce three distinct action-conditioned video generation models. Our first two Learned Action Prior or LeAP models are stochastic video generation frameworks in which the action or camera movement is learned through a latent Gaussian distribution. However, the methods by which these action priors are learned differ significantly from each other, based on distinct sets of assumptions regarding the interaction between action and video. With the third model, we introduce RAFI, our Random Action-Frame Conditioned Flow Integrating video generation model, based on the RIVER diffusion framework by Davtyan, Sameni, and Favaro (). RAFI showcases how camera actions can be seamlessly integrated into Flow Matching Lipman et al. () and the diffusion process to enhance video prediction quality.

In this paper, we denote the action of the robot or the platform on which the camera is mounted at timestep $t$ by $a_{t}\in{\mathbb{R}}^{n}$, where $n$ is the dimension of the action or actuation space of the robot/platform. We also assume actions are normalised meaning $a_{t}\in[0,1]$.

Figure: (a)
Refer to caption: extracted/5681396/figure/vleap_state_flow.png

### 3.1 Video generation with learnt action prior

Our first model Video generation with learnt action prior or VG-LeAP is built on the principles of stochastic video generation in Denton and Fergus (). However, unlike Denton and Fergus () where only images were considered as the observed state of the stochastic process, we introduce the notion of image-action pair $(x_{t},a_{t})$ as an augmented state of the extended stochastic process that models the image frames as well as the action of the robot. In scenarios where the camera is moving, the observed image frames are influenced by the past actions or movements of the camera. Additionally, in many cases, the future actions of a robotic agent or a car (on which the camera is mounted) depend on the images observed, particularly when obstacle avoidance modules are integrated into the platform’s motion planner. This interdependence between the image and action is also referred to as the partial observability problem in video prediction literature Villegas et al. (); Sarkar, Ghose, and Bala (). Thus modelling this process with the notion of system or robot action as a part of an extended state of the process provides a clear way of encapsulating these interdependent dynamics.

We assume that the extended image-action pair $\chi_{t}=(x_{t},a_{t})$ is generated from a latent unknown process $p(z_{t})$ of variable $z_{t}$ whose posterior is approximated with a recurrent neural architecture of parameter $\theta$ in the form $p_{\theta}(z_{t}|x_{1:t},a_{1:t})$. In order to learn this posterior distribution we employ a similar variational architecture as that of SVG-lp, however in our case we use the notion of extended state instead of just the image frames. We use the reparameterization trick from variational inference Kingma and Welling (), to approximate we approximate $p_{\theta}(z_{t}|x_{1:t},a_{1:t})$ as a Gaussion process such that $z_{t}\sim{\mathcal{N}}(\mu_{\theta}(z_{t}|\chi_{1:t-1}),\sigma_{\theta}(z_{t}|
\chi_{1:t}))$ where $\mu$ and $\sigma$ denotes the mean and variance respectively. The state diagram of the learned image-action prior model in Fig 1(a) depicts this relationship between learned latent variable $z_{t}$ and observed image-action pair $(x_{t},a_{t})$ with connecting blue arrows. We also use a recurrent module parameterised by $\phi$ to learn the image-action prior $p_{\phi}(z_{t}|x_{1:t-1},a_{1:t-1})$ to use during inference when the current image $x_{t}$ and action $a_{t}$ is not available. The architecture of the network can be expressed as follows and is pictorially represented in Fig 1(b):

$$ $x_{t}\overset{Enc}{\longrightarrow}h_{t},\hskip 20.00003pta_{t}\overset{Enc}{ \longrightarrow}\alpha_{t}$ (1) $$

$$ $\mu_{\theta}(t),\sigma_{\theta}(t)=\overset{\curvearrowleft}{RNN}_{\theta}(h_{ 0:t},\alpha_{0:t}),\hskip 20.00003ptz_{t}\sim{\mathcal{N}}(\mu_{\theta}(t), \sigma^{2}_{\theta}(t))$ (2) $$

$$ $x_{t-1}\overset{Enc}{\longrightarrow}h_{t-1},\hskip 20.00003pt\tilde{h}_{t}= \overset{\curvearrowleft}{RNN}_{\zeta_{1}}(h_{0:t-1},z_{1:t})$ (3) $$

$$ $a_{t-1}\overset{Enc}{\longrightarrow}\alpha_{t-1},\hskip 20.00003pt\tilde{ \alpha}_{t}=\overset{\curvearrowleft}{RNN}_{\zeta_{2}}(\alpha_{0:t-1},z_{1:t})$ (4) $$

$$ $\tilde{x}_{t}\overset{Dec}{\longleftarrow}\tilde{h}_{t},\hskip 20.00003pt \tilde{a}_{t}\overset{Dec}{\longleftarrow}\tilde{\alpha}_{t}$ (5) $$

In equation 1 we encode image frames to a low dimensional manifold with $h_{t}$ and we map action data to a higher dimensional state of $\alpha_{t}$. These encoded features are then fed to the posterior estimation network for eventual sampling of $z_{t}$ in equation 2. Please note the dependence in $z_{t}$ on the past data $(h_{0:t},\alpha_{0:t})$ arises from the recurrent LSTM components in the posterior network. This same dependence of the predicted image $\tilde{h}_{t}$ and action data $\tilde{\alpha}_{t}$ on the history of observed data $(h_{0:t-1},z_{0:t})$ and $(\alpha_{0:t-1},z_{0:t})$ in equation 3 and equation 4 respectively are modelled with the LSTM components in the image and action predictor networks $\overset{\curvearrowleft}{RNN}_{\zeta_{1}}$ and $\overset{\curvearrowleft}{RNN}_{\zeta_{2}}$ respectively. Finally the generated image $\tilde{x}_{t}$ and action $\tilde{a}_{t}$ are decoded with their respective decoder architectures in equation 5. The action conditioned prior $p_{\phi}(z_{t}|x_{1:t-1},a_{1:t-1})$ is learned as $\mu_{\phi}(t),\sigma_{\phi}(t)=\overset{\curvearrowleft}{RNN}_{\phi}(h_{0:t-1}
,\alpha_{0:t-1})$.

Loss: We use a modified variational lower bound or ELBO loss in equation 6 to train our VG-LeAP architecture.

$$ $\displaystyle\underset{\theta,\phi,\zeta_{1},\zeta_{2}}{\max}$ $\displaystyle{\mathcal{L}}_{\theta,\phi,\zeta_{1},\zeta_{2}}(x_{1,T},a_{1:T})= \sum_{t=1}^{T}\big{[}\mathbb{E}_{p_{\theta}(z_{1:t}|x_{1:t},a_{1:t})}(\text{ln }q_{\zeta_{1}}(x_{t}|x_{1:t-1},z_{1:t})+$ (6) $\displaystyle\beta_{a}\text{ln}q_{\zeta_{2}}(a_{t}|a_{1:t-1},z_{1:t}))-\beta D _{KL}(p_{\theta}(z_{t}|x_{1:t},a_{1:t})||p_{\phi}(z_{t}|x_{1:t-1},a_{1:t-1})) \big{]}$ $$

The first and the third components in equation 6 refer to the widely used reconstruction and KL divergence loss of variational frameworks Denton and Fergus (); Villegas et al. (); Chatterjee, Ahuja, and Cherian (), however, the second expectation term comes as a natural expansion of the extended state of $(x_{t},a_{t})$ that incorporates action $a_{t}$. In equation 6, $q_{\zeta_{1}}(x_{t}|\cdots)$ and $q_{\zeta_{2}}(a_{t}|\cdots)$ represents the likelihood functions of predicting $x_{t}$ and $a_{t}$ by $\overset{\curvearrowleft}{RNN}_{\zeta_{1}}$ and $\overset{\curvearrowleft}{RNN}_{\zeta_{2}}$ respectively and are approximated with the $L_{\text{p}}$ where $p\in\{1,2\}$ norm loses between the ground truth and predicted values. The hyper-parameters $\beta_{a}$ and $\beta$ are selected based on the numerical stability of the training and their selection process is discussed in detail in the supplementary material.

Figure: (a)
Refer to caption: extracted/5681396/figure/causal_leap_state_flow.png

### 3.2 Causal video generation with learned action prior

In our Causal-LeAP or Causal Learned Action Prior model, instead of treating the image-action pair $(x_{t},a_{t})$ as an extended state of the generative process, we adopt a causal approach. We assume a causal relationship between the action, $a_{t}$ taken by the moving platform or robot at time-step $t$ and the observed image frame $x_{t}$. This approach aligns with most motion planning algorithms, which plan actions based on the current observed state, following Markovian models. Consequently, the action taken at time $t$ influences the image frame observed at $t+1$, $x_{t+1}$ and this causal chain continues sequentially. We assume that image $x_{t}$ is generated from a latent unknown process $p(z_{t})$ of variable $z_{t}$ whose posterior is approximated with a recurrent neural architecture of parameter $\theta$ in the form $p_{\theta}(z_{t}|x_{1:t},a_{1:t-1})$ (This is similar to SVG-lp video generator part except for the conditioning of $z_{t}$ on past observed action $a_{1:t-1}$). Action $a_{t}$ is assumed to be generated from the latent process $p(u_{t}|x_{t})$ where we have already observed $x_{t}$. We approximate the posterior of $u_{t}$ with an LSTM module of parameter $\psi$ of the form $p_{\psi}(u_{t}|a_{1:t},z_{1:t})$. The causal relationship between the image latent state $z_{t}$ and the action latent variable is depicted with the blue connecting lines in the state flow diagram in Fig 2(a).

We reparameterize Kingma and Welling (), $p_{\theta}(z_{t}|x_{1:t},a_{1:t-1})$ and $p_{\psi}(u_{t}|a_{1:t},z_{1:t})$ as Gaussion processes such that $z_{t}\sim{\mathcal{N}}(\mu_{\theta}(z_{t}|x_{1:t},a_{1:t-1}),\sigma_{\theta}(x
_{1:t},a_{1:t-1}))$ and $u_{t}\sim{\mathcal{N}}(\mu_{\psi}(u_{t}|a_{1:t},z_{1:t}),\sigma_{\psi}(a_{1:t}
,z_{1:t}))$ respectively. Here $\mu$ and $\sigma$ signify the mean and variance of the distributions. With Causal-LeAP we train two recurrent modules parameterised by $\phi$ and $\varphi$ to learn the image prior $p_{\phi}(z_{t}|x_{1:t-1},a_{1:t-1})$ and causal action prior $p_{\varphi}(u_{t}|a_{1:t-1},z_{1:t-1})$ respectively. $p_{\phi}(z_{t}|\cdots)$ and $p_{\varphi}(u_{t}|\cdots)$ are used at the time of inference when the current image $x_{t}$ and action $a_{t}$ are not available. The architecture of the network can be expressed as follows and is depicted in Fig 2(b):

$$ $x_{t}\overset{Enc}{\longrightarrow}h_{t},\hskip 20.00003pta_{t}\overset{Enc}{ \longrightarrow}\alpha_{t}$ (7) $$

$$ $\mu_{\theta}(t),\sigma_{\theta}(t)=\overset{\curvearrowleft}{RNN}_{\theta}(h_{ 1:t}),\hskip 20.00003ptz_{t}\sim{\mathcal{N}}(\mu_{\theta}(t),\sigma^{2}_{ \theta}(t))$ (8) $$

$$ $\mu_{\psi}(t),\sigma_{\psi}(t)=\overset{\curvearrowleft}{RNN}_{\psi}(\alpha_{1 :t},z_{1:t}),\hskip 20.00003ptu_{t}\sim{\mathcal{N}}(\mu_{\psi}(t),\sigma^{2}_ {\psi}(t))$ (9) $$

$$ $x_{t-1}\overset{Enc}{\longrightarrow}h_{t-1},\hskip 20.00003pt\tilde{h}_{t}= \overset{\curvearrowleft}{RNN}_{\zeta_{1}}(h_{1:t-1},z_{1:t},\alpha_{1:t-1})$ (10) $$

$$ $a_{t-1}\overset{Enc}{\longrightarrow}\alpha_{t-1},\hskip 20.00003pt\tilde{ \alpha}_{t}=\overset{\curvearrowleft}{RNN}_{\zeta_{2}}(\alpha_{1:t-1},u_{1:t})$ (11) $$

$$ $\tilde{x}_{t}\overset{Dec}{\longleftarrow}\tilde{h}_{t},\hskip 20.00003pt \tilde{a}_{t}\overset{Dec}{\longleftarrow}\tilde{\alpha}_{t}$ (12) $$

We encode image frames and actions to $h_{t}$ and $\alpha_{t}$ respectively in equation 7 which is similar to equation 1 in 3.1. The encoded image vectors $h_{t}$ are then fed to the posterior estimation network $\overset{\curvearrowleft}{RNN}_{\theta}$ for the eventual sampling of $z_{t}$ in equation 8. Please note the dependence in $z_{t}$ on the past data $(h_{1:t},\alpha_{1:t-1})$ in equation 8 arises from the recurrent LSTM components in the posterior network and in equation 8 $z_{t}$ does not depend upon $a_{t}$ like in equation 2. Equation 9 encapsulates the causal relationship between $x_{t}$ and $a_{t}$ as the image latent variable is fed to $\overset{\curvearrowleft}{RNN}_{\psi}$ to generate $u_{t}$. The predicted image $\tilde{h}_{t}$ and action data $\tilde{\alpha}_{t}$ are dependent on the history of observed data $(h_{1:t-1},z_{1:t},\alpha_{1:t-1})$ and $(\alpha_{1:t-1},u_{1:t})$ in equation 10 and equation 11 respectively and these dependencies are modelled with LSTM modules in the image and action predictor networks $\overset{\curvearrowleft}{RNN}_{\zeta_{1}}$ and $\overset{\curvearrowleft}{RNN}_{\zeta_{2}}$ respectively. Finally the generated image $\tilde{x}_{t}$ and action $\tilde{a}_{t}$ are decoded with their respective decoder architectures in equation 12. The action conditioned image prior $p_{\phi}(z_{t}|x_{1:t-1},a_{1:t-1})$ is learned as $\mu_{\phi}(t),\sigma_{\phi}(t)=\overset{\curvearrowleft}{RNN}_{\phi}(h_{1:t-1})$ and the causal learned action prior $p_{\varphi}(u_{t}|a_{1:t-1},z_{1:t})$ is learned as $\mu_{\varphi}(t),\sigma_{\varphi}(t)=\overset{\curvearrowleft}{RNN}_{\varphi}(
\alpha_{1:t-1},z_{1:t-1})$.

Loss: We derive our variational lower bound or ELBO loss in equation 13 to train our Causal-LeAP architecture.

$$ $\displaystyle\underset{\theta,\phi,\psi,\varphi,\zeta_{1},\zeta_{2}}{\max}$ $\displaystyle{\mathcal{L}}_{\theta,\phi,\psi,\varphi,\zeta_{1},\zeta_{2}}(x_{1 ,T},a_{1:T})=\sum_{t=1}^{T}\big{[}\mathbb{E}_{p_{\theta}(z_{1:t}|x_{1:t})} \text{ln}q_{\zeta_{1}}(x_{t}|x_{1:t-1},z_{1:t},a_{1:t-1})-$ (13) $\displaystyle\beta D_{KL}(p_{\theta}(z_{t}|x_{1:t})||p_{\phi}(z_{t}|x_{1:t-1}) )+\beta_{a}\mathbb{E}_{p_{\psi}(u_{1:t}|z_{1:t},a_{1:t})}\text{ln}q_{\zeta_{2} }(a_{t}|a_{1:t-1},u_{1:t},z_{1:t})$ $\displaystyle\hskip 120.00018pt-\gamma D_{KL}(p_{\psi}(u_{t}|a_{1:t},z_{1:t})| |p_{\varphi}(u_{t}|a_{1:t-1},z_{1:t}))\big{]}$ $$

In equation 13, the first two components represent the conventional reconstruction and KL divergence loss components of variational learning. The third and fourth components come from maximizing the log-likelihood of $p(a_{t}|x_{t})$ or $\text{ln}p(a_{t}|x_{t})$.
The likelihood functions $q_{\zeta_{1}}(x_{t}|\cdots)$ and $q_{\zeta_{2}}(a_{t}|\cdots)$ for predicting $x_{t}$ and $a_{t}$ by $\overset{\curvearrowleft}{RNN}_{\zeta_{1}}$ and $\overset{\curvearrowleft}{RNN}_{\zeta_{2}}$ respectively are approximated with the $L_{\text{p}}$ where $p\in\{1,2\}$ norm loses between the ground truth and predicted values.
The hyper-parameter $\gamma$ relating to the KLD loss associated with the action prior function is chosen according to the numerical stability of the problem. In this case, the action predictor is a much smaller model compared to the image predictor and thus tends to converge much quicker which can lead to numerical instability in case of large learning rates or very small $\beta$ values. The selection criteria for all the three hyper-parameters $\beta,\beta_{a}$ and $\gamma$ are discussed in the supplementary.

### 3.3 Random Action-Frame Conditioned Flow Integrating video generation model

Our diffusion model Random Action-Frame Conditioned Flow Integrating video generator or RAFI is based on the sparsely conditioned flow matching model of RIVER by Davtyan, Sameni, and Favaro (). Like RIVER, we also encode our image states in the latent space of a pre-trained VQGAN Esser, Rombach, and Ommer (). However, unlike RIVER, we then join the latent image state $z_{t}$ from the VQGAN network with the action vectors to generate the extended image-action state $\tilde{z}_{t}$ as shown in the fourth step in Algo. 1. Following $\tilde{z}_{t}$, we follow the steps similar to RIVER to train the flow vector regressor Lipman et al. () using gradient descent. The step by step algorithm for RAFI is given in Algo. 1 and for more details about the implementation of the algorithm please refer to our Models and Training section of supplementary.

Figure: Algorithm 1 Training Procedure for RAFI

## 4 Dataset and Experiments:

### 4.1 RoAM dataset:

RoAM or Robot Autonomous Motion dataset is a synchronised and timestamped image-action pair sequence dataset, recorded with a Turtlebot3 Burger robot with a Zed mini stereo camera. The dataset was first introduced by Sarkar et al. () to establish the connection between the generated image frames and the robot action data. RoAM is recorded indoors capturing corridors, lobby spaces, staircases, and laboratories featuring frequent human movement like walking, sitting down, getting up, standing up, etc. However, the original RoAM in Sarkar et al. () only contained 25 long video sequences out of which only 20 were used for training. This dataset was very small for training stochastic frameworks such as SVG or RIVER. So we have recorded another 25 long video frames in various indoor environments with varying lighting conditions. Now the dataset is segregated into 45 long training video sequences and 5 sequences are kept for testing. Using these 45 video sequences we have used the Tensorflow Abadi et al. () Dataset API to generate 3,07,200 video sequences of length 40 and image size $64\times 64\times 4$. It also contains the corresponding action values from the robot’s motion to capture the movement of the camera. The dimension of the action data in RoAM is $m=2$ featuring forward velocity along the body $x$-axis and turn rate about the body $z$-axis of the robot’s centre of mass.

More details on the process of RoAM dataset recording and our training pipeline are discussed in the Dataset section of our supplementary file.

### 4.2 Experimental Setup:

Out of the 25 frames in each sequence, we used 5 random frames to condition our networks VG-LeAP, Causal-LeAP and SVG on the past data. In the case of RIVER and RAFI we only condition them on the past two frames and in all 5 models we generated the next 10 frames in the future during training. In order to test the networks, we have created 1024 randomly generated video sequences of length 40 from the origin 5 test sequences in RoAM and tested all the 5 networks against the quantitative performance metrics such as: Peak Signal-to-Noise Ratio (PSNR), VGG16 Cosine Similarity Simonyan and Zisserman (), and Fréchet Video Distance (FVD) Unterthiner et al. () and Learned Perceptual Image Patch Similarity or LPIPS metricZhang et al. (). Among these metrics, FVD is based on the Fréchet Inception Distance (FID) that is commonly used for evaluating the quality of images from generative frameworks and measures the spatio-temporal distribution of the generated videos as a whole, with respect to the ground truth. We use VGG16 cosine similarity index, LPIPS and PSNR for frame-wise evaluation. The VGG16 cosine similarity index uses the pre-trained VGG16 network Simonyan and Zisserman () to measure the cosine similarity between the generated and ground truth video frames. Recently perceptual similarity metric LPIPS Zhang et al. () which uses pretrained AlexNet as its image feature generator, has emerged as a popular measure Franceschi et al. () for its human-like perception of similarity between two image frames. In case of VGG16 Cosine Similarity and PSNR values, closer resemblance to the ground truth images is indicated by higher values whereas in LPIPS and FVD scores, superior performance is associated with lower values.

## 5 Results and Discussion:

During inference, we tested all our 6 models: VG-LeAP, Causal LeAP, RAFI along with SVG-lp, RIVER and ACPNet on predicting 20 future frames conditioned on the past 5 image frames and the plots for LPIPS, VGG Cosine Similarity and PSNR are shown in Fig 3. From all the figures in Fig 3, we can easily see that our Causal-LeAP and VG-LeAP models easily outperform the SVG-lp model on RoAM dataset. While all these three models Causal-LeAP, VG-LeAP and SVG-lp share similar image predictor architectures, it can be easily concluded that the improved behaviour is a direct result of modelling the combined image-action dynamics in the case of VG-LeAP and Causal-LeAP. Comparing the behaviour between SVG and VG-LeAP, where both the networks share almost identical architecture and size of the parametric space, we can very clearly see that VG-LeAP outperforms SVG in Fig. 3(a), Fig 3(b), and Fig 3(c). The mean FVD score of VG-LeAP is around 481.15 which is better than the 539.29 from SVG.

In between VG-LeAP and Causal-LeAP, Causal-LeAP outperforms VG-LeAP in almost every quantitative metric (Fig. 3) except for FVD score shown in Table 1 (Causal LeAP has a FVD score of 514.65). In the case of our two diffusion-based models RIVER and RAFI, we can see that both of them initially perform much better than Causal-LeAP and VG-LeAP (Fig 3(a),3(b)), however as time increases their performance gets worse. However, in terms of FVD scores, RIVER and RAFI generate the best results with mean scores of 284.46 and 288.23 respectively (Table 1).

We have also plotted the comparative L2 norm errors in the predicted action data from Causal-LeAP, VG-LeAP and RAFI in Fig 4 and here we see that RAFI performs better than all the other two models when it comes to predicting the forward velocity. However, for turn rate, RAFI does not provide reliable predictions as compared to both Causal-LeAP and VG-LeAP in Fig 4(c). It produces erroneous turn rates which we believe also has an adverse effect in the generated images by RAFI. Since RAFI treats Image-Action as an extended state, the rotation in action also results in generated images that are rotated and thus the prediction accuracy decreases.

We can also observe from Fig 4 that until $t=13$, both VG-LeAP and Causal-LeAP generated similar $\text{L}_{2}$ norm error in forward velocity, however after $t=12$, in the case of VG-LeAP, the action error starts to accumulate exponentially whereas, in case of Causal-LeAP, the action error loss remains almost constant.

The additional KL divergence term from the action network in the loss function for Causal-LeAP also acts as an additional regularize to the causal training process which may lead to numerical instability during training. However, we found keeping a small $\gamma$ value in equation 13 can easily reduce the scenarios. The numerical instability in the training of the Causal-LeAP model also arises from the disproportionate size of the parametric spaces of the Image and the Action prediction networks. Since the Action prediction framework converges faster than the Image prediction module, we found keeping a low $\gamma$ is essential for the training.

In the case of ACPNet which is the only deterministic model in our empirical study, we have found that even though ACPNet initially starts generating good prediction (3(a),3(b)), it quickly suffers from blurring effects that are common in deterministic architectures. The blurring effect is also the reason why ACPNet produces the best PSNR result Zhang et al. (); Franceschi et al. () among all the 6 models. The FVD score for ACPNet is 908 (Table 1).

Additional results on generated raw image frames from all the 6 networks can be found on our project page: [https://meenakshisarkar.github.io/Motion-Prediction-and-Planning/dataset/](https://meenakshisarkar.github.io/Motion-Prediction-and-Planning/dataset/)

Figure: (a)
Refer to caption: extracted/5681396/figure/lpips_paper.png

Figure: (a)
Refer to caption: extracted/5681396/figure/at1_fina.png

**Table 1: FVD Score**
| Model | Score |
| --- | --- |
| Causal-LeAP | 514.65 $\pm$ 3.37 |
| VG-LeAP | 481.15 $\pm$ 2.39 |
| SVG-lp | 539.29 $\pm$ 1.94 |
| RIVER (BEST) | 284.46 $\pm$ 3.21 |
| RAFI | 288.23 $\pm$ 4.39 |
| ACPNET | 908.36 |

## 6 Concusion:

We have presented three new stochastic video generative frameworks based on the mathematical premise of incorporating action into the video generation process. We have also established a causal relationship between the image and camera actions in the partially observable scenarios where the camera is moving with our Causal-LeAP model and have shown with our detailed empirical studies that not only image-action models improve the efficacy of the prediction framework but also provides a way to learn and model the system dynamics by simply observing and modelling the interaction between the image-action pair. Our Causal model learned an action prior conditioned on the latent image state $p_{\varphi}(u_{t}|a_{1:t-1},z_{1:t})$ which has direct applications to the field of robotics and RL. Our model diffusion model RAFI also shows how easily we can extend the concepts of image-action state pair to existing diffusion approaches leading to good results.

## Appendix A Appendix

### A.1 Variational Lower Bound for Video Generation with Learned Action Prior

Here we are trying to maximize the joint likelihood of $(x_{1:t},a_{1:T})$ which is equivalent to maximizing $\text{ln}q_{\zeta}(x_{1:T},a_{1:T})$ or $\text{ln}q_{\zeta}(x,a)$ for better readability. Let’s assume $z=[z_{1},\cdots,z_{T}]$ denotes the latent $z$ variable across all the time-steps and they are independent of each other across time.

$$ $\displaystyle\text{ln}q_{\zeta}(x_{1:T},a_{1:T})$ $\displaystyle\equiv\text{ln}q_{\zeta}(x,a)=\text{ln}\int_{z}q_{\zeta}(x,a|z)p(z)$ (15) $\displaystyle=\text{ln}\int_{z}q_{\zeta}(x,a|z)p(z)\frac{p_{\theta}(z|x,a)}{p_ {\theta}(z|x,a)}$ (16) $\displaystyle=\text{ln}\left(\mathbb{E}_{p_{\theta}(z|x,a)}q_{\zeta}(x,a|z) \frac{p(z)}{p_{\theta}(z|x,a)}\right)$ (17) $\displaystyle\geq\mathbb{E}_{p_{\theta}(z|x,a)}\left(\text{ln}q_{\zeta}(x,a|z) \frac{p(z)}{p_{\theta}(z|x,a)}\right)$ (18) $\displaystyle=\mathbb{E}_{p_{\theta}(z|x,a)}\text{ln}q_{\zeta}(x,a|z)-\mathbb{ E}_{p_{\theta}(z|x,a)}\left(\text{ln}\frac{p_{\theta}(z|x,a)}{p(z)}\right)$ (19) $\displaystyle=\mathbb{E}_{p_{\theta}(z|x,a)}\text{ln}q_{\zeta}(x,a|z)-D_{KL}(p _{\theta}(z|x,a)||p(z))$ (20) $$

Given that we assumed in our model that $x$ and $a$ and conditionally independent given $z$, thus $q_{\zeta}(x,a|z)=q_{\zeta_{1}}(x|z)q_{\zeta_{2}}(a|z)$, where $\zeta=\{\zeta_{1},\zeta_{2}\}$. Thus equation 20 can be written as :

$$ $\displaystyle\text{ln}q_{\zeta}(x,a)$ $\displaystyle\geq\mathbb{E}_{p_{\theta}(z|x,a)}\text{ln}q_{\zeta_{1}}(x|z)+ \mathbb{E}_{p_{\theta}(z|x,a)}\text{ln}q_{\zeta_{2}}(a|z)-D_{KL}(p_{\theta}(z| x,a)||p(z))$ (21) $$

Similar to SVG, we use RNN architectures in VG-LeAP to recursively predict image frames and at each time-step $t$, $\overset{\curvearrowleft}{RNN}_{\zeta_{1}}$ takes the encoded past image $x_{t-1}$ and $z_{t}$ as input. With the recursive behaviour of $\overset{\curvearrowleft}{RNN}_{\zeta_{1}}$, we can express $\text{ln}q_{\zeta_{1}}(x|z)$ as:

$$ $\displaystyle\text{ln}q_{\zeta_{1}}(x_{1:T}|z_{1:T})$ $\displaystyle\equiv\text{ln}q_{\zeta_{1}}(x|z)=\text{ln}\prod_{t}q_{\zeta_{1}} (x_{t}|x_{1:t-1},z_{1:T})$ (22) $\displaystyle=\sum_{t}\text{ln}q_{\zeta_{1}}(x_{t}|x_{1:t-1},z_{1:t})$ (23) $$

In the case of action predictor in VG-LeAP, we use a similar RNN architecture $\overset{\curvearrowleft}{RNN}_{\zeta_{2}}$ which takes the past action $a_{t-1}$ and $z_{t}$ as input. Thus $lnq_{\zeta_{2}}(a|z)$ can be expressed as:

$$ $\displaystyle\text{ln}q_{\zeta_{2}}(a_{1:T}|z_{1:T})$ $\displaystyle\equiv\text{ln}q_{\zeta_{2}}(a|z)=\text{ln}\prod_{t}q_{\zeta_{2}} (a_{t}|a_{1:t-1},z_{1:T})$ (24) $\displaystyle=\sum_{t}\text{ln}q_{\zeta_{2}}(a_{t}|a_{1:t-1},z_{1:t})$ (25) $$

In the case of the posterior and learned prior networks $\overset{\curvearrowleft}{RNN}_{\theta}$ and $\overset{\curvearrowleft}{RNN}_{\phi}$ respectively, we recursively feed the action $a_{t}$ and image $x_{t}$ to approximate $z_{t}$ (in case of learned prior we feed $x_{t-1}$ and $a_{t-1}$). Cause $z_{t}$s are independent across time, $p_{\theta}(z|x,a)$ can be expressed as:

$$ $\displaystyle p_{\theta}(z_{1:T}|x_{1:T},a_{1:T})$ $\displaystyle\equiv p_{\theta}(z|x,a)=\prod_{t}p_{\theta}(z_{t}|x_{t},a_{t})$ (26) $$

We assume the extended image-action state as $\chi=(x,a)$ for better readability and compact expressions in long equations. Since $z_{t}$s are independent across time, we can rewrite $D_{KL}(p_{\theta}(z|x,a)||p(z))$ or $D_{KL}(p_{\theta}(z|\chi)||p(z))$as:

$$ $\displaystyle D_{KL}(p_{\theta}(z|x,a)||p(z))$ $\displaystyle\equiv D_{KL}(p_{\theta}(z_{1:T}|x_{1:T},a_{1:T})||p(z_{1:T}))= \int_{z}p_{\theta}(z|x,a)\text{ln}\frac{p_{\theta}(z|\chi)}{p(z)}$ (27) $\displaystyle=\int_{z_{1}}\cdots\int_{z_{T}}p_{\theta}(z_{1}|\chi_{1})\cdots p _{\theta}(z_{T}|\chi_{1:T})\text{ln}\frac{p_{\theta}(z_{1}|\chi_{1})\cdots p_{ \theta}(z_{T}|\chi_{1:T})}{p(z_{1})\cdots p(z_{T})}$ (28) $\displaystyle=\int_{z_{1}}\cdots\int_{z_{T}}p_{\theta}(z_{1}|\chi_{1})\cdots p _{\theta}(z_{T}|\chi_{1:T})\sum_{t}\text{ln}\frac{p_{\theta}(z_{t}|\chi_{1:t}) }{p(z_{t})}$ (29) $\displaystyle=\sum_{t}\int_{z_{1}}\cdots\int_{z_{T}}p_{\theta}(z_{1}|\chi_{1}) \cdots p_{\theta}(z_{T}|\chi_{1:T})\text{ln}\frac{p_{\theta}(z_{t}|\chi_{1:t}) }{p(z_{t})}$ (30) $$

Since $\int_{z}p_{\theta}(z)=1$ we can further simplify equation 30 as

$$ $\displaystyle D_{KL}(p_{\theta}(z|x,a)||p(z))$ $\displaystyle=\sum_{t}\int_{z_{t}}p_{\theta}(z_{t}|\chi_{1:t})\text{ln}\frac{p _{\theta}(z_{t}|\chi_{1:t})}{p(z_{t})}$ (31) $\displaystyle=\sum_{t}D_{KL}(p_{\theta}(z_{t}|\chi_{1:t})||p(z_{t}))=\sum_{t}D _{KL}(p_{\theta}(z_{t}|x_{1:t},a_{1:t})||p(z_{t}))$ (32) $$

Thus combining equation 23, equation 25and equation 32, with equation 21 we get the variational lower bound as:

$$ $\displaystyle\text{ln}q_{\zeta}(x,a)$ $\displaystyle\geq\mathbb{E}_{p_{\theta}(z|x,a)}\text{ln}q_{\zeta_{1}}(x|z)+ \mathbb{E}_{p_{\theta}(z|x,a)}\text{ln}q_{\zeta_{2}}(a|z)-D_{KL}(p_{\theta}(z| x,a)||p(z))$ (33) $\displaystyle=\sum_{t}[\mathbb{E}_{p_{\theta}(z_{1:t}|x_{1:t},a_{1:t})}(\text{ ln}q_{\zeta_{1}}(x_{t}|x_{1:t-1},z_{1:t})+\text{ln}q_{\zeta_{2}}(a_{t}|a_{1:t- 1},z_{1:t}))$ $\displaystyle\hskip 160.00024pt-D_{KL}(p_{\theta}(z_{t}|x_{1:t},a_{1:t})||p(z_ {t}))]$ (34) $$

### A.2 Variational Lower Bound for Causal Video Generation with Learned Action Prior

Here we are trying to maximize the joint likelihood of $(x_{1:t},a_{1:T})$ which is equivalent to maximizing $\text{ln}q_{\zeta}(x_{1:T},a_{1:T})$ or $\text{ln}q_{\zeta}(x,a)$ for better readability. Let’s assume $z=[z_{1},\cdots,z_{T}]$ denotes the image latent $z$ variable across all the time-steps and $u=[u_{1},\cdots,u_{T}]$ denotes the action latent $u$ variable across all the time-steps. Both $z_{t}$s and $u_{t}$s are independent of across time.
From the Causal relationship between $a_{t}$ and $x_{t}$ we get:

$$ $\displaystyle\text{ln}q_{\zeta}(x_{1:T},a_{1:T})$ $\displaystyle\equiv\text{ln}q_{\zeta}(x,a)=\text{ln}q_{\zeta_{2}}(a|x)q_{\zeta _{1}}(x)$ (35) $\displaystyle=\text{ln}q_{\zeta_{2}}(a|x)+\text{ln}q_{\zeta_{1}}(x)$ (36) $$

from equation 36, we can derive the lower bound for $\text{ln}q_{\zeta_{1}}(x)$ as:

$$ $\displaystyle\text{ln}q_{\zeta_{1}}(x)$ $\displaystyle=\text{ln}\int_{z}q_{\zeta_{1}}(x|z)p(z)$ (37) $\displaystyle=\text{ln}\int_{z}q_{\zeta_{1}}(x|z)p(z)\frac{p_{\theta}(z|x)}{p_ {\theta}(z|x)}$ (38) $\displaystyle=\text{ln}\left(\mathbb{E}_{p_{\theta}(z|x)}q_{\zeta_{1}}(x|z) \frac{p(z)}{p_{\theta}(z|x)}\right)$ (39) $\displaystyle\geq\mathbb{E}_{p_{\theta}(z|x)}\left(\text{ln}q_{\zeta_{1}}(x|z) \frac{p(z)}{p_{\theta}(z|x)}\right)$ (40) $\displaystyle=\mathbb{E}_{p_{\theta}(z|x)}\text{ln}q_{\zeta_{1}}(x|z)-\mathbb{ E}_{p_{\theta}(z|x)}\left(\text{ln}\frac{p_{\theta}(z|x)}{p(z)}\right)$ (41) $\displaystyle=\mathbb{E}_{p_{\theta}(z|x)}\text{ln}q_{\zeta_{1}}(x|z)-D_{KL}(p _{\theta}(z|x)||p(z))$ (42) $$

Similar to VG-LeaP, we use RNN architectures $\overset{\curvearrowleft}{RNN}_{\zeta_{1}}$ in Causal-LeAP to recursively predict image frames at each time-step $t$. $\overset{\curvearrowleft}{RNN}_{\zeta_{1}}$ takes the encoded past image $x_{t-1}$, action $a_{t-1}$ and $z_{t}$ as input. With the recursive behaviour of $\overset{\curvearrowleft}{RNN}_{\zeta_{1}}$, we approximate $\text{ln}q_{\zeta_{1}}(x|z)\approx\text{ln}q_{\zeta_{1}}(x_{1:T}|z_{1:T},a_{1:
T-1})$ as:

$$ $\displaystyle\text{ln}q_{\zeta_{1}}(x|z)\approx\text{ln}q_{\zeta_{1}}(x_{1:T}| z_{1:T},a_{1:T-1})$ $\displaystyle=\text{ln}\prod_{t}q_{\zeta_{1}}(x_{t}|x_{1:t-1},z_{1:T},a_{1:t-1 },\cancel{a_{t:T-1}})$ (43) $\displaystyle=\sum_{t}\text{ln}q_{\zeta_{1}}(x_{t}|x_{1:t-1},z_{1:t},a_{1:t-1})$ (44) $$

In the case of the posterior and learned prior of the image prediction networks $\overset{\curvearrowleft}{RNN}_{\theta}$ and $\overset{\curvearrowleft}{RNN}_{\phi}$ respectively in Causal-LeAP, we recursively feed the action image $x_{t}$ to approximate $z_{t}$ (in case of learned prior we feed $x_{t-1}$ ). Cause $z_{t}$s are independent across time, $p_{\theta}(z|x)$ can be expressed as:

$$ $\displaystyle p_{\theta}(z_{1:T}|x_{1:T})$ $\displaystyle\equiv p_{\theta}(z|x)=\prod_{t}p_{\theta}(z_{t}|x_{t})$ (45) $$

We can rewrite $D_{KL}(p_{\theta}(z|x)||p(z))$ as:

$$ $\displaystyle D_{KL}(p_{\theta}(z|x)||p(z))$ $\displaystyle\equiv D_{KL}(p_{\theta}(z_{1:T}|x_{1:T})||p(z_{1:T}))=\int_{z}p_ {\theta}(z|x)\text{ln}\frac{p_{\theta}(z|x)}{p(z)}$ (46) $\displaystyle=\int_{z_{1}}\cdots\int_{z_{T}}p_{\theta}(z_{1}|x_{1})\cdots p_{ \theta}(z_{T}|x_{1:T})\text{ln}\frac{p_{\theta}(z_{1}|x_{1})\cdots p_{\theta}( z_{T}|x_{1:T})}{p(z_{1})\cdots p(z_{T})}$ (47) $\displaystyle=\int_{z_{1}}\cdots\int_{z_{T}}p_{\theta}(z_{1}|x_{1})\cdots p_{ \theta}(z_{T}|x_{1:T})\sum_{t}\text{ln}\frac{p_{\theta}(z_{t}|x_{1:t})}{p(z_{t })}$ (48) $\displaystyle=\sum_{t}\int_{z_{1}}\cdots\int_{z_{T}}p_{\theta}(z_{1}|x_{1}) \cdots p_{\theta}(z_{T}|x_{1:T})\text{ln}\frac{p_{\theta}(z_{t}|x_{1:t})}{p(z_ {t})}$ (49) $$

Since $\int_{z}p_{\theta}(z)=1$ we can further simplify equation 49 as

$$ $\displaystyle D_{KL}(p_{\theta}(z|x)||p(z))$ $\displaystyle=\sum_{t}\int_{z_{t}}p_{\theta}(z_{t}|x_{1:t})\text{ln}\frac{p_{ \theta}(z_{t}|x_{1:t})}{p(z_{t})}$ (50) $\displaystyle=\sum_{t}D_{KL}(p_{\theta}(z_{t}|x_{1:t})||p(z_{t}))$ (51) $$

The lower bound of $\text{ln}q_{\zeta_{2}}(a|x)$ is derived as follows:

$$ $\displaystyle\text{ln}q_{\zeta_{2}}(a|x)$ $\displaystyle=\text{ln}\int_{u}q_{\zeta_{2}}(a|u,x)p(u|x)$ (52) $\displaystyle=\text{ln}\int_{u}q_{\zeta_{2}}(a|u,x)p(u|x)\frac{p_{\psi}(u|a,z) }{p_{\psi}(u|a,z)}$ (53) $\displaystyle=\text{ln}\left(\mathbb{E}_{p_{\psi}(u|a,z)}q_{\zeta_{2}}(a|u,x) \frac{p(u|x)}{p_{\psi}(u|a,z)}\right)$ (54) $\displaystyle\geq\mathbb{E}_{p_{\psi}(u|a,z)}\left(\text{ln}q_{\zeta_{2}}(a|u, x)\frac{p(u|x)}{p_{\psi}(u|a,z)}\right)$ (55) $\displaystyle=\mathbb{E}_{p_{\psi}(u|a,z)}\text{ln}q_{\zeta_{2}}(a|u,x)- \mathbb{E}_{p_{\psi}(u|a,z)}\left(\text{ln}\frac{p_{\psi}(u|a,z)}{p(u|x)}\right)$ (56) $\displaystyle=\mathbb{E}_{p_{\psi}(u|a,z)}\text{ln}q_{\zeta_{2}}(a|u,x)-D_{KL} (p_{\psi}(u|a,z)||p(u|x))$ (57) $$

Now combining equation 36, equation 42 and equation 57 we get:

$$ $\displaystyle\text{ln}q_{\zeta}(x,a)\geq\mathbb{E}_{p_{\theta}(z|x)}\text{ln}q _{\zeta_{1}}(x|z)+\mathbb{E}_{p_{\psi}(u|a,z)}\text{ln}q_{\zeta_{2}}(a|u,x)-$ $\displaystyle D_{KL}(p_{\theta}(z|x)||p(z))$ (58) $\displaystyle-D_{KL}(p_{\psi}(u|a,z)||p(u|x))$ $$

In the case of action predictor in Causal-LeAP, to predict $a_{t}$ we use RNN architecture $\overset{\curvearrowleft}{RNN}_{\zeta_{2}}$ which takes the past action $a_{t-1}$ and $u_{t}$ as inputs at time $t$. Thus recursively it builds dependence upon all past actions $a_{1:t-1}$ and action latent variable $u_{1:t}$. Please note in the case of the action predictor network we do not feed the last image $x_{t}$ as input. We found that even without the image $x_{t}$ as input, the action predictor generates accurate approximation of future actions. Thus $\text{ln}q_{\zeta_{2}}(a|u,x)\approx\text{ln}q_{\zeta_{2}}(a|u)$ in can be expressed as:

$$ $\displaystyle\text{ln}q_{\zeta_{2}}(a_{1:T}|u_{1:T})$ $\displaystyle\equiv\text{ln}q_{\zeta_{2}}(a|u)=\text{ln}\prod_{t}q_{\zeta_{2}} (a_{t}|a_{1:t-1},u_{1:T})$ (59) $\displaystyle=\sum_{t}\text{ln}q_{\zeta_{2}}(a_{t}|a_{1:t-1},u_{1:t})$ (60) $$

In the case of the posterior and learned prior of the action prediction networks $\overset{\curvearrowleft}{RNN}_{\psi}$ and $\overset{\curvearrowleft}{RNN}_{\varphi}$ respectively in Causal-LeAP, we recursively feed the action $a_{t}$ and the image latent variable $z_{t}$ to approximate $u_{t}$ (in case of learned prior we feed $(a_{t-1},z_{t-1})$ ). Cause $u_{t}$s are independent across time, $p_{\theta}(u|a,z)$ can be expressed as:

$$ $\displaystyle p_{\psi}(u_{1:T}|a_{1:T},z_{1:T})$ $\displaystyle\equiv p_{\psi}(u|a,z)=\prod_{t}p_{\psi}(u_{t}|a_{t},z_{t})$ (61) $$

We can rewrite $D_{KL}(p_{\psi}(u|a,z)||p(u|x))$ as:

$$ $\displaystyle D_{KL}$ $\displaystyle(p_{\psi}(u|a,z)||p(z))\equiv D_{KL}(p_{\psi}(u_{1:T}|a_{1:T},z_{ 1:T})||p(u_{1:T}|x_{1:T}))=\int_{u}p_{\psi}(u|a,z)\text{ln}\frac{p_{\psi}(u|a, z)}{p(u|x)}$ (62) $\displaystyle=\int_{u_{1}}\cdots\int_{u_{T}}p_{\psi}(u_{1}|a_{1},z_{1})\cdots p _{\psi}(u_{T}|a_{1:T},z_{1:T})\text{ln}\frac{p_{\psi}(u_{1}|a_{1},z_{1})\cdots p _{\psi}(u_{T}|a_{1:T},z_{1:T})}{p(u_{1}|x_{1})\cdots p(u_{T}|x_{T})}$ (63) $\displaystyle=\int_{u_{1}}\cdots\int_{u_{T}}p_{\psi}(u_{1}|a_{1},z_{1})\cdots p _{\psi}(u_{T}|a_{1:T},z_{1:T})\sum_{t}\text{ln}\frac{p_{\psi}(u_{t}|a_{1:t},z_ {1:t})}{p(u_{t}|x_{t})}$ (64) $\displaystyle=\sum_{t}\int_{u_{1}}\cdots\int_{u_{T}}p_{\psi}(u_{1}|a_{1},z_{1} )\cdots p_{\psi}(u_{T}|a_{1:T},z_{1:T})\text{ln}\frac{p_{\psi}(u_{t}|a_{1:t},z _{1:t})}{p(u_{t}|x_{t})}$ (65) $$

Since $\int_{u}p_{\psi}(u)=1$ we can further simplify equation 65 as

$$ $\displaystyle D_{KL}(p_{\psi}(u|a,z)||p(u|x))$ $\displaystyle=\sum_{t}\int_{u_{t}}p_{\psi}(u_{t}|a_{1:t},z_{1:t})\text{ln} \frac{p_{\psi}(u_{t}|a_{1:t},z_{1:t})}{p(u_{t}|x_{t})}$ (66) $\displaystyle=\sum_{t}D_{KL}(p_{\psi}(u_{t}|a_{1:t},z_{1:t})||p(u_{t}|x_{t})$ (67) $$

Now combining equation 44, equation 60, equation 51 and equation 67 with equation 58 we get the final expression for the variational lower bound as:

$$ $\displaystyle\text{ln}q_{\zeta}(x,a)$ $\displaystyle\geq\mathbb{E}_{p_{\theta}(z|x)}\text{ln}q_{\zeta_{1}}(x|z)+$ $\displaystyle\hskip 50.00008pt+\mathbb{E}_{p_{\psi}(u|a,z)}\text{ln}q_{\zeta_{ 2}}(a|u)-D_{KL}(p_{\theta}(z|x)||p(z))-D_{KL}(p_{\psi}(u|a,z)||p(u|x))$ (68) $\displaystyle=\sum_{t}[\mathbb{E}_{p_{\theta}(z_{1:t}|x_{1:t})}\text{ln}q_{ \zeta_{1}}(x_{t}|x_{1:t-1},z_{1:t},a_{1:t-1})+\mathbb{E}_{p_{\psi}(u_{1:t}|a_{ 1:t},z_{1:t})}\text{ln}q_{\zeta_{2}}(a_{t}|a_{1:t-1},u_{1:t})$ $\displaystyle\hskip 50.00008pt-D_{KL}(p_{\theta}(z_{t}|x_{1:t})||p(z_{t}))-D_{ KL}(p_{\psi}(u_{t}|a_{1:t},z_{1:t})||p(u_{t}|x_{t})]$ (69) $$

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: - How does the model perform on long-horizon video prediction problems. 
- How is the performance in different robot settings? (for instance, robot manipulators, drones, etc.)
- Were any other latent conditioning methods evaluated?
soundness: 3
strengths: The problem of partially observable video prediction is quite interesting and has applications in autonomous vehicles that have an onboard camera (such as autonomous cars/taxis), drones (with an onboard camera), and robot manipulators (that have wrist-mounted cameras). 

Tackling video prediction under partially observable settings (where the acting agent is not visible on the camera) can be benefit robot applications (for instance pedestrian intent detection and prediction could influence autonomous driver decisions, video prediction networks as world models for robot manipulators etc.)

This work tackles the interesting idea of incorporating (robot) actions (as a learned latent) into the video prediction/generation task that have been traditionally conditioned on just image frames. 

The problem statement is interesting the authors motivate it well. The paper is also well-presented and articulated except for a few spelling errors (for instance lossses pg 6, divergance pg 6).
summary: The authors tackle the problem of partial observability video prediction, which deals with video prediction problems in which the camera is also moving, in which case the video is influenced by both the scene dynamics and the camera's motion -- common in autonomous vehicles and robot manipulators. 

This work explicitly models camera motion dynamics by extending the observed image state (existing settings) by introducing three models build upon prior works. Two models are based upon SVG-lp that learn image-action priors (LeAP) -- 
- (i) vg-leap (imagen-action pairs generated using a single stochastic process), and 
- (ii) causal-leap (causal relationship between image and action), and 
- (iii) RAFI -- that augments the image-action state pair of an existing flow-matching model RIVER. 

For learning the latent action prior, two variational approaches are presented -- (i) combined image-action prior derived from both the observed image and action, with the assumption that image and action states are conditionally independent, and (ii) two separate posterior priors learnt for the image and action latent variables assuming causal interlinking. 

The experimental evaluations are performed on the RoAM dataset which consists of synchronized image-action pairs recorded with a Turtlebot robot using a stereo camera setup. The dataset has 45 training videos and 5 testing sequences (300k videos sequences of 25 frames each used for training). The models are evaluated on the task of video generation for generating 10 frames conditioned of randomly sampled 5 previous consecutive frames. Perceptual and semantic quantitative metrics are used for evaluation.
weaknesses: Although sound, I find the contributions (incorporating actions) to be minimal additions to the existing frameworks. For instance, in VG-leap, the extended image-action state pair is used to condition the SVG-lp model instead of just the images, and the latent posterior approximated with recurrent modules. 
Similarly, in Causal-leap, two stochastic posteriors are learned -- one each for image and action, learnt using recurrent modules. 
For RAFI, the image latent is concatenated with the action state along the image latent's channel dimension of an existing RIVER model. Given the extensive literature on latent conditioning methods, I would have liked to see comparisons/discussions with different latent conditioning methods for incorporating actions. 

I find the experimental evaluations quite weak. The experiments are performed just on the RoAM dataset which has just 45 video sequences for training and 5 for testing/inference I would have liked to see comprehensive comparisons -- for instance on the robot manipulation settings in which case the camera is mounted on the robot's wrist. The Causal-leap model should be evaluated on such a setting -- which has a larger action space (7dof instead of the 2-dimensional in RoAM) that was used to motivate the problem. A similar comparison could have been done on drones for instance. 

Evaluations on short sequences. Predicting just 10 frames or evaluating on 25 frame length long video sequences does not quantify as long-term video prediction (which was used to motivate the manuscript). In such short horizon prediction problems, it is harder to quantify the effect of the moving camera. I would have liked to see examples of video predictions that are influenced by external factors (for instance the car turning right due to an obstacle on the left -- collision avoidance being one of the motivations). The quantitative metrics used in the paper evaluate semantic/perceptual quality of the generated videos -- fvd, lpips, vgg-16 etc., and don't necessarily motivate incorporating actions into the video prediction problem. TLDR; how obstacles or other hinderances influence video predictions is quite unclear as the metrics primarily evaluate video quality and not decisions. This would also strengthen the applications of such systems.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: * How are discussions on diffusion-based video prediction (line 131 - 135) related to camera controls (line 103)? 
* What's the action space in RoAM? Is it continuous or discrete? Does the method scale to high-dimensional action space?
soundness: 2
strengths: * The paper adapts action-free VAE and flow-based formulation from video prediction literature to action-conditioned learning. 
* Empirical results show benefit of incorporating action information in training in terms of video prediction accuracy.
summary: The paper proposes three action-conditioned video prediction training frameworks and shows empirical results of their efficacy on a robot video dataset.
weaknesses: * Missing related works. The paper revisits the literature on VAE-based and flow-matching-based video prediction models, without discussing the latter in the section of prior works. 
* The method restricts cameras to be static (line 148). This assumption does not hold in general for casual videos outside of the training data being used and limits the applicability of the method. 
* The assumption of causality between actions $a_t$ and observed framer $x_t$ is again specific to robot manipulation tasks with fixed cameras as in the RoAM dataset used in this paper. This assumption does not hold in generic videos. 
* The paper lacks a discussion of the potential benefit of incorporating actions into video modeling frameworks in general. Action-free video prediction training does not require action annotations and is much more scalable. If the downstream task is robot manipulation, an alternative approach is to train action-free video prediction models and extract robot actions via inverse dynamics [1]. More discussions would help strengthen the paper. If the goal is accurate video prediction itself, then how does the method compare to state-of-the-art video prediction architectures? 
* What's the relation of the proposed 3 distinct models? A much more extensive discussion on this would help clarify the motivation of developing three separate frameworks in the paper.  
* The paper claims results on incorporating camera motion (line 014) but empirically only evaluate on datasets with fixed cameras. 

[1] Learning Universal Policies via Text-Guided Video Generation.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: Please check the questions in the section of the weaknesses.
soundness: 2
strengths: - It shows that combining the camera motion dynamics with visual dynamics will help the video prediction as well as the action prediction. This idea is straightforward but is useful in many cases, such as the design of the world models in embodied agents.
- The paper is easy to follow, and the supplementary materials seem comprehensive.
summary: This paper extended the stochastic video generation method with explicit encoding of camera motion dynamics, and then proposes three models , 1) SVG-LP extended with image-action joint states, 2) SVG-LP with disentangled image and action states (with a predefined causal dependency), and 3) RIVER jointly conditioned by the image-action states. This paper compares the proposed models in a action-labeled robotic video dataset named RoAM, and shows certain improvements than existing approaches that do not include camera motion dynamics.
weaknesses: Novelty
- The main weakness of this paper is lack of novelty. Indeed, applying camera motion dynamics to condition video generation is not a new story. Recent studies have even tried to customize video generation with user-directed camera movement (Direct-a-video, siggraph'24), or abstract textual motion descriptions (LEGO, eccv'24). In this case, multimodal training of actions and images to the basemodel such as SVG-LP may not meet the expectations of the audience in recent research communities. However, the modification of RIVER is quite trivial and the gain is quite marginal. It was expected that a good action interaction scheme with visual space would lead to a powerful world model that can predict faithful future actions along with physically reliable future frames. The proposed method did not show such kind of potential, according to its results.

Dataset Limitations 
- Limited diversity: The RoAM dataset is just about several indoor scenes with a specific robot and features mainly corridors, lobby space, staircases with human movements like walking and sitting. This limited setting may not fully represent the wide variety of real-world scenarios with moving cameras. It would be beneficial to test on a more diverse set of datasets that include a broader range of environments and camera movements. 
- Data size and complexity: While the dataset contains a significant number of video sequences (more than 300k), each sequence has only 25 frames of image size 64 × 64 × 4 (again, the diversity of scenarios is limited). This relatively small frame size and limited sequence length (just 1 second may not capture large motions) may not capture all the complex spatio-temporal dynamics that occur in real videos. Larger frame sizes and longer sequences could provide more detailed information for the models to learn from, potentially leading to better performance and more accurate video generation.

Comparison with State-of-the-Art
- Poor visual quality and overfitting issues. The reported qualitative results are of poor visual quality, making it hard to convince that the proposed action prior indeed works. Moreover, the action prediction errors are surprisingly small. Are the GT action values small, or the overfitting issue just happened? 
- Incomplete benchmarking: The paper compares the proposed models with a few existing methods such as SVG-lp, RIVER, and ACPNet (cannot find its reference though). However, there are many other recent and relevant state-of-the-art video generation models that are not included in the comparison. For example, some of the latest diffusion-based models or other advanced variational frameworks may have different approaches to handling camera motion and image-action dynamics. 
- Lack of ablation studies on key components: While an ablation study is conducted to compare the performance of Causal-LEAP, VG-LeAP, and SVG, more ablation studies on other key components of the models could be useful. For example, analyzing the impact of different encoding and decoding strategies, the role of the latent variables in capturing image-action dynamics, or the effect of the conditional flow matching in RAFI on the overall performance would provide a deeper understanding of the models and help in identifying areas for improvement.

Minor Issue
- The citation in the paper may use the wrong latex command, using \citep{} instead of \cite{}.
- The font sizes in the plots are too small, especially Fig. 4 and 5, 7.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. What are the differences between the proposed methods and prior works listed in Weakness? What are the advantages of the proposed method?
2. What are the connections between the three proposed models? What is the purpose of proposing three models? What are the advantages and disadvantages of each model?
3. Can the proposed method generalize to video data without explicit camera action annotations?
soundness: 3
strengths: 1. The paper addresses an important and interesting problem in video generation—partially observable video prediction.
2. The paper introduces three models based on variational frameworks and conditional flow matching.
summary: This paper tackles the partially observable video prediction problem, where the camera is in motion, by incorporating both camera movement and action. The authors propose three models: (1) VG-LeAP, which treats the image-action pair as a state from a single latent stochastic process; (2) Causal-LeAP, which learns a separate action prior conditioned on the observed images and action history; and (3) RAFI, which integrates the augmented image-action state with a conditional flow matching framework. Empirical results on the RoAM dataset demonstrate the effectiveness of these models in addressing partially observable video generation.
weaknesses: 1. The paper lacks a comparison with prior works that model camera motion in video generation. Several existing studies incorporate camera motion information [1-6]. The authors should discuss how their methods differ from these works and clarify the advantages of their approach.
2. Experiments are limited to the RoAM dataset, which includes camera action annotations. Testing the models on additional datasets and demonstrating generalization to video data without explicit camera action annotations (like A2D2 [7]) would strengthen the effectiveness of proposed models.
3. The rationale for proposing three distinct models is unclear. The authors should explain the specific advantages, disadvantages and computational complexities of each model and provide guidance on when to use each one.
4. The frames shown in Figures 6 and 8 are too blurry, hindering a proper assessment of video prediction quality. Providing zoomed-in key areas or video samples would improve evaluation clarity.

[1] Guo, Yuwei, et al. "AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning." The Twelfth International Conference on Learning Representations.

[2] Wang, Xiang, et al. "Videocomposer: Compositional video synthesis with motion controllability." Advances in Neural Information Processing Systems 36 (2024).

[3] Wang, Zhouxia, et al. "Motionctrl: A unified and flexible motion controller for video generation." ACM SIGGRAPH 2024 Conference Papers. 2024.

[4] He, Hao, et al. "Cameractrl: Enabling camera control for text-to-video generation." arXiv preprint arXiv:2404.02101 (2024).

[5] Yang, Shiyuan, et al. "Direct-a-video: Customized video generation with user-directed camera movement and object motion." ACM SIGGRAPH 2024 Conference Papers. 2024.

[6] Xu, Dejia, et al. "CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation." arXiv preprint arXiv:2406.02509 (2024).

[7] Geyer, Jakob, et al. "A2d2: Audi autonomous driving dataset." arXiv preprint arXiv:2004.06320 (2020).

Meta-review: The paper addresses long-term stochastic video generation with moving cameras, which introduces complex spatio-temporal dynamics and partial observability. Traditional methods focus on pixel-level reconstruction without modeling camera motion. To address this, the authors propose a multi-modal learning framework that incorporates camera actions into the image state. They introduce three models: VG-LeAP: Combines image-action pairs into an augmented state using a single latent stochastic process with variational inference. Causal-LeAP: Establishes a causal link between actions and image frames, learning separate action priors conditioned on image states. RAFI: Integrates image-action states with a conditional flow matching framework, extending to transformer-based architectures.

Strengths:

-- Tackles the important issue of video prediction with moving cameras, applicable to autonomous vehicles and robotics.

-- Effectively integrates camera motion with visual data, extending existing video generation models.

-- Shows performance improvements on the RoAM dataset, highlighting the benefits of action incorporation.

Weakness:

-- The integration of camera motion is incremental and lacks significant innovation compared to recent studies.

-- Evaluations are restricted to the RoAM dataset, limiting the demonstration of generalizability.

-- Does not adequately compare with a wide range of state-of-the-art models or discuss the distinct advantages of the proposed methods.

Despite addressing a relevant problem and showing some empirical benefits, the paper suffers from narrow experimental validation, and inadequate comparative analysis.  After carefully reading the paper, the reviews and rebuttal discussions, the AC agrees with the reviewers on recommending to reject the paper.

Decision: Reject