Time-Agnostic Prediction: Predicting Predictable Video Frames

1 INTRODUCTION . Imagine taking a bottle of water and laying it on its side . Consider what happens to the surface of the water as you do this : which times can you confidently make predictions about ? The surface is initially flat , then becomes turbulent , until it is flat again , as shown in Fig 1 . Predicting the exact shape of the turbulent liquid is extremely hard , but its easy to say that it will eventually settle down . Prediction is thought to be fundamental to intelligence ( Bar , 2009 ; Clark , 2013 ; Hohwy , 2013 ) . If an agent can learn to predict the future , it can take anticipatory actions , plan through its predictions , and use prediction as a proxy for representation learning . The key difficulty in prediction is uncertainty . Visual prediction approaches attempt to mitigate uncertainty by predicting iteratively in heuristically chosen small timesteps , such as , say , 0.1s . In the bottle-tilting case , such approaches generate blurry images of the chaotic states at t = 0.1s , 0.2s , . . . , and this blurriness compounds to make predictions unusable within a few steps . Sophisticated probabilistic approaches have been proposed to better handle this uncertainty ( Babaeizadeh et al. , 2018 ; Lee et al. , 2018 ; Denton & Fergus , 2018 ; Xue et al. , 2016 ) . What if we instead change the goal of our prediction models ? Fixed time intervals in prediction are in many ways an artifact of the fact that cameras and monitors record and display video at fixed frequencies . Rather than requiring predictions at regularly spaced future frames , we ask : if a frame prediction is treated as a bet on that frame occurring at some future point , what should we predict ? Such time-agnostic prediction ( TAP ) has two immediate effects : ( i ) the predictor may skip more uncertain states in favor of less uncertain ones , and ( ii ) while in the standard approach , a prediction is wrong if it occurs at t± rather than at t , our formulation considers such predictions equally correct . Recall the bottle-tilting uncertainty profile . Fig 1 depicts uncertainty profiles for several other prediction settings , including both forward/future prediction ( given a start frame ) and intermediate prediction ( given start and end frames ) . Our time-agnostic reframing of the prediction problem targets the minima of these profiles , where prediction is intuitively easiest . We refer to these minima states as “ bottlenecks. ” At this point , one might ask : are these “ easy ” bottlenecks actually useful to predict ? Intuitively , bottlenecks naturally correspond to reliable subgoals—an agent hoping to solve the maze in Fig 1 ( e ) would do well to target its bottlenecks as subgoals . In our experiments , we evaluate the usefulness of our predictions as subgoals in simulated robotic manipulation tasks . Our main contributions are : ( i ) we reframe the video prediction problem to be time-agnostic , ( ii ) we propose a novel technical approach to solve this problem , ( iii ) we show that our approach effectively identifies “ bottleneck states ” across several tasks , and ( iv ) we show that these bottlenecks correspond to subgoals that aid in planning towards complex end goals . 2 RELATED WORK . Visual prediction approaches . Prior visual prediction approaches regress directly to future video frames in the pixel space ( Ranzato et al. , 2014 ; Oh et al. , 2015 ) or in a learned feature space ( Hadsell et al. , 2006 ; Mobahi et al. , 2009 ; Jayaraman & Grauman , 2015 ; Wang et al. , 2016 ; Vondrick et al. , 2016b ; Kitani et al. , 2012 ) . The success of generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ; Mirza & Osindero , 2014 ; Radford et al. , 2015 ; Isola et al. , 2017 ) has inspired many video prediction approaches ( Mathieu et al. , 2015 ; Vondrick et al. , 2016a ; Finn & Levine , 2017 ; Xue et al. , 2016 ; Oh et al. , 2015 ; Ebert et al. , 2017 ; Finn et al. , 2016 ; Larsen et al. , 2016 ; Lee et al. , 2018 ) . While adversarial losses aid in producing photorealistic image patches , prediction has to contend with a more fundamental problem : uncertainty . Several approaches ( Walker et al. , 2016 ; Xue et al. , 2016 ; Denton & Fergus , 2018 ; Lee et al. , 2018 ; Larsen et al. , 2016 ; Babaeizadeh et al. , 2018 ) exploit conditional variational autoencoders ( VAE ) ( Kingma & Welling , 2013 ) to train latent variable models for video prediction . Pixel-autoregression ( Oord et al. , 2016 ; van den Oord et al. , 2016 ; Kalchbrenner et al. , 2016 ) explicitly factorizes the joint distribution over all pixels to model uncertainty , at a high computational cost . Like these prior approaches , we too address the uncertainty problem in video prediction . We propose a general time-agnostic prediction ( TAP ) framework for prediction tasks . While all prior work predicts at fixed time intervals , we aim to identify inherently low-uncertainty bottleneck frames with no associated timestamp . We show how TAP may be combined with conditional GANs as well as VAEs , to handle the residual uncertainty in its predictions . Bottlenecks . In hierarchical reinforcement learning , bottlenecks are proposed for discovery of options ( Sutton et al. , 1999 ) in low-dimensional state spaces in ( McGovern & Barto , 2001 ; Şimşek & Barto , 2009 ; Bacon , 2013 ; Metzen , 2013 ) . Most approaches ( Şimşek & Barto , 2009 ; Bacon , 2013 ; Metzen , 2013 ) construct full transition graphs and apply notions of graph centrality to locate bottlenecks . A multi-instance learning approach is applied in ( McGovern & Barto , 2001 ) to mine states that occur in successful trajectories but not in others . We consider the use of our bottleneck predictions as subgoals for a hierarchical planner , which is loosely related to options in that both aim to break down temporally extended trajectories into more manageable chunks . Unlike these prior works , we use predictability to identify bottlenecks , and apply this to unlabeled high-dimensional visual state trajectories . Concurrently with us , Neitz et al . ( 2018 ) also propose a similar idea that allows a predictor to select when to predict , and their experiments demonstrate its advantages in specially constructed tasks with clear bottlenecks . In comparison , we propose not just the basic time-agnostic loss ( Sec 3.1 ) , but also improvements in Sec 3.2 through 3.5 that allow time-agnostic prediction to work in more general tasks such as synthetic and real videos of robotic object manipulation . Our experiments also test the quality of discovered bottlenecks in these scenarios and their usefulness as subgoals for hierarchical planning . 3 TIME-AGNOSTIC PREDICTION OF BOTTLENECK FRAMES . In visual prediction , the goal is to predict a set of unobserved target video frames given some observed context frames . In forward prediction , the context is the first frame , and the target is all future frames . In the bidirectionally conditioned prediction case , the context is the first and the last frame , and the frames in between are the target . In Fig 1 , we may wish to predict future images of the tilting bottle , or intermediate images of an agent who traverses the maze successfully . 3.1 MINIMUM-OVER-TIME LOSS In standard fixed-time video prediction models ( Ranzato et al. , 2014 ; Oh et al. , 2015 ; Mathieu et al. , 2015 ; Vondrick et al. , 2016a ; Walker et al. , 2016 ; Finn & Levine , 2017 ; Xue et al. , 2016 ; Oh et al. , 2015 ; Ebert et al. , 2017 ; Finn et al. , 2016 ; Lee et al. , 2018 ; Denton & Fergus , 2018 ) , a frame xτ ( video frame at time τ ) is selected in advance to be the training target for some given input frames c. For instance , in a typical forward prediction setup , the input may be c = x0 , and the target frame may be set to xτ = x1 . A predictor G takes context frames c as input and produces a single frame G ( c ) . G is trained as : G∗ = argmin G L0 ( G ) = argmin G E ( G ( c ) , xτ ) , ( 1 ) where E is a measure of prediction error , such as ‖G ( c ) − xτ‖1.1 This predictor may be applied recursively at test time to generate more predictions as G ( G ( c ) ) , G ( G ( G ( c ) ) ) , and so on . We propose to depart from this fixed-time paradigm by decoupling prediction from a rigid notion of time . Instead of predicting the video frame at a specified time τ , we propose to predict predictable bottleneck video frames through a time-agnostic predictor ( TAP ) , as motivated in Sec 1 . To train this predictor , we minimize the following “ minimum-over-time ” loss : G∗ = argmin G L ( G ) = argmin G min t∈T E ( G ( c ) , xt ) , ( 2 ) where the key difference from Eq 1 is that the loss is now a minimum over target times of a timeindexed error Et = E ( · , xt ) . The target times are defined by a set of time indices T. For forward prediction starting from input c = x0 , we may set targets to T = { 1 , 2 , ... } . Fig 2 depicts this idea schematically . Intuitively , the penalty for a prediction is determined based on the closest matching ground truth target frame . This loss incentivizes the model to latch on to “ bottleneck ” states in the video , i.e. , those with low uncertainty . In the bottle-tilting example , this would mean producing an image of the bottle after the water has come to rest . One immediate concern with this minimum-over-time TAP loss might be that it could produce degenerate predictions very close to the input conditioning frames c. However , as in the tilting bottle and other cases in Fig 1 , uncertainty is not always lowest closest to the observed frames . Moreover , target frame indices T are always disjoint from the input context frames , so the model ’ s prediction must be different from input frames by at least one step , which is no worse than the one-step-forward prediction of Eq 1 . In our experiments , we show cases where the minimum-over-time loss above captures natural bottlenecks successfully . Further , Sec 3.2 shows how it is also possible to explicitly penalize predictions near input frames c. This minimum loss may be viewed as adaptively learning the time offset τ , but in fact , the predictor ’ s task is even simpler since it is not required to provide a timestamp accompanying its prediction . For example , in Fig 1 ( e ) , it need only specify which points in the maze the agent will go through ; it need not specify when . Lifting the requirement of a timestamped prediction relieves TAP approaches of a significant implicit burden . Recursive TAP . TAP models may also be trained for recursive prediction , by minimizing the following loss : G∗ = argmin G Lrec ( G ) = argmin G ∑ r min t∈T ( r ) E ( G ( c ( r ) ) , xt ) , ( 3 ) 1We omit expectations over training samples throughout this paper to avoid notational clutter . where c ( r ) and T ( r ) are the input and target set at recursion level r , both dynamically adapted based on the previous prediction . The input c ( r ) may be set to the previous prediction G ( c ( r − 1 ) ) , so that the sequence of predictions is ( G ( c ( 0 ) ) , G ( G ( c ( 0 ) ) ) , . . . ) . T ( r ) is set to target all times after the last prediction . In other words , if the prediction at r = 0 was closest to frame x5 , the targets for r = 1 are set to T ( 1 ) = { 6 , 7 , ... } . While we also test recursive TAP in Sec 4 , in the rest of this section , we discuss the non-recursive formulation , building on Eq 2 , for simplicity . Bidirectional TAP . Finally , while the above description of TAP has focused on forward prediction , the TAP loss of Eq 2 easily generalizes to bidirectional prediction . Given input frames c = ( x0 , xlast ) , fixed-time bidirectional predictors might target , say , the middle frame xτ = xlast/2 . Instead , bidirectional TAP models target all intermediate frames , i.e. , T = { 1 , 2 , ... , last − 1 } in Eq 2 . As in forward prediction , the model has incentive to predict predictable frames . In the maze example from Fig 1 , this would mean producing an image of the agent at one of the asterisks .

Summary: The paper reformulates the task of video prediction/interpolation so that a predictor is not forced to generate frames at fixed time intervals, but instead it is trained to generate frames that happen at any point in the future. The motivation for such approach is that there might be future states that are highly uncertain – and thus, difficult to predict – that might not be useful for other tasks involving video prediction such as planning. The authors derive different loss functions for such Time-Agnostic Prediction (TAP), including extensions to the Variational AutoEncoders (VAE) and Generative Adversarial Networks (GAN) frameworks, and conduct experiments that suggest that the frames predicted by TAP models correspond to ‘subgoal’ states useful for planning.