Motion Graph Unleashed: A Novel Approach to Video Prediction

Authors: yiqi  zhong
luming  liang
bohan  tang
ilya  zharkov
ulrich  neumann
  microsoft
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 07:47:03

Abstract: We introduce motion graph, a novel approach to address the video prediction problem, i.e., predicting future video frames from limited past data. The motion graph transforms patches of video frames into interconnected graph nodes, to comprehensively describe the spatial-temporal relationships among them. This representation overcomes the limitations of existing motion representations such as image differences, optical flow, and motion matrix that either fall short in capturing complex motion patterns or suffer from excessive memory consumption. We further present a video prediction pipeline empowered by motion graph, exhibiting substantial performance improvements and cost reductions. Extensive experiments on various datasets, including UCF Sports, KITTI and Cityscapes, highlight the strong representative ability of motion graph. Especially on UCF Sports, our method matches and outperforms the SOTA methods with a significant reduction in model size by 78% and a substantial decrease in GPU memory utilization by 47%.

Main paper: 
 introduction: 
 Video prediction aims at predicting future frames given a limited number of past frames. This technology has potential for numerous applications, including video compression, visual robotics, and surveillance systems. A critical aspect of designing an effective video prediction system is the precise modeling of motion information, which is essential for its successful deployment. At first glance, video prediction appears as a sequence prediction problem, with conventional approaches using advanced sequential data modeling techniques such as 3D convolution [1, 2] , recurrent neural networks [3, 4, 5, 6, 7] , and transformers [8, 9, 10] . These methods implicitly model motion and image appearance to help reason sequence evolution. However, the implicit approach to motion modeling can result in inefficient learning, which stems from the inherent difference between video prediction and typical sequence prediction: in typical sequence prediction, features tend to evolve uniformly; whereas video prediction involves not only static elements like object appearance but also dynamic elements intricately embedded in the sequence, e.g., motion-related information, including object pose and camera movement. Effectively managing both the static and dynamic aspects of the hidden feature space presents a great challenge.Instead of treating video prediction purely as a sequence prediction problem, an alternative approach is to explicitly model inter-frame motion, and redefine the task primarily as a motion prediction problem [11, 12, 13, 10, 2] . For this approach, it is crucial to select an appropriate motion representation, which should exhibit expressiveness by accurately encapsulating motion with detailed information like direction and speed. However, as Section 2 and Table 1 will elaborate, existing video motion representations have limitations. Some, as Figure 1 (A) demonstrates, lack representative ability for complex scenarios. For example, optical flow is unable to handle motion blur and object distortion/deformation resulting from perspective projection. Others over-sacrifice memory efficiency to model complex motion patterns, such as the motion matrices proposed in MMVP [2] .To fill the gap in existing research, we propose a novel motion representation: the motion graph, which treats image patches of the observed video frames in the downsample space as interconnected graph nodes based on their spatial and temporal proximity; See Figure 1 (B ) for an illustration. This approach achieves both goals of representativeness and compactness by generating accurate predictions while conserving memory and computational resources. For the representative ability, our motion graph enriches each node with dynamic information, including multiple feasible weighted flows toward the subsequent frame. This method captures a broader range of motion patterns compared to single-direction representations like optical flow and enhances error tolerance. Additionally, the motion graph excels in compactness by avoiding computation-heavy operations such as stacked convolutional layers. To reduce computational complexity, we also limit the connections in the motion graph to a fixed number of neighbors for each node. Consequently, compared to previous representations (e.g., motion matrix, keypoint heatmap), the motion graph offers a more sparse and memory-efficient modeling of motion, effectively transcending the limitations of 2D image structures.Based on the motion graph we have introduced, we present a novel video prediction pipeline. The representative ability and compactness of the motion graph allow our system to have achieved notable enhancements in computational efficiency. We test various scenarios with different motion patterns on three well-known datasets, UCF Sports, KITTI, and Cityscapes. Across all three datasets, our approach consistently exhibits performance that aligns with or surpasses the state-of-the-art (SOTA) methods with significantly lower GPU comsumption. 
related works: 
 In this section, we mainly discuss the existing video prediction systems that apply to the setting of short-term high resolution video prediction, please see more details about the setting in Section A. 5 .Based on whether they explicitly model motion, existing video prediction methods can be put into two categories. Methods that do not explicitly model motion usually consider video prediction purely as a sequence prediction problem [3] . They use advanced sequential modeling techniques such as recurrent neural networks [4, 3, 5, 14, 15, 16, 17, 18] and transformers [8, 10, 19, 20] to estimate the evolution of image features in the hidden space along the temporal dimension. However, video prediction holds a unique property that is unlike text generation or other sequence prediction problems: that is, most features in video sequences (e.g., object appearance, scene layout, texture of the background) usually remain unchanged, or have no major swift change, from frame to frame. Yet, sequential modeling techniques treat the whole image features uniformly; using them for video prediction thus requires modification in the inner structure of certain sequential modeling architectures to accommodate this property of video prediction. Some works choose to enhance the spatial consistency by adding complex residual shortcuts [4, 17, 21] , or by decomposing temporal and spatial information into two separate hidden spaces, hoping that the network will figure out which to maintain and which to change [15, 16] . Although such accommodations sometimes help systems reach impressive prediction accuracy (given the powerful sequential modeling capability of the advanced techniques), these methods tend to have larger model sizes and complicated architectures.To improve the efficiency of motion modeling in systems, this paper extends the idea shared by the second type of methods, which treat video prediction as a motion prediction problem, and explicitly model the motion hidden in image sequences. Such methods either reason future motion patterns using certain representations with image features as the input, or solely use motion representations as the input for prediction. There are four lines of work on motion representations. First, image difference [11, 10] uses the subtraction of two consecutive frames to represent the dynamic of image sequences. It requires almost no computational cost to calculate but does not directly describe motion. Second, optical flow/ voxel flow [22, 23, 24, 13] describes the pixel-level motion. But, it usually requires auxiliary or out-of-shell models to generate, and is limited to only modeling one-to-one relationships [2] , which can be a disadvantage, especially for scenarios with ambiguous pixel-to-pixel correspondence (e.g., motion blur). Third, key point trace [12] serves highly structured scenarios (e.g., videos with static backgrounds and dominated by human motions) but can be inefficient for more complex content, including crowded scenes with multiple objects. Fourth, the recent motion matrix [2] describes the all-pair relationship between consecutive frames to overcome the drawback of optical flow. While motion matrix supports a more efficient and compact system of a notably smaller size, concerns were over the computational and space complexity of its operations [2] .Table 1 : Motion representation comparison. We assess the representative ability by judging how accurate the motion can be described, especially for the hard cases demonstrated in Fig 1(A) .Out-of-shell Model Representative Ability Space Complexity Image difference [11, 10] n/a low O(n) Keypoint trace [12] required medium O(n) Optical flow/ voxel flow [22, 23, 24, 13] Some required medium O(n) Motion matrix [2] n/a highO(n 2 ) Motion graph (ours) n/a high O(n)Besides the above motion representations, we notice that graph, as a sparse data structure that can accurately describe the relationships between entities, has been widely adopted in ordinary motion prediction systems, such as trajectory forecasting [25, 26, 27] and human motion prediction [28, 29, 30] . Inspired by these successful adoptions, we propose to represent the motion in videos using graph structures, named motion graph in this work.The major difference between motion graph and previous motion representations is that motion graph can describe many-to-many temporal and spatial correspondence with low space complexity. Such sparse representations not only enhance representative ability but also advantageously i) replace most 2D convolution operations by graph operations which are mostly implemented by linear layers, leading to faster and more parameter-efficient systems; ii) enable convenient spatial and temporal interactions among feature patches, promoting higher learning efficiency and prediction accuracy.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
methodology: 
 In this work, video prediction is approached as a motion prediction problem, utilizing a novel representation called the motion graph. This representation is designed to effectively capture the intricate motion dynamics within video sequences. Leveraging the motion graph, we develop a video prediction pipeline that achieves high performance with reduced model size and GPU memory requirements. Section 3.1 details our problem formulation and notation. Section 3.2 explains the construction of the motion graph from input video sequences. Finally, Section 3.3 delves into our video prediction pipeline, illustrating how the motion graph is used for effective video prediction.Given a video sequence with T frames {I t ∈ R H×W ×3 |t = 0, 1, ..., T -1}, a video prediction system aims to predict the next T ′ frames {I t ′ ∈ R H×W ×3 |t ′ = T, T + 1, ..., T + T ′ -1}. We approach this task by regarding video prediction as a pixel-level motion prediction problem. Consider a pixel located at (x, y) in a given video frame I t . Our system is designed to predict k dynamic vectors starting from this pixel, pointing to the possible locations in the target future frame I t ′ . These vectors represent the pixel's anticipated motion from its current position in I t to its future position in I t ′ . Mathematically, we express the dynamic vectors of all pixels across the observed frames as P ∈ R T ×H×W ×k×3 , where P t,x,y = [∆x 1 , ∆y 1 , w 1 ], ..., [∆x k , ∆y k , w k ] with both the motion direction (∆x k , ∆y k ) and the weighted component (w k ). The transformation of previous frames into future frames is performed by a pixel-level image warper, denoted as -→ W. The warping is based on the predicted dynamic vectors and is formally defined by the following equation:I T =-→ W(P, I 0 , ..., I T -1 ).(1) This formulation allows us to capture and translate the intricate motion dynamics within the video sequence into future frame predictions with enhanced accuracy.Inspired by advancements in general motion prediction systems [25, 26, 30] , we posit that modeling semantic correlations among image patches within observed frames is essential for improving current motion representations in video prediction. To capture such correlations, we first model the semantic information of each image patch across a video with T observed frames. This is achieved by multiscale image patch feature mappings, F = {f (1) , • • • , f (M ) }, generated by inputting each observed frame into a shared image encoder g enc and applying a pixel unshuffle technique [31] to reshape all M feature maps to the resolution of the smallest features. Here, the smallest resolution of the features generated by g enc is H s × W s , and f (m) ∈ R T ×Hs×Ws×Cm represents the feature map in the m-th scale containing the C m -dimensional features of T × H s × W s distinct image patches.Based on F, we construct a multi-view graph, named motion graph, to capture the semantic correlations among T × H s × W s image patches. This graph is represented as: G = {V, E (1) , • • • , E (M ) }, where V = {v 0 , • • • , v T HsWs-1 }denotes the set of nodes each corresponding to an image patch, and E (m) denotes the edge set for the m-th view capturing the correlations driven by the semantic information within f (m) . We detail the graph construction and the graph-related operations as follows.For each node v i in the m-th view, we initialize its motion featurev f (m) i ∈ R d (m)mf with two components: the tendency featurev tf (m) i ∈ R d (m) tf and the location feature v lf (m) i ∈ R d (m)lf . The tendency feature v tf (m) i captures the node's motion-related attributes relative to nodes in the subsequent frame and the location feature v lf (m) i models the normalized absolute location of each node in a frame.Inspired by prior works [2, 32] , we generate the tendency feature v tf (m) i in three steps. First, for each node v i , we form a node list L . By doing so, we mitigate the risk of false positives and better accommodate complex motion patterns, especially in downscaled spaces where an image patch may exhibit multiple potential movements. Second, we define the dynamic vector for each node v i as d(m) i = [∆x 1 , ∆y 1 , w 1 , • • • , ∆x K , ∆y K , w K ] ∈ R 3K, where ∆x k , ∆y k indicate the motion direction and w k is the cosine similarity score between v i and the k-th node in L (m) i . Here, an exception is made for nodes in the last observed frame I T -1 , where we apply zero-padding to the dynamic vectors, as I T is unknown. Finally, we generate the tendency feature v tf (m) i for each node by feeding each dynamic vector d (m) i into a multilayer perceptron (MLP) g tdc (.) followed by a max-pooling operation φ agg (.), as shown in:EQUATIONThe location feature v lf (m) i encodes the position of a pixel in a frame. Pixel positions influence motion patterns. For instance, pixels on the sides of a frame may appear to move differently than pixels in the center for street views collected by wide-range moving cameras, due to the perspective projection effect. We use another MLP g loc (.) to extract the location feature and define it as:EQUATIONFinally, the motion features of each node v f (m) i are initialized as follows:EQUATIONwhere ⊕(•) denotes the concatenation operation. Figure 2 details the node motion feature initialisation.After initializing the node features, we construct edges in the motion graph to capture the semantic relationships between image patches in the observed frames. We further represent the edge set of the m-th view asE (m) = {E S (m) , E B (m) , E F (m) }, where E S (m) is spatial edges, E B (m) is backward edges, and E F (m)is forward edges. Generally, spatial edges are posit on that neighboring image patches in a frame likely influence each other's future motion, and backward and forward edges connect nodes across adjacent frames indicating potential motion paths. Notably, nodes in the first frame are not assigned backward edges, and nodes in the last frame are not assigned forward edges.We construct these edges in two steps: 1) finding the neighbors of each node connected by spatial, backward, and forward edges respectively; and 2) generating the three types of edges by connecting the neighboring nodes found. For the first step, we denote N S (m) i , N B (m) i , and N F (m) i as sets containing neighbors of v i connected by the spatial, backward and forward edges, respectively. Then, the neighbors of each node v i are found by solving the following optimization problems:EQUATIONwhere C (m) ∈ R T HsWs×T HsWs contains cosine similarity scores between nodes in V computed byf (m) , k ∈ Z is a hyperparameter, S i = {v j ∈ V : ⌊ i HsWs ⌋ = ⌊ j HsWs ⌋}, B i = {v j ∈ V : ⌊ i HsWs ⌋ = ⌊ jHsWs ⌋ + 1}, andF i = {v j ∈ V : ⌊ i HsWs ⌋ = ⌊ j HsWs ⌋ -1}. For the second step, we construct E S (m) , E B (m), andE F (m) by connecting each v i with nodes in N S (m) i , N B (m) i, and N F (m) i respectively.After constructing the nodes and establishing the edges in the motion graph, we enable information flow within the m-th view of the graph via the message-passing operation g mp defined as follows: whereEQUATIONv (m) ∈ R T HsWs×d (m) is the input node features, v ′ (m) ∈ R T HsWs×d (m)is the updated node features, and E in (m) denotes an edge set. The operation g mp facilitates the information exchange between nodes connected by edges in E in (m) . In practice, the spatial message passing is implemented as a 2D convolution layer, while for the temporal message passing we use graph neural network to implement g mp . See implementation details in the appendix.With the message-passing operation g mp , we introduce a motion graph interaction module, denoted as Φ and illustrated in Figure 3 . The goal of Φ is to ensure that even nodes in the last observed frame are informed about the motion dynamics from the first observed frame, and vice versa. To achieve this comprehensive information flow, we implement T -1 rounds of full information transition, where T is the total number of observed frames. Each round of transition includes twice the spatial message passing, once the temporal forward message passing, and once the temporal backward message passing. The combination of spatial and temporal interaction in Φ ensures holistic information integration for accurate and comprehensive motion prediction in video sequences. The design of Φ allows for thorough and balanced dissemination of motion information throughout the motion graph. By constructing the motion graph, we create a tool to extract motion information from the observed video frames. The following passage describes how to conduct video prediction using the constructed motion graph and its related operations. As Figure 4 indicates, to predict the unknown video frame involves three main steps, which are elaborated in the following three subsections.Multi-view motion feature update. After having the motion graph G = {V, E (1) , • • • , E (M ) }, with the node motion feature v f (m)initialized by Eq. ( 4) in the m-th view, we use the motion graph interaction module introduced in Section 3.2.4 to update v f (m) via the following formula:EQUATIONwhere vf (m) ∈ R T HsWs×d (m) mf is the updated node features in the m-th view of the motion graph.Multi-view motion feature fusion. Once the node feature in each view of the motion graph is updated, we concatenate the node features from each view and apply a fusion module Ψ f use to integrate these multi-view features into a unified representation as follows:EQUATIONwhere f f use ∈ R T ×Hs×Ws×C node , with C node denoting the dimension of the node features.In this step, we transform the fused multi-view motion feature f f use into a 2D structure with a resolution of H s × W s . This feature is then upscaled to match the resolution of the original video frames (H × W ) using a motion upsampler, Θ SR . The architecture of Θ SR is inspired by ResNetbased image super-resolution networks [33] , which progressively refine features from lower to higher resolutions until reaching the original image size. This upsampling process is key to predicting pixel-level motion features for all observed frames, which extend toward the next future frame. By incrementally adjusting the resolution, Θ SR effectively bridges the gap between the multi-view motion information and the high-resolution requirements of accurate pixel-level motion prediction.Upon obtaining the pixel-level motion feature f SR f use , we use a motion decoder, Ω dec , to convert it into pixel-level dynamic vectors P. As defined in Section 3.1, P contains k potential motion directions with associated probability scores for each pixel. This decoding output is compatible with the motion features in the motion graph, which are generated by the k dynamic vectors of image patches.After having the dynamic vectors through the decoding process, we use them to warp the observed frames into the future frame. Unlike traditional methods that often use optical-flow-based backward warping, our approach follows the design logic of the motion graph feature learning as well as the output format to use a multi-flow forward warping technique. This method, drawing from the image splatting concepts in prior works [34, 35] , is visually detailed in Figure 4 . Forward warping allows each predicted dynamic vector to directly contribute to the construction of the future frame. To aggregate the contributions of multiple vectors at each pixel location in the synthesized frame, we apply a normalization operation φ norm . It normalizes the weight of each vector based on the sum of their weights, ensuring an even and balanced contribution to the final pixel value in the future frame. 
experiments: 
 For evaluation, we trained our video prediction pipeline in an end-to-end fashion on three public datasets: i) UCF Sports [36] (i.e. 150 video sequences emphasizing various sports scenes, with frame resolution, and two different data splits to date-one from STRPM [15] , one from MMVP [2] evaluation is on both splits); ii) KITTI [37] (i.e. 28 driving videos with a resolution of 375 × 1242; following previous works [38, 13] , the image frames are resized to 256×832); and iii) Cityscapes [39] (i.e. 3,475 driving videos with 2,945 in the training set and 500 in the validation set). For each dataset, by default, we set k = max(10, 1% × H s × W s ) which is 1% of the smallest feature map's resolution and no larger than 10 for efficiency consideration. Table 2 shows the configuration for each dastaset. More training and implementation details are in the appendix. On the UCF Sports STRPM split, we evaluate the method on two metrics following previous research [15] : peak signal-to-noise ratio (PSNR) and learned perceptual image patch similarity (LPIPS) [40] . The proposed method is compared with existing methods for their results of the 1st and 6th future frames (t = 5, t = 10 in Table 3 ). Table 3 shows that our method matches the SOTA performance on the PSNR metric and outperforms all previous methods on the LPIPS metric.On the UCF Sports MMVP split, the validation dataset has been divided into three categories: the easy (SSIM ≤ 0.9), intermediate (0.6 ≤ SSIM < 0.9), and hard subsets (SSIM < 0.6), which take up 66%, 26%, and 8% of the full set respectively. We evaluate the methods on PSNR, LPIPS, and Structural Similarity Index Measure (SSIM). Table 4 shows that the proposed method outperforms all other methods. [39] and KITTI [37] datasets. "RGB", "F", "S" and "I" denote video frames, optical flow, semantic map, and instance map; t + 3 and t + 5 respectively indicate the average performance of the next 3 and 5 frames. Results with * are copied from [13] .Method Input In Table 5 , we compare the proposed method with existing methods on the Cityscapes and KITTI datasets. Following previous research, we report the Multi-scale Structural Similarity Index Measure (MS_SSIM) and LPIPS of the first future frame (t+1), the average numbers of the next three future frames (t+3) and the average numbers of the next five future frames (t+5).Cityscapes KITTI MS_SSIM(×10 -2 )↑ LPIPS(×10 -2 )↓ MS_SSIM(×10 -2 )↑ LPIPS(×10 -2 )↓ t + 1 t + 3 t + 5 t + 1 t + 3 t + 5 t + 1 t + 3 t + 5 t + 1 t + 3 t +The UCF Sports dataset features sports scenes and thus contains a large amount of fast movements and motion blurs. The leading performance in both Table 3 and 4 validates that the proposed motion graph helps the network better interpret the motion in the input video sequence and facilitate more accurate prediction. Meanwhile, the qualitative result in Figure 5 demonstrates the method's ability to capture and restore intricate image details during the prediction.For KITTI and Cityscapes datasets, videos are captured from outdoor, large-scale traffic scenarios through cameras with perspective projection, which results in drastic object distortions on both sides of images as well as unstable lighting conditions. Our method matches SOTA performance in terms of quantitative evaluation (see Table 5 ). In Figure 6 , we conducted qualitative comparison on these datasets with two SOTA methods, OPT [53] and DMVFN [13] , which are all optical-flow-based.We noticed that as a non-generative model, our proposed video prediction system may face challenges with occlusions that require the generation of unseen objects. However, it excels in scenarios with scenarios involving occlusions of known objects, as showcased in white walls of the third column in Figure 6 . Notably, in scenes with partial obstructions-such as the white wall behind the cyclist, our system adeptly employs multiple motion vectors per pixel to reconstruct occluded areas. This feature Figure 5 : On the UCF Sports dataset, our method recovers richer image details than MMVP [2] .Figure 6 : Qualitative comparisons with OPT [53] and DMVFN [13] . Our method maintains the object structures better than both methods while holding a higher motion prediction accuracy. also supports precise management of object expansion due to perspective projection, as exemplified by the green car in Figure 6 's first column.Moreover, our approach is adept at managing objects exiting the camera's view. By explicitly modeling the motion of image patches, the motion graph predicts when features are about to leave the scene. Thus, any image patches projected to move out of view are not included in the final prediction. This is demonstrated in the 1st, 3rd, and 4th columns of Figure 6 , where objects like a cyclist and the front of a blue truck are shown as moving out of frame. Our method successfully and uniquely captures and represents these movements, unlike how the other methods evaluated.To assess the compactness of the motion graph, we conduct a model consumption analysis. We compare the proposed method with SOTA methods on each dataset in terms of the model size and maximum running GPU memory (the system configurations are identical to the models that generate the numbers in the corresponding tables above). Examination of Table 6 reveals that our method shows robust advantages than SOTA methods in minimizing model consumption and saving resources. We conduct ablation studies on three aspects of the UCF Sports MMVP split: i) the choice of k, which defines three attributes of the system, i.e., the number of dynamic vectors each node initially embeds, the number of temporal forward/backward edges for each node, and the number of dynamic vectors to be decoded from the upsampled motion features; ii) the number of views used for multi-view motion graph construction; and iii) the composition of the node's initial features. The choice of k: Adjustments to the value of k and subsequent evaluations reveal a consistent, monotonic increase in both performance and GPU memory usage, as documented in Table 7 . Notably, the gain rate diminishes as k increases. This observation aligns with the intuition that increasing the value of k will bring more temporal correspondences to the system. However, once the motion information nears saturation, the incremental benefit of further increasing k becomes less significant. More ablation studies related to k can be found in Section A.4.The number of graph views: As Section 3.2 mentioned, using multi-view graphs can enhance motion prediction accuracy. To validate this claim, we reduce the number of views used in the system and test the system on the UCF Sports dataset. The first three rows in Table 8 show that increasing the number of graph views indeed improves the system performance.Location feature f loc : We incorporate the tendency feature f tdc (directly related to motion) and a location feature f loc to initialize the node features. This inclusion hypothesizes that an image pixel's motion patterns may also be influenced by its spatial position. Empirical evidence supporting this hypothesis is documented in the last two rows of Table 8 . There is a notable decline in performance when the location feature f tdc is excluded from the node feature initialization. In Section A.7, we further visualize the location features and observe that such feature may reflect the camera projection pattern, providing substantial information to the following motion reasoning task. 
conclusion & limitation: 
 This work presents the motion graph for video prediction and a novel pipeline built upon it. We focus on balancing representative ability and compactness to optimize efficiency. The motion graph encapsulates complex motion information hidden in video sequences into a more manageable format, and achieves encouraging results: it matched or outperformed SOTA on three well-known datasets with a considerably smaller model size and less GPU running memory. Beyond the results, the motion graph and its associated video prediction pipeline set a foundation for further research and optimization in the field of video prediction, thus suggesting a promising direction for researchers seeking to balance performance and resource efficiency in the evolving domain of video processing.Limitation. There is no significantly improvement in terms of the inference speed. The current version of our model is slower than DMVFN, which is specifically optimized for this aspect. Future efforts will focus on accelerating inference while maintaining the model's compactness and efficiency. Additionally, video prediction remains particularly challenging in scenarios involving sudden or unpredictable motions, which our system occasionally fails to capture, as highlighted in Appendix A.6. These instances, where the model struggles with abrupt actions not readily discernible from the input frames, underscore the need for further enhanced video understanding abilities. Addressing these challenges presents a valuable direction for future research.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: The data is all public and the code is provided in the supplementary zip file.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Please see Appendix A.1Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: Please see Appendix A.1, A.2 and the code comments provided in the supplementary file.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We implement the video prediction system using PyTorch [54] and conduct end-to-end training on a single NVIDIA A100 GPU. We use AdamW optimizer [55] during the training. The initial learning rate is set to 1e -3 and decayed to 1e -5 following a cosine decay scheduler [56] . There are only a few hyperparameters to adjust for the system when training on different datasets. The adjustments are mainly based on the resolution of the frame. Those hyper-parameters include i) image feature length (Image feat.), which is the parameter for the image encoder; ii) Tendency feature length (Tendency feat.), which is the length of the tendency feature in node initialization; iii) location feature length (Location feat.), which is fixed to 4 for all datasets, indicating the length of the location feature in node initialization; iv) the number of graph views, indicating view number in motion graph feature learning; v) k, indicates the number of the dynamic vectors embedded in each node, the number of the temporal edges per node and the output dynamic vectors per pixel; vi) training epoch is the training related parameters; vii) the reconstruction loss, which follows the popular setting of the SOTA methods on each dataset. In Table 9 , we demonstrate the hyper-parameter setting for each dataset.Please note that we did not especially tune the parameters for each dataset. When adjusting the parameters, we consider more about the training efficiency instead of the performance. Therefore, our setting is likely not the optimal choice. For example, in DMVFN [13] , the training on Cityscapes and KITTI are both 300 epochs, we observe that our system can achieve comparable performance with only 100 and 200 epochs respectively, we thus stay with this configuration. The proposed video prediction system includes three major components, the image encoder, the motion graph interaction module, and the motion upsampler. Here we demonstrate the detailed architecture of each component for reproduction needs.Image Encoder: Figure 7 shows the inner structure of the image encoder in the proposed system. C img is related to the image feature length in Table 9 . Each convolution layer will come with a Leaky ReLU layer [57] as the activation layer. In Figure 8 we demonstrate the inner structure of the spatial and temporal message passing in the motion graph interaction module. C node equals the sum of the tendency feature length and the location feature length in Table 9 . For comparison convenience of the future works, we list the extra metrics of cityscapes and KITTI datasets in Table10, i.e. ssim and psnr.Table 10 : Evaluation on Cityscapes [39] and KITTI [37] datasets. "RGB", "F", "S" and "I" denote video frames, optical flow, semantic map, and instance map; t + 3 and t + 5 respectively indicate the average performance of the next 3 and 5 frames. Results with * are copied from [13] . In this section, we add two ablation studies to help the audience better interpret the design of the motion graph.Number of the predicted dynamic vectors per pixel: In the proposed system, we set the number of the predicted dynamic vectors per pixel to k, which is identical to the number of the dynamic vectors embedded by each node and the temporal edge of each node. This design ensures the flexibility of the predicted motion to have multiple modes compared to the optical-flow-based method which only allows each pixel to have a single future motion. The comparison between the first two rows of Table 11 showcase that allowing multiple predicted dynamic vectors can largely improve the performance. Meanwhile, if we control the number of the predicted dynamic vectors, as demonstrate by the comparison between the first and third row of the Table 11 , we see that when the motion graph embeds more past motion modes, the performance will also has significant improvements. Table 11 : Ablation study on the number of the predicted vectors. The experiments are conducted on UCF Sports MMVP splits. The listed results are from the models trained for 100 epochs (models were all trained for 300 epochs in the main manuscript).The design of the motion graph interaction module are following the intuition that both spatial connection and temporal connection should benefit the graph learning.Here we also show the experimental results in 12 : Ablation study on graph interaction module. The experiments are conducted on KITTI and metrics show evaluation on the t + 1 results.Our research on video prediction has identified key differences between systems designed for short-term and long-term predictions. Short-term systems typically use fewer frames to predict the immediate next few frames, while long-term systems are tasked with forecasting an extensive sequence of future frames. The design logic and objectives of these systems, thus, vary significantly. Table 13 SIMVP [1] , MMVP [2] , STRPM [15] , STIP [16] , DMVFN [13] MVCD [58] , MaskViT [8] ,VIDM [59] ,ExtDM [60] High resolution videos, pixel-level accuracy, real-time application Conditional video generation, semantic-level accuracy Table 13 : Experiment setting comparison between short-term and long-term video prediction task long-term video prediction usually involve with diffusion-based generative models, which is designed to feed the need of predicting long videos. The evaluation of such methods usually emphasize on if the predicted video is semantically correct given the input video frame(s). While in this study, we emphasize our method's advanced motion modeling and significant reduction in computational costs, essential for short-term prediction of high-resolution videos. Our system is tailored for short-term video prediction, with evaluations conducted along this line of work.We plan to further exploit the motion graph's potential as an efficient motion representation tool and develop advanced, motion graph-based systems for long-term video generation in our next work.The video prediction is always a challenging problem. Especially for those video sequences with abrupt motion which can be hardly indicated by the previous video frames. The proposed method formulates the video prediction as a motion prediction problem and outperforms most of the existing methods by using motion graph to better capture the motion hints from the input frames. However, when evaluating the qualitative results, we still find some failure cases that require additional research efforts to solve. In Figure 10 , we showed typical failure cases in UCF Sports dataset. We notice that most of the failures cases are in the action of kicking and diving, which usually include fast, unpredictable motion that requires stronger video understanding capability of the model.To better understand the initialization of the node embedding, here we visualize the tendency feature and the location feature. We first extract the tendency feature and location feature of each node in the motion graph and apply a K-means clustering to the extracted features. For the tendency feature, we set the cluster number to 2; and for the location feature, we set the cluster number to 4 for better visualization.From Figure 11 , we see that using the learned tendency feature, the system should be able to distinguish the dynamic areas from the static areas. If we further enlarge the cluster number, we can see more clearly that the tendency features embed the different motion patterns of each feature patch in the frame. For the location feature, in the paper, we have shown that removing the location feature from the node initialization will result in a performance drop. From Figure 12 we observe that the location feature may contain information that is related to the camera projection mode. For cityscapes and the KITTI, which use wide-range cameras, the clustering pattern of the location feature is very different from the UCF Sports whose projection mode is possible to be orthogonal projection. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Equal contributions. The work is mostly done during Yiqi Zhong's internship at Microsoft. 38th Conference on Neural Information Processing Systems (NeurIPS2024). 


Decision: Initially, the reviewers raise some questions regarding the robustness to different backbones, the prediction quality with respect to resolutions, occlusion and out-of-view cases, video prediction evaluation setting, lack of component-wise ablation, sensitivity in graph construction, diversity of metric evaluation, etc. Most of these questions are addressed in the rebuttal and recognized by reviewers. Eventually with the merit of the work, this paper receives two accepts and two borderline accepts, with agreement for acceptance. The AC recommends to accept. Authors are encouraged to revise the paper according to the reviews, including adding a discussion about occlusion, long-term video prediction, and the comparison with other methods, etc.

Reviews: review: paper_summary: paper_summary: The authors propose a Motion Graph for predicting future video pixels. To achieve this, they introduce three modules: 1. Motion Graph Node Construction; 2. Edge Construction; 3. Graph Interaction Module; 4. Video Prediction Model. The first three modules encode patches and their interactions in a frame, while the fourth module decodes future frame pixels. Experiments on UCF Sport, KITTI, and CityScape demonstrate the method's high efficiency and construction quality.
strength_weakness: strength_weakness: 1. It's better to present an overview figure for the framework. It's not easy to grasp the overall structure by reading about the four separate modules.

2. Figures 2-4 have fonts that are too small, making them hard to read.
questions: questions: 1. Which backbone are used to extract features during motion graph construction. Does the method robust to different backbones?
2. Does the prediction quality decrease with respect to resolutions? is it possible to analyze prediction quality with large resoultion settings.
limitations: limitations: The authors discuss and partially address the limitations of the paper.
paper_summary: paper_summary: This paper proposes a motion-based method for video prediction. They design a new motion representation named motion graph that transforms patches of video frames into interconnected graph nodes. The proposed video prediction pipeline, empowered by the motion graph, exhibits substantial performance improvements and cost reductions. Experiments on UCF Sports, KITTI, and Cityscapes are conducted.
strength_weakness: strength_weakness: - Motivation and Occlusion / Out-of-View cases. As the authors claimed in lines 33-37 and Figure 1, existing methods struggle to effectively handle occlusion cases. However, the paper does not provide a corresponding evaluation to demonstrate the effectiveness of the proposed motion graph in complex situations involving occlusion or out-of-view scenarios. This is a significant omission, as these challenges are prevalent in real-world video prediction tasks. Furthermore, based on my understanding of the work, the proposed motion graph may not adequately address these problems. Specifically, occluded and out-of-view object/background do not appear to be considered in the graph, as the connections and features captured in the motion graph are derived from the visible parts of the targets. This raises concerns about the method's ability to maintain accurate predictions when dealing with occlusions or objects moving out of the camera's field of view. The authors should provide a thorough discussion that includes both the theoretical principles and empirical evidence.

- Video prediction evaluation setting. While the proposed motion graph method shows promising results in predicting a few frames (up to 10), the paper lacks an evaluation of its performance in predicting a larger number of frames. Other video prediction works [A, B, C] have evaluated their methods on longer prediction horizons, providing a more comprehensive assessment of their models' robustness and accuracy.

[A] ExtDM: Distribution Extrapolation Diffusion Model for Video Prediction, CVPR24

[B] Efficient Video Prediction via Sparsely Conditioned Flow Matching, ICCV23

[C] MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation, NeurIPS22

- Comparison with MMVP. The proposed motion graph method appears to be quite similar to the motion matrix used in MMVP, which also provides a dense connection among frames. Although I appreciate the discussion provided in the related work, the authors do not sufficiently clarify the differences and advantages of the proposed method compared to MMVP. This lack of distinction makes it difficult to understand the unique contributions and potential improvements offered by the motion graph.

- Comparison with other motion-cues-guided video prediction methods. The paper lacks a comparison with other motion-cues-guided video prediction methods, such as MotionRNN, which also leverages a vector-based motion representation. The authors should explain how the feature warping mechanism in MotionRNN compares to the feature processing in the motion graph. What are the potential benefits or drawbacks of each approach? Are there specific scenarios where the motion graph's approach to feature handling provides a clear advantage?

- Lack of a component-wise ablation. The reviewer noticed that there is no component-wise ablation study for motion feature learning, upsampling, decoding, image wrapping, and interaction modules. It makes it difficult for the audience to identify which parts of the methodology are effective and contribute most to the overall performance. The authors should conduct a thorough ablation study that isolates and evaluates the impact of each component of their proposed method.

- Complexity of the proposed method and presentation. Actually, the proposed method is somehow non-trivial, making this work hard to follow. Accurate presentation can make the work easier to understand.
questions: questions: See weakness
limitations: limitations: The authors indicate limitations in the conclusion section.
paper_summary: paper_summary: The paper introduces a graph-based methods to predict video frames through motion prediction. The proposed motion graph captures complex spatial-temporal relationships by transforming video frame patches into interconnected graph nodes. This method improves performance and reduces computational costs compared to SOTA methods on UCF Sports, KITTI and Cityscapes.
strength_weakness: strength_weakness: 1. Slower inference speed compared to optimized methods like DMVFN.
2. Struggles with predicting abrupt or unpredictable motions.
3. Unclear if the motion graph captures hierarchical / semantic feature of motions.
4. Does not address the stochastic nature of human videos.
5. Limited diversity in evaluation datasets, focusing mainly on sports and driving scenarios.
questions: questions: 1. Do the authors see a future where such work might be useful for improving video generative models?
2. Can the authors explain what the weight components are for? It seems that the dynamic vectors can represent motion even without the weighted component. If it does represent something meaningful, can the authors show what they represent after training? For example, might it correspond to the change in depth?
3. How does the motion graph approach compare to recent transformer-based video prediction methods in terms of performance and efficiency?
4. How sensitive is the method to the choices made in graph construction (e.g., number of neighbors, edge types)?
5. How does the motion graph approach scale with longer video sequences? Is there a practical limit to the number of frames it can handle efficiently?
limitations: limitations: The authors briefly mention limitations regarding inference speed and handling sudden motions. However, they could provide a more comprehensive discussion of potential limitations, such as scalability to longer videos or generalization to diverse video types.
paper_summary: paper_summary: The paper proposes the motion graph method in predicting the video frames by exploring the spatio-temporal relations among frames from the limited data.
strength_weakness: strength_weakness: Please provide more details about the setup of the datasets such as KITTI and Cityscapes that are not originally used for testing video prediction.

The results on Cityscapes and KITTI datasets based on LPIPS, SSIM, PSNR are needed to be explained in detail in the evaluation methods either in the main text or Appendix.
questions: questions: .
limitations: limitations: The authors address the limitations briefly.

Comments: title: Official Comment by Reviewer Mw8x
comment: We deeply appreciate your suggestions and will incorporate a discussion on occlusion, long-term video prediction, and comparisons with other methods into the revised manuscript.
Thanks for the rebuttal. Most of my concerns are addressed.

I'd like to see the discussions about occlusions, long-term VP, and other related methods in the revised version.
Thank you for the valuable suggestion. We recognize that few existing studies report PSNR and SSIM metrics for evaluations on the KITTI and Cityscapes datasets. To support future research in the video prediction community, we will add a new table and discussion to Appendix A.3, detailing these metrics for our method. Additionally, we will compile and compare available results from other methods. We also plan to release the result images from these two datasets to enable more convenient quantitative and qualitative comparisons for future researchers.
I have read the rebuttal of the authors.

I have seen the authors' main point on generating their literature from the previous work using KITTI and Cityscapes datasets for video prediction. I appreciate the effort put into this and hope that further researchers can benefit from that literature in the future and the related works multiply based on video prediction.

As I tend to accept this work now, I hope to see the evaluation methods of LPIPS, SSIM, and PSNR on Cityscapes and KITTI datasets in the appendix in your revised version of the manuscript.