Motion Graph Unleashed: A Novel Approach to Video Prediction
Authors: Yiqi Zhong
Luming Liang
Bohan Tang
Ilya Zharkov
Ulrich Neumann
Venue: NeurIPS 2024 poster (2024)
Keywords: Video
Motion
Low-level computer vision
Frame synthesis
Primary area: machine_vision
TL;DR: This work proposed a novel motion representation called motion graph to facilitate more accurate video prediction with significantly lower GPU consumptions and a smaller model size.

Abstract: We introduce motion graph, a novel approach to address the video prediction problem, i.e., predicting future video frames from limited past data. The motion graph transforms patches of video frames into interconnected graph nodes, to comprehensively describe the spatial-temporal relationships among them. This representation overcomes the limitations of existing motion representations such as image differences, optical flow, and motion matrix that either fall short in capturing complex motion patterns or suffer from excessive memory consumption. We further present a video prediction pipeline empowered by motion graph, exhibiting substantial performance improvements and cost reductions. Extensive experiments on various datasets, including UCF Sports, KITTI and Cityscapes, highlight the strong representative ability of motion graph. Especially on UCF Sports, our method matches and outperforms the SOTA methods with a significant reduction in model size by 78% and a substantial decrease in GPU memory utilization by 47%.

Motion Graph Unleashed: A Novel Approach to Video Prediction
Yiqi Zhong 13+ Luming Liang'
Bohan Tang² Ilya Zharkov! University of Oxford UUniversity of Southern California
UIrich Neumann
MMicoosoft
Lyiiqizhong, lulian, zharkov) wwwcemmicttttooooooooo com
bohan. twwgleng..nnn... ax.ac.uk
3 Eyiqizhon, uneumann]
Abstract
We introduce motion graph, novel approach to the video prediction problem, which predicts future video frames from limited past data. The motion graph transforms patches of video frames into interconnected graph nodes, to comprehen- sively describe the spatial-temporal relationships among them. This representation overcomes the limitations of existing motion representations such image differ- ences, optical flow, and motion matrix that either fall short capturing complex motion patterns or suffer from excessive memory consumption. We further present a video prediction pipeline empowered by motion graph, exhibiting substantial performance improvements and cost reductions. Experiments various datasets, including UCF Sports, KITTI and Cityscapes. highlight the strong representative ability of motion graph. Especially outperforms the SOTA methods with a significant reduction in model size by 78% and substantial decrease in GPU memory utilization by 47%. Please refer to this link for the official code.
UCF Sports, method matches and
Introduction
Video prediction aims at predicting future frames given a limited number of past frames. This technology has potential for numerous applications, including video compression, visual robotics, and surveillance systems. A critical aspect of designing an effective video prediction system is the precise modeling of motion information, which is essential for its successful deployment.
A. Hard cases for exiating motion representations
B. Proposed Graph based Motion Representation
Graph
Graph Nodo
Tamporsl Etige
Detoraatoon
Conditution
Cookis
Eepaning
Figure 1: (A) Hard cases which cannot be properly modeled by most existing representations. (B) Motion graph transforms single-frame patches into interconnected nodes, describing the spatial- temporal relationships. Future per-pixel motion dynamic vectors are then predicted on this graph.
'quual contributions. The work mostly done during Yiqi Zhong's internship at Microsoft.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

At first glance, video prediction appears as sequence prediction problem. with conventional ap- proaches using advanced sequential data modeling techniques such as 3D convolution 2 recurrent neural networks 3
71 and transformers R 9 10 These methods implicitly model motion and image appearance to help reason sequence evolution. However, the implicit approach to motion modeling can result in inefficient learning, which stems from the inherent difference between video prediction and typical sequence prediction: in typical sequence prediction, features tend to evolve uniformly; whereas video prediction involves not only static elements like object appearance but also dynamic elements intricately embedded the sequence, c.g., motion-related information, including object pose and camera movement. Effectively managing both the static and dynamic aspects of the hidden feature space presents a great challenge.
Instead of treating video prediction purely as a sequence prediction problem, an alternative approach to explicitly model inter-frame motion, and redefine the task primarily as a motion prediction 10, 2] For this approach, it is crucial to select appropriate motion repre- sentation, which should exhibit expressiveness by accurately encapsulating motion with detailed information like direction and speed. However, as Section 2 and Table I will elaborate, existing video motion representations have limitations. Some, as Figure A) demonstrates, lack representative ability for complex scenarios. For example, optical flow is unable to handle motion blur and object distortion/deformation resulting from perspective projection. Others over sacrifice memory efficiency to model complex motion patterns, such as the motion matrices proposed in MMVP (2).
problem [11 12 13
To fill the gap in existing research, we propose novel motion representation: the motion graph. which treats image patches of the observed video frames in the downsample space interconnected
graph nodes based on their spatial and temporal proximity; See Figure T (B) for an illustration. This approach achieves both goals of representativeness and compactness by generating accurate predictions while conserving memory and computational resources. For the representative ability, our motion graph enriches each node with dynamic information, including multiple feasible weighted flows toward the subsequent frame. This method captures a broader range of motion patterns compared to single-direction representations like optical flow and enhances error tolerance. Additionally, the motion graph excels in compactness by avoiding computation-heary operations such as stacked convolutional layers. To reduce computational complexity, motion graph to a fixed number of neighbors for each node. Consequently, compared to previous representations (e.g., motion matrix, keypoint heatmap), the motion graph offers more sparse and memory-efficient modeling of motion, effectively transcending the limitations of 2D image structures.
also limit the connections in the
Based on the motion graph we have introduced, we present novel video prediction pipeline. The representative ability and compactness of the motion graph allow our system to have achieved notable enhancements in computational efficiency. We test various scenarios with different motion patterns on three well-known datasets, UCF Sports, KITTI, and Cityscapes. Across all three datasets, our approach consistently exhibits performance that aligns with or surpasses the state-of-the-art (SOTA) methods with significantly lower GPU comsumption.
2 Related Works
In this section, we mainly discuss the existing video prediction systems that apply to the setting of short-term high resolution video prediction, please see more details about the setting in Section A..
Based on whether they explicitly model motion, existing video prediction methods can be put into two categories. Methods that do not explicitly model motion usually consider video prediction purely as sequence prediction problem (3] They use advanced sequential modeling techniques such as recurrent
estimate the evolution of image features in the hidden space along the temporal dimension. However, video prediction holds a unique property that is unlike text generation or other sequence prediction problems: that is, most features in video sequences (e.g., object appearance, scene layout, texture of the background) usually remain unchanged, or have no major swift change, from frame to frame. Yet, sequential modeling techniques treat the whole image features uniformly: using them for video prediction thus requires modification in the inner structure of certain sequential modeling architectures to accommodate this property of video prediction. Some works choose to enhance the spatial consistency by adding complex residual shortcuts 4 17 21, or by decomposing temporal and spatial information into two separate hidden spaces, hoping that the network will figure out which to maintain and which
neural networks [4] 5 14
15
16
17 [8] and transformers [8]
10
19 20

---

Although such accommodations sometimes help systems reach impressive prediction accuracy (given the powerful sequential modeling capability of the advanced techniques), these methods tend to have larger model sizes and complicated architectures.
to change 11
16].
To improve the efficiency of motion modeling in systems, this paper extends the idea shared by the second type of methods, which treat video prediction as motion prediction problem, and explicitly model the motion hidden in image sequences. Such methods either reason future motion patterns using certain representations with image features as the input, or solely use motion representations as the input for prediction. There are four lines of work on motion representations. First, image difference [O] uses the subtraction of two consecutive frames to represent the dynamic of image sequences. It requires almost no computational cost to calculate but does not directly describe motion. Second, optical flow/ voxel flow [2 23 24 [3] describes the pixel-level motion. But, it usually requires auxiliary or out-of-shell models to generate, and is limited to only modeling one-to-one relationships (2). which can be disadvantage, especially for scenarios with ambiguous pixel-to-pixt correspondence (e.g., motion blur). Third, key point trace [2] serves highly structured scenarios (e.g., videos with static backgrounds and dominated by human motions) but can be inefficient for more complex content, including crowded scenes with multiple objects. Fourth, the recent motion matrix 2 describes the all-pair relationship between consecutive frames to overcome the drawback of optical flow. While motion matrix supports a more efficient and compact system of a notably smaller size, concerns were over the computational and space complexity of its operations (2).
Table 1:
Motion representation comparison. We assess the representative ability by judging how accurate the motion can be described, especially for the hard cases demonstrated in Fig A).
Motion Representation
Out-of-shell Model
Representative Ability
Space Complexity
Image difference [1]
n/a required Some required n/a
low
O[n]
Keypoint trace [12
medium
O[n)
Optical flow/ voxel flow (22 24
medium high
((n)
Motion matrix 2
O(r²)
Motion graph (ours)
n/a
high
O(n)
Besides the above motion representations, we notice that graph, as a sparse data structure that can
accurately describe the relationships between entities, has been widely adopted in ordinary motion
prediction systems, such as trajectory forecasting [25] 26 27 and human motion prediction [28] 29
30). Inspired by these successful adoptions, we propose to represent the motion in videos using graph structures, named motion graph in this work.
The major difference between motion graph and previous motion representations is that motion graph can describe many-to-many temporal and spatial correspondence with low space complexity. Such sparse representations not only enhance representative ability but also advantageously i) replace most 2D convolution operations by graph operations which are mostly implemented by linear layers, leading to faster and more parameter-efficient systems; ii) enable convenient spatial and temporal interactions among feature patches, promoting higher learning efficiency and prediction accuracy.
3 Methodology
In this work, video prediction is approached as motion prediction problem, utilizing novel representation called the motion graph. This representation is designed to effectively capture the intricate motion dynamics within video sequences. Leveraging the motion graph, we develop video prediction pipeline that achieves high performance with reduced model size and GPU memory requirements. Section 3.1 details our problem formulation and notation. Section 3.2 explains the construction of the motion graph from input video sequences. Finally, Section 3.3 delves into our video prediction pipeline, illustrating how the motion graph is used for effective video prediction.
3.1 Problem Formulation
Given a video sequence with T frames (I, E B = 0,,,,, 1). video prediction system aims to predict the next T' frames (Ir E xxx T,T T....,T+ T' -1]. We approach this task by regarding video prediction pixel-level motion prediction problem.
3

---

Y) 1%
Dynamic Vectors
cosiin similartty
(m))
uay sMaxpooting
Conratenation
9enc
(m)
[+1
!!
L+1
(x,y)
Sloe
Figure 2: Motion graph node construction: Cosine similarity, denoted by (.), between patch features in consecutive frames is computed to further choose top directions for each patch. Tendency () and location features v (f((m) then generated based on these k vectors and the patch location.
Consider a pixel located at (x, y) in given video frame 1, Our system is designed to predict dynamic vectors starting from this pixel, pointing to the possible locations in the target future frame Ir. These vectors represent the pixel's anticipated motion from its current position in I, to its future position in Iv. Mathematically, we express the dynamic vectors of all pixels across the observed frames as E x xxx3 where P Pt.... = [[x1] Ay,, W1]. [Ary, Ayk: wy]] with both the motion direction (ArR, Ayı: and the weighted component (wx). The transformation of previous frames into future frames performed by a pixel-level image warper, denoted as W. The warping is based on the predicted dynamic vectors and is formally defined by the following equation:
IT- (P, Io, ..., Ir-1)-
(1)
This formulation allows us capture and translate the intricate motion dynamics within the video sequence into future frame predictions with enhanced accuracy.
3.2
Motion Graph
3.2.1
Intuition
Inspired by advancements in general motion prediction systems 25 26 30), we posit that modeling semantic correlations among image patches within observed frames is essential for improving current motion representations in video prediction. To capture such correlations, information of each image patch across a video with T observed frames. This is achieved by multi- P(M),, generated by inputting each observed frame into shared image encoder Jenc and applying pixel unshuffle technique [31] reshape all M feature maps to the resolution of the smallest features. Here, the smallest resolution of the features
first model the semantic
scale image patch feature mappings, F= = =
generated by Jene is H, X We, and f(m) E R.. T xw..HxW..xCC. represents the feature map in the m-th scale containing the Cn -indennnsion features T x H, x W, distinct image patches.
Based on F. we construct a multi-view graph, named motion graph, to capture the semantic correla- tions among T H, W. image patches. This graph represented as: = (V, g(1)
g(M) where V- = - ... , Vrn,,,-11 w.-1) denotes the set of nodes each corresponding to an image patch, and (mm) denotes the edge set for the m-th view capturing the correlations driven by the semantic information within p(m). We detail the graph construction and the graph-related operations as follows.
3.2.2 Node Motion Feature Initialization
(mm)
For each node v; in the m-th view, we initialize its motion feature vs IR
with two components: The tendency feature
the tendency feature yrrm) <R and the location feature ! ER/
(m)
(m))
v (m))
captures the node's 's motion-related attributes relative to nodes in the subsequent frame and the
location feature v ffmm) models the normalized absolute location of each node in a frame.
(mm) in three steps. First, for each
Inspired by prior works 2 32
we generate the tendency feature v
node vú we form a node list c!²) by choosing top- K scoring image patches from the subsequent
frame based on the cosine similarities computed by p p((m) and ((m)
B.w, Jn J+1 By doing so, we mitigate

---

the risk of false positives and better accommodate complex motion patterns, especially in downscaled spaces where an image patch may exhibit multiple potential movements. Second, we define the dynamic vector for each node v; as d (T))) = |Ax Ayi, wi,... ..... ArK, AYk wK] E IR where Arg, Ayk; indicate the motion direction and wi is the cosine similarity score between v and the k-th
node in c. Here, an exception is made for nodes in the last observed frame Ir-1, where we apply zero-padding to the dynamic vectors, as I is unknown. Finally, we generate the tendency feature for each node by feeding each dynamic vector a!m) into multilayer perceptron (MLP) Itde(.) followed by max-pooling operation fagg(.), as shown in: ((m) Pagg Pagg(9tdc (d.
v veflm)
The location feature vi motion patterns. For instance, pixels on the sides of frame may appear to move differently than pixels in the center for street views collected by wide-range moving cameras, due to the perspective projection effect. We use another MLP gloe (.) to extract the location feature and define it as:
encodes the position of a pixel in a frame. Pixel positions influence
(m)
y
gloc
W
H
(mm)
Finally, the motion features of each node
are initialized as follows:
(((m) (m)
(4)
where ((-) denotes the concatenation operation. Figure details the node motion feature initialisation.
2
3.2.3 Edge Construction
After initializing the node features, we construct edges in the motion graph to capture the semantic relationships between image patches in the observed frames. We further represent the edge set of the mnth view as {(m) = {{S g gF(m) }, where gs(m) is spatial edges, (B(m) is backward edges, and g(m) is forward edges. Generally, spatial edges are posit on that neighboring image patches a frame likely influence each other's future motion, and backward and forward edges connect nodes across adjacent frames indicating potential motion paths. Notably, nodes in the first frame are not assigned backward edges, and nodes the last frame are not assigned forward edges.
We construct these edges in two steps: 1) finding the neighbors of each node connected by spatial, backward, and forward edges respectively; and 2) generating the three types of edges by connecting
the neighboring nodes found. For the first step, we denote // AAB((") and Appmm) as sets containing neighbors of 8; connected by the spatial, backward and forward edges, respectively. Then, the neighbors of each node v; are found by solving the following optimization problems:
(m)
arg max
1.0
|1..1
s.f. Ω C S, [|]] k,
arg max
C
(m)
|1..1
s.t. Ω C B, ||] k,
(5)
Ng arg max [C
11.1
s.t. Ω g F,, |S]| = k.
10
where ((m) E TH, contains cosine similarity scores between nodes in V computed by f(m), k E Z is hyperparameter, S, [u,
b,,, = =L ]). B, = {u, V L W, - 1}. For the second step, we construct (m))
W. + and F; = (v, E V HnW =
gB(m)) and p( by connecting each v; with nodes in Asstm) _ABB(") and Mpr(m)) respectively.
3.2.4 Motion Graph Interaction Module
After constructing the nodes and establishing the edges in the motion graph, we enable information flow within the m-th view of the graph via the message-passinn operation Imp defined as follows:
((m)
9mp (v(m), gin(m))
(6)
5

---

Motion Graph Interaction Module + (m)
Spatial: (mm)
xT
Spatial
Temporal Forward
Spatial
Temporal
Backward
Spatial
Temporal Forward
v (m))
gro (vllin geniny
Motion Graph In the -th Scale
Motion Graph In the mn +h Scale
Temporal
((w)
Impl Amp(yinn) gaam
Backward
Inside the interaction module for the m-th view f(m), The spatial and temporal message passing are iteratively conducted and repeated T - times, where T is the observed frame number.
Figure 3:
where ((m) E @TH..W,xdlm is the input node features, ((m) E TR.'H. is the updated node features, and Ein (m)) denotes an edge set. The operation gmp facilitates the information exchange between nodes connected by edges in gin(m) In practice, the spatial message passing is implemented as 2D convolution layer, while for the temporal message passing we use graph neural network to implement 9mp- See implementation details in the appendix.
With the message-passing operation Imp, we introduce motion graph interaction module, denoted as & and illustrated in Figure 3 The goal of & is to ensure that even nodes in the last observed frame informed about the motion dynamies from the first observed frame, and vice versa. To achieve this comprehensive information flow, we implement T-1 rounds of full information transition, where T is the total number of observed frames. Each round of transition includes twice the spatial message passing, once the temporal forward message passing, and once the temporal backward message passing. The combination of spatial and temporal interaction in T ensures holistic information integration for accurate and comprehensive motion prediction in video sequences. The design of + allows for thorough and balanced dissemination of motion information throughout the motion graph.
3.3
grotooegrapph-empooere Video Prediction
Step II: Motion Feature Upsampling Decoding
Interaction
Multi-flow Forward Warping
"OCERT spend C
Interacion
V
+ SR
Bas
Interaction
FSH
b
Step Image Warping
Motion Feature Learning
Figure 4: Pipeline overview. After decoding per-pixel motion features into dynamic vectors, we perform multi-flow forward warping for future frame generation.
By constructing the motion graph, we create a tool to extract motion information from the observed video frames. The following passage describes how to conduct video prediction using the constructed motion graph and its related operations. As Segure/iiniee indiicates, Figuraajidiiatts, to predict the unknown video frame involves three main steps, which are elaborated in the following three subsections.
3.3.1 Motion Feature Learning
Multi-view motion feature update. After having the motion graph G = (V, (((M)))))))))))) with the node motion feature v ((m) initialized by Eq. (4) in the m-th view, we use the motion graph interaction module introduced in Section 3.2.4 to update v (m)) via the following formula:
f.f (m)
(vm)) g(()
where (m)) T...H. x is the updated node features in the m-th view of the motion graph.
Multi-view motion feature fusion. Once the node feature in each view of the motion graph is updated, we concatenate the node features from each view and apply a fusion module v Juse to integrate these multi-view features into a unified representation as follows:
ffuse V Juse (yocat ( m 1,.., M)),
(8)

---

where fluse E RRTHH,WW, MW,,,,WWW,,excee XOrude with Codee denoting the dimension of the node features.
3.3.2 Motion Feature Upsampling and Decoding
In this step, we transform the fused multi-view motion feature fase into a 2D structure with a resolution of H₃ x W,, This feature is then upscaled to match the resolution of the original video frames (H x W) using a motion upsampler, OSR. The architecture @ is inspired by ResNet- based image super-resolution networks [33] which progressively refine features from lower higher resolutions until reaching the original image size. This upsampling process is key to predicting pixel-Ievel motion features for all observed frames, which extend toward the next future frame. By incrementally adjusting the resolution, S effectively bridges the gap between the multi-view motion information and the high-resolution requirements of accurate pixel-level motion prediction.
Upon obtaining the pixel-level motion feature fuse. we use a motion decoder, Qdecs to convert it into pixel-level dynamic vectors P. As defined Section 3.1 P contains potential motion directions with associated probability scores for each pixel. This decoding output is compatible with the motion features in the motion graph, which are generated by the &s dynamic vectors of image patches.
3.3.3 Image Warping
After having the dynamic vectors through the decoding process, use them warp the observed frames into the future frame. Unlike traditional methods that often use optical-flow-basee backward warping, our approach follows the design logic of the motion graph feature learning as well as the output format to use a multi-flow forward warping technique. This method, drawing from the image splatting concepts in prior works [34 35), is visually detailed in Figure 4 Forward warping allows each predicted dynamic vector to directly contribute to the construction of the future frame. To aggregate the contributions of multiple vectors each pixel location in the synthesized frame, we apply a normalization operation (norm- It normalizes the weight of each vector based on the sum of their weights, ensuring an even and balanced contribution to the final pixel value the future frame.
Experiments
For evaluation, we trained our video prediction pipeline in end-to-end fashion on three public
datasets: i) UCF Sports [36] (i.c. 150 video sequences emphasizing various sports scenes, with
frame resolution, and two different data splits date- one from STRPM [15), one from MMVP [2] evaluation is on both splits): ii) KITTI (37 (i.e. 28 driving videos with a resolution of 375 x 1242;
following previous works 38 13). the image frames are resized to 256 x 832); and iii) Cityscapes (3
(i.e. 3,475 driving videos with 2.945 in the training set and 500 in the validation set). For each dataset, by default, we set & mar(10. 1% x H x W.) which is 1% of the smallest feature map's resolution and no larger than 10 for efficiency consideration. Table 2 shows the configuration for each dastaset. More training and implementation details are the appendix.
Table 2: Dataset configurations
Dataset
Resolution
H.
W,
Input Frame
Ourput Frame
Training Loss
k
UCF Sports
x
32
32
mean--uuure error (following [5]
2)
KITTI
256 x 832
16
52
11 Perceptual loss(following
Cityscapes
512 1024
32
64
I1 Perceptual loss (following 13]
10
4.1 Public Benchmark Comparison
On the UCF Sports STRPM split, we evaluate the method on two metrics following previous research [15]: peak signal-to-noise ratio (PSNR) and learned perceptual image patch similar- ity (LPIPS) [4]]. The proposed method is compared with existing methods for their results of the 1st and 6th future frames (t 5, t = 10 in Table B). Table B shows that our method matches the SOTA performance on the PSNR metric and outperforms all previous methods on the LPIPS metric. On the UCF Sports MMVP split, the validation dataset has been divided into three categories: the easy (SSIM < 0.9), intermediate (0.6 < SSIM < 0.9), and hard subsets (SSIM < 0.6), which take up 66%, 26%, and 8% of the full set respectively. We evaluate the methods on PSNR, LPIPS, and Structural Similarity Index Measure (SSIM). Table 4 shows that the proposed method outperforms all other methods.
7

---

Table 3: Performance comparison UCF Sports STRPM split
5
10
Method
PSNR
LPIPS x
PSNR f
LPIPS 1000
Convl STM (NeurIPS 2015)
26.43
32.20
17.80
58.78
BeyondMSE (ICLR 2016 [1
26.42
29.01
18.46
55.28
PredRNN (NeurIPS 2017)
27.17
28.15
19.65
55.34
PredRNN=+4 (ICML 2018)
27.26
26.80
19.67
56.79
SAVP (arXiv 2018)
27.35
25.45
19.90
49.91
SV2P HCLR 2018)
27.44
25.89
E3D-LSTM (ICLR 2019) [17
27.98
25.13
20.33
47.76
CycleGAN (CVPR 2019)
[44
22.95
CrevNet (ICLR 2020) [45]
28.23
23.87
20.33
48.15
NN (CVPR 2021) 7
27.67
24.23
20.01
49.20
STRPM (CVPR 2022)
28.54
20.69
20.59
41.11
STIP (arXiv 2022) [16
30.75
12.73
21.83
39.67
DMVEN (CVPR 2023)
[3]
30.05
10.24
22.67
22.50
MMVP (ICC 2023)
31.68
7.88
23.25
22.24
Ours
31.70
6.61
23.27
19.94
Table 4: Performance comparison on the UCF Sports MMVP split.
Method
SSIM
Easy (SSIM
LPIPS
Insermediste (0.6
1
LPPSS
Hard (SSIM
0.6)
LPIPS
Model
STIP L6
0.8817
23.17
0,1626
0.491
30.65
0.1056
0.8351
23.97
02271
0.4673
13.97
0.4450
18.5M
SimVP L
0.9189
29.97
0.132
00664
32.87
0.0584
0.8855
25.79
0.1951
0.6367
18.99
0.5600
3.47M
MMVP
0,9300
30.35
0.062
0.9744
33.05
0,0580
0.8970
26.29
0.1569
0.7303
20.84
0.3510
2.79M
Curs
0.934
30.49
0J823
03685
35.35
0.444
0.978
26.36
0.1348
0.764
2033
0.233
0.0M
Table 5: Evaluation on Cityscapes (39 and KITTI 37 datasets. "RGB", "F", "S" and "I" denote video frames, optical flow, semantic map, and instance map: t 3 and t respectively indicate the average performance of the next and frames, Results with are copied from [3]
Cityscapes
KITTI
Method
Inpot
MS. MS_MM Ftf
LPIPS(X
1PIPS(x1001 1
MS 10--y-
SSIMSS
LPIPS 10 1O
(
t
Viizvid (NowrPPS 2018)*
46
RGB4S
88.16
80.55
75.13
10.58
15.92
20.14
NIA
N/A
N/A
N/A
N/A
N/A
SegEvid (CVPR
47
88.32
NIA
N/A
25.99
NIA
FVS (CVPR 2020)*
RGB4S+]
89.1
81.13
75.68
8.3
12.98
16.5
79.28
67.65
60.77
18.48
24.61
30.49
SADM (CVPR 2021>*
49
RGB--SFF
95.99
I/A
83.51
7.67
NIA
14.93
83.06
72.44
64.72
14.41
24.58
$1.16
ProdNet (ICI 2017y"
50
RGB
$4.03
79.25
75.21
25.99
29.99
36
56.26
$1.47
47
55.35
58.66
62.95
(ICLR
DVF (ICCV 2017)*
RGB
83.85
76.23
71.11
17.37
24.05
28.79
53.93
46.99
42.62
32.47
37.43
41.59
CoeWWise (CVPR 2022
S2
RGB
92.8
NIA
83.9
8.5
NIA
15
82
N/A
6..7
17.2
N/A
25.9
OPT (CVPR 2022)*
[3
RGB
94.54
86
80.4
6.46
12.5
17.83
82.71
69.5
61.09
12.34
20.29
26.35
DMVEN (CVPR 2023y
Ours
RGB
91.35
87.82
82.11
4.13
8.12
11.70
$7.70
77.15
69.72
9.30
16.94
22.90
In 5 compare the proposed method with existing methods the Cityscapes and KITTI datasets. Following previous research, we report the Multi-scale Structural Similarity Index Mea- sure (MS SSIM) and LPIPS of the first future frame (t++1), the average numbers of the next three future frames (f++) and the average numbers of the next five future frames (t++5).
The UCF Sports dataset features sports scenes and thus contains large amount of fast movements and motion blurs. The leading performance both Table and validates that the proposed motion graph helps the network better interpret the motion in the input video sequence and facilitate more accurate prediction. Meanwhile, the qualitative result in Figure demonstrates the method's ability capture and restore intricate image details during the prediction.
For KITTI and Cityscapes datasets, videos are captured from outdoor, large-scale traffic scenarios through cameras with perspective projection, which results in drastic object distortions on both sides of images as well as unstable lighting conditions. Our method matches SOTA performance in terms of quantitative evaluation (see Table 5). In Figure
conducted qualitative comparison these
datasets with two SOTA methods, OPT 13 and DMVEN [13]. which are all optical-flow flww-floowweeeee based.
We noticed that as non -en-eeerertive model, our proposed video prediction system may face challenges with occlusions that require the generation of unseen objects. However, excels scenarios with scenarios involving occlusions of known objects, as showcased in white walls of the third column in F Notably, in scenes with partial obstructions such as the white wall behind the cyelist, our system adeptly employs multiple motion vectors per pixel to reconstruct occluded areas. This feature

---

MMVP
Ouurs
GT
Figure 5: On the UCF Sports dataset, our method recovers richer image details than MMVP
2
(a) KITTI Daraset
(b) Cityscupes Dataset
List Frame
OPT
DMVFN
Ours
Figure 6:
Qualitative comparisons with OPT (53) and DMVEN [13] Our method maintains the
object structures better than both methods while holding higher motion prediction accuracy.
also supports precise management of object expansion due to perspective projection, exemplified by the green car in Figure [6s first column.
Moreover, approach is adept at managing objects exiting the camera's view. By explicitly modeling the motion of image patches, the motion graph predicts when features are about to leave the scene. Thus, any image patches projected to move out of view are not included in the final prediction. This is demonstrated in the 1st, 3rd, and 4th columns of Figure where objects like a cyclist and the front of blue truck are shown moving out of frame. Our method successfully and uniquely captures and represents these movements, unlike how the other methods evaluated.
To assess the compactness of the motion graph, we conduct a model consumption analysis. We compare the proposed method with SOTA methods on each dataset in terms of the model size and maximum running GPU memory (the system configurations are identical to the models that generate the numbers in the corresponding tables above). Examination of Table 6 reveals that our method shows robust advantages than SOTA methods in minimizing model consumption and saving resources.
Table 6: Model consumption analysis on three datasets, compared with the SOTA methods.
Dataset
Image Resolution
Input Frame
Method
Model Size
GPU Memory
MMVP 2
2.79M
4.53GB
UCF Sports
512 x 512
4
Ours
0.60M
2,38GB
DMVFN 13
3.56M
3.79GB
KITTI
256 x 832
2
Ours
0.97M
0.97GB
DMVFN
13
3.56M
7.41GB
Cityscapes
512 1024
Ours
0.97M
1.25GB
4.2
Ablation Studies
We conduct ablation studies on three aspects of the UCF Sports MMVP split: i) the choice k. which defines three attributes of the system, i.c., the number of dynamic vectors each node initially embeds,

---

the number of temporal forward/backward edges for each node, and the number of dynamic vectors to be decoded from the upsampled motion features; ii) the number of views used for multi-view motion graph construction; and iii) the composition of the node's initial features.
Table 8: Ablation study on the number of motion
Table 7: Ablation study on the value of k.
graph views and on the location feature floe-
SSIM
PSNR
LPIPS 1
Memory L
0.9159
29.84
0.1888
11988
View Number foe
SSIM 1
PSNR* 1
LPIPS
0.9273
30.19
0.0966
161G
0.9385
0.0934
2.07G
0,9058
28.53
0.1205
10
0.93114
30.49
0,0823
2.38G
0.9158
29.47
0. 10604
0.9319
30.58
0,0809
3,96G
0.9247
30.27
0.1008
The choice of k: Adjustments to the value of l and subsequent evaluations reveal a consistent, monotonic increase in both performance and GPU memory usage, as documented in Table Notably, the gain rate diminishes as increases. This observation aligns with the intuition that increasing the value of k will bring more temporal correspondences to the system. However, once the motion information nears saturation, the incremental benefit of further increasing & becomes less significant. More ablation studies related to k can be found in Section A.4
The number of graph views:
As Section 3.2 mentioned, using multi-view graphs can enhance motion prediction accuracy. To validate this claim, we reduce the number of views used in the system and test the system on the UCF Sports dataset. The first three rows in Table g show that increasing the number of graph views indeed improves the system performance.
Location feature floe: We incorporate the tendency feature fide (directly related to motion) and location feature fle to initialize the node features. This inclusion hypothesizes that an image pixel's motion patterns may also be influenced by its spatial position. Empirical evidence supporting this hypothesis is documented in the last two rows of Table X There is a notable decline in performance when the location feature fude is excluded from the node feature initialization. In Section A.7 we further visualize the location features and observe that such feature may reflect the camera projection pattern, providing substantial information to the following motion reasoning task.
5 Conclusion & Limitation
This work presents the motion graph for video prediction and a novel pipeline built upon it. We focus on balancing representative ability and compactness to optimize efficiency. The motion graph encapsulates complex motion information hidden in video sequences into a more manageable format, and achieves encouraging results: it matched or outperformed SOTA on three well-known datasets with a considerably smaller model size and less GPU running memory. Beyond the results, the motion graph and its associated video prediction pipeline set a foundation for further research and optimization in the field of video prediction, thus suggesting a promising direction for researchers seeking to balance performance and resource efficiency in the evolving domain of video processing.
Limitation. There is no significantly improvement in terms of the inference speed. The current version of our model is slower than DMVFN, which is specifically optimized for this aspect. Future efforts will focus on accelerating inference while maintaining the model's compactness and efficiency. Additionally, video prediction remains particularly challenging in scenarios involving sudden or unpredictable motions, which our system occasionally fails to capture, as highlighted in Appendix [A66] These instances, where the model struggles with abrupt actions not readily discernible from the input frames, underscore the need for further enhanced video understanding abilities. Addressing these challenges presents valuable direction for future research.
References
[1] Zhangyang Gao. Cheng Tan, Lirong Wu, and Stan Z Li. Simvp: Simpler yet better video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3170-3180, 2022.
[2] Yiqi Zhong, Luming Liang, Ilya Zharkov, and Ulrich Neumann. Mnvp: Motion-matrix dadioo-matri--based -bsed video prediction. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 4273-4283 2023.
10

---

[3] Xingjian Shi, Zhourong Chen, Hao Wang, Yan Yeung, Wii-Kin Wong, and Wang-chun Woo. Convolutional Istm network: A machine learning approach for precipitation noweasting. Advances in neural information processing systems, 28, 2015.
[4] Yunbo Wang, Mingsheng Long, Jianmin Wang, Zhifeng Gao, and Philip S Yu. Predrnn: Recurrent neural networks for predictive learning using spatiotemporal Istms. Advances in neural information processing systems, 30, 2017.
[5] Yunbo Wang, Zhifeng Gao, Mingsheng Long, Jianmin Wang, and S Yu Philip. Predrn+++: Towards a resolution of the deep-in-time dilemma in spatiotemporal predictive learning. In International Conference on Machine Learning, pages 5123-5132. PMLR, 2018.
[6] Yunbo Wang, Jianjin Zhang, Hongyu Zhu, Mingsheng Long, Jianmin Wang, and Philip S Yu. Memory in memory: A predictive neural network for learning higher-order non-stationarity from spatiotemporal dynamics. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9154-9162, 2019.
[7] Haixu Wu, Zhiyu Yao, Jianmin Wang, and Mingsheng Long. Motionrnn: flexible model for video prediction with spacetime-varying motions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15435-15444, 2021.
[8] Agrim Gupta, Stephen Tian, Yunzhi Zhang, Jiajun Wu, Roberto Martín-Martin and Li Fei-Fei. Maskvit: Masked visual pre-training for video prediction. arXiv preprint arXiv:2206. 1186,, 2022.
[9] Shuliang Ning, Mengeheng Lan, Yanran Li, Chaofeng Chen, Qian Chen, Xunlai Chen, Xi- aoguang Han, and Shuguang Cui. Mimo is all you need: a strong multi in-multi-out baseline for video prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1975-1983, 2023.
[10] Mingzhen Sun, Weining Wang. Xinxin Zhu, and Jing Liu. Moso: Decomposing motion, scene and object for video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 18727-18737, 2023.
[11] Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. arXiv preprint arXiv:1706.08033 2017.
[12] Xiaojie Gao, Yueming Jin, Qi Dou, Chi-Wing Fu, and Pheng-Ane Ann Heng. Accurate grid keypoint learning for efficient video prediction. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5908-5915. IEEE, 2021.
[13] Xiaotao Hu, Zhewei Huang. Ailin Huang, Jun Xu, and Shuchang Zhou. dynamic multi-scale voxel flow network for vidco prediction. arXiv preprint arXiv:2303.00 098855 2023.
[14] Lluis Castrejon, Nicolas Ballas, and Aaron Courville. Improved conditional vmnss for video prediction. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7608-7617, 2019.
[15] Zheng Chang, Xinfeng Zhang, Shanshe Wang, Siwei Ma, and Wen Gao. Strpm: A spatiotem- poral residual predictive model for high-resolution video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13946-13955, 2022.
[16] Zheng Chang, Xinfeng Zhang, Shanshe Wang, Siwei Ma, and Wen Gao. Stip: A spatiotemporal information-pee eroornation-presevving and praception-augmentee model for high-resolution video prediction. arXiv preprint arXiv:2206..m 04381, 2022.
[17] Yunbo Wang, Lu Jiang, Ming-Hsuan Yang, Li-Jia Li, Mingsheng Long, and Li Fei-Fei. Fidetic 3d Istm: model for video prediction and beyond. International conference on learning representations, 2018
[18] Jun-Ting Hsieh, Bingbin Liu, De-An Huang, Li F Fei-Fei, and Juan Carlos Niebles. Learning to decompose and disentangle representations for video prediction. Advances neural information processing systems, 31, 2018.

---

[19] Xi Ye and coillaume-Alexande Bilodeau. Vptr: Efficient transformers for video prediction. In 2022 26th International Conference on Pattern Recognition (ICPR), pages 3492-3499, IEEE, 2022.
[20] Rohit Girdhar and Kristen Grauman. Anticipative video transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 13505-13515, 2021.
[21] Nal Kalchbrenner, Aäron Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. In International Conference on Machine Learning, pages 1771-1779. PMLR, 2017.
[22] Xiaodan Liang, Lisa Lee, Wei Dai, and Eric P Xing. Dual motion gan for future-flow embedded video prediction. In proceedings of the IEEE international conference on computer vision, pages 1744-1752, 2017.
[23] Yung-Har Ho, Chuan- Yuan Cho, Hei-H Peng. and Guo-Lun Jin. Sme-net: Sparse motion estimation for parametric video prediction through reinforcement learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10462-10470, 2019.
[24] Hang Gao, Huazhe Xu, Qi-Zhi Cai, Ruth Wang, Fisher Yu, and Trevor Darrell. Disentangling propagation and generation for video prediction. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9006-9015, 2019.
[25] Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun. Learning lane graph representations for motion forecasting. In Computer EssionEECVV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 11 16, pages 541-556. Springer, 2020.
[26] Maosheng Ye, Tongyi Cao, and Qifeng Chen. Tpen: Temporal point cloud networks for motion forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11318-11327, 2021.
[27] Roger Girgis, Florian Golemo, Felipe Codevilla, Martin Weiss, Jim Aldon D' D'Souza, Samira Ebrahimi Kahou, Felix Heide, and Christopher Pal. Latent variable sequential set transformers for joint multi-agent motion prediction. arXiv preprint arXiv:2104.00563 2021.
[28] Maosen Li, Siheng Chen, Yangheng Zhao, Ya Zhang, Yanfeng Wang, and Qi Tian. Dynamic multiscale graph neural networks for 3d skeleton based human motion prediction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 114-223, 2020.
[29] Lingwei Dang, Yongwei Nie, Chengjiang Long, Qing Zhang, and Guiqing Li. Msr-gen: Multi- scale residual graph convolution networks for human motion prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11467-11476 2021.
[30] Chongyang Zhong, Lei Hu, Zihao Zhang, Yongjng Ye, and Shihong Xia. Spatio-temporal gating-adjacency gen for human motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6447 6456, 2022.
[31] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1874-1883, 2016.
[32] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In European conference on computer vision, pages 402-419. Springer, 2020.
[33] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2472-2481, 2018.
[34] Simon Niklaus and Feng Liu. Softmax splatting for video frame interpolation. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, pages 5437-5446. 2020.
12

---

[35] Simon Niklaus, Ping Hu, and Jiawen Chen. Splatting based synthesis for video frame interpola- tion. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 713-723, 2023.
[36] Mikel D Rodriguez, Javed Ahmed, and Mubarak Shah. Action mach a -emporll maximum average correlation height filter for action recognition. In 2008 IEEE conference on computer vision and pattern recognition, pages 1 1-8. IEEE, 2008
[37] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11):1211-1237, 2013.
[38] Yue Wu, Rongrong Gao, Jaesik Park, and Qifeng Chen. Future video synthesis with object motion prediction. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 5539-5548, 2020.
[39] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213-3223, 2016.
[40] Richard Zhang, Phillip
Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[41] Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. arXiv preprint arXiv:1511.0544 2015.
[42] Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint 1rXiv:180.001550 2018.
[43] Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine. Stochastic variational video prediction. arXiv preprint arXiv: 7200. 1252, 2017.
[44] Yong-Hoon Kwon and Min-Gyu Park. Predicting future frames using retrospective cycle gan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1811-1820, 2019.
[45] Wei Yu, Yichao Lu, Steve Easterbrook, and Sanja Fidler. Efficient and information preserving future frame prediction and beyond. 2020.
[46] Ting-Chun Wang, Ming-Yo Yu Liu, Jun-Yar Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to-video synthesis. arXiv preprint 17X1v:8808.06600 2018.
[47] Junting Pan, Chengyu Wang, Xu Jia, Jing Shao, Lu Sheng, Junjie Yan, and Xiaogang Wang. Video generation from single semantic label map. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3733-3742, 2019.
[48] Yue Wu, Rongrong Gao, Jaesik Park, and Qifeng Chen. Future video synthesis with object motion prediction. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 5539-5548, 2020.
[49] Xinzhu Bei, Yanchao Yang, and Stefano Soatto. Learning semantic-aware dynamics for video prediction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 902-912, 2021.
[50] William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video 2016.
prediction and unsupervised learning. arXiv preprint
[51] Ziwei Liu, Raymond A Ych, Xiaoou Tang, Yiming Liu, and Aseem Agarwala. Video frame synthesis using deep voxel flow. In Proceedings of the IEEE international conference on computer vision, pages 4463-4471, 2017.
[52] Daniel Geng, Max Hamilton, and Andrew Owens. Comparing correspondences: Video pre- diction with correspondencewwiee losses. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3365-3376, 2022.
13

---

[53] Yue Wu, Qiang Wen, and Qifeng Chen. Optimizing video prediction via video frame interpola- tion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17814-17823, 2022.
[54] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, -ehffoerfrrmeec deep learning library. Advances in neural information processing systems, 32, 2019.
[55] Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
[56] nya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 03983, 2016.
[57] Andrew I. Maas, Awni Y Hannun, Andrew Y Ng, et al. Rectifier nonlinearities improve neural network acoustic models. In Proc. icml, volume 30, page 3. Atlanta, GA, 2013.
[58] Vikram Voleti, Alexia Marttnnaa,,a and Chris Pal. Mevd-masker conditional video dif- fusion for prediction, generation, and interpolation. Advances in neural information processing systems, 35:23371 255333,,--3338,,... 2022.
[59] Kangfu Mei and Vishal Patel. Vidm: Video implicit diffusion models. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 91 9117-9125, 2023.
[60] Zhicheng Zhang, Junyao Hu, Wentao Cheng, Danda Paudel, and Jufeng Yang. Extdm: Distri- bution extrapolation diffusion model for video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19310-19320, 2024.
14

---

A Appendix
A.1
Implementation Details
We implement the video prediction system using PyTorch (54) and conduct end-to-end training single NVIDIA A 100 GPU. We use AdamW optimizer [55] during the training. The initial learning rate is set to -e and decayed to le- following cosine decay scheduler [56]. There are only few hyperparameters to adjust for the system when training on different datasets. The adjustments are mainly based on the resolution of the frame. Those (Image feat.), which is the parameter for the image encoder; ii) Tendency feature length (Tendency feat.), which is the length of the tendency feature in node initialization; iii) location feature length (Location feat.), which is fixed to 4 for all datasets, indicating the length of the location feature in node initialization; iv) the number of graph views, indicating view number in motion graph feature learning; v) k, indicates the number of the dynamic vectors embedded in each node, the number of the temporal edges per node and the output dynamic vectors per pixel: vi) training epoch is the training related parameters; vii) the reconstruction loss, which follows the popular setting of the SOTA methods on each dataset. In Table we demonstrate the hyper-parameter setting for each dataset.
include i) image feature length
Please note that did not especially tune the parameters for each dataset. When adjusting the parameters, we consider more about the training efficiency instead of the performance. Therefore, setting is likely nor the optimal choice. For example, in DMVFN [3], the training Cityscapes and KITTI are both 300 epochs, we observe that our system can achieve comparable performance with only 100 and 200 epochs respectively, thus stay with this configuration.
Dataset
Image feat.
Tendency feat.
Location feat.
Number of Graph views
k
Epoch
Loss
UCF Sports
16
10
300
MSE
Cityscapes
16
32
4
10
100
L1 Lpips
KITTI
16
32
4
8
200
L1 Lpips
Table 9: Hyper-parameter setting for each dataset.
A.2 Network Architecture
The proposed video prediction system includes three major components, the image encoder, the motion graph interaction module, and the motion upsampler. Here we demonstrate the detailed architecture of each component for reproduction needs.
Image Encoder: Figure 7 shows the inner structure of the image encoder in the proposed system. Cimg is related to the image feature length in Table Each convolution layer will come with a Leaky ReLU layer [57] as the activation layer.
I
x
ResBlock Downsample
C H' W
Pixel Unshuffle
ResBlock Downsample
Conv 2D 3X3, stride 2
Conv 2D 3X3
ResBlock Downsample
Conv 2D 3X3
Downsample
Sum
ResBlock Downsample
H'
W
f 800mg x H5 Ws
2C x
2
X
2
Figure Architecture of the image encoder
15

---

Motion Graph Interaction Module In Figure we demonstrate the inner structure of the spatial and temporal message passing the motion graph interaction module. Cnode equals the sum of the tendency feature length and the location feature length in Table D
Spatial:
V Coode Restaps
V: Guods Hg Ws
Conv2D
V N Guode
Temporal:
Vprea' : Mpred X
Crole
Linear Projection
pred Npred x Cacte
XW pred Ngred
Vnee x Casde
Linear Layer
V
X Crode
Figure 8: Inner structure of spatial and temporal block in motion graph interaction module
Motion Upsampler Figure iillsstates the inner structure of the motion upsampler as well as the motion decoder. The implementation of the decoder a single 2D convolution layer with a kernel size of 1.
ffuee : Cnodee x Hs x Ws
ResBlock Upsample
ResBlock Upsample
H W
Crode
2
ResBlock Upsample
Conv 9X3
ResBlock Upsample
Conv 2D 3X3
Upsample
FSR Juse
Grode x H x W
Sum
Conv 2D 1X1
Cnose x II' x W
P W
Figure 9: Inner structure of the motion upsampler and the motion decoder.
A.3 Additional Quantitative Evaluation
For comparison convenience of the future works, we list the extra metries of cityscapes and KITTI datasets in To) i.e. ssim and psnr.
Table 10: Evaluation on Cityscapes 39 and KITTI 37 datasets. "RGB", "F" "S" and "I" denote
video frames, optical flow, semantic map, and instance map; t + and 5 respectively indicate the average performance of the next and 5 frames. Results with are copied from [13]
Chyscapes
KITTI
Method
Input
SSIM -)
PSNR" 1+3.
SSIM 10
PSNRT
(+5
(+1
(+5
(+1
L+S.
DMIVFN (CVPK 2023)
RGB
87.15
78.75
6335
18.81
15.17
13601
72.09
02.52
37.05
22.74
20.16
18.70
A.4
Extensive Ablation Study
In this section, we add two ablation studies to help the audience better interpret the design of the motion graph.
Number of the predicted dynamic vectors per pixel: In the proposed system, we set the number of the predicted dynamic vectors per pixel to k, which is identical to the number of the dynamic vectors embedded by each node and the temporal edge of each node. This design ensures the flexibility of the predicted motion to have multiple modes compared to the optical-flove flow-based method which
16

---

only allows each pixel to have single future motion. The comparison between the first two rows of Table IT showcase that allowing multiple predicted dynamic vectors can largely improve the performance. Meanwhile, if we control the number of the predicted dynamic vectors, as demonstrate by the comparison between the first and third row of the Table we see that when the motion graph embeds more past motion modes, the performance will also has significant improvements.
Full set
Hard (SSIM < 0.6)
Predicted Vectors #
Memory I
SSIM
PSNR
LPIPS
SSIM t
PSNR
LPIPS 1
1
0.8742
26.34
0.1527
0.5394
17.34
0.5032
0,98G
10
0.9199
29.87
0.1042
0.6408
19.44
0.3706
1.97G
0.9212
30.00
0.0877
0.6546
19.63
0.3261
1.38G
Table 11: Ablation study the number of the predicted vectors. The experiments conducted on UCF Sports MMVP splits. The listed results are from the models trained for 100 epochs (models were all trained for 300 epochs in the main manuscript).
Motion Graph Interaction Module The design of the motion graph interaction module are following the intuition that both spatial connection and temporal connection should benefit the graph learning. Here we also show the experimental results in Table 12 that both spatial and backward edges are beneficial to the final performance.
Spatial
Backward
PSNR f
MS-SSIM x 1000 1
LPIPSx100 +
21.55
87.06
9.85
21.64
87.25
9.83
21.71
87.70
9.50
Table 12: Ablation study on graph interaction module. The experiments are conducted on KITTI and metrics show evaluation on the / results.
A.5 Extensive Discussion on Experiment setting
Our research on video prediction has identified key differences between systems designed for short-term and long-term predictions. Short-term systems typically use fewer frames to predict the immediate next few frames, while long-term systems are tasked with forecasting an extensive sequence of future frames. The design logic and objectives of these systems, thus, vary significantly. Table 13 demonstrates the differences with more details. It is easy to notice that the recent works of
Prediction
Long- term Video Prediction
Prodiction Longth
Limited, usually frame
Much longer,
Viden Resolution
Up to 4K
Usually smaller, up to 258 256
Dataset
UCF UCF Sports, KITTI, Cityscapes, SPTU4K, Vimco, DAVIS
KTH, moving MNIST BAIR, cropped Ciryscapes
Recent works
SIMVPD MMVP2) STRPM [S) STIPLM DMVEN [3]
MVC Mask &J VID (50 Exi (X)
Objectives
High resolation videos, pixel level accurncy, real time application
Canditionl video generation, semantic level accuracy
Table 13: Experiment setting comparison between short-term and long-term video prediction task
long-term video prediction usually involve with diffusion- generative models, which is designed to feed the need of predicting long videos. The evaluation of such methods usually emphasize on if the predicted video semantically correct given the input video frame(s). While this study, we emphasize our method's advanced motion modeling and significant reduction in computational costs, essential for short-term prediction of high-resolution videos. Our system is tailored for short-term video prediction, with evaluations conducted along this line of work.
We plan to further exploit the motion graph's 's potential as an efficient motion representation tool and develop advanced, motion graph-based systems for long-term video generation in our next work.
A.6 Failure case demonstration
The video prediction is always a challenging problem. Especially for those video sequences with abrupt motion which can be hardly indicated by the previous video frames. The proposed method
17

---

Frame I 4
Frame 5 (GT)
Prediction
Figure 10: Failure cases in UCF Sports Dataset
formulates the video prediction motion prediction problem and outperforms most of the existing methods by using motion graph to better capture the motion hints from the input frames. However, when evaluating the qualitative results, we still find some failure cases that require additional research efforts to solve. In Figure I0 we showed typical failure cases in UCF Sports dataset. We notice that most of the failures cases are in the action of kicking and diving, which usually include fast, unpredictable motion that requires stronger video understanding capability of the model.
A.7 Node Feature Visualization
To better understand the initialization of the node embedding, here we visualize the tendency feature and the location feature. We first extract the tendency feature and location feature each node in the motion graph and apply K-means clustering to the extracted features. For the tendency feature, we set the cluster number to 2; and for the location feature, we set the cluster number to 4 for better visualization.
From Figure 11 we see that using the learned tendency feature, the system should be able to distinguish the dynamic areas from the static areas. If we further enlarge the cluster number, we can see more clearly that the tendency features embed the different motion patterns of each feature patch in the frame. For the location feature, in the paper, we have shown that removing the location feature from the node initialization will result in a performance drop. From Figure 12 we observe that the location feature may contain information that is related to the camera projection mode. For cityscapes and the KITTI, which use wide-range cameras, the clustering pattern of the location feature is very different from the UCF Sports whose projection mode is possible be orthogonal projection.
18

---

Frame
Freme
Tendency 2luster=2 2
Tendency Cluster=3
Figure Tendency feature visualization using KITTI dataset
Cityscapes
UCP Sports
Figure 12: Locaiton feature visualization on three datasets.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Section describes the detailed methodology and the Section validates the claim.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
19

---

Justification:
Please see Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please see Section
Appendix
A.I and Appendix A.2
Guidelines:
The answer NA means that the paper does not include experiments.
20

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The data all public and the code is provided in the supplementary zip file.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ 7/nips. /ttppp.cc/pbllc/guideeee http://oeeuuiiiiiooooooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
21

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see Section 4 and Appendix A.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eтror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see Appendix A.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
22

---

The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics attpss//neuripp::: ottps//ttts.ss///.....ooo/o////p
Answer: [Yes]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. degloymeen of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
23

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Please see Appendix A.I
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pwpprswithcaeoooo thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer[[Yes)
Justification: Please see Appendix A.I mentary file.
A.2 and the code comments provided in the supple-
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemmodel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
24

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "cOwavgFpiU", "reviewer": "Reviewer_gkX2", "summary": "The authors propose a Motion Graph for predicting future video pixels. To achieve this, they introduce three modules: 1. Motion Graph Node Construction; 2. Edge Construction; 3. Graph Interaction Module; 4. Video Prediction Model. The first three modules encode patches and their interactions in a frame, while the fourth module decodes future frame pixels. Experiments on UCF Sport, KITTI, and CityScape demonstrate the method's high efficiency and construction quality.", "questions": "1. Which backbone are used to extract features during motion graph construction. Does the method robust to different backbones?\n2. Does the prediction quality decrease with respect to resolutions? is it possible to analyze prediction quality with large resoultion settings.", "limitations": "The authors discuss and partially address the limitations of the paper.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The author propose to predict future pixels from motion graph perspective, to achieve this, they propose motion graph constuction and graph decoding modules.\n\n2. The method demonstrate high computation efficieny and good reconstruction quality on few video benchmarks.", "weaknesses": "1. It's better to present an overview figure for the framework. It's not easy to grasp the overall structure by reading about the four separate modules.\n\n2. Figures 2-4 have fonts that are too small, making them hard to read.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Question 1: Which backbone are used to extract features during motion graph construction. Does the method robust to different backbones?\n\nAnswer 1: ResNet was used as the backbone of our image encoder to extract features during motion graph construction. In the manuscript submission, Figure 7 of Appendix A.2 illustrates the network architecture of the image encoder. Our experiments demonstrate the robustness of our design across different image encoder architectures. We selected a comparatively lightweight model to ensure both model compactness and system computational efficiency. \n\n---\n\nQuestion 2: Does the prediction quality decrease with respect to resolutions? is it possible to analyze prediction quality with large resolution settings.\n\nAnswer 2: To evaluate the model's performance on high-resolution images, we tested our method on the SJTU4K dataset, which has a resolution of $2160\\times3840$. Due to the absence of publicly available data splits for direct comparison with existing state-of-the-art (SOTA) methods, we created our own training and validation splits, allocating 80% of the sequences to training and 20% to validation.\n\n\n| Dataset | Resolution | LPIPS $\\downarrow$ | MS_SSIM $\\uparrow$ |\n|:---:|:---:|:---:|:---:|\n| KITTI | $256\\times 832$ | 9.50 | 87.70 |\n| Cityscapes | $512\\times 1024$ | 4.13 | 94.85 |\n| SJTU4K | $2160\\times 3840$ | 8.22 | 90.70 |\n\n\nWhile the model's performance on the SJTU4K dataset surpasses its performance on KITTI, which has a much lower resolution, it does not match the performance on Cityscapes, which also has a smaller resolution than SJTU4K. Our findings suggest that factors such as motion complexity, frame rate, and scene complexity have a greater influence on prediction performance than frame resolution alone.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZplvGdZWOb", "reviewer": "Reviewer_Mw8x", "summary": "This paper proposes a motion-based method for video prediction. They design a new motion representation named motion graph that transforms patches of video frames into interconnected graph nodes. The proposed video prediction pipeline, empowered by the motion graph, exhibits substantial performance improvements and cost reductions. Experiments on UCF Sports, KITTI, and Cityscapes are conducted.", "questions": "See weakness", "limitations": "The authors indicate limitations in the conclusion section.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "+ Motion-based video prediction is quite an interesting topic and benefits the research community with an affordable training cost.\n+ The evaluation of both efficiency and effectiveness is conducted.\n+ The paper is well-structured.", "weaknesses": "- Motivation and Occlusion / Out-of-View cases. As the authors claimed in lines 33-37 and Figure 1, existing methods struggle to effectively handle occlusion cases. However, the paper does not provide a corresponding evaluation to demonstrate the effectiveness of the proposed motion graph in complex situations involving occlusion or out-of-view scenarios. This is a significant omission, as these challenges are prevalent in real-world video prediction tasks. Furthermore, based on my understanding of the work, the proposed motion graph may not adequately address these problems. Specifically, occluded and out-of-view object/background do not appear to be considered in the graph, as the connections and features captured in the motion graph are derived from the visible parts of the targets. This raises concerns about the method's ability to maintain accurate predictions when dealing with occlusions or objects moving out of the camera's field of view. The authors should provide a thorough discussion that includes both the theoretical principles and empirical evidence.\n\n- Video prediction evaluation setting. While the proposed motion graph method shows promising results in predicting a few frames (up to 10), the paper lacks an evaluation of its performance in predicting a larger number of frames. Other video prediction works [A, B, C] have evaluated their methods on longer prediction horizons, providing a more comprehensive assessment of their models' robustness and accuracy.\n\n[A] ExtDM: Distribution Extrapolation Diffusion Model for Video Prediction, CVPR24\n\n[B] Efficient Video Prediction via Sparsely Conditioned Flow Matching, ICCV23\n\n[C] MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation, NeurIPS22\n\n- Comparison with MMVP. The proposed motion graph method appears to be quite similar to the motion matrix used in MMVP, which also provides a dense connection among frames. Although I appreciate the discussion provided in the related work, the authors do not sufficiently clarify the differences and advantages of the proposed method compared to MMVP. This lack of distinction makes it difficult to understand the unique contributions and potential improvements offered by the motion graph.\n\n- Comparison with other motion-cues-guided video prediction methods. The paper lacks a comparison with other motion-cues-guided video prediction methods, such as MotionRNN, which also leverages a vector-based motion representation. The authors should explain how the feature warping mechanism in MotionRNN compares to the feature processing in the motion graph. What are the potential benefits or drawbacks of each approach? Are there specific scenarios where the motion graph's approach to feature handling provides a clear advantage?\n\n- Lack of a component-wise ablation. The reviewer noticed that there is no component-wise ablation study for motion feature learning, upsampling, decoding, image wrapping, and interaction modules. It makes it difficult for the audience to identify which parts of the methodology are effective and contribute most to the overall performance. The authors should conduct a thorough ablation study that isolates and evaluates the impact of each component of their proposed method.\n\n- Complexity of the proposed method and presentation. Actually, the proposed method is somehow non-trivial, making this work hard to follow. Accurate presentation can make the work easier to understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1. Motivation and occlusion / out-of-view cases.\n\nA1: As a non-generative model, our proposed video prediction system may face challenges with occlusions that require the generation of unseen objects. However, it excels in scenarios with scenarios involving occlusions of known objects, as showcased in the third column of Figure 6, where our method outperforms existing SOTA methods. Notably, in scenes with partial obstructions—such as the white wall behind the cyclist—our system adeptly employs multiple motion vectors per pixel to reconstruct occluded areas. This feature also supports precise management of object expansion due to perspective projection, as exemplified by the green car in Figure 6's first column.\n\nMoreover, our approach is adept at managing objects exiting the camera’s view. By explicitly modeling the motion of image patches, the motion graph predicts when features are about to leave the scene. Thus, any image patches projected to move out of view are not included in the final prediction. This is demonstrated in the 1st, 3rd, and 4th columns of Fig 6, where objects like a cyclist and the front of a blue truck are shown as moving out of frame. Our method successfully and uniquely captures and represents these movements, unlike how the other methods evaluated.\n\n---\n\nQ2. Video prediction evaluation setting.\n\nA2: Thank you for initiating this discussion. Our research on video prediction has identified key differences between systems designed for short-term and long-term predictions. Short-term systems typically use fewer frames to predict the immediate next few frames, while long-term systems are tasked with forecasting an extensive sequence of future frames. The design logic and objectives of these systems, thus, vary significantly. For a more detailed comparison, we have outlined these distinctions in the table below:\n|Task type|Output length|Resolution|Dataset|Recent work|Objectives|\n|---|---|---|---|---|---|\n|Short-term|Short|Up to 4K|UCF 101, UCF Sports, KITTI,  Cityscapes, SJTU4k, Vimeo, DAVIS|SIMVP, MMVP, STRPM,  STIP, DMVFN|High resolution videos, pixel-level accuracy, real-time application|\n|Long-term|Long ($\\gg 10$)|Up to $256\\times 256$|KTH, moving MNIST,  BAIR, cropped Cityscapes|MVCD, MaskViT,VIDM,ExtDM|Conditional video generation, semantic-level accuracy|\n\nIn this study, we emphasize our method's advanced motion modeling and significant reduction in computational costs, essential for short-term prediction of high-resolution videos. Our system is tailored for short-term video prediction, with evaluations conducted along this line of work. We plan to further exploit the motion graph's potential as an efficient motion representation tool and develop advanced, motion graph-based systems for long-term video generation in our next work. \n\n---\n\nQ3: Comparison with MMVP.\n\nA3: We highlight in the manuscript three major differences to MMVP. \n|Method|Motion representation|Prediction module architecture|Motion & Appearance Composition method|\n|---|---|---|---|\n|MMVP|Motion Matrix   ($H\\times W\\times H\\times W$)|3D Convolution based network|Downsampled-feature-level matrix multiplication|\n|Motion Graph (Ours)|Motion Graph   ($H\\times W\\times k\\times 3$) |Graph convolution network|Pixel-level image forward warping|\n\nThe advantages of our method are evidenced by its performance in various tests:\n\na) Our sparser motion representation allows the system to process videos with **higher resolutions using fewer computational resources**. For instance, Table 6 indicates that GPU consumption for our method is 52% of that required by MMVP.\n\nb) Our graph-based motion prediction module **aggregates relational information from motion embeddings more efficiently**, resulting in a lighter and more effective model. As shown in Tables 4 and 6, our method requires only 21% of the model size of MMVP while improving the LPIPS metric by 22%.\n\nc) **Pixel-level image warping** maintains critical appearance information from existing frames, significantly boosting detail recovery. Figure 5 demonstrates this by comparing the clarity of details like the horse’s facial features, the rider’s face, and the athlete’s feet, with our method outperforming others in detail retention.\n\n---\n\nQ4: Comparison with other motion-cues-guided video prediction methods.\n\nA4:  Although MotionRNN also utilizes motion vectors, there are several fundamental differences:\n\na) MotionRNN employs local $3\\times3$ motion filters, which are limited in handling large-scale motion;\n\nb) MotionRNN does not account for the interaction of motion across different spatial and temporal regions, a feature that our motion graph learning process emphasizes;\n\nc) MotionRNN performs feature-level warping and our approach conducts pixel-level warping, providing finer detail.\n\nThese distinctions contribute to the lesser robustness of MotionRNN in handling high-resolution videos with complex motion scenarios, as detailed in Tab 3 of the manuscript. We have reviewed various motion-based video prediction methods in Sec 2. Moving forward, we plan to conduct a further literature review to expand and update this section accordingly.\n\n---\n\nQ5: Lack of component-wise ablation study.\n\nA5: The components of motion upsampling, motion decoding, and image warping modules are each essential to the functionality of the system. So we have opted not to isolate each component for ablation studies. Instead, we focus the ablation analysis on the composition of motion features: please see details in Table 8 of the manuscript. Appendix A.4 also provides additional ablation studies, with Table 12 specifically exploring the effects of various interaction modules. \n\n---\n\nQ6. Presentation of this complex method.\n\nA6: Thank you for this comment. To more effectively present our proposed method, for it to be more comprehensible and accessible, in the revised manuscript we worked on the clarity of descriptions and incorporated additional visual explanations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R3ewd8BeDi", "reviewer": "Reviewer_Vf8f", "summary": "The paper introduces a graph-based methods to predict video frames through motion prediction. The proposed motion graph captures complex spatial-temporal relationships by transforming video frame patches into interconnected graph nodes. This method improves performance and reduces computational costs compared to SOTA methods on UCF Sports, KITTI and Cityscapes.", "questions": "1. Do the authors see a future where such work might be useful for improving video generative models?\n2. Can the authors explain what the weight components are for? It seems that the dynamic vectors can represent motion even without the weighted component. If it does represent something meaningful, can the authors show what they represent after training? For example, might it correspond to the change in depth?\n3. How does the motion graph approach compare to recent transformer-based video prediction methods in terms of performance and efficiency?\n4. How sensitive is the method to the choices made in graph construction (e.g., number of neighbors, edge types)?\n5. How does the motion graph approach scale with longer video sequences? Is there a practical limit to the number of frames it can handle efficiently?", "limitations": "The authors briefly mention limitations regarding inference speed and handling sudden motions. However, they could provide a more comprehensive discussion of potential limitations, such as scalability to longer videos or generalization to diverse video types.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper introduces a novel motion graph representation, combining graph theory with video prediction to overcome existing limitations.\n2. It has comprehensive evaluations across multiple datasets and achieve superior performance than prior methods.\n3. The manuscript is well organized and has detailed experimental results.", "weaknesses": "1. Slower inference speed compared to optimized methods like DMVFN.\n2. Struggles with predicting abrupt or unpredictable motions.\n3. Unclear if the motion graph captures hierarchical / semantic feature of motions.\n4. Does not address the stochastic nature of human videos.\n5. Limited diversity in evaluation datasets, focusing mainly on sports and driving scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Question 1: Do the authors see a future where such work might be useful for improving video generative models?\n\nAnswer 1: Yes. This work uses video prediction task as an example to validate the efficiency and effectiveness of motion graph as a comprehensive video motion representation. In the future, we will explore motion graph's potential as an efficient motion representation tool, developing advanced motion graph-based systems for long-term video generation. \n\n---\n\nQuestion 2: Can the authors explain what the weight components are for? It seems that the dynamic vectors can represent motion even without the weighted component. If it does represent something meaningful, can the authors show what they represent after training? For example, might it correspond to the change in depth?\n\nAnswer 2:  After training, the graph construction process uses weights from the cosine similarity scores between pairs of image patches to determine their connectivity. Similarly, the weights output by the motion upsampler function as confidence scores for each predicted motion vector. This weighting strategy offers several advantages:\n\na) Graph Construction Robustness: In the motion graph, each image patch ideally connects to $k$ patches in the subsequent frame. If a patch is occluded in the next frame, selecting the top $k$ connections could lead to random, inaccurate connections. The weighting strategy helps identify and minimize these erroneous connections.\n\nb) Selective Information Aggregation: During graph learning, this approach allows the model to focus on high-confidence correspondences and reduce the influence of ambiguous connections, enhancing the overall learning effectiveness.\n\nc) Enhanced Frame Synthesis: During the synthesis of future frames, multiple source pixels may correspond to a single pixel in the future frame. The weighted approach allows for strategic aggregation of these source pixels, improving the quality of the resulting image.\n\n---\n\nQuestion 3: How does the motion graph approach compare to recent transformer-based video prediction methods in terms of efficiency and performance? \n\nAnswer 3: We appreciate the reviewer initiating this discussion. Both the motion graph approach and transformer-based video prediction methods (e.g. MaskViT) share the fundamental goal of predicting unknown relational information by modeling existing temporal correspondences between adjacent frames. However, significant differences in efficiency, performance, and adaptability set them apart:\n\na) Efficiency: Transformer-based methods create a complete graph for all feature patches between two adjacent frames, employing complex models that require a large number of parameters. This often leads to high computational demands, making these methods less suitable for high-resolution video processing. In contrast, the motion graph approach constructs a sparser graph, with edge weights determined by feature similarities from a CNN-based image encoder. This results in a more lightweight model. Furthermore, motion graphs leverage Graph Convolutional Networks (GCNs) composed of stacked linear layers, enabling efficient learning from the graph structure.\n\nb) Performance: The efficiency differences make direct performance comparisons with transformer-based methods challenging. Currently, few transformer-based approaches are tested on the high-resolution datasets discussed in this manuscript, leaving performance comparisons inconclusive.\n\nc) Adaptability to Different Video Resolutions: A major drawback of transformer-based methods is their lack of flexibility with varying input resolutions during testing. They struggle to process videos of different resolutions effectively. Conversely, our motion graph approach employs a CNN-based image encoder and motion upsampler, offering greater adaptability to various video resolutions. This adaptability enhances the suitability of the motion graph method for real-world applications.\n\n---\n\nQuestion 4: How sensitive is the method to the choices made in graph construction (e.g., number of neighbors, edge types)?\n\nAnswer 4: Our ablation study (detailed in Table 7) reveals a monotonic performance increase as the value of $k$ increases, although the rate of gain diminishes with larger $k$ values. This suggests that performance is more sensitive to changes in $k$ when $k$ is small, as it more closely resembles an optical flow-based method. However, as $k$ increases and motion information approaches saturation, the impact of $k$ lessens. Additionally, Table 12 in Appendix A.4 demonstrates that including both spatial and backward edges enhances model performance. \n\n---\n\nQuestion 5: How does the motion graph approach scale with longer video sequences? Is there a practical limit to the number of frames it can handle efficiently?\n\nAnswer 5:  Longer video sequence can add more nodes and edges to the motion graph, which will result in an increase in the parameter number and computational overhead of motion graph learning. However, in terms of graph construction and feature extraction, the impact of video sequences length will be less, since we extract each image frame individually through an image encoder. Lastly, the practical limit to the number of frames a motion graph approach can handle efficiently is typically dependent on the hardware constraint. The available GPU memory significantly influences how many frames can be effectively managed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ej1eQPeZxk", "reviewer": "Reviewer_V8rg", "summary": "The paper proposes the motion graph method in predicting the video frames by exploring the spatio-temporal relations among frames from the limited data.", "questions": ".", "limitations": "The authors address the limitations briefly.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "It is significant to propose methods of few-shot prediction techniques in video inputs.", "weaknesses": "Please provide more details about the setup of the datasets such as KITTI and Cityscapes that are not originally used for testing video prediction.\n\nThe results on Cityscapes and KITTI datasets based on LPIPS, SSIM, PSNR are needed to be explained in detail in the evaluation methods either in the main text or Appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Question 1:  More details about the setup of the datasets such as KITTI and Cityscapes that are not originally used for testing video prediction.\n\nAnswer 1: We follow the experiment setup from previous works which originated from *Wu, Yue, et al. \"Future video synthesis with object motion prediction\". Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.* We use the data preprocessing code from this repo, https://github.com/hzwer/CVPR2023-DMVFN/blob/main/utils/, to generate the train/val data.\n\n---\n\nQuestion 2: The results on Cityscapes and KITTI datasets based on LPIPS, SSIM, PSNR are needed to be explained in detail in the evaluation methods either in the main text or Appendix.\n\nAnswer 2: Thank you for the suggestion. We will add the calculation details in the appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are grateful for the thoughtful suggestions and comments from the reviewers.

In response, we have implemented several enhancements to the manuscript, including, but not limited to:


a) Enhancing the visual presentation throughout, such as adding an overview figure;


b) Adjusting the text font size of the figures for better readability;

c) Expanding the literature review to include more comprehensive discussions;

d) Clarifying the descriptions of our methods for improved understanding;

e) Providing additional details about the evaluation metrics and the development of the testing datasets.

Please see below for our point-by-point response to the reviewers' feedback. Thank you for your time.

Meta-review: Initially, the reviewers raise some questions regarding the robustness to different backbones, the prediction quality with respect to resolutions, occlusion and out-of-view cases, video prediction evaluation setting, lack of component-wise ablation, sensitivity in graph construction, diversity of metric evaluation, etc. Most of these questions are addressed in the rebuttal and recognized by reviewers. Eventually with the merit of the work, this paper receives two accepts and two borderline accepts, with agreement for acceptance. The AC recommends to accept. Authors are encouraged to revise the paper according to the reviews, including adding a discussion about occlusion, long-term video prediction, and the comparison with other methods, etc.
Decision: Accept (poster)