STREAM: Spatio-TempoRal Evaluation and  Analysis Metric for Video Generative Models
Authors: Pum Jun Kim
Seojun Kim
Jaejun Yoo
Venue: ICLR 2024 poster (2024)
Keywords: Generative Models
Video Generative Models
Evaluation
Fidelity
Diversity
Assessment
Primary area: generative models
TL;DR: Current video models lack robust metrics, focusing too much on spatial aspects. We introduce STREAM, a unique metric evaluating spatial and temporal aspects independently, offering improved insights for developing advanced video generative models.

Abstract: Image generative models have made significant progress in generating realistic and diverse images, supported by comprehensive guidance from various evaluation metrics. However, current video generative models struggle to generate even
short video clips, with limited tools that provide insights for improvements. Current video evaluation metrics are simple adaptations of image metrics by switching the embeddings with video embedding networks, which may underestimate the unique characteristics of video. Our analysis reveals that the widely used Frechet Video Distance (FVD) has a stronger emphasis on the spatial aspect than the temporal naturalness of video and is inherently constrained by the input size of the embedding networks used, limiting it to 16 frames. Additionally, it demonstrates considerable instability and diverges from human evaluations. To address the limitations, we propose STREAM, a new video evaluation metric uniquely designed to independently evaluate spatial and temporal aspects. This feature allows comprehensive analysis and evaluation of video generative models from various perspectives, unconstrained by video length. We provide analytical and experimental evidence demonstrating that STREAM provides an effective evaluation tool for both visual and temporal quality of videos, offering insights into area of improvement for video generative models. To the best of our knowledge, STREAM is the first evaluation metric that can separately assess the temporal and spatial aspects of videos. Our code is available at https://github.com/pro2nit/STREAM.

Published as a conference paper at ICLR 2024
STREAM: SPATIO-TEMPORAL EVALUATION AND ANALYSIS METRIC FOR VIDEO GENERATIVE MODELS
Pum Jun Kim, Seojun Kim, Jaejun Yoo Ulsan National Institute of Science and Technology (pumjun kin kim, seo jun. kim, jaejun, yoo) st kr
ABSTRACT
Image generative models have made significant progress in generating realistic and diverse images, supported by comprehensive guidance from various evalua- tion metrics. However, current video generative models struggle to generate even short video clips, with limited tools that provide insights for improvements. Cur- rent video evaluation metrics are simple adaptations of image metrics by switching the embeddings with video embedding networks, which may underestimate the unique characteristics of video. Our analysis reveals that the widely used Fréchet Video Distance (FVD) has a stronger emphasis on the spatial aspect than the tem- poral naturalness of video and is inherently constrained by the input size of the embedding networks used, limiting it to 16 frames. Additionally, it demonstrates considerable instability and diverges from human evaluations. To address the lim- itations, we propose STREAM, new video evaluation metric uniquely designed independently evaluate spatial and temporal aspects. This feature allows com- prehensive analysis and evaluation of video generative models from various per- spectives, unconstrained by video length. We provide analytical and experimental evidence demonstrating that STREAM provides an effective evaluation tool for both visual and temporal quality of videos, offering insights into area of improve- ment for video generative models. To the best of our knowledge, STREAM is the first evaluation metric that can separately assess the temporal and spatial aspects of videos. Our code is available at STREAM
INTRODUCTION
"Measure what is measurable, and make measurable what is not so." This quote by Galileo Galilei reflects the underlying philosophy of many scientific advancements.
similar vein, Peter Drucker famously stated, "If you cannot measure it, you cannot manage it." Recent breakthroughs in powerful generative models (Karras et al.) 2020 Sauer et al. 2022 Rombach al., 2022) have ushered in an of generating highly realistic images. largely due to insightful evaluation metries (Heusel al. 2017] Salimans et al., (0166] Kynkäännicmi et al.) 2019 Nacem et al.) 2020 Kim et al.] (023) that pivot to video generative models, there is pronounced gap. Many of contemporary video generative models often struggle to generate even concise video clips. Numerous recent studies have proposed variety of solutions, ranging from crafting new architectures or modules to oversee entire video frames (Tulyakov et al. 2018 Li et al. 2019), to the introduction of regularization techniques aimed ensuring temporal consistency (Sun et al.) 2020). While these innovations show promise, they come with an inherent limitation: the necessity for measurable and reliable metric gauge the actual extent of their intended improvements. The absence of comprehensive metrics for analyzing and evaluating these models leaves researchers without a clear guide to model enhancements, hindering further progress.
have shaped their progress. However,
At first glance, using image evaluation metrics on each frame of a video and computing an average might seem sufficient. This method, however, neglects an essential aspect: videos are more than mere collections of individual images. For video to feel genuinely natural, model needs to not only generate high-quality images for each frame but also to ensure that the content is temporally consistent, fluid, and seamless across frames. Seeking for more objective assessment, most current video generative models use Video Inception Score (VIS) (Saito et al.) 2020) and Fréchet Video Distance (FVD) (Unterthiner et al. 2019) as their go-to metrics. These are essentially derivatives of

---

Published as a conference paper at ICLR 2024
1. E Embedding space
2. Temporal auplitudes
3. STREAMS
Spatial Qualls
T-1
172
Frequency (
STREAM-
STREAM-D
(1) Fit to power law und
S.STREMM-T
measure skewness
(2) Construct listogram with skewness values:
Yr-1
Tamperd
7/2
Qualts
skew (f)
Frequency
Real vidra frome fosture
8 (t)) E a* at Irequency 4
Stewness vestor for slngle viteo sample
Fake video frome fosture
Roal ompltude st.5;
skow
Inconsistent feature
Pake ampltude at
(3) STREAM-T
ammHiSt,,HISty
1/2
Figure 1: An illustration of the proposed evaluation pipeline. We use image embedding space to evaluate video regardless of its length and to consider the spatial and temporal aspects of video independently (Section 2.2). Then, we use the Fast Fourier Transform (FFT) along the temporal variation over time for evaluation and to utilize the average at
axis of frame features to capture frequency zero for spatial evaluation (Section 2.1) Finally, we calculate STREAM-S and STREAM-
evaluate the video generative models (Section 2.4 and 2.3
the earlier Inception Score (IS) (Salimans
2016) and Fréchet Inception Distance (FID) (Heusel
et al. (2177). initially designed for image generative models. This adaptation entails replacing the core embedding network with one optimized for videos such as Inflated 3D Convnet (I3D) (Carreira & Zisserman 2017). However, based on our in-depth analyses, we find that there are considerable gaps in the existing metrics. For instance, the widely used FVD predominantly focuses on the spatial attributes of video content, with significant underemphasis on consistently evaluating temporal flow. This becomes a pressing concern as video generative models advance. Morcover, as these models develop capabilities generate longer videos, FVD remains constrained to evaluating videos with a mere 16 frames, heavily dependent on the constraints of the utilized embedding network; handle. Therefore, to develop properly functioning video
e.g., the input size that the network generative models, it is essential to evaluate videos of varying lengths, from short video clips to longer videos, and to have metrics that can assess both spatio-temporal quality separately.
Addressing the identified limitations, we propose (STREAM), a new metric designed to separately assess the temporal naturalness (STREAM-T) and the realism and diversity of videos (STREAM-S). STREAM-T evaluates the congruity in the overall trend of temporal changes between consecutive generated video frames in comparison those ob- served in real ones. STREAM-S comprises two evaluation components: STREAM-F, dedicated to analyzing the fidelity of videos, and STREAM-D, focused on evaluating the diversity within videos. Through a series of carefully designed experiments, we demonstrate the efficacy of STREAM in an- alyzing and evaluating the spatio-temporal aspects of video generative models. Our findings reveal the prevailing challenges in current video generative models, showcasing their struggle to generate video frames that are both realistic and diverse, particularly when the video length increases.
Evaluation and Analysis Metric
Our contributions are summarized follows:
Uniqueness: We propose STREAM, new video evaluation metric. STREAM is the first evalua- tion metric that can separately assess the temporal and spatial aspects of videos.
STREAM stands out as unique metric offering granular evaluation of videos and providing bounded values, notable advancement over FVD which only allows comparative as- sessment.
Boundness:
Universal Applicability: STREAM applies all video generative models. Furthermore, STREAM assesses video regardless of video length, offering a versatile solution to analyze a broad spectrum of videos.
Explainability: We demonstrate the efficacy STREAM under various scenarios, revealing the challenges and unsolved problems faced by current models, particularly in long video generation.
2

---

Published as a conference paper at ICLR 2024
2 SPATIO-TEMPORAL EVALUATION AND ANALYSIS METRIC (STREAM)
STREAM separately evaluates the spatial and temporal aspects of generated video dataset (Figure
D.
We first describe the embedding network for evaluation in Section 222 introduce STREAM-T,
metric for assessing the temporal aspect of videos in Section 2.3 Next, we introduce STREAM-S,
a metric for assessing the spatial aspect of videos in Section 2.4
2.1 NOTATION
Let real and fake video datasets be denoted as A' and D. respectively. We refer to the support of X' as supp(X)., We denote the a real video as X = {Xi, (2, .T. E X where T a number of frames or time. The projected real video Xproj through image embedding network f(-) is written as X Xrraj f(X) = (f(x)); (x, [(Xr)] E RRdxT, where d is the feature dimension. Throughout this paper, we refer to X as Xproj for simplicity. After applying FFT to X along time axis, the Fourier amplitude is defined as X = (So), ( ((,)...., (....,,,(r (CT/2)3 E RdxT/2 where is frequency with the order of Co <i ... Gr/2 and X (So = 0) is mean amplitude by definition. We use the notations for ) in the same manner as X.
2.2 EMBEDDING NETWORK
Evaluation metrics for generative models heavily rely on the embedding networks trained on large- scale dataset. This is because, embedding networks are known to extract features that align with human perceptual quality (Theis et al., 2015) and facilitate the handling of high-dimensional data. The choice of an embedding network is thus critical when assessing the quality of generated videos. However, existing video embedding networks are often unsuitable for effective evaluation yet, expe- riencing the intrinsic challenges of video embedding. These challenges include either the imperfect encoding of temporal information in video data (Li et al.) 2022, Wang et al., 2022) the mixed en- coding of spatio pemprrlt information coupled with rigid constraints on the number of input video frames (Wang al. 2023), significantly impeding long-term applicability and efficacy. To cireum- vent these limitations, we utilize a proficient image embedding network proposed by Caron et al. (2020) the foundation for our evaluation metric. By encoding each video frame independently, this approach enables separate consideration of spatial and temporal aspects of videos, unhindered by the number of video frames.
2.3 STREAM-T: EVALUATING THE TEMPORAL FLOW OF VIDEOS
To capture the temporal trend of independent frame features, we estimate power law distribution of frequency amplitudes and assess its skewness. By aggregating the skewness of both real and fake features and measuring their correlation, we compute the temporal flow score, "STREAM-T".
Distribution for video time-series signals The temporal variations of each feature within the em- bedding space can be perceived as variations in the amplitude frequencies, or the spectrum. From this standpoint, one might attempt to compare temporal flow between real and generated data by simply examining the mean or variance of these amplitudes. However, applying these measures di- rectly may introduce bias. This is often due to the dominance of power spectra at low frequencies, common characteristic observed in natural signals (West & Shlesinger [1990). Such dominance can obscure the genuine variations present in high-frequency components, leading to inaccurate evalua- tion of temporal flow differences. This phenomenon is referred to as "1/f fluctuation" of the spectral density, S(f), ). of stochastic process, having the form of S(f) = constant/ fo with f representing frequency. To better observe the differences in temporal flow, we transform this power law into the form of probability distribution. For a given arbitrary feature dimension and frequency Ç. (where i E [0, (/2). (x) can be approximated with power law C where a is a coefficient, and C' is normalization constant. Here, the coefficient C is simple scaling factor, and plays role in adjusting the overall slope of the power law. To approximate the parameters a R and C E IR, we solve X, ~ X = C <-a through least square estimation. Then the power law distribution (or amplitude distribution) is defined Px, = C Ç- where K is the normalization constant.
3

---

Published as a conference paper at ICLR 2024
Skewness of data distribution We calculate the difference in temporal flow of real and generated data by comparing the skewness of the power law distribution, Px, (6) = C K S rather than com- paring the mean or variance of Fourier amplitudes. First, for a random variable X with a known probability distribution, the skewness is defined as 7 E 1/23, where E[(X =4 is the third moment of the distribution, and D is the variance. To estimate the parameters in the above equation, utilize the widely known method of Moment Generating Function (MGF) (see Ap- pendix A.1 for details). By calculating the second and third moments of Px, (C) using the MGF, we can approximate all the necessary parameter and the skewness of power law distribution is computed as follows:
EI(S E Var(s)
V (
skewness 7x
VOE, (12-))
Evaluating temporal flow of videos As Al- gorithm [ for given real feature X and fake feature Y, we first perform FFT to obtain the real and fake amplitude features X E RRxx//2 and Y E RdxT/2, respectively. Through this, interpret real and fake temporal flows in the form of signals. For each dimension of the amplitude features, Xs 1XXT/2 and Y2 e R1x1/2 (A: & [1, d)), we approximate the power law distribution. This gives us real and fake power law distributions at feature dimension k. f(xA, oxx,, Cx ) and f(E,, ay, Cy,), respec- tively. In this step, we do not utilize the real and fake amplitude values when the frequency is O, because it leads STREAM-T to respond to both visual and temporal qualities (see Appendix A.3) From the given power law distributions, we measure the real skewness 7x,k and fake skew- ness 7 at feature dimension k. We repeat this process for all feature dimensions, which results in 7x and 7Y-
relation. Note that, We have configured the his- togram's bin size to 50. The bin size ranging from 50 to 100 have minimal impact on the per- formance, but larger bins can induce unstable performance with a limited data size (See Ap- pendix A.1II.
Algorithm 1 STREAM-T
Input: real feature X, fake feature Y Given: power law distribution f(-)
for X E X and Y E Y do
X < FFT(X) E IR R
Y + FFT(Y) RdXT/2
for k = 1 to do
Px
f(Xe, ax2 Cxa)
Pv, (C) (c)())
+ f(Vr, aY, Cr.)
7x.6 skewness (C)) ER
YY.k + skewness (Py, (5)) E end for
7x <- {Tx,1, 2x,;, (x,a) E R
Y + {nY.: TY. "..a) E Ridd
end for
7x <- (yx; all X E X) E Baxd yy + (fr3 all E y} E Raxd # cale histogram for each feature dimension d HISTx +- -his((yx) and HISTy + hist(ny) cale correlation for each feature dimension d px,y <- corr (e (HISTx: HISTy) E Rd return Ed px.y
To compare skewness between the real and fake at all feature dimensions, we construct his- tograms of real hist(rxx and fake hist(ry) for each feature dimension d. Then we calcu- late the correlation px.y between the real and fake histograms at all feature dimension d. The STREAM-T is defined as the average of the cor-
2.4 STREAM-S: EVALUATING THE SPATIAL QUALITY OF VIDEOS
Evaluating fidelity and diversity of images In the image generation task, Kynkäänniemi et al. (2019) have proposed improved precision and recall (P&R) metric separately evaluate the fidelity and diversity of the generated image quality. Given real dataset X' and fake dataset ), precision and recall are computed as
M precision(XX,)) = M Σ S(Y vV,X), recall(X(,))
N Σ (X)X))) X
N
where J(Y E )2 = 1 E suppa (X) and otherwise 0, and vice versa. Here, suppa (X) is the estimated support defined as the union of spheres, where each sphere has a sample point X as its center and radius equal to the distance to the k-nearest neighbors (k-NN) of X. Here, k is the parammee for k-NN, and is assigned an arbitrary value for use. Likewise, supp: (X) is defined as a collection of spheres with centers Y and radius equal to k-NN distances of Y.
4

---

Published as a conference paper at ICLR 2024
G Gaussian noise
Salt and pepper noise
Luminance shift
Color
Clest
Nedii
Stnce
Semap
Coean
voc
Medium
Stooe
Guean
Mediia
Stann
1.00
1.00
1.00
1.00
0.75
0.97
0.66
0.75
0.99
0.99
0.7
0.75
1.0
039
0.56
0.90
0.75
0.50
0.43
0.50
0.50
0.50
540
0.50
713
1270
0.00
0.25
28.1
254
0.00
0.25
40.7
70.0
311
0.00
0.25
1.3
23.3
81.8
101
0.00
0.25
844
90.9
0
Noise inteinsity"
Nolse Intensity"
Nolse intensity
Noise intersitty
STREAM-
STREAM
STREAM-D
Figure 2: Behavior of STREAM regarding noise affecting the "visual quality". All noise used in the experiment is equally added to entire video frames. Luminance shift decreases the contrast of all frames as the intensity increases. Color jitter applied by randomly sampling colors for each video sample, thereby affecting the overall color tone of the video.
Evaluating fidelity and diversity of videos Simply applying P&R video dataset has several issues. Firstly, our embedding network provides features for each frame of video. In the process of P&R support estimation, if we define a sphere with centered at sample point with radius defined by k-NN to estimate the support of data, we end up with support consisting of small spheres that may not even cover the features from a single video sample (See Appendix (A.4). Another approach is to use a larger value of k of k-NN than the number of video frames, but this approach also does not guarantee that a single sphere includes all the video frame features for a given video. To evaluate the fidelity and diversity of videos, mean amplitude, which corresponds to the amplitude at frequency of O. This aligns with the frame- wise average values. Given real and fake mean amplitude features (So) (So) and Y (So) E ('()), respectively, we define the STREAM-F for fidelity and STREAM-D for diversity
extend the P&R method for video data using the
M
N
1
STREAM-F
Σ
(((())) STREAM-D
Σ F(8(55)) ((Co
M
((oo
We set the hyperparameter for k-NN to k =
3 EXPERIMENTS
We assess the capability of STREAM to accurately evaluate spatial and temporal aspects of video data. We employ series of tests involving synthetic toy data and actual samples generated by video generative models to ensure a comprehensive evaluation of the effectiveness and reliability of the proposed metric in various scenarios. In all experiments, data. The results for all metrics are the average of five repeated measurements. In the accompanying figures, the shaded area represents the range of standard deviation.
consider total of 2,048 real and fake
3.1 TOY DATA EXPERIMENT
We use the synthetic CATER dataset (Girdhar & Ramanan, 2020) in our experiments. This ensures full control over the experimental conditions, allowing mitigate the influence of potential confounding factors such as degraded samples, temporally inconsistent samples, and blurry frames, which could compromise the integrity of our experimental setup. The CATER dataset consists of scenarios where multiple objects are positioned against static background, with only a select few demonstrating movement.
3.1.1 EVALUATING VISUAL QUALITY DEGRADATION IN VIDEO
In this experiment, we consider visual degradation using four types of noise, and the same noise is applied to all video frames. Therefore, the ideal behavior metrics evaluating the temporal natu- ralness should not respond to such noise, while metrics assessing the spatial aspects should exhibit a decreasing trend in response to noise intensity. In Figure 2 STREAM-T exhibits robust response to all noise intensities. Meanwhile, STREAM-S consistently decreases and converges zero the noise intensity increases. In contrast, FVD shows varying evaluation scores depending on the type of noise, exhibiting significant difference in these values. Note that there is upper limit in FVD.
5

---

Published as a conference paper at ICLR 2024
Example of local swap when number of swaps 2
Example of global swap when number of swaps are 3
Local swap
Global swap
Comparison of FVD scale
1.00
405
0.85
0.85
450
0.70
0.76
0.70
0.50
0.54
300
0.55
0.03
0.52
0.55
0.00
0.57
0.2
0.5
16.0
64.6
102
110
0.40
0.40
0-
4 Number swaps
Number swaps
Number of swaps
STREAM-T
STREAX-
STREAM-D
FVD Gooa swap)
FVD (ginbal swaol
Figure 3: Comparison of the behaviors of STREAM and FVD when changes are introduced to the "temporal flow" of video data. As in the example, local swap involves swapping the orders of two randomly selected frames within the video, while global swap entails exchanging a randomly chosen frame with a frame from another video.
Example of random translation with arbitrary shift: intensity
Random Translation
Replacement of video with stop scenes
Comparison of FVD scale
1.0
0.9
0.8
0.8
600
802
0.86
0.85
0.84
0.3
1000
0.6-
235
290
327
0.7
0.6
0.2
0.57
0.23
0.20
011
36
25.6
37.4
71.6
334
18
30
0.0
0.4
1.0
12
24
Translation Intensity STREAN-T
Ratio stop scenes
Intensity (ratio is equal to intensity/30)
STREAM
STREAM
(trasttioo)
(coae))
Figure 4: Behaviors STREAM and FVD in response to various temporal flow modifications. Random translation applies random directional shifts to each video frame. The translation intensity indicates the number of pixels to be shifted in random direction. Replacement of video with stop scenes replaces a certain proportion of videos in the dataset with video containing only still frames.
3.1.2 EVALUATING TEMPORAL FLOW DEGRADATION VIDEO
As shown in Figure position of two randomly selected frames (local swap) or by integrating frames from another video into the original video (global swap). For example, when local swap distortion where the order of ten video frames is altered. To induce such distortion global swap experiments, requires sampling ten frames. An ideal temporal flow metric should accurately detect the degradation in temporal quality in both settings, while an ideal spatial metric should remain unaffected by local swap but should discern the decrease in diversity caused by increased global swap, as this mixes frames between videos, creating more uniform outputs. Our results show that STREAM-T effectively captures the variations in the levels of temporal distortion, and STREAM- S accurately distinguishes the difference between local and global swap. In contrast, FVD shows inconsistent sensitivities across the experiments, being approximately five times more sensitive when there is reduction in diversity with temporal distortion (global swap).
manipulate the continuous movement of a video either by swapping the
five times, it leads to
3.1.3 EVALUATING OPATIO-TEMPONT DEGRADATION IN VIDEO
For "random translation". we randomly determine both the horizontal and vertical directions of movement for each video frame and then shift the frames parallel to these directions by the specified translation intensity. The temporal inconsistencies intensifies This degradation not only disrupts the temporal flow but also induces spatial distortions because the empty area of the video frame post-shift is filled with the surrounding pixels. The "replacement of
the level of degradation increase.

---

Published as a conference paper at ICLR 2024
Gaussian noise
Salt and pepper noise
Color
1.00
1.00
1.00
0.75
0:33
DCG
0:32
050
0.5
0.99
0.38
0.75
057
0.96
0.5
579
0.50
403
792
0.00
025
56.6
116
0.5
0.00
517
85.8
109
0.25
0.00
61.5
163
Noiso intersity
Noiso intersity
Nossointersiyy
STREAMT
STREAME
STREAMD
Figure 5: Behavior of STREAM when noise affecting the "visual quality" is applied to the real- world data (UCF-101). All noise used in the experiment is equally added to entire video frames. Color jitter is applied by randomly sampling color filters for each video, which alters the overall color tone of the video.
Random Translation
Replacement of video with stop scenes
Comparison of FVD scale
1.0
1.00
224 280
0.7
0.9
0.8
0.92 0.68
0.57
087
0.67
0.4
0.03
0.20
0.25
0.50
0.75
0.51
0.40
0.29
0.20
0.15
0:10
70
46.3
881
163
216
63.
00.
954
134
107
0.6
18 Franslation intensity STREAM-T
30
0.0
0.2
0.4
0.6
0.8
1.0
12
24
Ratio of stop scanes
Intensity (ratio is equal to intensity330) 330)
STREAM
STREAMD
(trestation
secnes)
Figure 6: Temporal distortion experiments using Kinetics-600 data. Identical to Section 3.1.3 this experiment compares the response of each metric when temporal information of video is distorted through (a) random translation, (b) stop scenes. We utilized translation intensity that preserves the realism of the video as much as possible (Figure TO).
video with stop scenes" replaces a certain proportion of videos with static scenes. At a replacement ratio of 1.0, the temporal flow within the video completely disappears. Therefore, an ideal temporal metric exhibit heightened sensitivity to stop scenes compared to random translations, whereas spatial metric should exclusively be responsive to random translation. Figure 4 shows that both STREAM-S and STREAM-T respond appropriately in all experiments. In contrast, FVD shows heightened sensitivity to random translation, presenting response that is more than three times as intense (reaching up to 822) in comparison to situations where temporal information is entirely absent. It is noteworthy that the real dataset does not contain any videos composed entirely of still scenes. This indicates that FVD places a higher emphasis on spatial quality over temporal aspects.
3.2 REAL DATA EXPERIMENT
We show the consistent performance of STREAM, using real dataset: Kinetics-600 (Carreira al. 2018) and UCF-101 (Soomro et al. 2012) datasets, aligning with results from the toy dataset (Sec- tion 3.2.T This consistency underlines the reliability of STREAM in evaluating the real video generative models (Section 3.2.2). Unlike FVD, constrained by its embedding network and unable to assess long video data, STREAM proves effective in evaluating long videos with more than 16 frames (Section 3.2.3
3.2.1 EVALUATING SPATIAL AND TEMPORAL DEGRADATION IN VIDEO
Section 3.1.1 presents an experiment conducted in a synthetic setting where the background remains static, and only simple geometric objects motion are present. To demonstrate STREAM maintains its effectiveness even when dealing with more intricate data involving moving backgrounds and vi- sual degradation, we have conducted experiments using the UCF-101 dataset (Figure(3). We applied the same gaussian noise, salt and pepper, and color jitter as in the experiment of Section 3.1.1 excluding the simpler degradation of luminance shift. As shown in Figure 3 STREAM-T remains unaffected by all types of visual or spatial noise, which are applied uniformly across all frames. Con- versely, STREAM-S, designed to assess spatial aspects, effectively responds to alterations in visual integrity, exhibiting a proportional decrement in response the escalating intensity of the visual noise applied. In contrast, FVD exhibits significantly different evaluation trends depending on the type of noise introduced. Additionally, we have conducted temporal degradation experiments using
experinne
7

---

Published as a conference paper at ICLR 2024
Table 1: Comparison and analysis of video generative models (unconditional). All the models
trained on the UCF-101 dataset generating 16 frame vidcos with 128 x 128 resolution. The numbers in parentheses next to evaluation scores represent the standard deviation of the scores, calculated through five repeated measurements.
VIS (t)
FVD (4)
STREAM-T (P)
STREAM-F (1)
STREAM-D (T)
MoCoGAN
16.64 (=0.09)
1174.3 (1336669
0.9683 (=00.001)
0.1595 (=0.023)
0.0000 (+:0.000)
DIGAN
24.32 (=9019)
763.64 (+28.82)
0.9743 (+000000)
0.3101 (=0.011)
0.0662 (+0.005)
TATS
34.39 (#0.30)
693.27 (#21.555)
0.9832 (+0.000)
0.9120 (=0.011)
0.0850 (+0.0055)
VideoGPT
30.35 (=00.55)
647.75 (5.34)3)
0.9782 (100.000
0.7806 (100.00)
0.3272 (110.005)
MeBT
64.54 (10.551)
504.21 (4.24.50)
0.9616 (100.01)
0.7441 (=10.006)
0.1852 (+00.09)
PVDM
60.02 (=0)82)
415.70 (+25.59)
0.9843 (+0.002)
0.6416 (=0.011)
0,3112 (+0.005)
Table 2: Comparison and analysis of video generative models which produce long video frames (unconditional). All the models are trained on UCF-101 dataset generating 128 frame vidcos with 128 x 128 resolution. sVIS and sFVD denotes the modified version of VIS and FVD measured for every 16 frames using sliding window. See Appendix aA. for the sample qualities.
sVIS (1)
sFVD (L)
STREAM-T (1)
STREAM-F (1)
STREAM-D (t)
MoCoGAN
11,8450
1454.1
0.3274
0.0615
0,0000
DIGAN
18.0075
1103.0
0.1327
0.1206
0,2656
TATS
40.3345
1008.0
0,0302
0.6284
0,2104
MeBT 33,9492
948.51
0.8265
0.6284
0.1601
Kinetics-600 in the setting analogous to Section 3.1.3 As shown in Figure 6 STREAM behaves as intended, while FVD shows even greater sensitivity to random translation.
3.2.2 EVALUATING VIDEO GENERATIVE MODELS
In Table we compare the video generative models using FVD, VIS, and our proposed metrics.
The models considered in the experiments are MoCoGAN-HD (Tul yaly et al. 2018 DIGAN Yu
al., 2022). TATS-bast (Ge et al. 2022), VideoGPT (Yan et al., 2021), MeBT (Yoo et al. 2023), and PVDM YYu et al. 2023). FVD and VIS, offering singular score, fall short of exposing the var- ied strengths and limitations these models. In contrast, our analysis through STREAM M reveals that while all considered models maintain reasonable consideration for temporal flow in generating short videos, they typically exhibit low diversity and a deficiency in realism. For example, relying solely on FVD and VIS benchmarks might give the perception that TATS has mediocre performance compared to the others. However, utilizing STREAM reveals that TATS surpasses other models like MeBT and VideoGPT in terms of realism and temporal naturalness in short video generation, despite its deficiency in diversity, which aligns well with the qualitative analysis (Appendix(A.) A.7). To validate the alignment of the results in Table with human perceptual quality, we evaluate the Spearman's rank correlation coefficient between human judgement scores and each metric on realism and tem- poral coherence. STREAM demonstrates effective representation of human perceptual quality with correlations of 0.9 for realism and 0.6 for temporal naturalness, while FVD remains less effective for both aspects (see Appendix 6 for further details). Therefore, to truly assess the performance of video generative model, it is important to consider the spatial and temporal aspects separately.
3.2.3 EVALUATING LONG VIDEO GENERATION
With the emerging advancements, video generative models now aim to produce longer video quences. Consequently, it becomes important to establish evaluation metrics capable of assessing long videos effectively. In Table 2 demonstrate that STREAM is the only metric that provides accurate evaluations regardless of video length. Since FVD and VIS cannot be directly applied for long-video evaluation, we slightly modify these by measuring it for every 16 frames using a sliding
8

---

Published as a conference paper at ICLR 2024
MoCoGAN-HD, DIGAN, and McBT mod- els. Our results show that, in the generation of 128 video frames, all these models present a notable decline in temporal naturalness compared to the generation of shorter. 16-frame video clips (Table D.. Notably, TATS-base demonstrates a significant drop in temporal coherence, which aligns well with the observations made in Yoo et al. (2023). This is readily apparent upon actual sampling (see Appendix AA.)). In addition, by applying STREAM-S, it becomes evident that current video gener- ative models are yet to overcome substantial challenges in generating realistic and diverse videos, particularly as the video length increases. These observations collectively highlight that there is still a long way to go for a natural, long video generation.
window, sFVD and sVIS. Here, we compare
sFVD
4 RELATED WORKS
Video Generative Models Video generative models need to learn additional temporal informa- tion compared to image generation models. Considering this, various network architectures and training methods have been proposed. Tulyakov et al.] (2018) and Li et al. (2019) have introduced a structure that combines a 2D convolutional network with an RNN to account for the temporal axis. Vondrick et al. (2016) have introduced a 3D convolutional structure that allows simultancous consideration of spatial and temporal information. Another approach involves training a network to generate low-quality videos initially and progressively increasing the network size during training to enhance video quality gradually (Karras et al. (017). Furthermore, networks based on two different architectures have been introduced to learn video content and motion separately, using one network to capture content and another to learn motion (Sun et al., 2020). All these prominent approaches rely on evaluation metrics to verify the effectiveness of networks in learning temporal information in videos. Therefore, in order to accurately assess the proposed methods, there is a need for new metrics that consider the spatial and temporal quality separately.
Video Evaluation Metrics In contrast to image generation, the video generation domain has only a limited number of metrics specifically designed for evaluation. Among these, Video In- ception Score (VIS) and Fréchet Video Distance (FVD) are most commonly used. They both utilize the Inflated 3D Convnet (I3D) (Carreira & Zisserman 2017p. Given real image dataset X' E Rd, generated image dataset g(Z) E K through a generator, and set of labels V IS com- putes the KL-divergence between the conditional real distribution p()),)') and marginal distribu- tion p())= Seez p(y E Vlg(z))dz by utilizing the softmax layer from the I3D network. On the other hand, FVD uses feature embeddings before the softmax layer of the I3D network, defining the real feature set X' E Rd and the fake feature set g E Rdd assuming that the distribu- tions of these features are Gaussian. FVD calculates the Wasserstein distance between the given real distribution X' Nuxx, Ex) and fake distribution g(Z) ((pg(2), (()g()))))(2))) as follows: gg( While these metrics offer initial insights, VIS and FVD each provide only a single, composite score, which limits their capa- bility to elucidate the diverse strengths and weaknesses inherent in video generative models. This obscures the nuanced variances and subtleties in model performances, yielding a potentially super- ficial evaluation of the models' generative capacities. This limitation becomes especially significant in comparison to metrics like Precision and Recall, as proposed by Kynkäänniemi et al. (2019) for image generation tasks. These metrics, by separately evaluating the fidelity and diversity of gener- ated image quality, allow for a more granular and nuanced understanding of model capabilities and shortcomings that is equally important in the domain of video generation.
FVD(X, ((Z)) := |px - Mg(2)) 2 - Σg(2)
CONCLUSION
Current evaluation metrics for video generative models simply extend the metrics originally de- signed for image generative models. Unlike images, however, video data requires careful considera- tion of temporal naturalness, which FVD fails to adequately address. In addition, the current state of performance measurement and analysis for video generative models is constrained by a limited set of metrics. To address this, we propose STREAM, a new video evaluation metric, which allows for the evaluation of realism, diversity, and temporal aspects of videos, independently. Through care- fully designed experiments, we have shown that STREAM is an effective metric that can evaluate and analyze the performance of video generative models in a broader perspective.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No.2022RI CIC1008496), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2020- 0-01336, Artificial Intelligence Graduate School Program (UNIST), No.2021-0-02066 Intelligence Innovation Hub, No. No.2202-0-00955,
Artificial Vision and Language for Decision Making, (Part Few-Shot Learning of Causal Inference in and Generation with Knowledge- Deep Logic Neural Network).
Comprehensive Video Understanding
REFERENCES
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912 9929449, 2020.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 6299-6308, 2017.
Joao Carreira, Eric Noland, Andras Banki- honki-Horvatth, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. arXiv preprint arXiv:1808.01344 2018.
Frederik Ebert, Chelsea Finn, Alex X Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections. CoRL, 12:16, 2017.
songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vggan and time-sensitive transformer. In European Conference on Computer Vision, pp. 102-118. Springer, 2022.
Rohit Girdhar and Deva Ramanan. CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning. In ICLR, 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Tobias Höppe. Arash Mchrjou, Stefan Bauer, Didrik Nielsen, and Andrea Dittadi. Diffusion models for video prediction and infilling. arXiv preprint arXiv:2206.0777 07666, 2022.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint 1rXiv:1710.101100 2017.
Tero Karras, Samuli Laine, Mika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 8 8110-8111 2020.
Pum Jun Kim, Yoojin Jang. Jisu Kim, and Jaejun Yoo. Topp' &r: Robust support estimation approach for evaluating fidelity and diversity in generative models. arXiv preprint arXiv:2306.080) 08013, 2023.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances Neural Information Processing Systems, 32, 2019.
Kunchang Li. Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Limin Wang. and Yu Qiao. Uni- formerv2: Spatiotemporal learning by arming image vits with video uniformer. arXiv preprint arXiv:2211 1.00552,, 2022.
Yitong Li, Zhe Gan, Yelong Shen, Jingjing Liu, Yu Cheng, Yuexin Wu, Lawrence Carin, David Carl- son, and Jianfeng Gao. Storygan: A sequential conditional gan for story visualization. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3329-6338, 2019.
10

---

Published as a conference paper at ICLR 2024
Muhammad Ferjad Naeem, Scong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In International Conference on Machine Learning. pp. 7176-7185. PMLR, 2020.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Masaki Saito, Shunta Saito, Masanori Koyama, and Sosuke Kobayashi. Train sparsely, generate densely: Memory-efficient unsupervised training of high-resolution temporal gan. International 2020.
Journal of Computer Vision,
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings, pp. 1-10, 2022.
Mark J. Schervish and Morris H. DeGroot. Probability and Statistics, volume 563. Pearson Educa- tion, London, UK, 2014.
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. Advances in neural information processing systems, 32, 2019.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah, Ucf101: dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 1X1v:1212.0402 2012.
Ximeng Sun, Huijuan Xu, and Kate Saenko. Twostreamvan: Improving motion modeling in video generation. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 2744 2753, 2020.
Lucas Theis, Aäron den Oord, and Matthias Bethge. A note the evaluation of generative models. arXiv preprint
2015.
Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1526-1535, 2018.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. 2019.
Vikram Voleti, Alexia Jolicoeur-Martincau and Chris Pal. Mevd-masked conditional video diffusion for prediction, generation, and interpolation. Advances in Neural Information Processing Systems, 05:23371-23385. 2022.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. Advances in neural information processing systems, 29, 2016.
Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14549-14560, 2023.
Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint
2022.
Bruce West and Michael Shlesinger. The noise in natural phenomena. American Scientist, 78(1): 40 1990.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104 10157, 2021.

---

Published as a conference paper at ICLR 2024
Jachoon Yoo, Semin Kim, Doyup Lee, Chiheon Kim, and Seunghoon Hong. Towards end-to-end generative modeling of long videos with memory-eficient bidirectional transformers. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22888- 22897, 2023.
Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Woo Ha, and Jinwoo Shin. Generating videos with dynamics- implicit generative adversarial networks. arXiv preprint arXiv:2202.10571, 2022.
Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18456-18466, 2023.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 MOMENT GENERATING FUNCTION
Moment Generating Function is stated as follows:
Definition A.1. (Moment Generating Function) (Schervish & DeGroot 2014). Given random vari- able X with probability density function f(X), the moment generating function of X,
(t) = { exx f(X)dz,
exists there is an h > O such that for all in - < h.
Taking the derivative of the MGF with respect t allows to calculate arbitrary-order moments of X, such as the first moment which is the mean, and by taking the second derivative, the second moment which the variance.
A.2 STREAM-T METRIC BASED ON THE VIDEO EMBEDDING NETWORK
In this section, we validate whether the STREAM-T metric behaves as intended when using a video embedding network instead of an image embedding network. In this experiment, we have used the Masked Video Auto-encoder V2 (Wang et al.) 2023) which is widely known to effectively capture temporal information of video input in their latent space. Since STREAM-T gives a score based on the video's 's temporal naturalness, we aim for STREAM-T to be robust to the visual distortions when there is sufficient temporal information in the video. On the other hand, in cases where the order of frames is shuffled as in local swaps or when different frames suddenly appear as in global swaps in terms of temporal aspects, we expect STREAM-T to gradually decrease in response based on the number of swaps. As in Figure 7 when visual distortion occurs, STREAM-T does not behave as intended. This can be attributed to the fact that video embedding mixes both visual and temporal information of the video into the latent space. Additionally, when local swap occurs, STREAM-T does not react sensitively, whereas it reacts significantly to global swaps. Through these results, we confirm once again that in feature space where visual and temporal information is not accurately distinguished, STREAM-T does not operate as intended.
Visual distortion
Temporal distortion
1.0
1.0
0.960
0,972
0.75
1.00
0.966
0.993
0.991
059
STREAM-T swap) STREAM-7 Iglitel
0.8
0.30
STREAM-T (saltand pappert
0.6
STRFAM-T (contastcchngge
0.606
0512
0.25
0.001
0.034
0.128
0.154
0,077
0.00
Noise intensity
Number swaps
Figure 7: The behavior of STREAM-T T when using video embedding network instead of image em- bedding network. Visual distortion and temporal distortion experiments, based on UCF-101 dataset, are conducted using the same methods described in Section 3.1.1 and 3.1.2] respectively. In each ex- periment for visual distortion, the maximum noise intensity is applied while preserving the temporal flow of the video. The numbers within the figures represent the scores of the metrics.
A.3 STREAM-T METRIC UTILIZING MEAN AMPLITUDE SIGNAL
When calculating STREAM-T, we do not use mean amplitude (i.e., 8(o = 0)), because it repre- sents the average values of simple amplitude signals and does not provide additional information of temporal signal variations. In addition, the use of mean amplitude values would actually hinder STREAM-T from detecting the overall temporal signal changes. Figure & STREAM-T, caleu- lated including the mean amplitude, appears to be more sensitive to visual distortion compared to the previous result in Section 3.1.I This indicates that mean amplitude leads the metric more depen- dent visual information. Additionally, in Figure STREAM-T becomes more sensitive to global swap when utilizing mean amplitude. In other words, considering the visual information provided by mean amplitude influences STREAM-T in way that aligns with the trend of reduced diversity. Therefore, to accurately evaluate temporal naturalness with STREAM-T T, it is better to exclude mean amplitude, which does not provide new information about temporal changes.
13

---

Published as a conference paper at ICLR 2024
Gaussian noise
and pepper noise
Luminance shift
Color jitter
1.00
1.00
1.00
1.00
1600
STREAM
0.75
0.75
0.75
0.75
1200
0.50
0.50
0.50
800
0.25
0.25
0.25
400
0
2
0.00
3
0.00
0.00
3 Noise intensity
Noise intensity
Noise intensity
Noise intensity
STREAM
STREAK
STREAM
PVD
Figure 8: Behaviors of STREAM to various types of visual degradation when utilizing the mean
amplitude signal. All the settings for this experiment is identical to the experiments Figure
Local swap
Global swap
Comparison of FVD scale
1.00
1.00
600
0.85
0.85
450
0.70
0.70
300
0.55
0.55
150
0:10
3 Number swaps
0
Number of swaps
Number of swaps
STREAM-T
STREAX-F
STREAM-D
FVD Gooa
FVD (glonal swop)
Figure 9: Behaviors of STREAM to various types of temporal degradation when utilizing the mean amplitude signal. All the settings for this experiment is identical to the experiments in Figure
A.4 WHY STREAM-S UTILIZES AMPLITUDE AT FREQUENCY 0
In STREAM-S, the amplitude at frequency 0 (i.e., mean amplitude) calculations for two main reasons. First, the mean amplitude encapsulates the average signal information of a video, offering a comprehensive overview of its spatial characteristics. Second, including amplitudes across all frequencies renders P&R calculation inaccurate.
More specifically, P&R is designed to approximate the support of real and fake distributions, typ- ically using the union of spheres centered on data points with radii equal to the distance to the k-nearest neighbor. These spheres are expected cover at least & data points (or image features) and collectively represent the population data support. However, videos, unlike images, are com- posed of multiple image frames. For instance, a 16-frame video yields 16 image features obtained by an embedding network. Applying FFT along the temporal axis, get 8 amplitude features for each video. In this context, using P&R directly on videos might result inaccurate data support estimation, as single sphere may not even encompass all data points from one video. Naturally, in such a scenario, the approximated data support may fail to represent the full scope of video data. To address this, we use the mean amplitude, which represents the average feature of a video, data point. This approach ensures a more accurate and representative approximation of real and fake video data support, enhancing the precision of STREAM-S measurements without limiting it the number of video frames as shown in our experiments.
single
14

---

Published as a conference paper at ICLR 2024
A.5
TEMPORAL DISTORTION EXPERIMENT USING KINETICS-600 DATASET
Example of Kinetics- 600 sample at random translation intensity
Example Kinetics- 600 sample at random translation intensity
Example Kinetics 600 sample at random translation intonsity
Example of Kinetics- 600 sample at random translation intonsity"
Figure 10: An example of Kinetics- 600 data (in Figure (a)) regarding the degree of distortion based on the random translation intensity. The information lost due translation was compensated for through reflection. Similar to Section 3.1.3 this experiment employs translation based on intensity in random directions.
A.6 SPEARMAN'S RANK CORRELATION COEFFICIENT WITH HUMAN JUDGEMENT SCORES
We have collected human judgement scores to quantify how well STREAM reflects human percep- tual quality. We asked 81 raters to evaluate video generative models Table for each realism and temporal naturalness aspect. Each rater reviewed five randomly sampled videos from each model and assigned scores for realism and temporal naturalness. The scoring criteria for video realism is as follows: video scenes are indistinguishable from real video (6 points), scenes that realistic and clearly interpretable (5 points), scenes are partially realistic (4 points), not realistic but recognizable scenes (3 points), only partially recognizable scenes (2 points), and complete inability to discern any- thing from the scenes (1 point). Additionally, the scoring criteria for temporal naturalness of videos as follows: the scene transitions are smooth and continuous (3 points), some scene transitions temporally inconsistent (2 points), and discontinuous scene transitions (1 point). Subsequently, we have measured the mean spearman's rank correlation coefficient (spearman's 's correlation) between model rankings based on human judgement scores for realism and temporal naturalness, and the model rankings from STREAM. From the results Table 3 STREAM shows spearman's correla- tions of 0.9 and 0.6 for realism and temporal naturalness, respectively, confirming that it provides scores that align well with human perceptual quality. On the other hand, FVD has spearman's 's cor- relation of 0.7 and 0.5 for realism and temporal naturalness, respectively, indicating that it does not reflect human perceptual quality as effectively as STREAM. Note that, evaluating video diversity through human assessment is challenging do fairly unless one reviews and remembers the features and types of a larger number of videos. Therefore, we do not proceed with additional experiments in this regard.
Table 3: Human evaluation results of video generative models listed in Table I Each spatial and temporal scrore denotes the sum of scores evaluated by 81 raters for each model, and the numbers in parentheses indicate the average scores for each model.
Models
MoCoGAN-HD
DIGAN
TATS-base
VidcoGPT
McBT
Spatial score
183 (2.25)
237 (2.92)
378 (4.66)
304 (3.75)
356 (4.39)
Temporal score
122 (1.50)
141 (1.74)
161 (1.98)
129 (1.59)
145 (1.79)
15

---

Published as a conference paper at ICLR 2024
A.7 EXAMPLE OF SAMPLE QUALITY FROM SHORT FRAME VIDEO GENERATIVE MODELS
(a) 128x128 UCF videos with 16 frames sampled from MoCoGAN HD
(b) 128x128 UCF videos with frames sampled from DIGAN
(c) 128x 128 UCF videos with 16 frames sampled from TATS
(d) 128x128 UCF videos with frames sampled from VideoGPT
(e) 128x UCF videos with 16 frames sampled from MeBT
Figure 1I: Example of sample quality sampled from video generative models trained UCF-101 dataset. All the models listed in this figure are compared through STREAM and FVD in Table
16

---

Published as a conference paper at ICLR 2024
A.8 LONG VIDEO SAMPLE QUALITY OF TATS- -BBSE
(a) Sample 128x UCF videos with 128 frames sampled from TATS-base
(b) Sample 2x8 videos frames sampled from S-TS-base
Sample 128x128 UCF videos with 128 frames sampled from TATS- base
(d) Sample 4: 128x128 UCF videos with 128 frames sampled from TATS-base
Figure 12: Example of sample quality of TATS-bar base trained UCF-101 dataset. All listed video samples have 128 frames with a resolution of 128 x 128. We have visualized frames from the initial frame to the 64th frame for each video sample. The sample qualities from TATS-base resemble the stop scene (sample 1), local swap (sample 2), and global swap (sample 3, 4) cases in our experiments.
17

---

Published as a conference paper at ICLR 2024
A.9 EXAMPLE OF SAMPLE QUALITY FROM LONG FRAME VIDEO GENERATIVE MODELS
The figures in this section illustrate the sample quality of video generative models compared in Table 2 Examining the sample quality of MoCoGAN-HD, it is able observe the model struggles to generate realistic frames but showing diverse frames with temporally continuous changes, which consistent with the scores obtained from STREAM-F and T. In Figure [4 the frames sampled from DIGAN does not vary much over time and each frame does not have realism, corresponding the low score of STREAM-F and T. The sample quality of MeBT, in Figure 15| shows relatively lower fidelity and temporal naturalness not having various changes over time, and this also aligns with the mediocre scores of STREAM-F and STREAM-T. By comparing the sample quality and STREAM score for each video generative model, STREAM seems to align with human perceptual quality.
Figure 13: Example of sample quality of MoCoGAN-HD trained UCF-101 dataset. The video
has 128 frames with resolution of x 128 x 128.
Figure 14: Example of sample quality of DIGAN trained on UCF-101 dataset. The video has 128 frames with resolution of x x128 x 128.
18

---

Published as a conference paper at ICLR 2024
Figure 15: Example of sample quality of MeBT trained on UCF-101 dataset. The video has 128 frames with resolution of x x288> x 128.
A. ABLATION STUDY ON THE DATA SIZE
Video data requires significantly larger memory capacity than images, highlighting the impor- tance of evaluation metrics that can perform ac- curately with limited data size. We conducted an ablation study to determine at which data size FVD and STREAM-T are capable to pro- vide consistent scores. For the experiment, we compared three types of distorted video data with real ones using local swap, global swap, and random translation. For each temporal dis- tortions, the noise intensity is randomly cho- sen. As shown in Figure STREAM-T shows consistent evaluation performance at approxi- mately a data size of 2,000.
500
FVD
STREAM-T
INOP
400
goba
neakez Uraddiin.
0.0
300
0.7
locil soop
200
0.5
100
2000
3000
4000
1000
2000
3000
Sample size (n)
Sample size (n)
Figure 16: Ablation study data size. The y- axis represents the scores of each metric, and the dashed lines indicate the metric values when the data size is 2,000. The noise intensity of local swap, global swap, and random translation randomly chosen.
A.11
ABLATION STUDY ON THE HISTOGRAM BIN SIZE
Gaussian noise
Salt and pepper noise
L Luminance shitt
Color jitter
1.00
1.00
1.00-
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0
Noise intensity
Noise intensity
Noise intensity
Noise intensity
bi size 10
hin size 100
bim
300
biin size 500
bin
700
550
Figure 17: Robustness of STREAM-T T against spatial noise across various bin sizes, Ablation study on bin size of STREAM-T. This investigation is built upon the same experimental setup in Section 3.1.I For each spatial noise scenario, we validate whether the evaluation tendencies change based on different settings of the bin size, which is a hyper-parameter of STREAM-T.
19

---

Published as a conference paper at ICLR 2024
Local swap
Global swap
Random translation
Stop scenes
1.00
1.00
1.0
1.0
0.85
0.9
0.70
0.8-
0.6-
0.55
0.55
0.7
0.4
0.40
0.40
0.6
6
8
10
12
18
24
30
0.0
0.2 0.4
066 088
1.0
Noise Intersity
Nolse Intersity
Noise intensity
Noise intensity
lin size = 10
bin size = 100
biin size 300
bin sizze = 500
bin size = 700
bin size 50
Figure 18: Consistency of STREAM-T against temporal distortions across various bin sizes. This investigation is built upon the same experimental setup as in Section B.1.2] and 3.1.3] For each tem- poral distortion, we validate whether the evaluation tendencies change based on different settings of the bin size, which is a hyper-paramett parameter of STREAM-T.
We have conducted an ablation study on histogram bin size which is a T. In Figure 17 STREAM-T exhibits robustness across various types of visual noise regardless of bin size. Furthermore, in the presence of various types temporal distortions (Figure (8). STREAM-T demonstrates consistent evaluations, except when the bin size becomes extremely small. We have selected the bin size (= 50) that exhibits the most optimal performance.
of STREAM-
A.12 EVALUATING VISUAL QUALITY DEGRADATION IN VIDEO WITH GAUSSTAN B..R
Gaussian noise
Salt and pepper
Luminance shift
Color jitter
Gaussian blur
1.00
1.00
1.00
1.00-
1.00
0.00
0.25
0.50
5.
16.9
10L
0.25
0.00
0.50
6.4
78
120
213
0.25
0.50
7.3
23,3
333
0.25
0.50
0.00
94.0
329
710
1270
0.50
0.25
6.6
26
533
3
4
2
3
3
5
Noise intensity
Noise intensity
Noise intensity
Noise intensity
Noise intenstly
STREAN-T
STREAM-
STREAM-
IVD
Figure 19: Comparison of metrics considering Gaussian blur distortion in addition to the experiment of Section B.I.I STREAM-T, which evaluates the temporal naturalness of videos, should demon- strate robust evaluation trend for all types of visual distortions. We measure the extent of visual distortion and decrement in video diversity through STREAM-F and STREAM-D, respectively.
Video generative models often produce blurry samples, necessitating development of effective eval- uation metrics capable of appropriately penalizing the visual quality of such samples. In Figure 19 we have observed that as the intensity of Gaussian blur increases, STREAM-S exhibits heightened sensitivity, ultimately converging to score of O. This responsiveness underscores STREAM-S's ability to discern and penalize the degradation in visual quality associated with intensifying blur.
A.13 VALIDATION OF STREAM-T BY COMPARING FAST AND SLOW VIDEO DATASET
Table 4: Experiment on the evaluation trends of STREAM-T with variations in Video Frame Per Second (FPS). Real video exhibit a fast change in frame rate at 1.0 FPS, while setting the fake video to have progressively decreasing speed of change from x 1.0 FPS to x0.2 FPS.
Fake FPS
x 1.0 FPS
x0.8 FPS
x0.6 FPS
x0.4 FPS
x0.2 FPS
STREAM-T 0.996 (+0.000) 0.765 (+000033) 0.764 (+0.003) 0.418 (+0.003)
0.352 (+00.04)
The objective of video generative models is to resemble the spatial and temporal characteristics of real dataset by estimating the real data distribution. Therefore, when there are differences in movements of real and fake video datasets, the video generative model has estimated the real distri- bution with a temporal difference. In Table 4 we have compared the real videos with fast FPS and generated videos with slow FPS using STREAM-T. STREAM-T has decreased value when the generated video has a relatively low FPS compared to the real video, which shows the effectiveness of STREAM-T in capturing the temporal difference.
20

---

Published as a conference paper at ICLR 2024
A.14 ADDITIONAL SYNTHETIC EXPERIMENT WITH USING UCF- DATASET
Table 5: Study on the stability of STREAM metric. To verify the stability of STREAM metric in few corner cases, consider two experimental settings, reverse and flip. The first experiment, 'reverse", involves reversing the order of video frames in each video sample. The 'flip' experiment verifies the realism of video dataset created by horizontally flipping all frames of each video. Lastly, the 'reverse flip' applies both the 'reverse' and 'flip' settings simultaneously.
VIS (1)
FVD (L)
STREAM-T (t)
STREAM-F (t)
STREAM-D (1)
Reference
79.0769
14.3820
0.9989
0.9833
0.9842
Reverse
78.3968
24.9380
0.9989
0.9854
0.9825
Flip
78.5281
36,0726
0.9954
0.9843
0.9812
Reverse & Flip
78.5338
40,0170
0.9954
0.9827
0.9827
Table 6: Comparison of evaluation trends between STREAM-T and FVD when the order of partial video sequence is reversed. This experiment involves selecting three consecutive frames randomly from 16 frames of a real video and reversing the order these three frames when the reverse intensity is set to 1. As the value of the reverse intensity increases, the process described for intensity is repeated for the specified number of times (i.e., reverse intensity). An ideal metric should give progressively worse scores with increasing reverse intensity.
Reverse intensity
2
4
6
8
10
12
STREAM-T
0.9948
0.9688
0.9262
0.8865
0.8451
0,8231
0.7916
FVD
74.203
85.229
83.641
93.113
98.363
115.98
113.09
Given the nature of our metric, which focuses on evaluating continuity and consistency through frequency analysis, STREAM-T remains agnostic to the direction of time. However, this unique characteristic allows STREAM-T to be sensitive not only to situations where only a portion of the video sequence is reversed but also to various types of temporal inconsistencies. However, it is important to note that the generation of a completely reversed video is an uncommon scenario, and in such cases all the metrics (STREAM-T, VIS, and FVD) may not provide optimal responses. Still, other than this rare case, STREAM demonstrates effectiveness in detecting partially reversed segments within videos that are otherwise temporally natural (Table 6). For example, when the video entirely reversed on the temporal axis, FVD shows a modest increase from 14.38 to 24.93. This change, while noticeable, is relatively small compared significant fluctuations (such those in the hundreds) observed in other cases (Figure 5 and 6 in Section 3.2.1). Thus, in practice, this minor variation would make it difficult to conclusively determine FVD significantly impacted by this aspect alone.
The spatially flipped videos, being the mirror image of each video frame, possess highly realistic spatial and temporal quality. Through the experiment, we have observed that STREAM and VIS remains unchanged, while FVD slightly increased from 14.38 to 36.07. When both reverse and flipping is applied, the metrics should provide scores indicating spatially and temporally realistic results. From the results, VIS and STREAM consistently exhibited scores identical to the reference score, whereas FVD showed increased value from the reference score (14.38) to 40.01.
21

---

Published as a conference paper at ICLR 2024
A.15
SENSITIVENESS OF VIS TOWARD SPATIAL AND TEMPORAL DEGRADATIONS
Causiaan noise
Salt and pepper
Color
1.00
1.00
1,00
0.75
0.75
200
0.75
005
45.5
0.30
400
0.25
189
0.5
025
709
601
0.00
705
3.50
0.00
000
655
1
4
2
3
5
2
5
Nofse intensity
Noise intenity
Noiseirtensi
STREAMER
Figure 20: Comparison of VIS and STREAM under various types of visual degradation on UCF-101 data. Gaussian noise, salt and pepper noise, and color jitter are conducted on the same experimental setup as Section 3.2.1
Local swap
Global swap
Stop scene
1.00
1.00
1.00
0.75
785
0.75
75.
0.75
21.6
56.
0.50
0.50
0.25
0.25
35.
0.25
9.38
0.00
0.00
0.00
1
3 Number of swaps
5
4
0.0
0.2
04
0.6
0.8
1.0
Number swaps
Ratio ol stop sccnes
STREAM-T
STREAM-F
STREAM-D
Figure 21: Comparison VIS and STREAM under various types of temporal degradation on UCF- 101 data, aiming to determine whether VIS respones to changes in temporal. Local and global swap conducted on the same experimental setup as Section 3.1.2 and stop scenes conducted under same conditions as
We conducted experiments in Section 3.2.1 using the UCF-101 dataset to compare STREAM and VIS when spatial distortions such as gaussian noise, salt and pepper noise, and color jitter are applied to videos. Similarly, for temporal distortion, we performed experiments using local swap, global swap, and stop scene in Section and.2.aan33.11
compare the evaluation trends of STREAM-S and VIS. In Figure 20] both STREAM and VIS react to spatial distortion. However, for temporal distor- tion, as shown in Figure 21. VIS cannot discern different levels of temporal distortions compared to STREAM-T.
A.16 ADDITIONAL SYNTHETIC EXPERIMENT WITH USING TAICHI DATASET
Table 7: Comparison and analysis of video generative models (unconditional). All the models are trained on the Taichi dataset (Siarohin al., 2019) generating 16 frame videos. The numbers in parentheses next to evaluation scores represent the standard deviation of the scores, calculated through ten repeated measurements. Note that LVDM generates videos resolution of 256x256; for comparison purposes, we have downscaled them to 128x resolution.
Resolution
FVD (L)
STREAM-T T (1)
STREAM-F (t)
STREAM-D (t)
DIGAN
128x 128
728.7 (+12.21)
0.7711 (+00.0550)
0.7258 (+:00693)
0.1901 (+0.0152)
TATS
128x 128
445.6 (=13.633
0.8712 (+00.045)
0.7197 (10.0882)
0.4024 (10.0244)
McBT
128x 128
441.7 (1:11.69
0.8561 (1.0.003)
0.6712 (1000672)
0.3557 (10.0186)
LVDM
128x 128
205.5 (+-6.619)
0.9356 (+0.0017)
0.8291 (+000575)
0.4829 (+0.0288)
22

---

Published as a conference paper at ICLR 2024
A.17 THE EFFECT OF FRAME RESOLUTION AND VIDEO LENGTH ON STREAM
In this section, we have conducted experiments to assess STREAM's performance across different video resolutions and lengths under visual temporal distortion. We have used gaussian blur (for visual distortion) and local swap (for temporal distortion).
STREAM-F
STREAM-D
STREAMTT
FVD
1.00
1.00
1.00
2500
0.75
0.30
0.50
0.50
1250
0.25
0.25
0.25
625
0.00
0.00
0.00
0
3 Noise Intensity
2
4
2
3
4
3 Noise Intensiity
4
4
5
Naise Intereity
Noise Intersity
240%
120x 120
Figure 22: Performance of STREAM for the visual distortions on 16 frame videos with "various resolutions". In this experiment, increase the strength of gaussian blur noise to verify whether STREAM-S exhibits a gradual decrease in scores, while STREAM-T maintains consistent scores. Since the intensity of noise applied to the video is same at each noise level, important information in videos with lower resolutions is destroyed more rapidly compared to higher resolutions. Therefore, STREAM should react more sensitively as videos have lower resolutions.
STREAM-F
STREAM-D
STREAM-T
FVD
1.00
1.00
1.1
400
0.75
0.9
300
0.50
200
0.25
0.5
100
0.00
0.00
0.3
0
12
16
16
Ratio swapped
Ratio swapped frames
Ratio of swapped frames
Ratio of swapped frames
240 240
180 x 189
120 120
60
Figure 23: Performance of STREAM for the temporal distortion on 16 frame videos with "various resolutions". In this experiment, using the same setup as the local swap in Section 3.1.2] we verify whether STREAM-T appropriately responds temporal distortion as the number swaps increase.
We tested video resolutions of 60x60, 120x 120, 180x 180, and 240x240. Applying a uniform gaussian blur across resolutions (in Figure 22). we noted a more pronounced degradation in lower resolution videos, where fewer pixels carry critical information. This trend is accurately captured by the metrics. In the local swap tests (in Figure 23). we shuffled video frames equally across reso- lutions. Since local swap affects only temporal quality without impacting the spatial quality of the video, an ideal metric should consistently provide low temporal scores as the number local swaps increases, regardless of resolution. Our results show that FVD and STREAM meet these criteria.
23

---

Published as a conference paper at ICLR 2024
STREAM-F
STREAM-D
STREAM-T
FVD
1.00
1.00
1.00
700
0.75
0.75
0.75
525
0.50
0.50
0.50
350-
0.25
0.25
175-
0.00
0.00
0.00
0
2
5
Noise intensity
Noise intensity
Noise intensity
intensity
125 video tramss
64 video Tramss
vidio Tranes
16 video Treres
Figure 24: Performance of STREAM for the visual distortions on videos with "various video length". In this experiment, we increase the strength of gaussian blur noise to verify whether STREAM-S exhibits a gradual decrease scores, while STREAM-T maintains consistent scores. Note that, FVD is measured for every 16 frames using a sliding window with stride of 16.
STREAM-F
STREAM-D
STREAM-T
FVD
1.00
1.00
1.00
250
0.75
0.75
0.75
200
0.50
0.50
0.50
150
0.25
100
0.00
0.00
0.00
50
0
16
12
16
12
16
Number of swaps
Number swaps
Number of swaps
Number of swaps
vibee formss
vodeo frimes
vi frimes
video fremes
Figure 25: Performance of STREAM for the temporal distortion on videos with "various video length". : In this experiment, using the same setup as the local swap in Section 3.1.2. we verify whether STREAM-T appropriately responds to temporal distortion as the number of swaps increase. Given that local swapping is uniformly applied to videos irrespective of video length, shorter videos exhibit greater temporal degradation when swapping occurs compared to longer videos. Therefore, ideal temporal metrics should adeptly capture the extent of temporal distortion with respect to video length. Note that, FVD is measured for every 16 frames using a sliding window with stride of 16.
We tested video lengths of 16, 32, 64, and 128 frames. When applying uniform gaussian blur (in Figure(2] the spatial quality reduction was consistent across lengths, without temporal quality degradation. Both STREAM and FVD effectively captured this behavior. For local swaps, we used a fixed number of swaps across different lengths (in Figure 25). Consequently, local swaps affect longer videos less, as the proportion of unshuffled frames increases. STREAM demonstrated better temporal consistency for longer videos, whereas FVD struggled to reflect this variation. Addition- ally, FVD exhibited increasing deviations in evaluation scores as the video length increases. This highlights FVD's s limitations in evaluating videos longer than 16 frames, as it tends to over or un- derestimate temporal distortions. Our findings show the credibility of STREAM in handling various video resolutions and lengths, and they provide valuable insights into the limitations of existing metries in such contexts.
A.18 EVALUATION OF VIDRO PREDICTION MODELS USING STREAM
Table 8: Comparison and analysis of video prediction models. All the models are trained on the BAIR dataset predicting 15 frame vidoes with 64 x 64 resolution from conditional frame. The numbers in parentheses next to evaluation scores represent the standard deviation of the scores, calculated through five repeated measurements.
FVD (1)
STREAM-T (t)
STREAM- (1)
STREAM-D (t)
RaMViD
160.1 (#:2.719)
0.9193 (+.0.000)
0.5562 (1.0.0151)
0.4709 (+.0.061))
MCVD
21.63 (=0551
0.9896 (#0.0002)
0.8355 (+t0.0076)
0.8885 (#0.0086)
In this section, we demonstrate that STREAM is not limited to solely comparing unconditional video generative models; rather, it is equally applicable to a broader range of tasks, including video prediction models. The objective of video prediction model is to learn representations of objects and their temporal changes from given past frames and predict the future video scenes. Therefore, utilize the dataset consisting of input-output pairs of video prediction models as real and fake
24

---

Published as a conference paper at ICLR 2024
datasets for evaluation. We have compared open-source models, RaMviiDHHppp al. 2022 and MCVD(Voleti et al. 2022). trained on (AIR(Eberr et al. 2017) dataset predicting 16 frame video with 64 x 64 resolution.
In Table 8 the FVD score indicate MCVD's superiority in generating better predictions than RaMViD. STREAM offers detailed analysis of each model's performance in terms of fidelity, diversity, and temporal naturalness. While MCVD excels all aspects, RaMViD lags, particularly in fidelity and diversity. In detailed assessment (in Figure 27 [6) observe that RaMViD tends to produce videos with blurry scenes containing artifacts, which are the source of its low fi- delity score. Moreover, shows temporally inconsistent transitions; e.g.. objects in the video tend to suddenly appear or disappear over time. On the other hand, MCVD's outputs are of higher fidelity and demonstrate smoother, more consistent transitions with fewer discrepancies object appear- ance during scene transitions.
Sample
videos ground-trutbtlkne predicted by RaMViD (right)
(b) Sample 2: 64 64 BAIR vidsos with 16 frames of ground-truth(lef) and predicted by RaMVID (right)
Figure 26: Example of samples predicted by the RaMViD model: predicted result (right), ground- truth (left). The red circle in (a) demonstrates the emergence of blurry artifacts. The red circle in (b) illustrates instances where form of the object undergoes transformation over times and sudden appearances.
(a) Sample
ground-rut(jeed predicted (right)
(b) Sample 2: 64 x 64 BAIR videus with 16 frames of (ruumd-rutt(lefl)) and predicted by MCVD (right)
Figure 27: Example of samples predicted by the MCVD model: predicted result (right), ground-truth (left). Through example, predicted samples exhibit similar spatial and temporal quality compared to real video. Additionally, the occurrence of blurry artifacts or sudden appearances and disappearances of objects in each video scene is less frequent compared to the sample quality of RaMViD
25


Reviews:
[{"review_id": "sb8A8al7G1", "reviewer": "Reviewer_QwpP", "summary": "While evaluation metrics for image generative models are relatively comprehensive, those for video are limited. This paper addresses the drawbacks of existing metrics, proposing a new metric, STREAM, which comprehensively evaluates the temporal and spatial aspects of videos. Experimental validation was conducted on several baseline video generative models.", "questions": "Despite some experimental limitations, the paper makes substantial theoretical contributions. The problem it addresses holds significant value. Currently, I am providing a borderline accept score, hoping the authors will address the experimental aspects mentioned in my review in their future research.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The writing is clear and easy to follow.\n- The paper's motivation is strong. Given the rapid development in the field of video generation, traditional metric FVD falls short in representing performance comprehensively. This paper bridges this gap.\n- The proposed metric allows independent evaluation of spatial and temporal domains, making it applicable for assessing the quality of long videos.", "weaknesses": "- The experimental section of the paper is relatively weak, being limited to a few older baseline models. Authors should focus more on current open-source Text-to-video models, which would provide more convincing results.\n- Although UCF-101 is a common benchmark dataset, as a contribution paper introducing a new evaluation metric, testing on a wider range of datasets and tasks, including T2V, video prediction, and the latest works in unconditional generation, would enhance the paper's credibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uSbFajr97R", "reviewer": "Reviewer_1pjF", "summary": "This paper proposed a novel method to evaluate the quality of generated videos. Specifically, it proposed STREAM-T and STREAM-S to assess temporal and spatial aspects of the videos respectively. STREAM-T is designed to measure temporal quality. It evaluates the continuity and consistence of videos by calculating the FFT of real and fake videos and comparing the difference of the frequences. STREAM-S evaluates the spatial quality by classifying the amplitude at frequency of 0 using KNN. Experiments show that the proposed method is able to measure video quality better than current methods such as FVD.", "questions": "see weaknessess", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. The proposed method proposed a novel method targeting a challenging task, video generation evaluation. \n3. Experiments show the effectiveness of proposed method in evaluating temporal quality in video generation.\n4. STREAM-T is a reasonable metric since it applies statistical method rather than simple l2 or l1 distance to compute loss of video frequency.", "weaknesses": "1. How will the performance be if most of the generated videos are still? Can they be correctly evaluated?\n2. Using P&R to compute fidelity and diversity of fake videos is reasonable, but why to use amplitude at frequency of 0 as sample points? I would like to see more explanations from the authors.\n3. As video quality is very subjective, a systematic evaluation by human raters is required to compare with current method FVD.\n4. The difference in the style of real and fake video datasets may have a huge effect on the result. For example, if real videos tend to have fast changes, the frequency may be concentrated on the high frequency, and the curve fitted may have a higher skewness. In such case, if the curve fitted by fake videos is less steep (fake videos change slowly), skewness may be small, and even if the fake video is of high quality, STREAM-T may be small.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pT6oNlQ8qw", "reviewer": "Reviewer_rxEA", "summary": "The paper presents a new metric to evaluate the generated videos. The authors present three new metrics to assess video quality namely 1) spatial fidelity 2) diversity and 3) temporal coherence. Authors construct various kind of perturbation and evaluate the videos.", "questions": "I would like to see how this metric performs in three scenarios.\n* When the generated video consists of only one frame repeating throughout the video segment\n* One of the main selling points of the FVD metric was it penalizes blur phenomena significantly higher than noise phenomena. This was a useful property because it correlates well with human vision. Additionally, the video generation methods tend to produce blurry samples which would score higher on traditional metrics like SSIM and PSNR. I would want to see how it performs on blurry videos (apply Gaussian blur). \n* What would be the results if the video is reversed temporally and evaluated? please do the same for flipping the videos spatially(take mirror images of the frames) and lastly, run the evaluation for both spatially and temporally flipped video sequences.\nI would like to see an evaluation of all these three scenarios before making my final decision.\n\nHow is the metric affected by the length and resolution of videos, and if it is affected, please provide the standardization of metric because people can game the metric utilizing this loophole.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* Paper is easy to follow\n* The most important contribution of the paper is the newly proposed metric is bounded between 0-1 as opposed to FVD, which is an unbounded metric.", "weaknesses": "* FVD is a single metric used to evaluate a video. With STREAM as a metric, you would have 3 sub-metrics to evaluate generated videos. This would result in $2^3$ scenarios when comparing two baselines, making it tedious to evaluate a new method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v4FeenPOah", "reviewer": "Reviewer_srFK", "summary": "The authors propose a set of metrics to evaluate generative models for video. The propose STREAM-T to assess temporal naturalness as well as STREAM-F for fidelity of videos and STREAM-D for the diversity of videos. They compare to the Frechet Video Distance (FVD) on both synthetic data and using a number of generative models.\n\nSTREAM-T is based on the idea of looking at FFT features of the video to assess temporal \"naturalness\", while STREAM-F and STREAM-D are based on Precision & Recall metrics, but include tweaks to make them work better for video data.", "questions": "* I would like to know how well FVD correlates with the various STREAM measures. Figures 3, 4 and 6 seem to indicate that the spearman correlation between STREAM-T and FVD is actually fairly high, which is completely contrary to the main text, which several times claims that FVD is not good at picking up temporal details.\n\n* What is the reason for not comparing against VIS?\n\n* STREAM-T uses a histogram comparison. How do binning sizes affect the outcome?\n\n* Will the authors provide source code for implementing STREAM? It seems nontrivial to implement.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Originality:** The Authors argue that STREAM can separately assess temporal and spatial aspects of video, and works regardless of video length (unlike FVD). To the best of my knowledge this is true. I like their idea of using FFT to look at spectral features to identify temporal consistency.\n\n**Quality:** The motivation for their method makes sense, and I think their empirical valuation is sensible. The authors only compare to FVD.\n\n**Clarity:** I was able to follow along nicely\n\n**Significance:** I think it is certainly nice to be able to have several dimensions upon which to evaluate generative methods.  However, it is unclear to me how useful that is in practice. Having these metrics can certainly help \"debug\" generative methods, but I would imagine that they will not be as useful as a \"one metric to judge overall quality\" that FVD provides. Especially since the human evaluation shows that FVD actually does fairly well (especially given that in contrast to the three measurements of STREAM it  is only a single number).\n\n\n**UPDATE AFTER READING THE REBUTTAL**\nOverall, my judgment is that the authors have convinced me that in principle this manuscript deserves publication: they attack a meaningful problem and their empirical work is solid. So I think this work scores high enough in Originality,  Quality, and Clarity to be publication. As far as signficance goes: I'm not 100% conviced ICLR is the best venue for this work (I'd imagine you find a more interested audience in conferences that are more focused on computer vision), which is why I still think the manuscript is only marginally above the acceptance threshold for ICLR.", "weaknesses": "* While I think it is useful to have metrics that focus on different aspects of the generation, I would imagine that while developing a method, most of the time it is more helpful to have a single metric to look at to judge progress (if I'm developing a new method and my newest change to the method improves STREAM-F but hurts STREAM-T, is it a good modification or not. Thus, I think the proposed STREAM metrics will be helpful, but will fail to actually replace FVD.  If the authors could find a good way to combine their measurements into a single number (e.g. akin to the F1-score to combine P&R), I think the paper would have more impact. \n* The paper mentions the Video Inception Score (VIS) several times as go-to metric for this task, but does not compare to it. The authors should motivate why they did not use this metric at all, despite it being an obvious competitor.\n* I find it confusing that the authors introduce the term \"STREAM-S\": Its name implies that it's one of the metrics that is being introduced (it follows the same naming convention), but it actually isn't. I'm absolutely unclear what that term actually denots.. I think the paper would improve in clarity if that term would be removed and instead clearly state that there are 3 new metrics that are being introduced (STREAM-T, STREAM-F, STREAM-D).\n\n## Typos and Minor points\nPage two: Spatio-TemproRal Evaluation and Analysis Metric(STREAM)  => TempoRal\nSection 2.1: \"Let real and fake video datasets as\"  => ... be denoted as...\nSeciton 2.2: we utilize SwAV (Caron et al., 2020), a proficient image embedding network => SwAV is an algorithm, not a network\n\n\"In the image generation task, Kynka¨anniemi et al. ¨ (2019) have proposed precision and recall (P&R) to separately evaluate the fidelity and diversity of the generated image quality.\" => I think this was first proposed by Sajjadi et al, (NeurIPS 2018), Assessing Generative Models via Precision and Recall.\n\nSection 2.4: \"If we define a sphere with centered at sample point\"  => I don't understand this sentence", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)