Title: Spatially-Guided Temporal Attention (SGuTA) and Shifted-Cube Attention (SCubA) for Video Frame Interpolation
Venue: NeurIPS 2023
Accepted: no
Keywords: Video frame interpolation

Transformer
Abstract: In recent years, methods based on convolutional kernels have achieved state-of-the-art performance in video frame interpolation task. However, due to the inherent limitations of their convolutional kernel size, it seems that their performances have reached a plateau. On the other hand, Transformers are gradually replacing convolutional neural networks as a new backbone structure in image tasks, thanks to their ability to establish global correlations. However, in video tasks, the computational complexity and memory requirements of Transformer will become more challenging. To address this issue, we employ two different Transformers, SGuTA and SCubA, in VFI task. SGuTA utilizes the spatial information of each video frame to guide the generation of temporal vector at each pixel position. Meanwhile, SCubA introduces local attention into the VFI task, which can be viewed as a counterpart of 3D convolution in local attention Transformers. Additionally, we analyze and compare different embedding strategies and propose a more balanced embedding strategy in terms of parameter count, computational complexity, and memory requirements. Extensive quantitative and qualitative experiments demonstrate that our models exhibit high proficiency in handling large motions and providing precise motion estimation, resulting in new state-of-the-art results in various benchmark tests. The source code can be obtained at https://github.com/esthen-bit/SGuTA-SCubA.

1

Abstract

In recent years, methods based on convolutional kernels have achieved state-of-theart performance in video frame interpolation task. However, due to the inherent limitations of their convolutional kernel size, it seems that their performances have reached a plateau. On the other hand, Transformers are gradually replacing convolutional neural networks as a new backbone structure in image tasks, thanks to their ability to establish global correlations. However, in video tasks, the computational complexity and memory requirements of Transformer will become more challenging. To address this issue, we employ two different Transformers, SGuTA and SCubA, in VFI task. SGuTA utilizes the spatial information of each video frame to guide the generation of temporal vector at each pixel position. Meanwhile, SCubA introduces local attention into the VFI task, which can be viewed as a counterpart of 3D convolution in local attention Transformers. Additionally, we analyze and compare different embedding strategies and propose a more balanced embedding strategy in terms of parameter count, computational complexity, and memory requirements. Extensive quantitative and qualitative experiments demonstrate that our models exhibit high proficiency in handling large motions and providing precise motion estimation, resulting in new state-ofthe-art results in various benchmark tests. The source code can be obtained at https://github.com/esthen-bit/SGuTA-SCubA .

1 Introduction

Video frame interpolation (VFI) is the process of reconstructing uncaptured intermediate frames during the exposure time by synthesizing adjacent frames, which can enhance its visual quality and smoothness of motion. As a fundamental problem in computer vision, it requires an understanding of both spatially and temporally consistence within the video frames, breaking the limitations of video sampling rate and lighting conditions. Its applications span across diverse domains, including virtual reality [1], video compression [2, 3, 4], and slow-motion generation [5, 6].

The majority of state-of-the-art techniques for VFI rely on convolutional neural networks (CNNs), 27 particularly those based on kernels [7, 8, 9, 10, 11, 12, 13, 14], which have gained increasing 28 popularity in recent years. Nevertheless, due to the inherent constraint imposed by the kernel 29 size, convolutional kernels seem to have reached their performance ceiling, even after undergoing 30 transformations such as 2D kernels, separable kernels, deformable kernels, and 3D kernels. It appears 31

that there is a limited potential for further improvement of VFI methods based on CNNs and their 32

associated kernels. Meanwhile, Transformers [15] have recently demonstrated its great potential 33

in various image tasks such as image classification [16, 17, 18], object detection [19, 20], spectral 34

reconstruction [21], and image restoration [22, 23, 24], due to their ability to capture long-range 35

A Embedded Token

A Partiton Cube

f1faee

Uncaptured

frames

(a)

(b)

H

W

T

S

T

I

0

I

1

SGuTA

dependencies and contextual relationships in sequences. However, extending the Transformer to 36

video tasks is not as straightforward as extending 2D convolutions to 3D convolutions, as it poses 37

challenges such as computational complexity and memory requirements. 38

This article introduces two distinct Transformer-based approaches, SCubA (Shifted-Cube Attention) 39

and SGuTA (Spatially-Guided Temporal Attention), which are integrated into a multi-stage multi-scale 40

framework for VFI task. Both methodologies exhibit linear computational complexity with respect to 41

the patch number, making them concise, efficient, and demonstrating exceptional performance. 42

It is revealed by [25] that there exists an inherent correlation between the spatial information and 43

temporal sequence of a video. Fig. 1(a) illustrates a simple example for this phenomenon. If we 44

exchange any spatial dimension (height or width) with the temporal dimension, a new video sequence 45

can be obtained in which the low-resolution version of the original spatial information is recurred. 46

Therefore, the higher-resolution original spatial information can provide powerful guidance for 47

improving the temporal resolution. Inspired by this, we propose SGuTA, a self-attention mechanism 48

that establishes intrinsic correlations between spatial information and temporal sequence. 49

Inspired by [16, 18, 26], as shown in Fig. 1(b), SCubA treats 3D-patches as tokens and partitions 50 them into cubes with a fixed size along the height, width, and time axis. Local self-attention is 51 computed within each cube, followed by shifted-cube mechanism to establish connections between 52 adjacent cubes. This approach enables the model to exploit spatiotemporal locality inductive bias and 53 achieve better performance than existing methods. 54

55

56

57

58

59

60

61

62

63

64

65

66

The main contributions of this work are listed as follow:

We present a novel Transformer called SGuTA, which is designed to establish the inherent correlations between the spatial characteristics and the temporal sequence within a video. SGuTA outperforms VFIT-B [14] in terms of PSNR by 0.58dB on vimeo-90k test set.

We propose a method called SCubA, which applies Video Swin Transformer [26] to VFI task. Compared to VFIT-B, SCubA achieves a PSNR improvement of 1.08dB while reducing both the number of parameters (Params) and computational complexity (FLOPs) by approximately 40%.

We conduct a analysis of existing embedding strategies, and put forth a novel half-overlapping embedding strategy. This method exhibits a more balanced performance in relation to Params, computational complexity, and memory usage.

2 Related Works

2.1 Video Frame Interpolation

The objective of VFI is to generate intermediate frames by combining adjacent frames that were not 67 captured during the exposure period. This longstanding and classical problem in video processing 68

is currently tackled through three prominent approaches: phase-based methods, optical flow-based 69

70

methods, and kernel-based methods.

Phase-based methods [27, 28] utilize Fourier theory to estimate motion by analyzing the phase 71 discrepancy between corresponding pixels in consecutive frames. These techniques generate interme72 diate frames by applying phase-shifted sinusoids. However, the 2 π -ambiguity problem can pose a 73 significant challenge in determining the correct motion. 74

75

76

77

78

79

80

Flow-based methods [5, 10, 29, 30, 31, 32, 33] utilize optical flow estimation to perceive motion information and capture dense pixel correspondence between frames. These methods use a flow prediction network to compute bidirectional optical flow that guides frame synthesis, along with predicting occlusion masks or depth maps to reason about occlusions. However, these methods are limited by the accuracy of the underlying flow estimator remaining challenging problems in real-world videos, especially when there is large motion and heavy occlusion.

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

Kernel-based methods have gained momentum in VFI since the emergence of AdaConv [7], a method that uses a fully convolutional network to estimate spatially adaptive convolution kernels. This is because it no longer requires motion estimation or pixel synthesis like flow-based methods. DSepConv [9] and AdaCoF [10] employ Deformable convolution to overcome the limitation of a fixed grid of locations in original convolution. CAIN [11] expands the receptive field size of convolution by utilizing Pixel Shuffle. SepConv [8] performs separable convolution, thereby reducing the Params and memory usage. Then, FLAVR [13] substitutes the 2D convolutions utilized in Unet with their 3D counterparts, while applying feature gating to each of the resultant 3D feature maps. This achieves the best performance among CNN-based methods at the cost of a large Params. However, these CNN-based architectures still cannot overcome their inherent limitation of using fixed-size kernels, which prevent them from capturing global dependencies to handle large motion and limit their further development for VFI task. Inspired by Depth-wise separable convolution [34], Zhihao Shi et al. introduce VFIT [14], a separated spatio-temporal multi-head self-attention mechanism, which outperforms all existing CNN-based approaches while significantly reducing the Params. Within the field of kernel-based methods, CNN backbones have undergone a developmental trajectory from 2D to separable and then to 3D kernels. Zhihao Shi et al. has proposed a space-time separation strategy [14] in Transformer methods. In this work, we introduce a 3D version of the local self-attention mechanism and a spatially-guided temporal self-attention mechanism to the VFI task.

2.2 Vision Transformer

The key innovation of the ViT [16] is its application of the Transformer architecture, originally developed for natural language processing, to computer vision tasks. This represents a notable departure from the standard backbone architecture of CNNs in computer vision. By dividing the image into a sequence of patches and leveraging the Transformer encoder to capture global dependencies between them, ViT achieves impressive performance on image classification benchmarks. This pioneering work has paved the way for subsequent research aimed at improving the utility of the ViT model, and underscores the potential of the Transformer architecture in computer vision applications. To mitigate the computational and memory challenges associated with ViT, Swin-Transformer [18] partitions the embedded patches into non-overlapping windows. Within each window, local self-attention is calculated by ViT. Subsequently, shifted-window self-attention is computed to establish the correlation among windows. This strategy has demonstrated remarkable performance in various image tasks, such as image classification [18, 35], object detection [20, 19], and image restoration [36, 24], achieving state-of-the-art results. Despite Swin-Transformer's success in image tasks, extending it to video tasks by simply expanding along the time dimension resurfaces thorny computational and memory issues [14]. To address these issues, Ze Liu et al. further proposed a new 3D shifted windows mechanism [26] that efficiently captures temporal information, reduces the computational and memory demands, and achieves state-of-the-art results on video action recognition tasks. This method makes Swin-Transformer a promising approach for video analysis tasks.

3 Proposed method

SCubA and SGuTA share this same network architecture. Fig. 2(a) depicts a multi-stage architecture 119 that utilizes N s cascaded Multi-Scale Transformer. Instead of selecting two or four adjacent frames 120 next to the reference frame I 0 . 5 as input, as in previous methods [11, 13, 14], our approach chooses 121 six adjacent frames to accurately estimate the motion of the interpolated frame. Moreover, a long 122 identity mapping is employed to mitigate the vanishing gradient problem. The desired interpolated 123

+

+

3D Conv 1x1x1

3D DownSample

Embedding

De-Embedding

MAB

MAB

MAB

MAB

MAB

3D UpSample

(b)Multi-Scale Transformer

+

Concatenate

×

MatMul

×

(f) SC-MAB

Embedded Tokens

×

Q

K

V

Linear Projection

Scale

×

Linear Projection

+

+

PE

(c) MSA

×

3D Out-Conv

(a)Multi-Stage Architecture

+

Trans-U Net

xN

s

3D Conv

GELU

3D Conv

GELU

3D Conv

(e) FFN

(d) G-MAB

G-MSA

+

LayerNorm

FFN

+

LayerNorm

SC-MSA

+

LayerNorm

FFN

+

LayerNorm

Element-Wise add

+

+

frame ˆ I 0 . 5 is finally obtained via a 3D convolution operation. Fig. 2(b) illustrates the network 124 structure of the Multi-Scale Transformer when N s = 1 . Specifically, in the embedding layer, patches 125 of frames are transformed into dense representations. The de-embedding layer performs the inverse 126 operation of the embedding layer, whereby the representations are restored to patches. To enable 127 multi-scale self-attention, it is essential to downsample the output of the Multi-Head Attention Block 128 (MAB) from the previous scale before each MAB layer in the encoder. Similarly, in the decoder, 129 the upsampling of the output of each MAB layer is first performed to restore the original spatial 130 resolution, before sending into the next scale. Moreover, skip connections are employed at same 131 scale, while a 1 × 1 × 1 convolutional operation is applied to halve the depth of the concatenated 132 feature maps. 133

134

135

136

137

138

139

140

141

142

SGuTA and SCubA are two transformer-based models that differ in their self-attention mechanisms. SGuTA is derived from the global self-attention mechanism, and its MAB module comprises solely the G-MAB module illustrated in Fig. 2(d). In contrast, SCubA is based on the local self-attention mechanism and its MAB module is constituted by the G-MAB module shown in Fig. 2(d), as well as the SC-MAB module depicted in Fig. 2(f), with the dotted line being solely applicable when utilizing SCubA. The composition of the feed-forward network (FFN) is presented in Fig. 2(e), whereas a concise procedure of multi-head self-attention (MSA) is portrayed in Fig. 2(c) (with some details omitted for concision). The disparity between SCubA and SGuTA is situated in the MSA module, which will be expounded upon in Section 3.1

143

144

145

146

147

148

3.1 Proposed MSA

3.1.1 SGuTA

Assuming an input tensor of shape X in ∈ R T × H × W × D , where D denotes the length of embedding vector , the MSA module of SGuTA first transposes and reshapes it into a 2D tensor X ∈ R HW × TD . This reshaped tensor X is then projected through linear transformations W Q , W K , and W V ∈ R TD × TD to obtain the query Q , key K , and value V ∈ R HW × TD , respectively:

Such a transformation enables MSA to leverage the interactions among different spatial features within 149 the input tensor, facilitating the capturing of complex dependencies in the subsequent processing. 150 Then, Q,K,V ∈ R HW × TD are divided into n heads: Q = [ Q 1 , ..., Q n ] , K = [ K 1 , ..., K n ] , 151

V = [ V 1 , ..., V n ] , so that each head has a dimension of d h = TD n . The remaining process of SGuTA 152 can be expressed as follows: 153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

3.2 Other MSAs

In general, Global MSA [16] and Feature MSA [21] follow a standard procedure: the input X in ∈ R T × H × W × D is reshaped and linearly transformed using W ′ Q , W ′ K , and W ′ V ∈ R D × D to obtain Q ′ , K ′ , and V ′ ∈ R THW × D , which are then divided into n heads. Specifically, Q ′ = [ Q ′ 1 , ..., Q ′ n ] , K ′ = [ K ′ 1 , ..., K ′ n ] , and V ′ = [ V ′ 1 , ..., V ′ n ] , with each head having a dimension of d h = D n .

For Global MSA and Feature MSA, The multi-head self-attention is obtained by:

The computational complexity for Global MSA and Feature MSA is respectively given by:

We validate the performance of the MSAs listed above, in addition to STS and Sep-STS [14], in the 183 VFI task. Specific results and analysis can be found in Section 4.3.2. 184

where d ∈ R 1 and W ∈ R TD × TD are learnable parameters. P ( V ) = 3 DConv ( Gelu (3 DCon ( v ))) to generate positional embedding. The output X out ∈ R T × H × W × D are obtained by reshaping the result of Eq. (2). Observing that SGuTA establishes correlations from space to time. Compared to the Global MSA method of establishing spatio-temporal correlations between all patches, SGuTA is capable of effectively alleviating memory requirements and computational complexity issues. The computational complexity of SGuTA can be easily obtained as follows:

3.1.2 SCubA

In accordance with [26], the input tensor X in ∈ R T × H × W × D is subjected to a process of partitioning into THW thw non-overlapping cubes of size t × h × w utilizing an even partitioning strategy, as presented in Fig. 1b. The resulting cubes are reshaped into x ∈ R thw × D by the MSA module of SCubA. Linear transformations, specifically w q , w k , and w v ∈ R D × D , are employed to produce the query q , key k , and value v ∈ R thw × D representations, respectively.

Similarly, q, k, v ∈ R thw × D are divided into n heads: q = [ q 1 , ..., q n ] , k = [ k 1 , ..., k n ] , v = [ v 1 , ..., v n ] , so that each head has a dimension of d h = D n . The multi-head self-attention operation is then conducted within each cube according to the following equation:

To establish connections among the cubes, each cube is shifted along the time, height, and width dimensions by t/ 2 , h/ 2 , and w/ 2 steps, respectively, as depicted in Fig. 1b. The SC-MSA (corresponding to Fig. 2f) is calculated within each new cube.

The process in Eq. (4) and Eq. (5) is calculated for THW thw times, and its computational complexity can be specifically expressed as:

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

3.3 Half Overlapping Embedding Strategy

We observe that the embedding strategies of Transformers can be mainly classified into two categories: the Non-overlapping [16] and Wide-overlapping [14] embedding strategy, which have no overlap and significant overlap between adjacent patches respectively. Specifically, if the patch size is set to t × h × w , the Non-overlapping and Wide-overlapping embedding strategies extract patches with strides of t × h × w , and 1 × 1 × 1 respectively. Clearly, on the one hand, different stride will significantly impact the number of tokens and further affect the memory requirements during training. On the other hand, the length of the representation will change Params. Both factors influence the final performance of the model. We found that different embedding strategies maintain comparable performance when the following equation is satisfied:

Here, D 1 and D 2 respectively represent the length of the embedding representation when patches are extracted with strides of t 1 × h 1 × w 1 and t 2 × h 2 × w 2 .

Hence, we propose a compromise solution - the Half-overlapping embedding strategy - where adjacent patches overlap by half of their area or the stride is set to t × h/ 2 × w/ 2 . The length of its embedding representation is set by Eq. (11). A detailed performance comparison and analysis of the three different embedding strategies can be found in Section 4.3.1.

4 Experiment

4.1 Implementation Details

Training: Consistent with [13], a basic l 1 loss is employed to train the networks: || I 0 . 5 -ˆ I 0 . 5 || . The training batch size is set to 4, and the cube size of SCubA is set to 2 × 4 × 4 . The Adam optimizer [37] is utilized with β 1 = 0 . 9 and β 2 = 0 . 99 . The learning rate is initialized to 2 e -4 , and a Cosine Annealing scheme is adopted over 100 epochs. Both SGuTA and SCubA employ Half-overlapping strategy with patch size setting to 1 × 4 × 4 pixels.

Dataset: In this study, we use the Vimeo-90K septuplet training set [38] for training, it includes 64,612 seven-frame sequences with a resolution of 448 × 256 . We selected the middle frame from each sequence as the ground truth, and pad one blank frame to the beginning and end of the remaining six frames. After random cropping, we obtain a video sequence of size 8 × 3 × 128 × 128 as input. We use the data augmentation method of FLAVR [13], which randomly applied horizontal and vertical flips and temporal flips to the input video sequence.

The performance of our models is accessed on widely-used datasets, including the Vimeo-90K septuplet test set [38], which comprises 7824 septuplets with a resolution of 448 × 256 ; the DAVIS dataset [39], containing 2849 triplets with a resolution of 832 × 448 ; and the SNU-FILM dataset [11], which is classified four categories based on the degree of motion: Easy, Medium, Hard, and Extreme. Each category comprises 310 triplets, primarily with a resolution of 1280 × 720 . we transform the DAVIS dataset and SNU-FILM dataset into septuplets while preserving the ground truth to accommodate our network requirements and ensure fairness in comparing various models.

4.2 Evaluation against the State of the Arts

Weconducted a comparative analysis of SGuTA and SCubA with competitive state-of-the-art methods, including SuperSlomo [31], SepConv [8], QVI [30], BMBC [32], CAIN [11], AdaCoF [10], FLAVR [13], VFIT-S [14], VFIT-B [14]. Tab. 1 reports the performance of each model in terms of Params, FLOPs, peak signal-to-noise ratio (PSNR), and structural similarity index (SSIM) on the Vimeo-90K and Davis datasets. Compared with the current SOTA method VFIT on the Vimeo-90K dataset, SGuTA achieves a significant performance improvement of 0.58dB with similar Params and FLOPs. Moreover, SCubA reduces the Params and FLOPs by 40% and 39%, respectively while achieving a notable performance improvement of 1.08dB . Fig. 3 illustrates the PSNR-FLOPs-Params comparison of these methods, which demonstrated that both SCubA and SGuTA are located in the upper-left region of the figure.

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

38-

SCubA

SGuTA

37-

VFIT-B

(dB

36

QVI

FLAVR

SNR

35-

AdaCoF

BMBC

S

CAIN

34

SepConv

33-

SuperSlomo

32-

10

15

20

25

30

35

40

45

Params(M)

Tab. 2 reports the performance of each model on the SNU-FILM dataset. Compared with the thirdbest model, SGuTA and SCubA achieve an average improvement of 0.67dB and 0.96dB, respectively, and a remarkable improvement of 1.14dB and 1.59dB in Hard scenario. This indicates that SGuTA and SCubA fully utilize the Transformer's ability to establish long-range correlations and prove their capability to handle challenging large-motion scenarios.

We provide qualitative results comparing our SGuTA and SCubA models to FLAVR [13] and VFIT [14]. As shown in Fig. 4. The first two rows fully demonstrate the ability of SGuTA and SCubA to provide accurate motion estimation (please carefully compare the rotation of the wheels and balls with the ground truth; other methods fail to restore the accurate rotation angles). The third row shows the performance of various models in non-rigid motion scenarios, where only SCubA clearly restores all the letters. In the fourth row, SGuTA and SCubA reconstruct clearer texture details. The last two rows again demonstrate the strong ability of our models to handle large motion scenarios.

4.3 Ablation Study

4.3.1 Embedding Strategy

In this section, we explore the relationship between various embedding strategies and the length of embedding vectors D . Taking SCubA as an example, given the input video size of T × H × W = 8 × 128 × 128 , we set N s = 2 and the patch size to 1 × 4 × 4 . The Wide, Half and Non-Overlapping Strategies extract patches with stride of 1 × 1 × 1 , 1 × 2 × 2 , 1 × 4 × 4 , respectively. As shown in Tab. 3, changing the Wide-Overlapping Strategy to the Non-Overlapping Strategy while keeping the size of D = 32 the same can reduce FLOPs and memory usage, but lower the performance.

Overlayed

FLAVR

VFIT-S

VFIT-B

SGuTA

SCubA

GT

ARI

F

Strategies that satisfy Eq. (11) perform similarly, thus we can use this equation to balance Params, 252 FLOPs, and memory usage. The rationale behind this phenomenon is that the correlation between 253 adjacent patches exhibits redundancy under the Wide-Overlapping Strategy, whereas it manifests 254 sparsity in the Non-Overlapping Strategy. Consequently, the latter requires a lengthier representation 255 to restore the comparable performance. The correlation of Half-Overlapping Strategy lies between 256 the previous two strategies, and the appropriate overlapping region can provide some inductive bias, 257 such as the relative positional information, to MSA. It is worth noting that due to the non-linear nature 258 of Eq. (11), the Half-Overlapping Strategy exhibits a distinct feature of high returns on investment, 259 with lower Params, FLOPs, and memory requirements compared to the average values of the Wide 260 Overlapping Strategy and Non-Overlapping Strategy at the same performance level.

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

4.3.2 Self-Attention Mechanism

In this section, We first replace all MSA blocks with two layers of 3D ResBlocks [40] to enable a comparative assessment of CNN-based methods with other Transformer-based methods. Subsequently, a thorough evaluation of the performance of different MSAs is conducted. The scrutinized MSA-based methods are enumerated as follows: 1) Baseline where the MSA modules are all removed from the multi-scale Transformer. 2) STS MSA [14] and 3) Sep-STS MSA [14] replaces our MSA modules with STS blocks and Sep-STS blocks [14] respectively. 4) Feature MSA [21] obtain self-attention from Eq. (8). 5) SGuTA and 6) SCubA are the methods proposed in this paper. Besides, Global MSA [16] employs Eq. (7) for self-attention, but its performance is unreported due to the excessively high computational complexity (587.93G) and memory requirements. To ensure fairness, all methods are configured with N s = 2 and adopt the half overlapping embedding strategy with D = 64 . Because the differences between models can be distinguished at the early stages of training, we report performance for all models trained for 20 epochs.

As shown in Tab. 4, on the one hand, compared to the 3D ResBlock method based on CNN, Transformers benefit from their ability to establish long-range dependencies, achieving improved performance with lower Params and FLOPs. On the other hand, compared to the Baseline, Feature MSA only provides a modest improvement in PSNR by 0.06dB, indicating that the self-attention for features has limited benefit for VFI task. STS MSA and Sep-STS MSA show PSNR improvements of 0.37dB and 0.60dB, respectively, with Sep-STS MSA acting similarly to depth-wise separable convolution [34], resulting in a lighter and more efficient STS-MSA. Compared to Feature MSA, SGuTA significantly improves PSNR by 0.54dB, demonstrating the effectiveness of SGuTA in establishing correlations between space and time. SCubA leverages the shifted-cube mechanism to fully exploit the power of local attention, achieving the best performance among all MSAs.

4.3.3 Stage

In this section, we explore the impact of the number of cascaded Multi-scale Transformers N s . Due to concerns regarding Params and FLOPs, we only consider the case when N s ≤ 3 . The results are presented in Tab. 5, where it can be observed that when N s = 3 , both SGuTA and SCubA perform the best. Additionally, it is worth noting that when N s = 1 , compared to VFIT-S, SCubA achieves a PSNR improvement of 0.24dB while reducing the Params and FLOPs by 23% and 58%, respectively. When N s = 2 , compared to VFIT-B, SCubA achieves a PSNR improvement of 0.52dB with 43% of its Params and 40% of the FLOPs.

5 Conclusions

In this paper, we employ two different Transformers, SGuTA and SCubA, to the VFI task. SGuTA is designed to establish intrinsic connections between video spatial and temporal information, while SCubA employs a 3D local self-attention mechanism. Both methods are integrated into a multi-stage multi-scale framework. Compared to previous state-of-the-arts, extensive experiments show that our methods achieve the best and second-best performance on multiple benchmarks, and particularly excel in handling large motion and providing accurate motion estimation. Additionally, we summarized the regularity between the patch extraction stride and the length representation when different embedding strategies maintain comparable performance. We will further verify the universality of this regularity, as well as extend our model to multi-frame interpolation in future work.

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

References

Robert Anderson, David Gallup, Jonathan T Barron, Janne Kontkanen, Noah Snavely, Carlos Hernández, Sameer Agarwal, and Steven M Seitz. Jump: virtual reality video. ACM Transactions on Graphics (TOG) , 35(6):1-13, 2016.

Chao-Yuan Wu, Nayan Singhal, and Philipp Krahenbuhl. Video compression through image interpolation. In Proceedings of the European conference on computer vision (ECCV) , pages 416-431, 2018.

Kai-Chieh Yang, Ai-Mei Huang, Truong Q Nguyen, Clark C Guest, and Pankaj K Das. A new objective quality metric for frame interpolation used in video compression. IEEE transactions on broadcasting , 54(3):680-11, 2008.

Jean Bégaint, Franck Galpin, Philippe Guillotel, and Christine Guillemot. Deep frame interpolation for video compression. In DCC 2019-Data Compression Conference , pages 1-10. IEEE, 2019.

Wenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang. Depth-aware video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3703-3712, 2019.

Youjian Zhang, Chaoyue Wang, and Dacheng Tao. Video frame interpolation without temporal priors. Advances in Neural Information Processing Systems , 33:13308-13318, 2020.

Simon Niklaus, Long Mai, and Feng Liu. Video frame interpolation via adaptive convolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 670-679, 2017.

Simon Niklaus, Long Mai, and Feng Liu. Video frame interpolation via adaptive separable convolution. In Proceedings of the IEEE international conference on computer vision , pages 261-270, 2017.

Xianhang Cheng and Zhenzhong Chen. Video frame interpolation via deformable separable convolution. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 10607-10614, 2020.

Hyeongmin Lee, Taeoh Kim, Tae-young Chung, Daehyun Pak, Yuseok Ban, and Sangyoun Lee. Adacof: Adaptive collaboration of flows for video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5316-5325, 2020.

Myungsub Choi, Heewon Kim, Bohyung Han, Ning Xu, and Kyoung Mu Lee. Channel attention is all you need for video frame interpolation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 10663-10671, 2020.

Whan Choi, Yeong Jun Koh, and Chang-Su Kim. Multi-scale warping for video frame interpolation. IEEE Access , 9:150470-150479, 2021.

Tarun Kalluri, Deepak Pathak, Manmohan Chandraker, and Du Tran. Flavr: Flow-agnostic video representations for fast frame interpolation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 2071-2082, 2023.

Zhihao Shi, Xiangyu Xu, Xiaohong Liu, Jun Chen, and Ming-Hsuan Yang. Video frame interpolation transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17482-17491, 2022.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. In Proceedings of the European conference on computer vision (ECCV) , pages 305-321, 2018.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

Ishan Misra, Rohit Girdhar, and Armand Joulin. An end-to-end transformer model for 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2906-2917, 2021.

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16 , pages 213-229. Springer, 2020.

Yuanhao Cai, Jing Lin, Zudi Lin, Haoqian Wang, Yulun Zhang, Hanspeter Pfister, Radu Timofte, and Luc Van Gool. Mst++: Multi-stage spectral-wise transformer for efficient spectral reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 745-755, 2022.

Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. Uformer: A general u-shaped transformer for image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17683-17693, 2022.

Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1833-1844, 2021.

Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5728-5739, 2022.

Liad Pollak Zuckerman, Eyal Naor, George Pisha, Shai Bagon, and Michal Irani. Across scales and across dimensions: Temporal super-resolution using deep internal learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VII 16 , pages 52-68. Springer, 2020.

Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3202-3211, 2022.

Simone Meyer, Oliver Wang, Henning Zimmer, Max Grosse, and Alexander Sorkine-Hornung. Phase-based frame interpolation for video. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1410-1418, 2015.

Simone Meyer, Abdelaziz Djelouah, Brian McWilliams, Alexander Sorkine-Hornung, Markus Gross, and Christopher Schroers. Phasenet for video frame interpolation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 498-507, 2018.

Ziwei Liu, Raymond A Yeh, Xiaoou Tang, Yiming Liu, and Aseem Agarwala. Video frame synthesis using deep voxel flow. In Proceedings of the IEEE international conference on computer vision , pages 4463-4471, 2017.

Xiangyu Xu, Li Siyao, Wenxiu Sun, Qian Yin, and Ming-Hsuan Yang. Quadratic video interpolation. Advances in Neural Information Processing Systems , 32, 2019.

Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang, Erik Learned-Miller, and Jan Kautz. Super slomo: High quality estimation of multiple intermediate frames for video interpolation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 9000-9008, 2018.

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

Junheum Park, Keunsoo Ko, Chul Lee, and Chang-Su Kim. Bmbc: Bilateral motion estimation with bilateral cost volume for video interpolation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16 , pages 109-125. Springer, 2020.

Liying Lu, Ruizheng Wu, Huaijia Lin, Jiangbo Lu, and Jiaya Jia. Video frame interpolation with transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3532-3542, 2022.

Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 , 2017.

Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML , volume 2, page 4, 2021.

Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12299-12310, 2021.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman. Video enhancement with task-oriented flow. International Journal of Computer Vision , 127:1106-1125, 2019.

Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung. A benchmark dataset and evaluation methodology for video object segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 724-732, 2016.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces two attention modules for video frame interpolation that achieve promising results in terms of quantitative and qualitative evaluation. The researchers conducted numerous experiments to provide a comprehensive evaluation of the proposed approach.\n\nSTRENGTHS:\nThe proposed attention mechanism enhances the interpolation quality of intermediate frames.\n\nWEAKNESSES:\n(1) While this work does not present many theoretical contributions or fundamental insights, it is technically sound and offers improvements and modifications to existing methods. For example, the shift windows attention applied to spatial-temporal aggregation is a relatively simple and incremental approach that builds upon similar ideas explored extensively in previous research.\n\n(2) Table 1/2 should specify the training setups of previous state-of-the-art video frame interpolation methods, as the proposed method uses six input frames. As far as I know, existing models are generally trained on Vimeo-Triplets, which only consist of 51K three-frame samples.\n\n(3) Additionally, the specific contributions of SGuTA are unclear, as SCubA significantly outperforms it with fewer parameters and lower multi-adds. Finally, while many previous works have demonstrated arbitrary interpolation ability, this point does not appear to have been discussed in this work.\n\nQUESTIONS:\n(1) The fairness of comparison in Table 1/2 should be carefully examined to avoid any potential biases or confounding factors that could affect the evaluation and conclusion.\n\n(2) The necessity of SGuTA should be clearly demonstrated with evidence, such as experiments or analyses.\n\n(3) The significance of the contributions of this work should be further discussed, highlighting the novel aspects and potential impact of the proposed approach in the field. This could include a comparison with other state-of-the-art methods and a discussion of potential future research directions.\n\nLIMITATIONS:\nPlease see the weakness and question parts.\nNo significant negative societal impact in this work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn video tasks, the computational complexity and memory requirements of Transformer is challenging. This paper employ two different Transformers in VFI task resulting in new state-of-the-art results. Furthermore, this paper conduct an analysis of existing embedding strategies, and put forth a novel half-overlapping embedding strategy. The author carried out the experimental analysis carefully.\n\nSTRENGTHS:\n1. I believe the author contributed efficient components to the multi-frame video transformer.\n2. The author has open sourced the relevant code.\n3. The resulting visuals look good. Overall, the paper has a good impression.\n\nWEAKNESSES:\n1. The author did not discuss and compare a series of works on single frame interpolation. CVPR20-SoftSplat, CVPR22-IFRNet, CVPR22-Many-to-many Splatting, ECCV22-RIFE, CVPR23-AMT. This greatly weakens the credibility of model evaluation. Overall, the authors cite very little recent relevant literature.\nThe model mainly compares the \"video frame interpolation transformer\", which is not as popular as SoftSplat or RIFE as far as I know, so I hope the author will add more comparison experiments to make the evaluation more solid.\n2. The video submitted by the author has only one 720p scene, and there is no other method to compare it.\n3. The author has discussed that this model currently does not support multi-frame interpolation.\n\nQUESTIONS:\n1. Please refer to AMT to supplement the results of SNU-FILM. \nAMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolation\n2. How did the authors achieve a fair comparison on SNU-FILM? Most of the methods on Table.2 cannot benefit from multi-frame input.\n3. How about the actual lentency of the model test, is it still a big improvement? Please report the test results for 720p and 1080p separately.\n4. How well does the model perform on 4k related datasets, such as Xiph-4k or XVFI-4k?\n\nLine 39 article->paper\nLine 218 we->We\n\nLIMITATIONS:\nPrevious video transformers often performed mediocrely at 2k or 4k resolutions, or had a large memory overhead, and it is unclear how well this work in HD scenes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to address the computational complexity and memory requirements of Transformers to enhance their suitability for the video frame interpolation (VFI) task. The authors propose two novel methods, SCuTA and SGubA, which are integrated into a multi-stage multi-scale framework. SCuTA leverages the correlation between spatial information and temporal sequences, while SGubA incorporates a 3D local self-attention mechanism. Through extensive experiments, the proposed methods demonstrate superior performance in terms of peak signal-to-noise ratio (PSNR) compared to other approaches, while also exhibiting a reduced parameter count.\n\nSTRENGTHS:\n1. Intuitive motivation: The paper provides a clear and intuitive motivation for modifying Transformers to better adapt to the VFI task, addressing the computational complexity and memory requirements.\n2. Reasonable design: The introduction of SCuTA and SGubA, which leverage the correlation between spatial information and temporal sequences, is a sensible approach. Additionally, SGubA's utilization of a 3D local self-attention mechanism aligns well with the requirements of the VFI task.\n3. Comprehensive experimentation: The paper presents a substantial number of experiments to support the proposed methods. The results consistently demonstrate that the proposed approaches achieve the highest PSNR while also reducing the parameter count, effectively addressing the computational and memory challenges of Transformers in the VFI task.\n\nWEAKNESSES:\n1. Lack of clarity in Figure 2: Figures 2d and 2f require further clarification. It is recommended that the authors explain G-MSA and SC-MSA before referring to these figures to enhance reader understanding.\n2. Inference time comparison: The paper should provide information on the inference time for all the methods evaluated. This will provide a more comprehensive evaluation of the proposed approaches in terms of both performance and computational efficiency.\n\nQUESTIONS:\nPlease refer to weaknesses.\n\nLIMITATIONS:\nThe authors do not address the limitations. Please refer to weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles video frame interpolation (VFI). It particularly aims to deploy a transformer architecture for VFI tasks. To address the computational complexity and memory requirements in transformers, it proposes two transformer networks, SGuTa and SCubA. While SGuTa uses the spatial (global) information of video frames to establish temporal correlation, SCubA focuses on local attention. Both methods exhibit linear computational complexities. The authors also introduce a half-overlapping embedding strategy to balance the trade-off between computational complexity and memory usage. Experimental comparisons are presented on several VFI benchmarks.\n\nSTRENGTHS:\n* The work attempts to tackle an important trade-off (performance vs. computation) in using transformers for the VFI task\n* The proposed transformers exhibit linear time complexity which makes them easily deployable\n* The qualitative results look good\n* The ablation studies are thorough and show the effect of the different design choices in the proposed networks\n* Code is shared in the supplementary material for reproducibility\n\nWEAKNESSES:\n* The motivation of the work is not convincingly justified\n\n   There are several works [2,3] that used a transformer architecture for VFI. However, the authors introduce two new transformer networks without drawing any motivation from the progress that has been made in this line of research. It is also not very clear why the authors need to introduce two different networks in one paper. The introduction part of the paper should be rewritten by positioning the proposed framework in comparison with existing literature and hence filling in the big jump in L38-39.\n\n* The technical novelty of the work is limited\n\n   The proposed modules are heavily copied from other relevant works such as VideoSwin Transformer [1].\n   \n   It is also not clear why the proposed multi-head self-attention (MSA) is very different from other MSAs in previous literature. As far as I understand, the key difference is in reshaping the input tensor (HW X TD instead of HWT X D) in SGuTa. The remaining operations are the same as other MSAs. The local attention in ScubA is also similar to the one used in [1]. \n\n* The experimental comparisons are limited and unfair\n\n   Experimental results are simply copied (quoted) from previous works. However, the experimental settings the authors used to conduct experiments are very different (6 adjacent frames, different patch sizes) from previous works. Hence, how can the authors convincingly justify that the performance gain is coming from the proposed method and not the different experiment settings?\n\n   Intuitively speaking, using more adjacent frames (6 versus 2 or 4) should provide more context during training, hence, it benefits the proposed method. A fair comparison would follow the common experiment protocol used in previous VFI works. \n\n   The benefit of half-overlapping in Table 3 is not clear. The performance gain is really negligible (0.04dB) compared to the computation overhead (significant increase in memory usage and FLOPs compared to non-overlapping baselines).  how did the authors come to the conclusion in Eq. 11? There has to be a more convincing explanation (proof) than simple observation, otherwise, the claim in L62-64 should be toned down. \n\n   The video presented in the supplementary file only shows one video result for the proposed method. The authors should provide a side-by-side video comparison of their method and competing approaches on different test videos.\n\n* The writing of the paper needs improvement \n  \n   The methodology part is very confusing with too many coined terms. It would be better to rewrite this part in a clearer manner.\n   \n   It is also not a recommended practice to introduce new acronyms in the title of the paper. \n\nReferences\n\n[1]. Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer, CVPR 2022\n\n[2]. Zhihao Shi, Xiangyu Xu, Xiaohong Liu, Jun Chen, and Ming-Hsuan Yang. Video frame interpolation transformer, CVPR 2022\n\n[3]. Liying Lu, Ruizheng Wu, Huaijia Lin, Jiangbo Lu, and Jiaya Jia. Video frame interpolation with transformer, CVPR 2022\n\nQUESTIONS:\nPlease refer to the \"Weaknesses\" section and address the raised concerns carefully.\n\nLIMITATIONS:\nThe authors do not discuss the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed two types of Transformer for video frame interpolation: Spatially-Guided Temporal Attention (SGuTA) and\nShifted-Cube Attention (SCubA). SGuTa merges the temporal dimension and the embedding dimension during the self-attention process to explore the inherent correlations between the spatial and temporal dimensions. On the other hand, SCubA is focused on reducing the computational complexity and adapts Video Swin Transformer for frame interpolation. Both approaches show significant performance improvements compared with the recent VFI models.\n\nSTRENGTHS:\n1. Good performance. Both of the proposed Transformer models---SGuTA and SCubA---outperform the existing works by a notable margin. As the authors have mentioned, given that the performance of the existing works for VFI is almost plateaued, it seems that the proposed architectures are quite effective.\n\n2. Writing is clear and easy to understand. Also, the figures are neat, and the experiments logically match the paper's claims. The motivations of the proposed approaches are sensible.\n\nWEAKNESSES:\n1. Main focus of the paper is a bit confusing.\n\nThis paper has two main contributions - SGuTA and SCubA - which are separate contributions, and how to combine these two ideas are not introduced, which makes the paper look incomplete. I would like to suggest three options to remedy this issue:\n\na) If both SGuTA and SCubA can be combined into a single framework, then this will be the best approach - each module is shown to be effective and can show synergies when combined together.\n\nb) If SGuTA and SCubA cannot be combined, then we should at least discuss when to use which method. It seems like SCubA is quite consistently better than SGuTQ in terms of PSNR/SSIM and also computational complexity - then what is the point of proposing SGuTA? Should we just use SCubA all the time?\n\nc) If neither a) nor b) option is available, then I think we should consider separating SGuTA and SCubA into two distinct papers.\n\n2. Use of more input frames and incapability of generating arbitrary intermediate time step.\n\nThe proposed method uses more number of input frames (thus, more information) to predict the middle frame. While methods like QVI [30] also uses more input frames, for truly fair comparison with the other works, I think the authors should also report the performance with triplet-based evaluations.\nAlso, I have one simple question: it is written in the paper that \"septuplets\" are used - does this mean that the from the frames 1~7, the input frames are [1, 2, 3, 5, 6, 7] and the GT is 4-th frame? Or, do you use [1, 3, 5, 7]-th frames as the input?\n\nAlso, most the existing flow-based works can generate arbitrary time-step intermediate frames, while the proposed SGuTA and SCubA cannot. In my opinion, at least a short discussion regarding this issue is needed.\n\nQUESTIONS:\nPlease see Weaknesses.\n\n+ one minor question on the validity of Eq.(11): Is it an empirical finding, or is there any proof for this equation?\n\nLIMITATIONS:\nThe limitations of the current work and its potential negative societal impact is not adequately discussed.\nThe authors are encouraged to think about these issues more seriously and write them, even if the potential societal impact seems to be minimal.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}