Recurrent Back-Projection Generative Adversarial Network for Video Super Resolution
Authors: Israa Sabry Fahmy
Marwah Hisham Sulaiman
Zahraa Shehabeldin
Mohamed Elnaggar
Dareen Hussein Hasan
Mohamed Yasser
Moustafa Youssef
Hesham Mohamed Eraqi
Venue: Submitted to ICLR 2023 (2023)
Keywords: Video Super Resolution
GANs
Temporal Coherence
Recurrent Projection.

Abstract: In this paper, we propose a new Video Super Resolution algorithm in an attempt to generate videos that are temporally coherent, spatially detailed, and match human perception. To achieve this, we developed a new generative adversarial network named RBPGAN which is composed of two main components: a generator Network that exceeds other models for producing very high-quality frames, and a discriminator which outperforms others in terms of temporal consistency. The generator of the model uses a reduced recurrent back-projection network that takes a set of neighboring frames and a target frame applies SISR (Single Image Super Resolution) on each frame, and applies MISR (Multiple Image Super Resolution) through an encoder-decoder Back-Projection based approach to concatenate them and produce x4 resolution version of the target frame. The Spatio-temporal discriminator uses triplets of frames and penalizes the generator to generate the desired results. Our contribution results in a model that outperforms earlier work in terms of perceptual similarity and natural flow of frames, while maintaining temporal coherence and high-quality spatial details. The algorithm was tested on different datasets to eliminate bias.

Under review as a conference paper at ICLR 2023
RBPGAN: BACK-PROJECTION GENERA- TIVE ADVERSARIAL NETWORK FOR VIDEO SUPER RES- OLUTION
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we propose generate vidcos that are temporally coherent, spatially detailed, and match human perception. To achieve this, we developed a new generative adversarial network named RBPGAN which is composed of two main components: generator Net- work which exceeds other models for producing very high-quality frames, and a discriminator which outperforms others in terms of temporal consistency, The generator of the model uses reduced recurrent back-projection network that takes set of neighboring frames and target frame, applies SISR (Single Image Super Resolution) on each frame, and applics MISR (Multiple Image Super Resolu- tion) through encoder-decoder Back- Paoc--rroecttion based approach concatenate them and produce x4 resolution version of the target frame. The Spatio-temporal discriminator uses triplets of frames and penalizes the generator generate the desired results. Our contribution results in model that outperforms earlier work in terms of perceptual similarity and natural flow of frames, while maintaining temporal coherence and high-quality spatial details, The algorithm was tested on different datasets to eliminate bias
Video Super Resolution algorithm in attempt
INTRODUCTION
Video Super Resolution (VSR) is the process of generating High Resolution (HR) Videos from Low Resolution (LR) Videos. Videos are of the most common types of media shared in our everyday life. From entertainment purposes like movies to security purposes like security camera videos, videos have become very important. As a result, VSR has also become important. There is need to modernize old videos or enhance their quality for purposes such as identifying faces in security footages, enhancing satellite captured videos, and having better experience watching old movies with today's quality.
Similar and older than VSR is ISR (Image Super Resolution), which is the process of generating single high-resolution image from single low-resolution image. Since a video is understood to be sequence of frames (images), Video Super Resolution can be seen as Image Super Resolution (ISR) applied to each frame in the video. While this is useful because many of the ISR techniques can be slightly modified apply VSR, however, there are major differences between VSR and ISR. The main difference is the temporal dimension in videos that does not exist in images. The relationship between a frame in a video and other frames in the video is the reason why VSR is more complex than ISR (Haris et al.) 2019).
In this research, various VSR methods will be explored. The methods are mainly clustered into two clusters, methods with alignment and methods without alignment (Liu et al.) 2022). We will compare between the different methods across different datasets and discuss the results. Out of the methods we studied, we chose models to be the base models for our research paper. We further explore these base models, experiment with them, and discuss possible enhancements. This paper aims to minimize the trade-off between temporal coherence, natural-to-the-ey perception, and quality of VSR. To achieve this, we propose a Generative Adversarial Network (GAN) that uses concepts from different state-of-art models and combines them in way to achieve our goal.

---

Under review as a conference paper at ICLR 2023
Our methodology, experimentation and results mentioned in this paper respectively. Finally, conclude the paper and propose future work recommendations.
2 RELATED WORK
Based on review of the literature, the Deep Learning based methods that target Video Super Resolution problem can be divided into 2 main categories: methods with alignment, and methods without alignment. Alignment basically means that the input LR video frames should be aligned first before feeding them into the model. Under the methods with alignment, existing models can be divided into two sub-categories: methods with Motion Estimation and Motion Compensation (MEMC), and methods with Deformable Convolution (DC). Under methods without alignment, existing models can be divided into 4 sub-categories: 2D convolution, 3D convolution. RNNs, and Non-Local based. In this section, the out-performing methods belonging to every category will be discussed.
2.1 METHODS WITH ALIGNMENT
2.1.1 MOTION ESTIMATION AND MOTION COMPENSATION (MEMC)
First, the Temporally Coherent Generative Adversarial Network (TecoGAN) (Chu et al. 2020) The network proposes a temporal adversarial learning method for a recurrent training approach that can solve problems like Video Super Resolution, maintaining the temporal coherence and consistency of the video without losing any spatial details, and without resulting in any artifacts or features that arbitrarily appear and disappear over time. The TecoGAN model is tested different datasets, in- cluding the widely used Vid4, and is compared to the state-of oatte-of-the-arts ENet, FRVSR, DUF, RBPN, and EDVR. TecoGAN is able to generate improved and realistic details in both down-sampled and captured images. However, one limitation of the model is that it can lead to temporally coherent yet sub-optimal details in certain cases such under-resolled faces and text.
Second, the recurrent back nook-prooection network (RBPN) (Haris et al. 2019). This architecture mainly consists of one feature extraction module, a projection module, and reconstruction module. The recurrent encoder-decoder module integrates spatial and temporal context from continuous videos. This architecture represents the estimated frame motion with respect to the target rather than explicitly aligning frames. This method is inspired by back-projection for MISR, which iteratively calculates residual images as reconstruction error between a target image and a set of its correspond- ing images. These residual blocks get projected back to the target image to improve its resolution. This solution integrated SISR and MISR in a unified VSR framework as SISR iteratively extracted various feature maps representing the details of target frame while the MISR were used to get set of feature maps from other frames. This approach reported extensive experiments for VSR and used different datasets with different specs conduct detailed evaluation of its strength and weaknesses. This model reported significant results in terms of the quality of produced videos.
2.1.2 DEFORMABLE CONVOLUTION METHODS (DC)
The Enhanced Deformable Video Restoration (EDVR) (Wang et al. 2019) is framework that per- forms different video super-resolution and restoration tasks. The architecture of EDVR composed of main modules known Pyramid, Cascading, and Deformable convolutions (PCD) and Temporal and Spatial Attention (TSA). The concept of deformable convolution is built on the idea having sampled frames, augmenting their spatial locations, and learning additional offsets to these loca- tions without supervision. This model results in very good results, however, it's size and number of parameters very high.
2.2 METHODS WITHOUT ALIGNMENT
2.2. 2D AND 3D CONVOLUTION
2D convolution based VSR mainly uses the classic convolution method to extract information from the frames of the video in only the spatial dimension and increasing the resolution accordingly (Lu-
2

---

Under review as a conference paper at ICLR 2023
al. 2019). However, 3d convolution is better convolution method in which the temporal di- mension Is taken into consideration, produsing videos with better consistency. One high- performing method based 3D convolution is Dynamic Upsampling Filters (DUF) (o al. 2018). The dynamic filter network can generate filters that take specific inputs and generate corresponding fea- tures. The structure of the dynamic upsampling filter and the spatio-temporal information learned from the 3D convolution led to comprehensive knowledge of the relations between the frames. However, main drawback of this method is the time complexity.
2.2.2 RCNNS
RCNNS very powerful network (Dieng al. 2019) developed a stochastic temporal con- volutional network (STCN) by incorporating a hierarchy of stochastic latent variables into TCNs, allowing them to learn representations over wide range of timescales. The network is divided into three modules: spatial, temporal, and reconstruction. The spatial module is in charge of extracting features from a series of LR frames. Temporal module is a bidirectional multi-scale convoluted ver- sion Motion estimation of LSTM that is used to extract temporal correlation between frames. The latent random variables STCN are organized in accordance with the temporal hierarchy of the TCN blocks, effectively spreading them across several time frames. As result, they generated new auto-regressive model that combines the computational advantages of convolutional architec- tures with the expressiveness of hierarchical stochastic latent spaces. The model in STCN is meant to encode and convey information across its hierarchy.
2.2.3 NON-LOCAL METHODS
There is progressive fusion network for VSR that is meant to make greater use of spatio-temporal information that has shown be more efficient and effective than existing direct fusion, slow fu- sion, and 3D convolution techniques through a technique known as Progressive Fusion Video Super- Resolution Networks in Exploiting Non-Local Spatio-Temporal Correlations (PFNL.). This is pre- sented in Progressive Fusion Video
Network via Exploiting Non-Local Spatio- Temporal Correlations (Yi et al. 2019). That enhanced the non-local operation in this progressive fusion framework circumvent the MEMC methods used in prior VSR techniques. This was done by adding a succession progressive fusion residual blocks (PFRBs). The suggested PFRB is de- signed to make greater of spatio-temporal information from many frames. So, the model be summarized into three major components: a non-local resblock, progressive fusion residual blocks (PFRB). and an upsampling block. The non-local residual blocks are used to extract opatio-tempoon characteristics, and PFRB is proposed to fuse them. Finally, the output of sub-pixel convolutional layer is added to the input frame, which is then upsampled using bicubic interpolation to produce the final super-resolution results.
OUR MODEL AND CONTRIBUTION
After studying the previous work done in the area of VSR, propose Generative Adversarial Network that has a Recurrent Back Projection network as the generator to achieve highest possi- ble accuracy without sacrificing the perceptual coherence.1. It offers superior results as it combines the benefits of the original MISR back nack-prroection approach with Deep back- nack-projection networks (DBPNs) which use the haok-projection to perform SISR through estimating the SR frame using the LR frame.The The discriminator of the model is inspired by the well-known TecoGAN paper to improve the temporal coherence of the generated frames. The architecture for the proposed architecture is shown in figure
3.1 GENERATOR
The Recurrent Back Projection network basically calculates the residual images reconstruction error between the target image and a set neighboring images, 6 neighboring frames. exploited temporal relationships between adjacent frames. The network mainly consists of three main mod- ules, one feature extraction module, a projection module, and a reconstruction module. The feature extraction module basically performs two operations, it extracts the features directly from the target
3

---

Under review as a conference paper at ICLR 2023
RBPN Generator
Fat.
LRpgi
LRe Fan
Concel
Concal
Concal
TecoGAN
Disoriminator
Comv
Conv
Conv
Gonerated Triplets
Recomnz
HR
9-t
0/1
L.R
Conv
Projection Module
Projection Modulo
Projection
Modulo
Bys
Triplets
bys
Bus
Traindd
Modelk
Intermeciates
Figure 1: Proposed Architecture.
frame also extracts the feature from the concatenation of the neighboring frame, and calculates the optical flow from the neighboring frame to the target frame. The second module which is the pro- jection module consists of an encoder and a decoder. The encoder composed super-reso resouutino (MISR), single image super-resolution, and residual blocks. For the decoder, it consists of a strided convolution and a residual block. The decoder takes the output the previous encoder input produce the LR features and then fed them to the encoder of the projection module. The last module, which is the reconstruction module, takes the output of the encoder in each projection module and concatenates them inputs to produce the final SR results.
multiple image
3.2 DISCRIMINATOR
The discriminator, denoted by D, is Spatio-temporal based and its main role to compare the gen- erated HR frames from the generator with the ground truth.
This network also purposes term temporal flow of the generated frames to make the results more natural without artifacts. The discriminator guides the generator learn the correlation between the LR input and the HR targets. In addition, it penalizes the generator if the generated frames contain less spatial or unrealistic artifacts in comparison to the target HR and the original LR frames.
loss function named "Ping-Pong" which focuses the long-
There is an issue that appears when super-resolving at large up-scaling factors and that is usually seen with CNNs Ledig et al. (2017). So, the proposed network chose this specific discriminator to mitigate the issue of lack of finer texture details. The discriminator is trained to differentiate between the generated high resolution images and high resolution ground truth images.
The main goal of our proposed architecture to recover precise photo-realistic textures and motion- based scenes from heavily down-sampled videos accordingly this improves the temporal coherence and perceptual quality.
DATASETS AND METRICS
4.1 DATASETS
We used the same datasets used by other VSR models for our results to be comparable to them. We used Vimeo-90k (Xue et al. 2019 and the training dataset created by TecoGAN publishers (we will refer to as Vimeo TecoGAN) (Chu et al. 2020) for training experiments, and we used Vid4 and ToS3 more details about the datasets in table
for testing. You
4

---

Under review as a conference paper at ICLR 2023
We used 4 down-sampling scale with bicubic interpolation (also known Gaussian Blur method) for training and testing, and we obtained comparable assessment findings on both testing datasets.
Table Comparison between different Datasets
Dataset
Resolutionn
Clips
Frames/clip
Frames
Vimeo90K
448x256
13, 100
91, 701
Vimeo Tecogan
Varies
265
120
31, 800
Vid4
Varies
Varies
684
ToS3
1280x534
233, 166, 150
549
4.2 EVALUATION METRICS
The usual criterion for evaluating the quality of super-resolution results mainly includes Peak signal- to-noise ratio (PSNR) and Structural index similarity (SSIM). PSNR is the ratio of an image' maxi- mum achievable power to the power of corrupting noise that affects the quality its representation. To calculate the PSNR of picture, it must be compared to an ideal clean image with the highest potential power (ground truth). Higher outcomes are preferable. single SR frame's PSNR can be calculated
MAX²
PSN R 10log 0 MSE
where MAX is the color value's maximum range, which is commonly 255 and MSE is the mean squared error. SSIM measures the similarity of structure between two corresponding frames using an uncompressed or distortion-free image as baseline. So, higher SSIM and PSNR values indi- cates higher quality. PSNR may be more sensitive to Gaussian noise, whereas SSIM may be more sensitive to compression errors. Their values, however, are incapable of reflecting video quality for human vision. That implies, even if video has very high PSNR value, it may still be unpleasant for humans. As a result, deep feature map- based measures like LPIPS (Zhang et al. 2018) can cap- ture more semantic similarities. It indicates the extent to which the generated video perceptually good and natural to the human-eye. The distance between picture patches calculated using LPIIS (Learned perceptual image patch similarity). Higher implies more distinct. Lower values indicate closer match. Additionally, ToF is used to calculate the pixel-wist difference of movements inferred from successive frames and it's very indicative measure of the temporal coherence of video frames. The lower (Of, the more temporally consistent is the video.
tOF = 110F b.) OF
(2)
5 LOSs FUNCTIONS
The loss functions used while training our model are as follows:
1. GAN Loss (min-max loss)
We use the Vanilla GAN loss (equation 3 which is the simplest form of the GAN loss, for the adversarial training. The generator tries to minimize it while the discriminator tries to maximize it.
min max Ez-mun log D(x) + Ez-n D(G(z))) D
(3)
Here, D(x)) the discriminator's estimate of the probability that real data instance x is real, and D(G(2)) is the discriminator's 's estimate of the probability that a fake instance is real. E the expected value over all data instances.
2. Pixel loss
Minimizes the pixel-wise squared differences between ground truth and generated frames (equation 4
1t
be l|
5

---

Under review as a conference paper at ICLR 2023
3. Ping Pong Loss
Proposed by TecoGAN model (Chu al. 2020), ping pong loss (equation 5 effectively avoids the temporal accumulation of artifacts, and targets generating natural videos that are consistent over time. PP loss uses sequence of frames with the forward order as well as its reverse. Using an input number of frames of length we can form symmetric sequence 01, 01:...0.11 On On- ...ai such that when feeding to the generator, the forward results should be identical to the backward result.
Σ 19: - ge/12
(5)
Here, the forward results represented with 9r and the backward results with ge!
4. Feature/perceptual Loss
represented by equation 6 it encourages the generator to produce features similar to the ground truth ones by increasing the cosine similarity of their feature maps. ensures more perceptually realistic and natural generated videos. Our discriminator features contain both spatial and temporal information and hence are especially well suited for the perceptual loss.
( ((I",) [ 31 [1
Where 1 {get Ib. 9t+1), (br1, b,, b
5. Warping Loss Used while training the motion estimation network (F) that produces the optical flow be- tween consecutive frames.
-(a--
equation 7 represents warping loss, where W() is the warping function, F() is the flow estimator, and a, the LR frame in position l.
EXPERIMENTS
In all our experiments, we focus on the 4x Super Resolution factor since provides satisfactory results and takes reasonable amount of training. Also, we used crop size of 32x32 and gaussian downsampling. All experiments were done using following specifications to enable the dense na- ture of the training phase: 64GB of DDR4 RAM, 2.80GHz Intel Core i9-109000 CPU, NIVIDIA GetForce RTX 3090 (1 x 24 GB) GPU, and Ubuntu 20.04.3 LTS operating system.
We will now present and explain the experiments we did in sequence, and later we will explain and discuss their results comparatively.
So, started by training and testing two base models (TecoGAN and RBPN) to ensure their correctness and reliability before integrating them and produce our model. Then, we integrated them as discussed in section 2. Later, performed some experiments our model with different parameters and loss functions till we reached the best outcome. The final model is later compared with the other state-of-the-ar models in terms of PSNR, SSIM, LPIPS, and ToF metries
6.1 EXPERIMENT 1: REDUCED RBPN MODEL SIZE
As discussed, RBPN is the base model we are using for our model's generator, and we started by training and testing it. The model size was very large, leading to high reference and training time, and therefore we decided to build our generator based reduced size of a recurrent back- projection network. We experimented with the number of neighbour frames passed to the projection modules in the model and decreased it without compromising much of the produced quality. This

---

Under review as a conference paper at ICLR 2023
Table Comparative analysis between all the conducted experiments on our model for Vid4 dataset
Metric Name
Experiment 3.1
Experiment 3.2
Experiment 3.3
PSNR
25.58
25.74
25.56
LPIPS
1.47
1.44
1.45
IOF
2.46
2.35
2.40
SSIM
0.756
0.762
0.751
resulted in decreased number of layers and smaller model. The training of this experiment took around hour/epoch and we trained for 150 epochs, using VimeoTecoGAN dataset and the loss functions mentioned in section 5.
6.2 EXPERIMENT 2: MODEL INTEGRATION
After we ensured the correctness, reliability and readiness of the two base models for the integration phase, we began integrating the generator with the spatio-temporal discriminator to create our GAN model and prepare it for some experiments. The integration was a challenging task since the two models had many differences functions' interfaces, dependencies used, training datasets and lack of generalization to fit any other dataset.
We will now discuss the experiments done on RBPGAN (our model) to monitor the model potential and test our hypothesis.
6.2.1 EXPERIMENT 3.1
We used all loss functions mentioned in the previous section (Ping Pong loss, Pixel loss, Feature loss, Warping loss, and GAN loss). We used neighbour (adjacent) frames per target frame, but due to the use of ping pong loss, this number is doubled create the backward and forward paths. Therefore, the generator was using neighbour frames per frame. We trained both the generator and discriminator simultaneously from the beginning, using the VimeoTecoGAN dataset, and training took around 3.5 days using the specs mentioned.
6.2.2 EXPERIMENT 3.2
We used the same loss functions as experiment 3.1 except the ping pong loss to observe effect on the results. We used 3 neighbour frames per target frame, started the training of generator and discriminator together, and used the same dataset and other parameters as experiment 3.1. The training took around days.
6.2.3 EXPERIMENT 3.3
This experiment is the same experiment 3.2 except that we firstly trained the generator solely for a number epochs and then started the training of the GAN using this pre-trained part. The training took aound days using the same dataset, number of neighbours and other parameters.
6.2.4 EXPERIMENT
We trained RBPN model with the same number of neighbours, crop size, dataset, and other unify- able parameters as we did for our model in the previous experiments to ensure fair comparison between it and our model.
7 RESULTS
Following the results and metrics evaluation for the experiments done and explained in previous section (tables and 3):

---

Under review as a conference paper at ICLR 2023
Table Comparative analysis between all the conducted experiments on our model for ToS3 dataset
Metric Name
Experiment 3.1
Experiment 3.2
Experiment 3.3
PSNR
32.89
32.85
32.78
LPIPS
0.78
0.69
0.75
IOF
1.60
1.64
1.62
SSIM
0.872
0.880
0.869
So, overall, experiment 3.2 yields the best results collectively, and therefore we will use to compare with the state-of-the-an afrtthe-rr models (tables and Also, you in figures 2 and 3 some examples from vid4 dataset for our model.
Figure 2: walk (Left:LR, Right: RBPGAN)
2Mm
28
29
30
2
3
1
28
29
30
31
1
2
3
10
6
7
8
0
10
Figure 3: calendar (Left:LR, Right: RBPGAN)
Table 4: Comparison between experiment and state-of-the-arts for Vid4 dataset
Metric
Experiment 3.2(Ours)
TecoGAN
RBPN (3 neighbors)
BIC
ENet
DuF
PSNR
25.74 74
25.57
26.71
23.66
22.31
27.38
LPIPS
1.44
1.62
2.0
5.04
2.46
2.61
I0F
2.35
1.90
2.19
5.58
4.01
1.59
SSIM
0.756
0.770
0.801
NA
NA
0.815
8

---

Under review as a conference paper at ICLR 2023
Table 5: Comparison between experiment and state-of-the-arts for ToS3 dataset
Metric
Experiment 3.2(Ours)
TecoGAN
RBPN (3 neighbors)
BIC
ENet
DuF
PSNR 32.85
32.65
34.32
29.58
27.82
34.6
LPIPS 0.69
1.09
1.10
4.17
2.40
1.41
tOF 1.64
1.34
1.54
4.11
2.85
1.11
SSIM
0.880
0.892
0.915
NA
NA
NA
DISCUSSION AND CONCLUSION
We can say that we managed to prove our hypothesis and achieved the highest results when it comes perceptual similarity, and very high results in temporal coherence and quality of frames. While current adversarial training produces generative models in a variety of fields, temporal correlations in the generated data have received far less attention. We, on the other hand, concentrated on im- proving learning goals and present a temporally self-supervised approach to bridge this gap. For the sequential generation tasks such as video super-resolution and unpaired video translation, natural temporal shifts are critical. The reduced Recurrent Back-Projection Network in our model treats each context frame as a separate source of information. These sources are combined in iterative refinement framework inspired by the idea of back-projection in multiple-image super-resolution. This aided by explicitly representing estimated inter-frame motion with respect to the target, rather than explicitly aligning frames. In this work, were trying to merge and advance the highly realistic output of RBPN with the more naturally-looking output of TecoGAN to have a smooth high quality generated videos from network. In the results section, we demonstrated the metrics of generated frames and how we achieve higher qualities, represented by SSIM and PSNR, when com- pared with TecoGAN, We also maintained high temporal consistency, represented by tOf, compared with all the models. When assessing our model with the LPIPS metric, which is responsible for measuring the perceptual similarity. we found that our model surpasses all models. including our two base models, TecoGAN and RBPN. This is because the discriminator helps the generator in learning further the correlations between sequential frames. Adding to that, we managed create the model with reasonable size and number of trained parameters compared to the other powerful models such as DuF, while producing distinguishing results.
There are some limitations we faced during the implementation: our model needs powerful compu- tational resources but we were only limited to a 2-GPU machine with 64GB memory which limited experiments since each experiment took longer than usual on the available machines. Moreover, while our technique produces extremely realistic results for a large variety of natural pictures, in some circumstances, such under-resolved faces and text in VSR, or jobs with dramatically dif- fering motion between two domains, our method can provide temporally coherent and natural but sub-optimal details. It would be interesting to both technique and motion translation from contemporaneous work in the latter instance (Wang et al. 2019).. Hence, we recommend using different downsampling methods introduce more generalization to the model, and may be train the model on more augmented datasets. In addition, there is continuing research on the usage of ac- celerator techniques boost the speed of network training and inference time, leading real- real-time VSR transition.
REFERENCES
Mengyu Chu, You Xie, Jonas Mayer, Laura Leal-Taixé, and Nils Thuerey. Learning temporal coher- ence via self-supervision for gan-based video generation. ACM Transactions on Graphics (TOG), 39-4) 2020.
Adji B Dieng, Yoon Kim, Alexander M Rush, and David M Blei. Avoiding latent variable collapse with generative skip models. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2397-2405. PMLR, 2019.
Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita. Recurrent back- net- work for video super-resolution. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3897-3906, 2019.

---

Under review as a conference paper at ICLR 2023
Younghyun Jo, Scoung Wug Oh, Jaeyeon Kang, and Scon Joo Kim. Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3224-3232 2018.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic sin- gle image super-resolution using a generative adversarial network. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4681-4690, 2017.
Hongying Liu, Zhubo Ruan, Peng Zhao, Chao Dong, Fanhua Shang, Yuanyuan Liu, Linlin Yang, and Radu Timofte. Video super-resolution based on deep learning: a comprehensive survey. Artificial Intelligence Review, pp. 1-55, 2022.
Alice Lucas, Santiago Lopez-Tapia, Rafael Molina, and Aggelos K Katsaggelos. Generative adver- sarial networks and perceptual losses for video super-resolution. IEEE Transactions on Image Processing, 28(7):3312-3322 2019.
Xintao Wang. Kelvin CK Chan, Ke Yu, Chao Dong. and Chen Change Loy. Edvr: Video restoration with enhanced deformable convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 0-0, 2019.
Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman. Video enhancement with task-oriented flow. International Journal Computer Vision (ICV), 127(8): 106-1066-1001 -125, 2019.
Peng Yi, Zhongyuan Wang, Kui Jiang, Junjun Jiang, and Jiayi Ma. Progressive fusion video super- resolution network via exploiting non-local spatio-temporal correlations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3106-311 3115, 2019.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as perceptual metric. In CVPR, 2018.
10


Reviews:
[{"review_id": "x7e1aWY2sqP", "reviewer": "Reviewer_jYRg", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new Video Super Resolution (VSR) method based on generative adversarial networks (called RBPGAN) to improve temporally coherent, spatially details, and human perception. In particular, the generator uses a reduced recurrent back-projection network, and the discriminator measures the temporal consistency. Some experiments are provided to demonstrate the effectiveness of the proposed method.", "summary_of_the_review": "The writing should be improved, the technical details and more experiments should be provided.", "strength_and_weaknesses": "Strength: This paper proposes a new GAN-based VSR method to address the important issues of temporally coherent, spatially details and human perception.\n\nWeaknesses: The writing of this paper should be improved. The details of the proposed method are not clear. The experiment section should be improved.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "w06ntgj1ryR", "reviewer": "Reviewer_LUmB", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper combines RBPN and TecoGAN to solve the Video Super-Resolution task. Experiments show that the performance is better than RBPN and TecoGAN.", "summary_of_the_review": "The paper is of poor quality. The idea is a combination. The experiments are not thorough and convincing enough.", "strength_and_weaknesses": "Weaknesses:\n1) The paper is poorly written. Many typos can be found in the paper. For example, in Fig. 1, there is a \"text\" appearing in the box of \"Projection Module\".\n2) The proposed method is just a combination of existing methods, i.e., RBPN and TecoGAN. It is too trivial.\n3) The performance is not compared with existing VSR methods except RBPN and TecoGAN. The comparisons with RBPN and TecoGAN should be regarded as a part of the ablation study.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "tIaynMw75m0", "reviewer": "Reviewer_aKDa", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper trains RBPN model for video super-resolution. The authors employ TecoGAN discriminator and their ping-pong loss to improve temporal consistency of the super-resolved video. The authors use perceptual loss and warping loss, too.\n", "summary_of_the_review": "I don’t find many reasons to accept this paper.\nThe paper lacks novelty and the experimental results are hard to claim the benefits over the previous methods. \nFurthermore, I seriously concern about plagiarism the issue. The authors copied sentences from the previous work which they bring the core architecture.\n", "strength_and_weaknesses": "- Novelty\n\nThere is not much novelty in this work.\nThe generator architecture is from RBPN, discriminator is from TecoGAN, the loss functions are widely used popular terms. \n\n- Experimental Results\n\nWhile the authors claim they achieve state-of-the-art performance in perceptual quality, there is no visual comparison with the other methods. In Table 5, the proposed method achieves good LPIPS and SSIM, however, I believe this is not a fair comparison. There is trade-off relation between fidelity-based metrics and perceptual quality metrics. Such perception-distortion trade-off is well known in image/video restoration community. The proposed method far lacks in PSNR.\n\nY. Blau and T. Michaeli, “The Perception-Distortion Tradeoff,” CVPR 2018\n\n\n\n- Concerns\n\nThe authors directly copy the sentences from RBPN paper.\nFor example, in section 8, \n\n“The reduced Recurrent Back-Projection Network in our model treats each context frame as a separate source of information. These sources are combined in an iterative refinement framework inspired by the idea of back-projection in multiple-image super-resolution. This is aided by explicitly representing estimated inter-frame motion with respect to the target, rather than explicitly aligning frames”\n\nThis is an except from the abstract of Haris et al., 2019. This is considered plagiarism.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "R2zhFs3lIBi", "reviewer": "Reviewer_6Vz2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new CNN-based method that deals with video super-resolution problems. The proposed method consists of a generator and a discriminator. The generator combines Single Image Super-Resolution and Multiple Image super-Resolution on target frame and its neighboring frames. The discriminator uses triplet loss to enhance the capability of discriminating scaled output from the generator. Experiments show that the proposed method outperforms other baselines. \n\n", "summary_of_the_review": "In summary, this paper fails to clearly describe the proposed method combining both texts and figures. Hence, it is quite difficult to judge the validity and the effectiveness of the work. The authors are suggested to provide a detailed explanation of the proposed method and also to give reasons of the network design. ", "strength_and_weaknesses": "Strength\n1. This paper provides a comprehensive literature review. \n\n\nWeakness\n1. In section 3.1 generator , the described encoder and decoder cannot be located in Figure 1. It is quite confusing how the proposed method arrange the SISR and MISR modules in the network. \n2. The Figure 1  does not provide the details of each projection module. By reading the figure and the method section, it is difficult to understand how the network should work. \n3. In Section 3, it does not validate the reason why the proposed network should work and to what extent it works. \n\n\n4. There are many typos in the submitted manuscript. \n(1). A comma is missing in Eq.(1) . The log with base 10 is not properly typed. \n(2). Almost all the equations in the article are missing punctuations. \n(3). Sentence after Eq.(5): Here the forward results are represented with g_t -> represented by g_t.\n(4). 4th point of Section 5 Loss functions, The first sentence is not properly written and formatted.\n(5). Section 6 Experiments, first paragraph, the past tense and present tense are mixed together. \n(6). The caption of Figure 1 should come with : The proposed architecture. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject