Kati8/ICLR_2023_2025 huggingface dataset:
authors: Jaemin Lee
authors: Minseok Seo
authors: Sangwoo Lee
authors: Hyobin Park
authors: Dong-Geol Choi
reviews: 
Comments: This paper proposes an implicit Multi-In-Single-Out (MISO) based VFI model designed to generate accurate intermediate frames that are robust against nonlinear motion. It predicts
a target intermediate frame by taking both a prior frame clip and a subsequent frame clip as input, enabling it to capture nonlinear motion effectively. Meanwhile, a video motion perceptual loss is proposed in this paper. The two contributions presented enable the state-of-the-art performance achieved in this paper.
Strengths: 1. The proposed structure is simple but effective and capable of achieving state-of-the-art performance.
2. The ablation experiments are more than sufficient to argue for the validity of the two proposed modules.
Weaknesses: 1. Will this multiple-input single-output framework have limitations in practical applications? For example, given a T-frame video, how is the first frame or the last stitch of the image processed?
2. EMA-VFI is incorrectly referenced.
3. The length of the METHOD section is too short. The model structure is not presented in the whole paper, and it is not enough to show it only in Figure 2.
4. I think the Table 1 comparison is unfair because the other methods in the table have not been trained on multi-frame video. Therefore Table 1 should not be used as the first table in the experimental chapter. Instead Table 2 will be a fairer comparison.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 2 fair
Clarity: 2 fair
reviews: 
Comments: This paper proposes multi-in-single-out MISO structure that targets at frame interpolation task without explicitly estimate optical flow. The target time t is inserted to the network as a time embedding. Based on such structure and a proposed video motion perceptual loss, the proposed model claims advantages on non-linear motions as well as the capability of handling occlusions between frames. Experiments are conducted on the leading benchmarks with ablation studies on the proposed loss function.
Strengths: The proposed contribution is clear, and the paper is easy to follow
Weaknesses: The network structure is straightforwrd, with the loss as the main technique contribution. The loss consists of a MSE loss, image perceptual loss  and motion perceptual loss, with the last one as the claimed contribution. Although these combined loss design seems reasonable, the motion perceptual loss is adapted from the image perceptual loss, which may be considered as delta modifications or improvements. 

I'm not directly work on VFI tasks, but I know that there are bunch of work, e.g., burst image processing, that adopts multi-in-sinle-out structure, such as burst image denoising, burst image super-resolution etc. Therefore, claiming multi-in-sinlge-out is not that novel. If it is the frist to the VFI, then it is ok to claim it. This is what I'm not sure. But even so, it is not that novel.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: This paper aims to tackle the challenge of multi-frame-based video frame interpolation. Based on an existing T-shaped architecture (Seo et al., 2023b), the author introduces a motion perceptual loss that measures the distance between 3D conv features. Experiments on datasets including UCF101, Vimeo90K, and Middlebury datasets are conducted to demonstrate the proposed method's effectiveness.
Strengths: - This paper is simple and easy to read.
- Very good performance are reported by the authors.
Weaknesses: - Hard to find the contributions of this paper. The model architecture is borrowed from an existing work ( though just on arxiv), it is hard to count as this paper's contribution in my opinion.  The only new thing in this work is the motion perceptual loss, which is too trivial and straightforward. Moreover, as shown in Tab.4, the improvement introduced by motion perceptual loss is too minor  (0.16 PSNR).

- Evaluations are not convincing. Tab-1 compares performance for multi-frame-based interpolation. Yet almost all the selected previous SOTA are not optimized for multi-frame input. The used datasets like Vimeo90K, Middlebury, and UCF101 are also very wired for this setting. In general, the reviewer believes the comparisons are unfair and less convincing.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness/correctness: 2 fair
Clarity: 3 good
reviews: 
Comments: This paper introduces a novel multi-frame video frame interpolation approach that incorporates explicit non-linear motion estimation. Additionally, the authors propose the use of a perceptual loss to enhance the visual quality of the interpolated frames.


I appreciate your response. Upon reviewing the supplementary results, I am pleased with their quality, particularly in challenging scenarios. As a result, I have raised my final score. However, I still find it challenging to accept this manuscript. I suggest that the authors revise their paper by offering more comprehensive explanations of their motivation and conducting additional analytical experiments.
Strengths: 1. The experimental results provide evidence of the superior performance of the proposed model when compared to existing methods.
2. The utilization of a different pre-trained model for motion perceptual loss enhances the overall effectiveness of the approach.
Weaknesses: 1. The approach of using an increased number of input frames to interpolate a single intermediate image may lack novelty in its overall concept.
2. The straightforward substitution of the pre-trained model for perceptual loss can be seen as a technical workaround, and there is a lack of clarity regarding the process of constraining a single interpolated image to achieve 3D-perceptual loss.
3. There is a lack of visual analysis provided for the non-linear motion estimation.
4. The quantitative comparison conducted on the Vimeo90K septuplet dataset is not commonly used. It is recommended to provide the results on the triplet benchmark using the 1-1-1 setup for a more standard comparison.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness/correctness: 2 fair
Clarity: 2 fair