Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper presents a multi-frame super-resolution method applied to satellite imagery. It first estimates a reference image for the multiple input LR images by median filtering. Then it pairwise encodes the reference image and each of the multiple images in a recursive fashion then fuses the corresponding feature maps with residual blocks and bottleneck layers until only one feature maps for the entire multiple images obtained. In other words, LR images are fused into a single global encoding. Then, it applies a standard upsampling network to obtain the super-resolved image this image is fed into a network that estimates only the translational shift, and the shifted image with the estimated translation parameters finally resampled. 

A major concern is the estimation of a single translational motion for the SR image at the end of the network after all multiple images are already fused. The fusion strategy disregards the underlying spatially varying motion. This explicitly assumes the images are on a flat surface, which perhaps an acceptable assumption for high-orbit satellite imagery where the ground surface depth variances might be negligible. Still, this is a very critical limitation of the method. Besides, I am not convinced that pair-wise fusion can handle significant translational fusion as the filters have shared parameters. How a single convolutional layer accomplishes a global encoding and compensates for any translation between any LR image pair is neither articulated nor convincing discussion and evaluations are provided. Of course, such a problematic approach needs at least some kind of motion compensation, which may explain the need for the ShiftNet layer at the end. Nevertheless, this seems quite problematic. 

Even assuming the method only applies to satellite imagery, it lacks mechanisms to compensate/distinguish cloud coverage and atmospheric distortions. Characterization of satellite imagery noise models (Weibull, etc.) common in such imagery as a prior also completely disregarded. For these reasons, the proposed method fails to be considered as a comprehensive approach for multi-image super-resolution of satellite imagery. 

Novelty-wise, there is very little as all modules have been commonly used for SR tasks. 
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper proposes a framework including recursive fusion to co-registration and registration loss to solve the problem that the super-resolution results and the high-resolution labels are not pixel aligned.  Besides, the method is able to achieve good performance in the Proba-V Kelvin dataset. However, I have some concerns about this paper:

1) This paper lacks many references. Recently, many works focus on multi-frame super-resolution containing video super-resolution and stereo image super-resolution via deep learning.  They are using multiple low-resolution image to construct high-resolution image. For example:

Stereo super-resolution:

Jeon, Daniel S., et al. "Enhancing the spatial resolution of stereo images using a parallax prior." *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*. 2018.

Wang, Longguang, et al. "Learning parallax attention for stereo image super-resolution." *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*. 2019.

Video super-resolution:

Tao, Xin, et al. "Detail-revealing deep video super-resolution." *Proceedings of the IEEE International Conference on Computer Vision*. 2017. 

FRVSR: Sajjadi, Mehdi SM, Raviteja Vemulapalli, and Matthew Brown. "Frame-recurrent video super-resolution." *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*. 2018.

FFCVSR: Yan, Bo, Chuming Lin, and Weimin Tan. "Frame and Feature-Context Video Super-Resolution." *Proceedings of the AAAI Conference on Artificial Intelligence*. Vol. 33. 2019.

EDVR: Wang, Xintao, et al. "Edvr: Video restoration with enhanced deformable convolutional networks." *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops*. 2019.

2) Recursive fusion is aimed to fuse multiple low-resolution image information. Recently, more and more work utilize different methods to fuse multiple low-resolution image. For example, Tao et al proposes SPMC (Sub-pixel Motion Compensation) to align image, FRVSR uses unsupervised flow network that predicts optical flow to warp image, FFCVSR directly concatenate low-resolution image as the input of 2D convolutional network to fuse the information, and EDVR fuses multiple image features via utilizing deformable convolution. Thus, what is the advantage of recursive fusion compared to the above methods? This paper should discuss the difference between recursive fusion and the above methods.

3) Registration loss is important in this paper and it can solve the problem the output SR is not pixel-wise aligned to the HR ground truth. Registration loss utilizes ShiftNet that is adapted from HomographyNet. Thus, what is the difference between ShiftNet and HomographyNet? This paper should add some details about ShiftNet and Lanczos interpolation. 

4) It is better to test more datasets and compare with more state-of-the-art methods. This paper only tests in a satellite image dataset. Some datasets can be considered such as VID4 dataset in video super-resolution.
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: This paper proposes an end-to-end multi-frame super-resolution algorithm, that relies on a pair-wise co-registrations and fusing blocks (convolutional residual blocks), embedded in a encoder-decoder network 'HighRes-net' that estimates the super-resolution image. Because the ground truth SR image is typically misaligned with the estimation SR image, the authors proposed to learn the shift with a neural network 'ShiftNet' in a cooperative setting with HighRes-net. The experiments were performed on the ESA challenge on satellite images, showing good results.

Overall, I found this paper interesting, and the method described is both clever and efficient. While some points need to be clarified, I am in favor of accepting this paper to ICLR.

Positive aspects:
- the paper is very clear and easy to read, with nice figures.
- a sensitivity analysis on many different parameters or types of inputs are made, which makes this paper an interesting research paper. For example, the tests on the type of reference image to stack at the beginning are very interesting.
- While I am not an expert on super-resolution, I do see a clever algorithm, that can be for example used with different number of input views. 
- The end-to-end framework is also quite interesting as it allows to be spread easily across the very large satellite images users, with the code aldready publicly available.
- Lastly, the results are good wrt to the state-of-the-art, as the algorithm was proposed during a 2019 challenge and was in the top ones on the private leaderboard.

Remarks and clarifications:
- After looking at the challenge website, it is quite strange to find that the results values in Table 1, in terms of public and private scores, differ from the actual leaderboards, even if the metric is still the same. Thus, I was not able to understand (i) if the authors really participate to the challenge (ii) what method they presented during the challenge as 3 or 4 are shown here (iii) what was their ranking in the real leaderboard. It is important to be precise. From what I see, they might be second on the private score (the one that matters), if so it can be clarified also in the abstract where the word 'topped' was used.
- cPSNR metric : (i) can you please explain the acronym; (ii) it was the competition metric, so it means it is not 'we use' but 'the challenge used' - which will also show that it was not a choice to satisfy your results, but a standard metric.
- Lanczos interpolation: can you please explain in a small sentence in what this interpolation differs from the others?
- ShiftNet: what is this network? We only know tha it is adapted from HomographyNet, but we don't have any information about how it is composed. We just know from App. 1 that it has a very large number of parameters (34M). Why is that? Why a rigid registration needs such a large number of parameters?
- median anchor image: this is one of the interesting points of the paper. Can you please just clarify that you take a naive median image? Such as: for pixel (i,j), med(i,j)= med( LR1(i,j), LR2(i,j) ,...) .
- You are saying 'each imageset is padded to 32 views'. What do you mean? I thought your network was able to be used with different numbers of views. How did you pad your imageset?
- you list 'several baselines': for me, a baseline is something to compare to - usually, it is even something easy, such as the ESA baseline. But in your list, you are mixing baselines, other state-of-the-art methods (some of which you don't beat, so it's not a 'baseline'...), and your methods. It is very difficult to know which ones are yours. Please make different lists and/or identify yours in the table 1.
- What is the Ensemble method? It appears that in the paper, you explain the 'HighRes-net + shiftNet' method, but actually, the 'Ensemble' is the better one, while not clearly described. I don't understand what outputs are averaged - I thought HighRes-net and shiftNet were performed simultaneously.
- How did you select the hyperparameters of your model?


Scientific questions: 
- Is it possible to have views of different sizes as inputs? Or views with missing parts?
- Usually, we have a high resolution image that is different in terms of type of acquisition (like a different type of satellite, but also true in medical images for ex.). Is your network ready for that? I mean: can we constrain the method even if it is not the same type of acquisition as ground truth? In that case, is that possible to have a super-resolution of the type of the input LR images?

Typos: 
- is comprised of -> is composed of
- in Table 7, the bold number should be the beta=infinity as it is the best one. It will be clearer, even if of course, a good train score does not mean a good method because of overfitting.

review_title: Official Blind Review #1

Meta-review: This paper proposes a multi-frame super-resolution method including recursive fusion for co-registration and registration loss to solve the problem where the super-resolution results and the high-resolution labels are not pixel-wise aligned. While reviewer #1 is positive about this paper, reviewer #2 and #3 rated weak reject and reject respectively. Both reviewer #2 and #3 have extensive experience in the topic of image super-resolution. The major concerns raised by the reviewers include the lack of many references, the comparison of recursive fusion with related work, limited test databases, using a single translational motion for the SR images, and limited novelty on the network modules.  The authors provided detailed response to the concerns, however they did not change the overall rating of the reviewers. While the ACs agree that this work has merits, given the various concerns raised by the reviewers, this paper can not be accepted at its current state.
Decision: Reject