Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper introduce a novel dataset for both low light enhancement and image super-resolution tasks in real world situation. The authors  manually set different under-exposure level and different focal length to capture low light and low resolution images.  Then they use auto-exposure, auto-white-balance, and auto-focus settings to capture images as ground truth. After that, authors manually filter out low quality images sequence and use Adobe Lightroom and crop to fix lens distortion. With comprehensive analysis and experiments, author prove the dataset can help further research on low light image super resolution task and show a new learning based method is needed to address this task.

strengths: 1. First large-scale dataset that address the low light image super resolution problem in real world.
2. Benchmark both low light enhancement and super resolution existing methods.
weaknesses: It's better to show some comparison with existing dataset and results of methods trained on previous dataset to illustrate the importance of the new dataset.
correctness: Nope.
clarity: Yes.
relation to prior work: Yes.
additional feedback: Nope.
review_title: Great Dataset for Both Low-light Enhancement and Image Super-Resolution in Real World.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presents the first dataset for training and testing Low-Light Image Super-Resolution networks. The dataset is captured using a carefully designed camera setup and is processed to eliminate artifacts.It contains images of the same scene capture using different focal lengths to attain different resolutions, and the same set of images captured under different exposure settings. The final dataset is provides paired Low-Light Low Resolution (LLLR) and Normal Light High Resolution (NLHR) images. The paper evaluates state-of-the-art methods on this dataset, showing the utility of the dataset.
strengths: + This dataset is the first one that provide paired Low-Light Low Resolution (LLLR) and Normal Light High Resolution (NLHR) images. It is useful for developing methods that perform both Low-light enhancement and Super-resolution. The paper shows that the dataset is useful by evaluating state-of-the-art methods on it. 
weaknesses: - It is unclear how well methods trained on this dataset perform on existing datasets such as [6,7,8,9]. 
- The paper emphasizes on joint Low-light enhancement and Super-resolution but does not explain why the two tasks should be jointly perform. It is not clear what is the benefit of doing them jointly.
correctness: + Technically correct
clarity: + Clearly written
relation to prior work: + Well discussed but should include results on other benchmarks as discussed above.
additional feedback: + Please see comments above.

====================================

Final comments: I am happy with the answers from authors. I am keeping my original rating.
review_title: A Solid Dataset for Low-Light Image Super-Resolution

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: In this paper, Aakerberg et al. provide an important dataset that could fill the gap between low-light image enhancement (LLE) and image super-resolution (SR). Unlike existing LLE datasets where the low-light (LL) and normal-light (NL) image pairs share the same spatial resolution, and existing SR datasets where the low-resolution (LR) and high-resolution (HR) image pairs have consistent illumination conditions, the RELLISUR dataset proposed in this work is the first dataset that simultaneously provides paired and aligned LL/LR and NL/HR images of a variety of content. Therefore, RELLISUR, for the first time, enables joint and end-to-end training from LL/LR to NL/HR images. Moreover, the authors benchmark the RELLISUR dataset extensively using 10 different state-of-the-art LLE and SR methods as well as novel joint training approaches, including ESRGAN and MIRNet with an upscaling module. Overall, this paper is very well written and easy to follow. It presents a much-needed dataset that would be highly valuable to the computer vision and image processing community. 
strengths: The RELLISUR is the first LL/LR to NL/HR dataset that allows for joint SR and LLE tasks. This dataset contains real-world LL/LR and NL/HR image pairs, as opposed to synthetic image pairs; therefore, it allows SR/LLE algorithms to generalize to practical applications. 

One unique advantage of this dataset is that the authors effectively process the image pairs with different resolution scales to allow pixel-wise alignment, which is a challenging task and a remarkable achievement. 

As the first of its kind, this dataset will be very useful to researchers aiming at directly inferring NL/HR images from LL/LR ones. 

After I download the dataset and examine its content, I found the images were all well marked and separated into the train, validation, and test sets. The images are all png files and can be easily used by most algorithms.

As stated by the authors, the dataset does not contain any personal information; therefore, it should not cause negative consequences as a result of using this dataset.
weaknesses: The name of this dataset, i.e., RELLISUR, is a little bit confusing. The authors did not provide the full name of the abbreviation and the name sounds strange. 

When capturing the images, the authors used a lot of "auto" settings, including auto-exposure, auto-white-balance, and auto-focusing. Due to the unpredictability of these settings, the acquired images might have significantly different exposure, white balance, and focus. These unpredictable imaging properties might have unforeseeable consequences when using these images to train a network.

According to the paper, there should be 850 distinct sequences, and therefore 850x3 = 2550 LR/HR image pairs. However, the number becomes 2250 in the paper.

The preprocessing pipeline in Section 3.2 is clearly a key contribution of this work. It allows images acquired with lenses of different focal lengths to have aligned pixels. However, the description in Section 3.2 is not very clear. The authors should add a diagram to better illustrate the pipeline of the preprocessing.


correctness: The claims in the paper seem to be correct. The dataset is constructed properly. The benchmark results are comprehensive and well presented.
clarity: The paper is very well written, well organized, and easy to follow.
relation to prior work: Prior works of LLE and SR have been properly cited in the introduction and compared in Tables 1 and 2.
additional feedback: The authors did not provide any code of their benchmarking methods. If possible, the authors are recommended to provide the code they used to evaluate different methods.
review_title: Great for end-to-end training from low-light low-resolution images to normal-light high-resolution images

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper presents the RELLISUR, a real dataset for the low-light image super-resolution task. The dataset contains 850 distinct LLLR/NLHR  sequences that are collected by controlling the camera lens and exposure time. Various SOTA models for LLE and SR are applied on RELLISUR for benchmarking.
strengths: - In general, this paper is well organized and easy to follow.
- The RELLISUR is created with realistic data and has a large size that is comparable to the existing dataset.
- Recent SOTA models are evaluated on the proposed dataset for benchmarking, which is beneficial for future research.

weaknesses: - Simply combining low-light enhancement (LLE) with super-resolution (SR) forms a very specific task and looks incremental to me. This paper fails to show/analysis the inherent relevance between those two tasks but just combining them together, hence I am concerned about its research value and interest for the community.

- All the samples are collected from static scenes and ignore dynamic content that is also very important in practice.

- Fig.5~7 should choose the same examples so as to support the authors' claim that joint task is better.

- The results in Tab.7 may not support the authors' claim that joint task is better. Even though under the Joint setting MIRNet achieves better performance, the ESRGAN achieves worse results. 

- A minor concern is that RELLISUR formulates a low-level computer vision problem more than a machine learning problem. It may be better suitable for some other venues rather than NeurIPS.

correctness: Yes
clarity: Yes
relation to prior work: Yes
additional feedback: No
review_title: A Real Low-Light Image Super-Resolution Dataset 

Meta-review: All reviewers agree that the proposed dataset fills in a gap in the important super-resolution research venue and the paper is well delivered. In addition, the reviewers also confirmed that their concerns are addressed. AC recommends accepting the paper.
Decision: Accept