Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This work aims at advancing nighttime flare removal and proposes a new dataset called Flare7K. Based on the proposed dataset, extensive experiments are conducted, which demonstrates that the proposed dataset can complement the diversity of existing flare datasets and push the frontier of nighttime flare removal.
strengths: This proposed dataset is mainly designed to improve the quality of nighttime images and might help the community to strengthen the research.
weaknesses: 1. Some no-reference metrics should be used in the Table 2 of the manuscript.
2. In th table 2, the benchmarking study is not convcing due to lack some the general methods for image restoration such as MPRNet, etc.    

correctness: Yes, this dataset is constructed in a sound way and I think this paper offers a good building process for future restoration benchmarks.
clarity: The paper is well written and well organized.

relation to prior work: It is introduced well in section 2 of the manuscript.
additional feedback: Stated in Weaknesses.
review_title: A Nighttime Flare Removal Dataset.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper presents the first synthetic dataset for nighttime flare removal. Authors collected diverse nighttime flare images with different types of cameras and light sources and then render flares based on the observations and statistics. These rendered flares can be added to flare-free nighttime images to form paired nighttime flare-corrupted and flare-free data. Compared to the only daytime flare removal dataset, flares generated in this paper look more similar to real flares in night scenes and more diverse. Annotations of flare components are also provided in this dataset which is helpful for other interesting applications related to nighttime flare images. Experimental results claims the effectiveness of this synthetic nighttime flare removal dataset.
strengths: 1. The first nighttime flare removal dataset with realistic-looking synthetic flares.
2. Additional annotation of flare components.
weaknesses: 1. As claimed in the abstract and conclusion, nighttime flare also degrades the performance of vision algorithms. Thus it may be better to include a simple experiment showing the benefits the dataset can bring to existing downstream vision algorithms, e.g., the flare-removed outputs result in better segmentation or detection performance.
2. It may be better to compare more existing flare removal methods rather than haze removal and image enhancement methods in the experiments part. It is good to include such methods for comparison but training more learning-based methods can better claim the effectiveness of the proposed dataset.
3. The writing and figures can be improved further, see the Clarify part below.
correctness: Yes.
clarity: The overall writing of this paper is easy to follow but there are still many issues to address for a better paper.

1. According to the template and guidance of NIPS paper format, only the first letter of section title should be uppercase.
2. The fonts in figures are not unified. For example, the fonts used in Figure 1 and 2. are obviously different from other figures. Please keep the fonts of figures the same and follow the instructions of official template, including those in the suppl.
3. There should be blank space before the references in Line 59 an Line 60.
4. 'are' should be 'is' in Line 63.
5. 'of' may be missing in Line 71, after the word 'dataset'.
6. 'unique' should be 'uniquify in Line 117.
7. 'datase' should be 'dataset' in Line 223.
8. A period is missing in Line 249.
9. 'remove' should be 'removing' in Line 266.

Anyway, the paper should be carefully checked and improved carefully, only some issues are listed above.
relation to prior work: Yes.
additional feedback: No.
review_title: A good synthetic dataset for nighttime flare removal

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors propose a  first-of-its-kind dataset, Flare7K, for nighttime flare removal. The dataset has 5,000 scattering flare images
and 2,000 reflective flare images. In the dataset, the scattering flares are of 25 types and the reflective flares are of 10 types. In addition, the dataset has more annotations with respect to light source, glare with shimmer, reflective flare, and streak than other flare removal datasets. The dataset is trained on U-net and the results are compared with previous datasets. 
strengths: It is the the first nighttime flare removal dataset.
More diversity in annotation with respect to light source, reflective flare and streak.
More number of images compared to previous datasets.

weaknesses: 1)Firstly, the manuscript has several typos and sentences are not well structured. Extensive proof reading required. 

2)A lot of citations are missing in sections 1 and 3 when explaining concepts. Even section 3 seems to be completely paraphrased from Wu et al.'s work but citations are missing. 

3)As there is only 1 other night flare dataset, comparisons are done with only that paper. It is difficult to find reasonable contributions when compared with only 1 paper.

4)The authors claim that compared to the previous dataset, Flare7K has more annotations. But the difference in size between the two datasets is not huge. Other datasets can easily overcome this size with proper data augmentation methods.

5)The authors have trained their dataset with only the network (U-net) from the paper Wu et al. There are several other state-of-the-art models that the authors could have experimented with in comparison tables to show the reliability of this dataset. In short, experiments are not extensive and clear.

6)In line 177: authors start with a new paragraph with the sentence "To solve these issues".  There is no reference to the said issues mentioned in the previous sections. It is not a good practice to skip referencing sections where there is clear verification of a claim or a fact.

7)In line 213: "compared with previous datasets", again missing references.

8)Table 2 comparison is explained in supplementary material but it is not clear as to how those models' parameters were selected for the evaluation and how Flare7K was deployed on those models. Was Flare7K trained or evaluated on all the models? It looks like only fair comparison is with Wu’s paper as they have used the same training model as Flare7K. But what about Zhang and Sharma? Is the Flare7K evaluated on all the models and that’s the result comparison in Table 2? In short, clear explanation answering these questions are needed for fair comparison.

9)Conclusion is vague. Needs to be re-written summarizing and highlighting specific contributions and results of the paper.
correctness: The Dataset's formation seems to be correct.
clarity: Paper needs major revision and extensive proof reading as mentioned in weaknesses. Also, check the comments on Additional Feedback.
The dataset generation section 4 is not easy to follow.
relation to prior work: Literature review and comparison to prior work is few. Some concepts of the paper look like inspired version of previous papers with no citations (Physics Part). There are additional contributions to those previous works as mentioned in the Strengths.
additional feedback: The authors have worked hard in building a meaningful dataset and the demo video is interesting, but the manuscript as a whole needs major revisions. It looks like the paper was written in a hurry, therefore, it is missing citations and references and has several typos. Moreover, the paper lacks a proper experimental method ( comparison of results not clear) and the experiments performed is not adequate.
Some example typos:
In line 63: street lights are significant lower than the sunshine---> significantly.
In line 117: To avoid confusing, we universally unique name of each component.--> confusion, universally choose?
In line 118: for better understanding our proposed dataset --> of our dataset
In line 223: Datase--> database
review_title: Only dataset available for nighttime flare removal but manuscript needs major revisions in terms of writing, presentation and experiments.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This work provides the first nighttime flare removal dataset, Flare7K.
Before this work, there was only a daytime flare dataset. 
The lack of the nighttime flare dataset hinders research on the task of nighttime flare removal.

The dataset contains 7K flare patterns that can be used to synthetically generate flare-corrupted and flare-free pairs.
The authors compare the patterns of the existing daytime dataset (by Wu [7])  and the proposed nighttime dataset (Flare7K) in Figure 3.
Flare7K seems to have more diverse patterns.

Figure 5 shows the effectiveness of Flare7K on the nighttime flare removal task: only replacing the training dataset with Flare7K improves results.
In my opinion, from the qualitative comparison in Figure 5, the difference between Wu's and the authors' is not small. 
However, from the quantitative comparison in Table 2, the difference between Wu's and the authors' seems relatively small. 
strengths: [1] The paper provides a detailed introduction, which makes readers easily understand why lens flare occurs, and why lens flare is a significant problem.

[2] Provide the first nighttime flare dataset, Flare7K, and show the effectiveness of Flare7K.
weaknesses: [1] Question: Are authors planning to provide the flare-corrupted and flare-free image pairs or provide the training code that generates paired images on the fly?
In my opinion, the same paired dataset used in the experiments is necessary to reproduce the results and fairly compare methods.
I downloaded and checked the datasets. 
The dataset only contains 7K flare patterns and test data, not flare-corrupted and flare-free image pairs. 

Furthermore, I cannot find the number of paired data. It would be important to grasp how many paired images are required to remove nighttime flare.
In supplementary materials, I find "To train our nighttime flare removal model, our paired flare-corrupted and flare-free images are generated on the fly." and "60 epochs on Flickr24K". 
Does it mean the number of paired data is 24K x 60? 

\

[2] Hard to understand section 4.2 dataset generation and Figure 4.
From L188-L191, "Figure 4 presents our scattering flare synthesis pipeline. From the reference flare images of each type, we can obtain the relationship between the RGB value of the pixel and its distance to the light source. Such a relationship can be viewed as a color correction curve" 

Question 1: Is the reference image in Figure 4 a "real" nighttime flare image?

Question 2: If so, how do we obtain the steak mask, 4 pattern parameters, and round gradients from the reference image? "we can obtain the relationship" seems not enough to explain.

In addition, a brief explanation of Optical Flares in Adobe After Effects is necessary.

\

[3] The evaluation set only contains 20 synthetic and 20 real paired data.
It is small. (but at least it seems enough to compare flare removal methods, as shown in Table 2.)
Increasing the size of the test set (especially real data) would be better.
Rewriting L223 to L239 would be better. 
It is hard to know that the test set consists of 20 real and 20 synthetic paired data at a glance.

\

[4] It would be better if there were more details about other methods Zhang [40] and Sharma [39].
From Table 2, their performances are worse than "input". I wonder why and how they work.
I find slightly more details in supplementary materials, but I still cannot roughly understand how they work.
In addition, in my opinion, when using Zhang's method, authors should fine-tune their model on Flair7K rather than using the pre-trained model. 
Please ignore this comment if Zhang's official code does not provide a proper training code.

\

[5] According to L107-L108, Qiao et al. [8] tried to remove lens flare with unpaired data using the cycle-GAN framework. 
If possible, I would like to see the results when the cycle-GAN framework is trained on unpaired data generated by Flare7K. 
Please ignore this comment if additional experiments are prohibited or somewhat burdensome.
correctness: In L223, datase -> dataset
clarity: In L177, what do "these issues" indicate?

In L227, it is better to notify that Huawei P40 is a smartphone, and the Sony α 6400 with Sigma 16mm F1.4 is a professional camera.
relation to prior work: Yes
additional feedback: Nice demo page and video. They help understand this work.
review_title: Interesting work. The authors introduce the task and dataset in detail.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper presents a dataset of synthetically generated flare images based on real-world flare images collected from different cameras with different lenses. Using the real-world reference as well as a strong background in the physics of flares, the synthetically generated dataset had a much greater diversity than prior works thus allowing models to generalize better on unseen data. The paper has shown that a model trained on their dataset has shown undeniable improvements than when trained on datasets from prior works.
strengths: The presented dataset has not only been shown to have already introduced significant boosts to model performance but has also opened up doors to other areas of research because of the accompaniment of annotation data (although it is not as immediately clear why it would be relevant to industrial usage). The paper has also done a very thorough literature review into the topic of image flare that will aid future studies.  

The dataset is also diverse across geographical regions. It isn't highlighted in the paper, but the website for the dataset showcases images from the dataset that are taken from countries outside of Singapore and in different environments.
weaknesses: The most immediate potential shortcoming I see to their method is that flare may have been synthetically generated in images where there would not otherwise be any flare even with any configuration of camera and lens. For example, there seem to be images in the dataset where flares were synthetically generated even though the original image does not have any lights or there were multiple lights in the image but only flares for one of them were generated. Just based on this disparity, there is a clear distribution shift between their dataset and real-world data which might have held back the models' abilities to generalize.

The paper could include details of methods to collect images that flares were synthetically generated for as well as details of how the Adobe After Effects tools worked under the hood.
correctness: The dataset was constructed in a sound way.
clarity: The paper was well written. 
relation to prior work: Yes. Prior work was clearly discussed.
additional feedback: Well done. Great contribution overall.
review_title: Significant improvement and thorough analysis

Meta-review: metareview: This paper propose a new nighttime flare hazard dataset. The synthetic dataset enables to obtain ground truth which is otherwise very difficult in real scenes. All reviewers are satisfied with the quality of the paper. Only one reviewer recommends marginal below acceptance, and the concerns are minor. The concerns has been carefully addressed by the authors through the discussion. All other concerns have also been carefully addressed and the paper has been rewritten accordingly. Therefore this paper is suitable for acceptance.
confidence: 5: The area chair is absolutely certain
Decision: Accept