Understanding and Tackling Scattering and Reflective Flare for Mobile Camera Systems

Authors: fengbo  lan
chang wen chen
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 13:45:04

Abstract: The rise of mobile devices has spurred advancements in camera technology and image quality. However, mobile photography still faces issues like scattering and reflective flares. While previous research has acknowledged the negative impact of the mobile devices' internal image signal processing pipeline (ISP) on image quality, the specific ISP operations that hinder flare removal have not been fully identified. In addition, current solutions only partially address ISP-related deterioration due to a lack of comprehensive raw image datasets for flare study. To bridge these research gaps, we introduce a new raw image dataset tailored for mobile camera systems, focusing on eliminating flare. This dataset encompasses over 2,000 high-quality, full-resolution raw image pairs for scattering flare, and 1,200 for reflective flare, captured across various real-world scenarios, mobile devices, and camera settings. It is designed to enhance the generalizability of flare removal algorithms across a wide spectrum of conditions. Through detailed experiments, we have identified that ISP operations, such as denoising, compression, and sharpening, may either improve or obstruct flare removal, offering critical insights into optimizing ISP configurations for better flare mitigation. Our dataset is poised to advance the understanding of flare-related challenges, enabling more precise incorporation of flare removal steps into the ISP. Ultimately, this work paves the way for significant improvements in mobile image quality, benefiting both enthusiasts and professional mobile photographers alike.

Main paper: 
 introduction: 
 Lens flare [12, 15, 17, 22] is a common optical artifact degrading image quality and visual appeal. It occurs when stray light enters the camera lens and interacts with the imaging sensor. This phenomenon is particularly prevalent in mobile computational imaging due to several factors. Firstly, mobile cameras often utilize plastic lenses, which generally exhibit lower quality compared to professionalgrade glass lenses. Secondly, the cost constraints of mobile devices often preclude the inclusion of expensive anti-reflective (AR) coatings [5] , further exacerbating the issue of lens flare.Early approaches to flare removal relied on image processing techniques [3, 6, 16, 25] . However, recent advancements in deep learning and the availability of flare datasets [8] [9] [10] 29] have spurred the development of learning-based methods for tackling this problem. Notably, researchers have identified tone-mapping as a critical factor affecting restoration performance [8, 32] . Tone-mapping is a non-linear and non-invertible process within the image signal processing (ISP) pipeline that can hinder accurate flare removal. However, the ISP pipeline encompasses multiple non-invertible transformations beyond tone-mapping, each contributing to the loss of original image information, as exemplified in Fig. 2 . For instance, denoising algorithms, while essential for mitigating noise amplified by high ISO settings and small sensor sizes (especially in nighttime photography), inevitably remove some image details as a trade-off. Additionally, user preferences often drive manufacturers to incorporate sharpening operations that enhance image details, potentially introducing artifacts. Finally, image compression, typically into lossy formats like JPEG, discards information to achieve manageable file sizes. These observations raise crucial questions regarding the impact of various ISP steps on flare removal: Do other non-invertible ISP operations, such as denoising, sharpening, and compression, hinder or benefit the effectiveness of flare removal? Does their influence vary for different types of flares (e.g., scattering vs. reflective)?Understanding the interplay between these operations is vital for optimizing the ISP pipeline and effectively tackling lens flare. Unfortunately, the lack of readily available raw image datasets has limited previous research to mimicking individual operations rather than conducting comprehensive investigations.This paper addresses these challenges by introducing a novel dataset and methodology specifically designed for lens flare removal. Our approach leverages the unique properties of raw images to decompose the non-linear, non-invertible processing steps within the ISP for detailed analysis. The dataset comprises over 2, 000 high-quality, full-resolution raw image pairs exhibiting scattering flare and 1, 200 pairs with reflective flare, all captured using mobile  
related works: 
 This section presents an overview of the flare removal problem and current available datasets, followed by a discussion on the utility of raw image datasets in mobile computational photography.Lens flare, a common issue in photography, arises from various sources such as light scattering within the lens system, reflections between lens elements, and the impact of dust, contaminants, or scratches on lens surfaces. It can significantly affect the quality of photographs by introducing unwanted artifacts. Lens flare is broadly categorized into two types: scattering flares and reflective flares, each with distinct characteristics and appearances. Scattering flares occur due to light interacting with microscopic imperfections within the lens system, leading to light scattering in different directions and resulting in visible artifacts such as veiling Exploiting the symmetrical nature of reflective flares, we capture two images with slightly different camera orientations. 3D registration is then used to calculate the difference between the images, which represents the flare itself. These images are then merged to create a ground truth image without flare.glare or a series of distortions in the image [13, 19, 21, 24] . The appearance of these flares depends on the distribution and nature of the imperfections. For example, dust on the lens surface may create bright spots or streaks, while scratches can lead to elongated artifacts. Multiple defects can result in complex overlapping flare patterns, degrading the image further.Reflective flares, on the other hand, are the result of light reflecting between lens elements in multi-element lens systems [6, 16, 18, 25] . These internal reflections can produce geometric shapes such as concentric rings or polygons, depending on the lens design and the positioning of the light source. Reflective flares are especially noticeable when the light source is near the optical axis or the lens comprises many elements. Furthermore, these flares can vary in shape and clarity based on their focus. In-focus reflective flares appear as sharp, defined patterns, whereas out-of-focus flares often manifest as diffuse, irregular shapes, influenced by lens design and aperture shape.Several significant datasets have been developed for flare removal research. Wu et al. introduced a dataset featuring both captured flare-only images and simulated scattering flare images [29] . Qiao et al. [20] collected unpaired flare-corrupted and flare-free images, suitable for networks that capture distribution differences but unsuitable for pixel-to-pixel neural networks due to the lack of paired data . The Flare7K dataset [8] includes synthesized scattering and reflective flare images, alongside real images for evaluation. Dai et al. later expanded this dataset to include real scattering flare images captured in a darkroom, addressing some limitations of their earlier datasets [9] . Zhou et al. proposed a dataset collected using electronic devices for evaluation purposes [32] .Despite these contributions, the issue of reflective flare, which is common in everyday photography, remains underexplored. Creating ground truth data for reflective flare is challenging due to its inherent presence in optical systems. Flare7K and Wu et al.'s datasets simulate reflective flares at specific angles without considering the image content context, such as the shape of the light source [8, 29] . The Bracket Flare dataset by Dai et al. addresses in-focus reflective flare with a night-time dataset using a novel composition method [10] .Our dataset offers a comprehensive resource for studying both reflective and scattering flares, covering in-focus and out-of-focus reflective flares, as well as local and global corruption in scattering flares. This diverse collection aims to deepen and broaden the scope of flare removal research.The Image Signal Processing (ISP) Pipeline is a critical component in digital cameras, specifically engineered to process the complex data captured by camera sensors. The main objective of an ISP is to transform raw sensor data into a visually appealing image format, such as JPEG [26] . This transformation process encompasses several steps, starting with a linear transform that includes demosaicing [14] , white balancing [2] , and color correction [30] . Subsequent steps involve non-linear transformations, such as denoising [7] , tone mapping [11] , and JPEG compression [26] . Additionally, in the realm of mobile photography, ISPs are tasked with advanced image enhancement techniques, like sharpening, to compensate for the limitations imposed by smaller sensors and the inherent processing pipeline, as illustrated in Fig. 3 .While existing datasets serve as essential resources for addressing the flare removal challenge, they fall short in providing the raw image data necessary to fully comprehend the impact of various ISP operations. This gap is particularly noticeable in operations like tone mapping, which significantly influence performance. Zhou et al. highlighted this issue, pointing out that methods like those employed in the Flare7K dataset [8] , which combine flare and scene images in a gamma-corrected space, fail to account for the nonlinear nature of tone mapping, leading to synthetic images with unrealistic contrast and color distortions, especially around bright light sources [32] . In the absence of paired raw data for flare-affected and clean images, these researchers have resorted to simulating tone-mapping effects within their synthesis pipeline through a pixel illuminance-based weighting scheme. While this approach represents a creative attempt to mimic the non-linear effects of tone mapping, it underscores the critical need for access to genuine raw data. This need is not only for improving the accuracy and generalizability of flare removal models but also because raw images are subject to a wider range of operations beyond tone mapping.Our proposed raw image dataset, therefore, presents a unique and valuable resource for examining the influence of tone mapping and other ISP operations on the manifestation of lens flare artifacts, setting the stage for more advanced research in the field. 
dataset construction: 
 This section details the creation of our dataset, covering the capturing devices and settings, followed by specific capturing schemes for scattering and reflective flare.We employed mobile phone models from various manufacturers as capturing devices to avoid potential similarities in lens flare within the same series. The chosen devices, detailed in Table 1 , are popular in mobile photography and represent a range of capabilities. For consistency, we used the main camera of each device with manual control over exposure and focus, whenever possible. Multi-frame fusion was disabled to ensure the best raw image quality. The statistics of the collected data is tabulated in Table 2 .To simulate a lens with defects, we introduced a stain-corrupted camera filter in front of the capturing devices. Varying the filter's location relative to the camera simulated different levels of corruption due to the varying defect levels across the filter. While the camera remained fixed on a tripod, minor vibrations during capture could cause misalignment between the flare-corrupted and flare-free image pairs. To address this, we performed sub-pixel registration using SURF feature extraction and matching [4] . Given the small movements, only translations along the vertical and horizontal axes were computed. The registration process was first applied to internally processed images and then converted for the raw image data with its integer pixel grid. For each registered pair, we identified areas with light sources and applied a detection algorithm to pinpoint their positions, as illustrated in Fig. 4a . Both raw and internally processed images were cropped into patches centered on these light source positions. The raw patches were then processed externally to generate high-quality RGB image pairs. Finally, before inclusion in the dataset, predefined metrics were used to filter out low-quality pairs, ensuring the overall dataset quality.To capture reflective data by utilizing the symmetrical property between the flare and the light source, we employed the following method. As illustrated in Fig. 4b , we rotated the camera to change the light source position, which resulted in a shifted flare image. Through image registration and warping, we aligned the two captured images. We then used image subtraction to identify the flare location and merged the images. This allowed us to use the corresponding area from one image to compensate for missing information due to the flare in the other. Repeating this process with the roles reversed for the two images provided two pairs of flare-corrupted and flare-free images from a single capture. Camera rotation and image registration necessitate image warping and interpolation. Direct interpolation on raw image data can introduce artifacts like color aliasing and disrupt the noise model. To avoid this, we provide the original raw images and perform registration only on the internally processed and externally generated RGB images. 
experiments: 
 This section evaluates the performance of flare removal models using images processed by both the internal ISP of a mobile phone and an external processing pipeline on a computer. We explore the differences in performance between these two processing methods by analyzing the impact of each processing step.Our evaluation includes both raw images and those processed by the mobile phone's internal ISP. For raw images, we convert flarecorrupted images and their corresponding ground truth pairs into RGB images using a custom MATLAB pipeline, RAW2RGB. This pipeline performs black level correction, white balancing, demosaicing, and color space conversion, but excludes denoising or postprocessing techniques like sharpening or compression. These images are referred to as RAW2RGB. Images processed by the internal ISP are denoted as ISPRGB. Both scattering and reflective flare images undergo these processing steps.For scattering flares, we utilize a light source detection algorithm to identify light sources and crop the high-resolution raw image pairs into 512 × 512 flare-corrupted pairs. This detection algorithm, adapted from [29] , includes enhancements to reduce background misclassification in images with prominent white areas. We manually mask the background in each ground truth image to improve accuracy, use morphological operations to refine the detection masks, and employ a multi-light source detection approach to better represent complex real-world scenes. For reflective flares, we prepare pairs at a resolution of 1024 × 1024. Both RAW2RGB and ISPRGB datasets are processed in this manner.We assess the performance of cuttingedge flare removal techniques. Training for scattering flares follows the network settings from prior studies [8, 10, 29] . Since the model from Wu et al.is unavailable, we train a new model using their released code and data. For the Flare7K [8] and Flare7K++ [9] datasets, we evaluate the pre-trained Uformer model [28] . Training for reflective flares involves models from Flare7K [8] and Bracket Flare [10] , using our collected data for separate treatments of scattering and reflective flares. Results are documented in Table 3 . 3 : Quantitative results on reflective and scattering flare removal for the two types of data. PSNR, SSIM [27] , and LPIPS [31] are used for evaluation. ↑ denotes higher is better, and ↓ denotes lower is better.Flare Type Metric Input Wu et al. [29] Flare7K [8] Bracket Flare [10] reflective and scattering flare data, also suffers from this problem, but it is improved in Bracket Flare [10] due to an enhanced training pipeline. However, due to the lack of out-of-focus data in Bracket Flare [10] , it can identify the in-focus area to some extent but struggles to resolve the transparent, out-of-focus reflective region, as highlighted in the third row of Fig. 5 . Our proposed model, which incorporates both in-the-focus and out-of-focus data for training, demonstrates better performance in addressing this issue. For scattering flares, recent models demonstrate the ability to restore local scattering flares, as shown in the third and fifth row of Fig. 6 . However, these models struggle to handle global flares that affect the overall contrast of images, as seen in the first and last rows of Fig. 6 , which are also common in daily life. The visual results from our model shows that, by introducing both local and global degradation data in the training process, we improve the generalization performance of existing approaches.We utilize peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM) [27] , and learned perceptual image patch similarity (LPIPS) [31] for quantitative comparisons. The results presented in Table 3 indicate that the overall image quality obtained with the internal ISP is lower than that of images processed using an external pipeline on raw data. Furthermore, these metrics highlight our proposed approach's enhanced generalization capabilities, particularly in handling global artifacts in scattering flare data and improving clarity in out-offocus areas for reflective flares.Reflecting on the discrepancy in restoration performance between images directly converted from raw data (RAW2RGB) and those processed through an ISP (ISPRGB), this study further investigates the influence of specific processing steps such as denoising, sharpening, and JPEG compression. We systematically introduce these steps into the RAW2RGB pipeline to evaluate their cumulative effect on image quality. The experiment employs the following processing steps:• Denoise: Denoising in image processing removes noise-caused by sensor flaws, poor lighting, or high ISO settings-to enhance image clarity and quality, which is often positioned as the initial step. In the experiments, denoising is performed using the medium-sized NAFNet [7] model, which is trained on the SIDD [1] dataset and optimized for a compromise between denoising efficacy and computational efficiency. • Sharpen: The sharpening step is typically placed at the end of an ISP pipeline to enhance the perceptual quality of images by increasing the visibility of edges and details that may have been softened during earlier processing stages. To approximate the varied sharpening approaches used by different smartphone manufacturers, a USM (Unsharp Masking) sharpening operator with a dynamic range of weights is utilized. • Compression: Acknowledging the widespread use of JPEG compression in mobile imaging, the pipeline incorporates DiffJPEG [23] for simulating compression. The quality factor in image compression, particularly in formats like JPEG, balances compression rate and image quality, where a higher factor preserves more details and increases file size, while a lower factor enhances compression efficiency at the expense of data loss. In the experiments, we use quality levels variably set between medium ranges.To dissect the impact of these processing steps on flare removal efficacy, we explore four distinct pipeline configurations, each altering the sequence of flare removal in relation to denoising, sharpening, and compression: 4 and 5 , several critical observations emerge: Denoising Benefit: Echoing previous findings, initiating the pipeline with denoising consistently enhances image quality across various metrics and both types of flare. This confirms the pivotal role of noise reduction in improving overall image clarity and facilitating more effective flare removal. Notably, the fourth configuration underscores the challenges posed by incomplete noise reduction, common in devices with constrained processing power or images taken under poor lighting, leading to significant quality degradation.Sharpening and Compression Degradation: The integration of sharpening and compression steps, regardless of their sequence, invariably diminishes image quality. This outcome highlights the delicate balance between the desire to enhance visual sharpness and the adverse effects of artifact introduction and loss of detail. The impact varies with the type of flare, with reflective flare being particularly vulnerable due to its localized nature, complicating the model's ability to accurately identify and correct affected regions, resulting in up to a 2.93dB degradation in performance. Conversely, scattering flares, which exhibit more global characteristics, face a different set of challenges. The global degradation effects caused by sharpening and compression mirror those inherent in scattering flares, resulting in a performance loss of 1.75dB.Optimal Flare Removal Placement: The analysis suggests that positioning flare removal after denoising but before sharpening and compression (as in configuration 1) tends to produce the best restoration outcomes in terms of PSNR, SSIM, and LPIPS for both flare types. This arrangement benefits from a cleaner input image, enhancing the efficacy of flare removal. Placing flare removal later in the pipeline, especially after sharpening and compression (configurations 2 and 3), appears less effective, likely due to the compounded artifacts and information loss from preceding steps.The examination of these pipeline configurations illuminates the complex interplay between different ISP processing steps and their collective impact on the quality of image restoration. It confirms that while denoising significantly improves image quality, subsequent processing steps like sharpening and compression can introduce negative effects that compromise both objective and perceptual quality measures. This comprehensive analysis not only clarifies the performance disparity observed between RAW2RGB and ISP-processed images but also offers actionable insights for optimizing ISP pipelines to achieve superior image restoration results. 
conclusion: 
 This paper tackled the significant challenge of lens flare in mobile computational photography by introducing a novel raw image dataset tailored for the analysis and removal of both scattering and reflective flares. By utilizing RAW data, we can investigate those non-invertible ISP operations, and provide new and critical insights for addressing the flare problem on real data, which is not feasible with previous methods. Through rigorous experimentation, we've illuminated the complex effects of various non-invertible Image Signal Processing (ISP) operations-namely denoising, sharpening, and compression-on the performance of flare removal algorithms. We've provided a nuanced understanding of how different ISP operations impact flare removal, particularly emphasizing the importance of denoising as a foundational step for enhancing image quality. This study's insights into the interplay between ISP steps and flare removal offer valuable guidance for optimizing image processing pipelines, ultimately facilitating better image restoration techniques and improving the visual quality of photographs captured with mobile devices. Our findings reveal the delicate balance between enhancing image quality and preserving essential details necessary for effective flare mitigation, highlighting the critical role of processing sequence in achieving optimal restoration results. 
 
 Appendix:  


Decision: After rebuttal, all the 4 reviews are positive, so I recommend the acceptance of this paper.

Reviews: review: paper_summary: paper_summary: This paper introduces a new raw image dataset tailored to eliminate flare for mobile camera systems. Experimental analysis supports the claim and results look significant.
strength_weakness: strength_weakness: 1.This paper proposes a high-quality raw image dataset for lens flare removal.

2.The motivation of the article is to incorporate the flare removal step into ISP for optimizing ISP pipelines and effectively solving lens flare, which is attractive and meaningful and may inspire other works.

3.Writing is good and the paper is easy to follow and understand.
limitations: limitations: 1.In the process of data capture, different devices may bring different visual effects for the same scene. How to ensure the optimal quality of data and reduce data discrepancies?

2.Lack of detailed experiments on out of distribution and real-world image generalization.

3.What would be the effect if the flare removal step is placed after the denoising step? It is recommended to further supplement experiments to demonstrate the impact of ISP operation on flare removal.

4.Suggest further explaining the differences between the proposed dataset and other datasets.
paper_summary: paper_summary: This paper introduced a novel dataset tailored for the analysis and removal of both scattering and reflective flares to solve the problem of lens flare in mobile computational photography. The extensive experiments shows its effectiveness.
strength_weakness: strength_weakness: 1. This paper proposed a high-quality raw image dataset for lens flare removal.
2. This paper investigated the impact of non-invertible ISP operations (denoising, compression, sharpening) on flare removal.
3. Enhances the generalizability of existing flare removal approaches by including both local and global flare corruptions data from the dataset.
limitations: limitations: 1. I think the authors should add a section to elaborate on the differences between the proposed dataset and existing flare datasets, such as [1][2]
[1] Flare7K: A Phenomenological Nighttime Flare Removal Dataset
[2] How to train neural networks for flare removal
2. In Chapter 4.1, how does the author determine the position of the light source through light source detection? What is the method of light source detection?
3. In the experimental section, why different methods are used for experiments on different datasets? I think this is unfair to other methods.
4. Reference formats need to be consistent, such as conference/journal abbreviations.
5. The author should provide detailed experimental settings, such as GPU type, image size during training, etc.
paper_summary: paper_summary: This paper introduces a novel approach to addressing scattering and reflective flares in mobile camera systems. It presents a comprehensive raw image dataset with over 2,000 scattering flare and 1,200 reflective flare images captured across diverse scenarios. By focusing on the effects of ISP operations such as denoising, sharpening, and compression on flare removal, the research identifies how these processes can either aid or obstruct flare mitigation. Detailed experiments demonstrate the varying impacts of these ISP steps, offering insights for optimizing ISP configurations.
strength_weakness: strength_weakness: 1. Provide a detailed examination of the impact of scattering and reflective flares on mobile camera image quality.

2. Offer critical insights into how different ISP operations, such as denoising, sharpening, and compression, influence flare removal.

3. Introduce a large-scale, high-quality dataset specifically tailored for studying and mitigating flare effects in diverse real-world scenarios.
limitations: limitations: 1. The paper does not address the full range of commonly used ISP operations on mobile devices, such as color correction, white balancing, and HDR processing, potentially limiting the comprehensiveness of the proposed flare removal solutions.

2. The paper does not provide a detailed evaluation of the computational efficiency and real-time performance of the proposed flare removal methods, which are important considerations for practical implementation on mobile devices.

3. The paper does not specify whether the introduced dataset will be made publicly available, which could hinder community-driven advancements and independent validation of the proposed methods.
paper_summary: paper_summary: The paper proposes the first raw image dataset for lens flare removal. The dataset contains lots of real-world flare-corrupted and clean image pairs. Furthermore, the paper investigates the impact of non-invertible ISP operations (denoising, compression, sharpening) on flare removal.
strength_weakness: strength_weakness: 1. The paper introduces a novel, high-quality raw image dataset for lens flare removal which contains lots of flare-corrupted and clean image pairs captured in real world.
2. The paper explores the impact of non-invertible ISP operations on flare removal, facilitating the task of flare removal with raw image data.
limitations: limitations: 1. From experimental results, the paper archives the best results. However, since other compared methods are trained based on different datasets with domain gaps, is such a comparison fair?
2. For the exploration of the Impact of non-invertible ISP operations, the experimental results is interesting. However, the analysis did not provide an explanation for the causes within. Can the authors provide a more persuasive analysis of the reasons?
3. From Fig. 2 and Fig. 3, different ISP processes have a significant impact on the final JPEG results. For flare removal, as shown in Table 3, your ISP pipeline reaches a better result compared to the internal pipeline. However, does your ISP pipeline have an impact on other aspects of image quality?
4. The way of utilizing raw data for flare removal seems to lack novelty.