Glare removal using dual cameras

Dual cameras that simultaneously capture RGB and IR images of a scene can be used to remove glare from the RGB image, transformed to a YUV image, by substituting a glare region in the luminance component of the YUV image with the pixel values in a corresponding region of the IR image. Further, color information in the glare region may be adjusted by averaging over or extrapolating from the color information in the surrounding region.

CLAIM OF PRIORITY

The present patent application claims the priority benefit of the filing date of Indian Application No. 202141024176 filed May 31, 2021, the entire content of which is incorporated herein by reference.

BACKGROUND

When a camera is used to capture video or take a photo, any presence of an active light source or high-intensity specular reflection from an object in the scene can cause an over- exposed region, called glare, which results in distortion of pixels and loss of detail in the image(s). One approach to addressing this problem involves taking images of the scene from two or more different perspectives, and post-processing the images to create a composite image from which the glare is removed, taking advantage of the fact that the glare caused by a given light source or specular reflection will occur at different locations in the images. The different images may be taken simultaneously with multiple cameras. Alternatively, if the image is to be taken of a still scene or object, the images may be taken sequentially with the same camera, moved to different locations and/or orientations relative to the scene. In a variation of this approach, it is also possible, in some application contexts, to take multiple images with the same camera perspective, but with the scene under illumination from different angles. Solutions that rely on sequentially taken images are generally inapplicable to glare removal from video of moving objects. Even if multiple cameras are used to image the scene from multiple angles simultaneously, the subsequent image processing is computationally expensive, which severely limits its feasibility for glare removal in real-time applications.

DESCRIPTION

Described herein are systems and methods to automatically remove glare from images captured with an RGB camera based on images simultaneously acquired with an IR camera. The disclosed approach exploits the fact that IR cameras do not capture light in the visible spectrum, and the IR images, therefore, do not include glare. A system for glare removal in accordance with various embodiment includes dual RGB and IR cameras that overlap in their respective fields of view and can be operated to simultaneously capture a pair of images of a scene, with associated electronic circuitry that processes the raw image data read out from the image sensors of the RGB and IR cameras to generate output images having the same field of view and the same resolution. Further, the system includes computing hardware and/or software that, after conversion of the RGB image to YUV format, processes the Y plane of the YUV image to detect any glare present in the image and identify its contours, which define the “glare region of interest (glare ROI)” or simply “glare region;” substitutes the glare ROI in the Y plane with the pixel values of a corresponding region in the IR image to remove the glare artifact; and (optionally) adjusts the color information of the pixels in the glare ROI based on the UV planes of the original YUV image. For example, the color information in the pixels surrounding the glare region can be averaged or extrapolated to determine the color information within the glare region. Alternatively, in some instances, the original UV plane values may be used as is.

The proposed approach lends itself to computationally efficient implementations that facilitate glare removal in real time, e.g., within 30 ms or less. Accordingly, glare removal in accordance herewith can be applied to video streams, for instance, to improve the user experience during video calls or video recording under various lighting conditions, or to improve the video provided by dashboard and other vehicle camera systems, to name just a couple of examples.

FIG.1is a flow diagram illustrating a method100for generating glare-free images based on pairs of RGB and IR images, in accordance with various embodiments. Two digital cameras, an RGB camera102capturing light in the visible wavelength range and an IR camera104capturing light in the infrared wavelength range, simultaneously acquire a pair of images of a scene. The pair of images may be, e.g., synchronized frames within video streams produced by the cameras102,104. The RGB camera102may include an image sensor overlaid with a color filter array (CFA); in this case, the image processing pipeline involves “de-mosaicing” the sensor output (act106), that is, digitally processing it based on the CFA to construct a full color image, or RGB image,108. The RGB image108is further transformed (in act110) to a YUV image112, which represents the image in terms of a luminance (physical brightness) component Y (114) and two chrominance components U (blue projection) and V (red projection) (115and116).

Glare removal operates on the Y component, or Y plane114, of the YUV image112. As shown, the method100involves determining whether glare exists in the Y plane114(act118), and if so (as decided at120), identifying the contours of the glare region (act122). Glare detection can generally be performed either by traditional, explicit image processing algorithms, or by machine learning models. Traditional algorithms usually rely on statistically defined thresholds and blob sizes to classify an image region as glare or not glare. Often, these algorithms identify glare regions of certain pre-defined shapes (e.g., circle, square, rectangle), which limits detection accuracy and can result in both false negatives (missed glare regions) and false positives (non-glare regions erroneously classified as glare). With machine learning models, the threshold and blob sizes are not statistically predefined, but tuned automatically based on the data. In addition, certain parameters of the machine learning model can be fine-tuned based on additional data even after deployment. In an example embodiment, a convolutional neural network (CNN) is employed to determine which pixels within the image are within the glare region. Note that, although glare detection and identification of the glare region are conceptually depicted inFIG.1as two separate steps118,122, they may, in some embodiments, constitute a single operation that identifies any glare region(s), thereby inherently determining whether glare exists in the images.

Once the glare ROI124has been identified within the Y plane114of the YUV image112, a corresponding region of pixels126is identified (in act128) in the IR image130. For computational simplicity, the determination of corresponding pixels is generally performed on a pair of an RGB/YUV image108/112and an IR image130that have been pre-processed, e.g., using spatial transformation and cropping, to share a common field of view and the same resolution. The pixels values in the glare region124of the Y plane are then replaced with the pixel values of the corresponding region126within the IR image130(act132), resulting in a glare-free Y-plane134. Further, the corrected, glare-free Y plane134is recombined with the chrominance components, that is, the UV planes115,116. In some embodiments, the color information within the glare ROI is adjusted based on the color values surrounding the glare ROI, taking advantage of the fact that color tends to be highly correlated (with neighboring pixels often having the same color value). For example, for glare regions including relatively few contiguous pixels, the UV pixel values within the glare region of the YUV image may be replaced by an average over UV pixel values of pixels surrounding the glare region (e.g., a weighted average, with weights being larger for pixels from the surroundings that are closer to a given pixel in the glare region). For larger glare regions, the UV planes may be fitted in the areas surrounding the glare region and extrapolated into the glare region. The glare-free image composed of the luminance information of the glare-free Y plane134and the chrominance information of the (e.g., color-adjusted) UV planes115,116may be sent to a display device136(e.g., a computer screen, phone screen, or video display screen in a vehicle) and/or stored in memory138. If no glare is detected in the Y plane (as determined at120), the Y plane114and the UV planes115,116are used as is in the stored or displayed image.

FIG.2is a schematic depiction of an image processing method200for glare removal from an RGB image using an IR image, as may be employed in the method100ofFIG.1, in accordance with various embodiments. The Y plane114from the RGB camera is input to a glare detection algorithm202that classifies every pixel as either a glare pixel or a normal pixel, creating a binary bit map (B)204where each glare pixel takes a value of one and each normal pixel takes a value of zero. As will be appreciated, the bit map204both reflects whether there is any glare in the image, and if there is, inherently defines the glare region. Thus, the glare detection algorithm202combines glare detection and identification of the glare region and its contours in one step.

The glare detection algorithm202may be explicitly programmed, or implemented by a machine learning model. In the latter case, the machine learning model may be, for instance, a neural network model such as a CNN or a fully connected network. CNNs, which are generally well-suited for image processing tasks, are particularly beneficial to implement the glare detection algorithm202in that they can provide high accuracy and performance. Neural network models for glare detection can be trained, e.g., in a supervised manner based on training data that includes many pairs of a Y plane and its corresponding binary bit map, with bit maps generated, e.g., manually by a user identifying the glare region contours in the image. As is known in the art, training the neural network involves iteratively adjusting network weights to optimize a cost function measuring the discrepancy between the output of the model (in this case, a binary bit map) and the ground-truth output provided with the training data (e.g., the manually generated binary bit map); a widely used algorithm for this optimization is backpropagation of errors.

Once the binary bit map204, which inherently identifies the glare region in the Y plane of the YUV image, has been generated, the corresponding region in the IR image (I)130can be identified by multiplying the IR image, at206, with the binary bit map204. This operation generates an intermediate image (I′)208in which all pixel values except in the glare region are zero. In the glare region, the intermediate image208has the same pixel values as the IR image130from which it is computed. Further, in act210, the binary bit map204is inverted by forming its 1's complement (B′)212, which is zero for all pixels within the glare region and one for all pixels outside the glare region. The Y plane114of the YUV image is multiplied, at214, with the 1's complement212of the binary bit map204. This operation generates another intermediate image (Y′)216, in which pixel values corresponding to the glare region become zero. The two intermediate images208,216are added, at218, to generate a single modified Y plane (Y″)220free of glare pixels, in which the glare pixels of the original Y plane114have been substituted by corresponding pixel values of the IR image130.

The method200, in an efficient implementation, can be computationally fast, allowing glare to be removed from an RGB image stream in real time, that is, within a timeframe that is unnoticeable to a human consumer of a video stream. For example, video captured at a frame rate of thirty frames per second (or more), corresponding to a new video frame every 33 milliseconds, is perceived by humans as continuous video. In various embodiments, the glare removal process is completed within less than 33 milliseconds, which enables producing glare-free video frames at the rate at which raw video frames are acquired.

FIG.3is a schematic diagram illustrating a system architecture for generating glare-free images in accordance with various embodiments. The system300may be implemented by a computing machine, e.g., as depicted schematically inFIG.4, that includes one or more hardware processors (e.g., one or more central processing units (CPUs) and/or graphic processing units (GPUs)), memory storing program instructions for execution by the processor(s) (including an operating system and one or more software applications), and various peripheral hardware devices, including, e.g., user input/output devices (such as a keyboard or touchpad, display device, or touch screen) and cameras. The computing machine may be, for example and without limitation, a personal desktop or laptop computer, tablet, or smartphone.

At the hardware level302, the system300includes an RGB camera102and an IR camera104. The cameras102,104generally have an associated digital signal processor (DSP), more specifically an image signal processor (ISP), that pre-processes the image sensor signals, e.g., to perform the de-mosaicing of RGB images, noise reduction, and the like. In dual-camera systems, pre-processing may also involve the image transformation and cropping operations used to create a pair of images with the same resolution and field of view. The cameras102,104have respective associated camera device drivers304,306, which generally run in the kernel space308of the operating system. At the application level310, the system300runs a software application312that provides video capabilities via access to the RGB camera102; hereinafter, such an application is also referred to as a camera application312. The camera application312may, for instance, be an online collaboration platform, such as Microsoft® Teams, that enables video calls. As another example, the camera application312may be a special smartphone app dedicated to taking pictures and recording video. The camera application312communicates with a user space driver314in the user space316of the operating system. The user space driver314, in turn, communicates with the camera device drivers304,306.

In the depicted example, glare removal is implemented in the user space316. That is, the user space driver314includes processing components, operating on pre-processed YUV and IR images received from the RGB and IR camera drivers304,306, to detect and find the contours of any glare region(s) in the Y plane of the YUV image (driver component318) and find the corresponding pixels in the IR image (driver component320), replace the glare pixel values in the Y plane with the corresponding IR pixel values (driver component322), and adjust the color information of the substituted pixels based on chrominance components of the YUV image in the glare region (driver component324). The glare-free, color-adjusted YUV image326can then be stored in memory328(e.g., double data rate (DDR) random-access memory) allocated to the camera application312, and/or output, by the camera application312, to a hardware display device330.

As will be readily appreciated by those of ordinary skill in the art, the glare removal functionality can generally be implemented both in the user space316or, alternatively, in the kernel space308. Implementation as part of the user space driver314, as illustrated inFIG.3, is beneficial in that it insulates the kernel space308from any errors that may occur during glare removal, and thus avoids the need for a system restart or reloading of the camera device drivers304,306in case of an error.

It is also possible, in an alternative embodiment, to provide for glare removal at the hardware level302in an ISP, e.g., as part of a system-on-chip (SoC) for a dual-camera system. For example, various existing dual-camera systems already perform image alignment between two images in the ISP (to obtain images having a shared field of view). For example, the Snapdragon™ SoC from Qualcomm® has an on-chip ISP, Spectra 580, which supports dual-camera systems. The Exynos® SoC from Samsung® also supports dual camera systems. In accordance with various embodiments, these and similar systems, if used with RGB and IR cameras, may be enhanced by further processing the images in the ISP to remove glare from the RGB images.

In yet another embodiment, glare removal as described herein (e.g., with reference toFIG.2) may be implemented in software at the application level310. This approach allows for the use of two individual cameras (one RGB and one IR camera), in place of a combined dual-camera system. Image processing software may receive raw YUV and IR data from the two cameras, and process them as needed to achieve the same resolution and a common field of view. For example, the image processing software may utilize image registration techniques to identify corresponding pixels in the two images. The image-processing method200for glare removal may then be applied to the properly aligned images.

Software implementation in principle also enables remote, cloud-based glare removal. For example, images from the RGB and IR cameras of a device connected to the Internet via a suitable wired or wireless communication channel may be sent to a remote server computer for glare removal, and the server may then return the processed, glare-free image. Such remote glare removal can be used when post-processing images, absent stringent time constraints. When processing video streams in real time, remote processing is feasible if communication rates are sufficient for real-time image transfer back and forth between the camera-equipped device and the server computer.

FIG.4is a block diagram of an example machine400with which glare removal in accordance with various embodiments may be performed. The machine may implement, in whole or in part, the method100, including the example image processing method200, and may form all or part of the system300. In alternative embodiments, the machine400may operate as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine400may operate in the capacity of a server machine, a client machine, or both in server-client network environments. In an example, the machine400may act as a peer machine in peer-to-peer (P2P) (or other distributed) network environment. The machine400may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile telephone, a smartphone, a web appliance, a network router, switch or bridge, a server computer, a database, conference room equipment, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein, such as cloud computing, software as a service (SaaS), other computer cluster configurations.

Machine (e.g., computer system)400may include a hardware processor402(e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), a main memory404and a static memory406, some or all of which may communicate with each other via an interlink (e.g., bus)408. The machine400may further include a display unit410, an alphanumeric input device412(e.g., a keyboard), and a user interface (UI) navigation device414(e.g., a mouse). In an example, the display unit410, input device412and UI navigation device414may be a touch screen display. The machine400may additionally include a storage device (e.g., drive unit)416, a signal generation device418(e.g., a speaker), and a network interface device420. Further, the machine400includes two or more cameras421to acquire still images or video. The machine400may include an output controller428, such as a serial (e.g., universal serial bus (USB), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection to communicate or control one or more peripheral devices (e.g., a printer, card reader, etc.).

The storage device416may include a machine-readable medium422on which are stored one or more sets of data structures or instructions424(e.g., software) embodying or utilized by any one or more of the techniques or functions described herein. The instructions424may also reside, completely or at least partially, within the main memory404, within static memory406, or within the hardware processor402during execution thereof by the machine400. In an example, one or any combination of the hardware processor402, the main memory404, the static memory406, or the storage device416may constitute machine-readable media.

The term “machine-readable medium” may include any medium that is capable of storing, encoding, or carrying instructions for execution by the machine400and that cause the machine400to perform any one or more of the techniques of the present disclosure, or that is capable of storing, encoding or carrying data structures used by or associated with such instructions. Non-limiting machine-readable medium examples may include solid-state memories, and optical and magnetic media. Specific examples of machine-readable media may include: non-volatile memory, such as semiconductor memory devices (e.g., Electrically Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; Random Access Memory (RAM); Solid State Drives (SSD); and CD-ROM and DVD-ROM disks. In some examples, machine-readable media are non-transitory machine readable media. In particular, in some examples, machine-readable media are media other than transitory propagating signals.

The following numbered examples are illustrative embodiments.

Example 1. A method for generating a glare-free image includes acquiring an RGB image of a scene in a visible wavelength range, acquiring an infrared (IR) image of the scene in an IR wavelength range, and processing the RGB image in conjunction with the IR image. As part of the processing, the RGB is transformed to a YUV image. Based on a luminance component of the YUV image, a glare region is identified in the YUV image, and then a region corresponding to the glare region in the YUV image is identified in the IR image. In the luminance component of the YUV image, pixel values in the glare region are substituted with pixel values of the corresponding region in the IR image. The result is a glare-free image. Beneficially, using an IR image in this manner to remove glare from an RGB image is computationally low-cost, which saves hardware resources and enables fast processing times, including, in some embodiments, real-time processing.

Example 2. The method of example 1 may be enhanced by adjusting, in a chrominance component of the YUV image, pixel values in the glare region based on pixel values in a region surrounding the glare region.

Example 3. In the method of example 1 or example 2, identifying the glare region based on the luminance component of the YUV image may include generating a binary bit map indicating for each pixel in the YUV image whether it is a glare pixel.

Example 4. In the method of example 3, the bitmap may be generated with a machine learning model. Beneficially, with a machine learning model, a pixel-value threshold and blob sizes of the glare region are not predefined, but can be tuned automatically based on data, and fine-tuned based on additional data even after deployment. This flexibility and data-driven optimization of the model helps achieve high accuracy of the glare, no-glare classification.

Example 5. In the method of example 4, the machine learning model may include a convolutional neural network (CNN). CNNs are highly suited for image-processing applications, and may result in particularly high performance.

Example 6. In the method of any of examples 3-5, the region in the IR image corresponding to the glare region in the YUV image may be identified by multiplying the IR image with the binary bit map to create a first intermediate image.

Example 7. In the method of example 6, the pixel values in the glare region in the luminance component of the YUV image may be substituted by the pixel values of the corresponding region in the IR image by multiplying the luminance component of the YUV image with a complement of the binary bit map to create a second intermediate image, and adding the first and second intermediate images to create the glare-free image.

Example 8. In the method of any of the preceding examples, the RGB and IR images may be synchronous frames of first and second video streams captured with an RGB camera and an IR camera, respectively.

Example 9. In the method of example 8, glare-free images may be generated from additional synchronous frames of the first and second video streams in real time.

Example 10. In the method of any of the preceding examples, the glare-free image may be displayed on a display device.

Example 11. A system for generating a glare-free image includes one or more hardware processors for processing an RGB image of a scene in a visible wavelength range in conjunction with an infrared (IR) image of the scene in an IR wavelength range. The processing operations include transforming the RGB image to a YUV image; based on a luminance component of the YUV image, identifying a glare region in the YUV image; identifying, in the IR image, a corresponding region to the glare region in the YUV image; and substituting, in the luminance component of the YUV image, pixel values in the glare region with pixel values of the corresponding region in the IR image. The system outputs a glare-free image, and may facilitate real-time glare removal.

Example 12. A system as in example 11 may further include an RGB camera configured to acquire the IR image and an IR camera configured to acquire the IR image.

Example 13. In the system of example 11 or example 12, the processing operations may further include adjusting, in a chrominance component of the YUV image, pixel values in the glare region based on pixel values in a region surrounding the glare region.

Example 14. In the system of any of examples 11-13, the one or more hardware processors may include one or more general-purpose processors configured by instructions stored in memory to perform the processing operations.

Example 15. In the system of example 14, the instructions may be part of a user space driver associated with RGB and IR cameras and executed within a user space of an operating system.

Example 16. In the system of any of examples 11-13, the one or more hardware processors may include a hardware-implemented image signal processor associated with the RGB and IR cameras.

Example 17. One or more machine-readable media store processor-executable instructions for processing pairs of synchronously acquired YUV and IR images to generate a glare-free image. The instructions, when executed by a hardware processor, cause the hardware processor to perform the following operations: based on a luminance component of the YUV image, identifying a glare region in the YUV image; identifying, in the IR image, a corresponding region to the glare region in the YUV image; and substituting, in the luminance component of the YUV image, pixel values in the glare region with pixel values of the corresponding region in the IR image.

Example 18. In the one or more machine-readable media of example 17, the operations may further include adjusting, in a chrominance component of the YUV image, pixel values in the glare region based on pixel values in a region surrounding the glare region.

Example 19. In the one or more machine-readable media of claim17or claim18, the glare region may be identified based on the luminance component of the YUV image by generating a binary bit map indicating for each pixel in the YUV image whether it is a glare pixel, the corresponding region in the IR image may be identified by multiplying the IR image with the binary bit map to create a first intermediate image, and the pixel values in the glare region in the luminance component of the YUV image may be substituted with the pixel values of the corresponding region in the IR image by multiplying the luminance component of the YUV image with a complement of the binary bit map to create a second intermediate image, and adding the first and second intermediate images to create the glare-free image.

Example 20. In the one or more machine-readable media of example 19, the bitmap may be generated with a machine learning model that includes, e.g., a convolutional neural network (CNN).