Patent Description:
<CIT> discloses a method according to the preamble of claim <NUM> and a system according to the preamble of claim <NUM>.

<CIT> discloses an image processing apparatus. An acquisition unit acquires a first image captured with visible light and a second image captured with non-visible light. A correction unit corrects luminance of the second image so as to bring luminance distribution of the second image close to luminance distribution of the first image. A generation unit combines the second image whose luminance is corrected by the correction unit and chromaticity information of the first image to generate a composite image.

<CIT> relates to a color night vision system and operation method thereof. The color night vision system includes a single-<NUM>-color image sensor configured to acquire a red, green, blue (RGB) image and an infrared (IR) image by processing RGB light signals and an IR light signal for each wavelength, and a processor configured to determine an exposure state of the RGB image by analyzing a brightness distribution of the RGB image, to decide at least one of an exposure compensation level of the RGB image, a denoising level of the RGB image, or a synthesis ratio between the RGB image and the IR image based on the determination result, and to create an output image based on the decision result that is made using the RGB image and the IR image.

<CIT> is directed to systems and methods for image processing. The system may obtain low-frequency component of a first image. For each element of the first image, the system may adjust a luminance of the element in response to determining that the luminance of the element is less than a predetermined luminance threshold. The system may determine a first luminance weight map corresponding to the first image based on the adjusted luminance of each element of the first image. The system may obtain the low-frequency component of a second image and determine a second luminance weight corresponding to the second image based on the luminance of each element of the second image. The system may further determine a fused image based on the low-frequency component of the first image, the first luminance weight map, the low-frequency component of the second image, and the second luminance weight map.

<CIT> discloses glare reduction in captured images. Techniques to improve the quality of captured images by reducing the effects of undesired objects are disclosed. The technique may involve the use of face detection to localize the likely position of screen glare within the captured images, as well as an awareness of the content that is being displayed in a display screen at the moment of capture of the respective image. The technique may then model the position, size, and/or distortion of the screen contents (or other projected lights) reflected by the user's eyeglasses (or other reflective surface the captured seen environment). Once the experience of the undesired screen glare has been modeled in the captured image, the techniques may perform an image modification operation to remove or reduce the undesired glare from the originally-acquired image in an efficient manner.

<NPL>, relates to detection and inpainting of specular reflection in colposcopic images with exemplar-based method.

Various systems and methods for glare removal from images are described herein with reference to the accompanying drawings, in which:.

Described herein are systems and methods to automatically remove glare from images captured with an RGB camera based on images simultaneously acquired with an IR camera. The disclosed approach exploits the fact that IR cameras do not capture light in the visible spectrum, and the IR images, therefore, do not include glare. A system for glare removal in accordance with various embodiment includes dual RGB and IR cameras that overlap in their respective fields of view and can be operated to simultaneously capture a pair of images of a scene, with associated electronic circuitry that processes the raw image data read out from the image sensors of the RGB and IR cameras to generate output images having the same field of view and the same resolution. Further, the system includes computing hardware and/or software that, after conversion of the RGB image to YUV format, processes the Y plane of the YUV image to detect any glare present in the image and identify its contours, which define the "glare region of interest (glare ROI)" or simply "glare region;" substitutes the glare ROI in the Y plane with the pixel values of a corresponding region in the IR image to remove the glare artifact; and (optionally) adjusts the color information of the pixels in the glare ROI based on the UV planes of the original YUV image. For example, the color information in the pixels surrounding the glare region can be averaged or extrapolated to determine the color information within the glare region. Alternatively, in some instances, the original UV plane values may be used as is.

The proposed approach lends itself to computationally efficient implementations that facilitate glare removal in real time, e.g., within <NUM> or less. Accordingly, glare removal in accordance herewith can be applied to video streams, for instance, to improve the user experience during video calls or video recording under various lighting conditions, or to improve the video provided by dashboard and other vehicle camera systems, to name just a couple of examples.

<FIG> is a flow diagram illustrating a method <NUM> for generating glare-free images based on pairs of RGB and IR images, in accordance with various embodiments. Two digital cameras, an RGB camera <NUM> capturing light in the visible wavelength range and an IR camera <NUM> capturing light in the infrared wavelength range, simultaneously acquire a pair of images of a scene. The pair of images may be, e.g., synchronized frames within video streams produced by the cameras <NUM>, <NUM>. The RGB camera <NUM> may include an image sensor overlaid with a color filter array (CFA); in this case, the image processing pipeline involves "de-mosaicing" the sensor output (act <NUM>), that is, digitally processing it based on the CFA to construct a full color image, or RGB image, <NUM>. The RGB image <NUM> is further transformed (in act <NUM>) to a YUV image <NUM>, which represents the image in terms of a luminance (physical brightness) component Y (<NUM>) and two chrominance components U (blue projection) and V (red projection) (<NUM> and <NUM>).

Glare removal operates on the Y component, or Y plane <NUM>, of the YUV image <NUM>. As shown, the method <NUM> involves determining whether glare exists in the Y plane <NUM> (act <NUM>), and if so (as decided at <NUM>), identifying the contours of the glare region (act <NUM>). Glare detection can generally be performed either by traditional, explicit image processing algorithms, or by machine learning models. Traditional algorithms usually rely on statistically defined thresholds and blob sizes to classify an image region as glare or not glare. Often, these algorithms identify glare regions of certain pre-defined shapes (e.g., circle, square, rectangle), which limits detection accuracy and can result in both false negatives (missed glare regions) and false positives (non-glare regions erroneously classified as glare). With machine learning models, the threshold and blob sizes are not statistically predefined, but tuned automatically based on the data. In addition, certain parameters of the machine learning model can be fine-tuned based on additional data even after deployment. In an example embodiment, a convolutional neural network (CNN) is employed to determine which pixels within the image are within the glare region. Note that, although glare detection and identification of the glare region are conceptually depicted in <FIG> as two separate steps <NUM>, <NUM>, they may, in some embodiments, constitute a single operation that identifies any glare region(s), thereby inherently determining whether glare exists in the images.

Once the glare ROI <NUM> has been identified within the Y plane <NUM> of the YUV image <NUM>, a corresponding region of pixels <NUM> is identified (in act <NUM>) in the IR image <NUM>. For computational simplicity, the determination of corresponding pixels is generally performed on a pair of an RGB/YUV image <NUM>/<NUM> and an IR image <NUM> that have been pre-processed, e.g., using spatial transformation and cropping, to share a common field of view and the same resolution. The pixels values in the glare region <NUM> of the Y plane are then replaced with the pixel values of the corresponding region <NUM> within the IR image <NUM> (act <NUM>), resulting in a glare-free Y-plane <NUM>. Further, the corrected, glare-free Y plane <NUM> is recombined with the chrominance components, that is, the UV planes <NUM>, <NUM>. In some embodiments, the color information within the glare ROI is adjusted based on the color values surrounding the glare ROI, taking advantage of the fact that color tends to be highly correlated (with neighboring pixels often having the same color value). For example, for glare regions including relatively few contiguous pixels, the UV pixel values within the glare region of the YUV image may be replaced by an average over UV pixel values of pixels surrounding the glare region (e.g., a weighted average, with weights being larger for pixels from the surroundings that are closer to a given pixel in the glare region). For larger glare regions, the UV planes may be fitted in the areas surrounding the glare region and extrapolated into the glare region. The glare-free image composed of the luminance information of the glare-free Y plane <NUM> and the chrominance information of the (e.g., color-adjusted) UV planes <NUM>, <NUM> may be sent to a display device <NUM> (e.g., a computer screen, phone screen, or video display screen in a vehicle) and/or stored in memory <NUM>. If no glare is detected in the Y plane (as determined at <NUM>), the Y plane <NUM> and the UV planes <NUM>, <NUM> are used as is in the stored or displayed image.

<FIG> is a schematic depiction of an image processing method <NUM> for glare removal from an RGB image using an IR image, as may be employed in the method <NUM> of <FIG>, in accordance with various embodiments. The Y plane <NUM> from the RGB camera is input to a glare detection algorithm <NUM> that classifies every pixel as either a glare pixel or a normal pixel, creating a binary bit map (B) <NUM> where each glare pixel takes a value of one and each normal pixel takes a value of zero. As will be appreciated, the bit map <NUM> both reflects whether there is any glare in the image, and if there is, inherently defines the glare region. Thus, the glare detection algorithm <NUM> combines glare detection and identification of the glare region and its contours in one step.

The glare detection algorithm <NUM> may be explicitly programmed, or implemented by a machine learning model. In the latter case, the machine learning model may be, for instance, a neural network model such as a CNN or a fully connected network. CNNs, which are generally well-suited for image processing tasks, are particularly beneficial to implement the glare detection algorithm <NUM> in that they can provide high accuracy and performance. Neural network models for glare detection can be trained, e.g., in a supervised manner based on training data that includes many pairs of a Y plane and its corresponding binary bit map, with bit maps generated, e.g., manually by a user identifying the glare region contours in the image. As is known in the art, training the neural network involves iteratively adjusting network weights to optimize a cost function measuring the discrepancy between the output of the model (in this case, a binary bit map) and the ground-truth output provided with the training data (e.g., the manually generated binary bit map); a widely used algorithm for this optimization is backpropagation of errors. Once the binary bit map <NUM>, which inherently identifies the glare region in the Y plane of the YUV image, has been generated, the corresponding region in the IR image (I) <NUM> can be identified by multiplying the IR image, at <NUM>, with the binary bit map <NUM>. This operation generates an intermediate image (I') <NUM> in which all pixel values except in the glare region are zero. In the glare region, the intermediate image <NUM> has the same pixel values as the IR image <NUM> from which it is computed. Further, in act <NUM>, the binary bit map <NUM> is inverted by forming its <NUM>'s complement (B') <NUM>, which is zero for all pixels within the glare region and one for all pixels outside the glare region. The Y plane <NUM> of the YUV image is multiplied, at <NUM>, with the <NUM>'s complement <NUM> of the binary bit map <NUM>. This operation generates another intermediate image (Y') <NUM>, in which pixel values corresponding to the glare region become zero. The two intermediate images <NUM>, <NUM> are added, at <NUM>, to generate a single modified Y plane (Y") <NUM> free of glare pixels, in which the glare pixels of the original Y plane <NUM> have been substituted by corresponding pixel values of the IR image <NUM>.

The method <NUM>, in an efficient implementation, can be computationally fast, allowing glare to be removed from an RGB image stream in real time, that is, within a timeframe that is unnoticeable to a human consumer of a video stream. For example, video captured at a frame rate of thirty frames per second (or more), corresponding to a new video frame every <NUM> milliseconds, is perceived by humans as continuous video. In various embodiments, the glare removal process is completed within less than <NUM> milliseconds, which enables producing glare-free video frames at the rate at which raw video frames are acquired.

<FIG> is a schematic diagram illustrating a system architecture for generating glare-free images in accordance with various embodiments. The system <NUM> may be implemented by a computing machine, e.g., as depicted schematically in <FIG>, that includes one or more hardware processors (e.g., one or more central processing units (CPUs) and/or graphic processing units (GPUs)), memory storing program instructions for execution by the processor(s) (including an operating system and one or more software applications), and various peripheral hardware devices, including, e.g., user input/output devices (such as a keyboard or touchpad, display device, or touch screen) and cameras. The computing machine may be, for example and without limitation, a personal desktop or laptop computer, tablet, or smartphone.

At the hardware level <NUM>, the system <NUM> includes an RGB camera <NUM> and an IR camera <NUM>. The cameras <NUM>, <NUM> generally have an associated digital signal processor (DSP), more specifically an image signal processor (ISP), that pre-processes the image sensor signals, e.g., to perform the de-mosaicing of RGB images, noise reduction, and the like. In dual-camera systems, pre-processing may also involve the image transformation and cropping operations used to create a pair of images with the same resolution and field of view. The cameras <NUM>, <NUM> have respective associated camera device drivers <NUM>, <NUM>, which generally run in the kernel space <NUM> of the operating system. At the application level <NUM>, the system <NUM> runs a software application <NUM> that provides video capabilities via access to the RGB camera <NUM>; hereinafter, such an application is also referred to as a camera application <NUM>. The camera application <NUM> may, for instance, be an online collaboration platform, such as Microsoft® Teams, that enables video calls. As another example, the camera application <NUM> may be a special smartphone app dedicated to taking pictures and recording video. The camera application <NUM> communicates with a user space driver <NUM> in the user space <NUM> of the operating system. The user space driver <NUM>, in turn, communicates with the camera device drivers <NUM>, <NUM>.

In the depicted example, glare removal is implemented in the user space <NUM>. That is, the user space driver <NUM> includes processing components, operating on pre-processed YUV and IR images received from the RGB and IR camera drivers <NUM>, <NUM>, to detect and find the contours of any glare region(s) in the Y plane of the YUV image (driver component <NUM>) and find the corresponding pixels in the IR image (driver component <NUM>), replace the glare pixel values in the Y plane with the corresponding IR pixel values (driver component <NUM>), and adjust the color information of the substituted pixels based on chrominance components of the YUV image in the glare region (driver component <NUM>). The glare-free, color-adjusted YUV image <NUM> can then be stored in memory <NUM> (e.g., double data rate (DDR) random-access memory) allocated to the camera application <NUM>, and/or output, by the camera application <NUM>, to a hardware display device <NUM>.

As will be readily appreciated by those of ordinary skill in the art, the glare removal functionality can generally be implemented both in the user space <NUM> or, alternatively, in the kernel space <NUM>. Implementation as part of the user space driver <NUM>, as illustrated in <FIG>, is beneficial in that it insulates the kernel space <NUM> from any errors that may occur during glare removal, and thus avoids the need for a system restart or reloading of the camera device drivers <NUM>, <NUM> in case of an error.

It is also possible, in an alternative embodiment, to provide for glare removal at the hardware level <NUM> in an ISP, e.g., as part of a system-on-chip (SoC) for a dual-camera system. For example, various existing dual-camera systems already perform image alignment between two images in the ISP (to obtain images having a shared field of view). For example, the Snapdragon™ SoC from Qualcomm® has an on-chip ISP, Spectra <NUM>, which supports dual-camera systems. The Exynos® SoC from Samsung® also supports dual camera systems. In accordance with various embodiments, these and similar systems, if used with RGB and IR cameras, may be enhanced by further processing the images in the ISP to remove glare from the RGB images.

In yet another embodiment, glare removal as described herein (e.g., with reference to <FIG>) may be implemented in software at the application level <NUM>. This approach allows for the use of two individual cameras (one RGB and one IR camera), in place of a combined dual-camera system. Image processing software may receive raw YUV and IR data from the two cameras, and process them as needed to achieve the same resolution and a common field of view. For example, the image processing software may utilize image registration techniques to identify corresponding pixels in the two images. The image-processing method <NUM> for glare removal may then be applied to the properly aligned images.

Software implementation in principle also enables remote, cloud-based glare removal. For example, images from the RGB and IR cameras of a device connected to the Internet via a suitable wired or wireless communication channel may be sent to a remote server computer for glare removal, and the server may then return the processed, glare-free image. Such remote glare removal can be used when post-processing images, absent stringent time constraints. When processing video streams in real time, remote processing is feasible if communication rates are sufficient for real-time image transfer back and forth between the camera-equipped device and the server computer.

<FIG> is a block diagram of an example machine <NUM> with which glare removal in accordance with various embodiments may be performed. The machine may implement, in whole or in part, the method <NUM>, including the example image processing method <NUM>, and may form all or part of the system <NUM>. The machine <NUM> may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile telephone, a smartphone, a web appliance, a network router, switch or bridge, a server computer, a database, conference room equipment, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine.

The machine <NUM> may additionally include a storage device (e.g., drive unit) <NUM>, a signal generation device <NUM> (e.g., a speaker), and a network interface device <NUM>. Further, the machine <NUM> includes two or more cameras <NUM> to acquire still images or video. The machine <NUM> may include an output controller <NUM>, such as a serial (e.g., universal serial bus (USB), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection to communicate or control one or more peripheral devices (e.g., a printer, card reader, etc.).

The storage device <NUM> may include a machine-readable medium <NUM> on which are stored one or more sets of data structures or instructions <NUM> (e.g., software) embodying or utilized by any one or more of the techniques or functions described herein. In an example, one or any combination of the hardware processor <NUM>, the main memory <NUM>, the static memory <NUM>, or the storage device <NUM> may constitute machine-readable media.

While the machine-readable medium <NUM> is illustrated as a single medium, the term "machine-readable medium" may include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) configured to store the one or more instructions <NUM>.

The term "machine-readable medium" may include any medium that is capable of storing, encoding, or carrying instructions for execution by the machine <NUM> and that cause the machine <NUM> to perform any one or more of the techniques of the present disclosure, or that is capable of storing, encoding or carrying data structures used by or associated with such instructions. Nonlimiting machine-readable medium examples may include solid-state memories, and optical and magnetic media. Specific examples of machine-readable media may include: non-volatile memory, such as semiconductor memory devices (e.g., Electrically Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magnetooptical disks; Random Access Memory (RAM); Solid State Drives (SSD); and CD-ROM and DVD-ROM disks. In some examples, machine-readable media are non-transitory machine readable media. In particular, in some examples, machine-readable media are media other than transitory propagating signals.

The instructions <NUM> may further be transmitted or received over a communications network <NUM> using a transmission medium via the network interface device <NUM>. The machine <NUM> may communicate with one or more other machines utilizing any one of a number of transfer protocols (e.g., frame relay, internet protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc.). Example communication networks may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), mobile telephone networks (e.g., cellular networks), Plain Old Telephone (POTS) networks, and wireless data networks (e.g., Institute of Electrical and Electronics Engineers (IEEE) <NUM> family of standards known as Wi-Fi®, IEEE <NUM> family of standards known as WiMax®), IEEE <NUM>. <NUM> family of standards, a Long Term Evolution (LTE) family of standards, a Universal Mobile Telecommunications System (UMTS) family of standards, peer-to-peer (P2P) networks, among others. In an example, the network interface device <NUM> may include one or more physical jacks (e.g., Ethernet, coaxial, or phone jacks) or one or more antennas to connect to the communications network <NUM>. In an example, the network interface device <NUM> may include a plurality of antennas to wirelessly communicate using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) techniques. In some examples, the network interface device <NUM> may wirelessly communicate using Multiple User MIMO techniques.

Examples, as described herein, may include, or may operate on, logic or a number of components, modules, or mechanisms (all referred to hereinafter as "modules"). In an example, the software may reside on a machine-readable medium.

Claim 1:
A method (<NUM>) for generating a glare-free image, the method comprising:
acquiring an RGB image (<NUM>) of a scene in a visible wavelength range; and
acquiring an infrared, IR, image (<NUM>) of the scene in an IR wavelength range;
characterized in that it further comprises:
transforming (<NUM>) the RGB image (<NUM>) to a YUV image (<NUM>);
based on a luminance component (<NUM>) of the YUV image (<NUM>), identifying (<NUM>) a glare region (<NUM>) in the YUV image (<NUM>);
identifying (<NUM>), in the IR image (<NUM>), a corresponding region to the glare region (<NUM>) in the YUV image (<NUM>); and
substituting (<NUM>), in the luminance component (<NUM>) of the YUV image (<NUM>), pixel values in the glare region (<NUM>) with pixel values of the corresponding region (<NUM>) in the IR image (<NUM>).