Patent Description:
The present disclosure relates to a system for enhancing underexposed photographic images using a convolutional neural network. In particular, a method reducing required open shutter time, reducing required aperture size, or lowering sensor sensitivity, with minimal loss in image quality, is described.

Low light environments often do not provide enough available light to properly expose a camera sensor and provide a usable image. Such low light pictures can be improved by increasing sensor light sensitivity, increasing aperture size, extending exposure time, or providing flash or other artificial lighting. Unfortunately, each of these solutions have well known problems. Increasing sensor sensitivity amplifies sensor noise. Aperture size increase can require large, heavy, and costly lenses. Extending exposure time can result in motion blur from camera shake, shearing artifacts in rolling shutter systems, and object motion blur. Flash lighting can be expensive, difficult to deploy, and often results in unnatural appearing overexposure of people or objects.

A variety of computer processing techniques have been applied to improve appearance of low light images. For example, simple techniques such as histogram equalization and gamma correction can often increase brightness of dark regions with limited effect on bright regions. Denoising and deblurring can be used to respectively reduce noise and motion artifacts. Picture level analysis and processing, using for example the inverse dark channel prior for image dehazing, wavelet transform processing, or illumination map estimation, can all improve low light images.

Various attempts have also been made to salvage unsatisfactory or poorly shot low light camera images using machine intelligence post-processing. For example, a paper by <NPL>), describes the use of a fully convolutional neural network for direct single image processing of low light images.

For a more complex machine intelligence processing example, a paper by <NPL>), describes the use of low dynamic range images that are over or underexposed and processed using a neural network to synthesize a natural appearing high dynamic range image. Similarly, <NPL>, camera underexposure/overexposure correction with a dual neural network system to reconstruct poorly exposed images. <CIT> discloses methods, systems, and apparatus, including computer programs encoded on computer storage media, for image processing using deep neural networks.

Non-limiting and non-exhaustive embodiments of the present disclosure are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various figures unless otherwise specified.

All image sensing systems and sensors will have some associated noise that is created during typical operation. In noisy environments such as is common in low light environments (e.g. low environment lux, fast shutter, or small aperture) this noise will become a dominant portion of the digitized signal. Unfortunately, many traditional and modern computer vision algorithms (i.e. object or facial identification, visual odometry, visual SLAM, or image stabilization) can fail in high noise environments. Algorithms and systems that reduce image noise and recover image details are needed to enable these algorithms to work in environments where they would typically fail.

Reducing noise can also benefit machine intelligence based processing of images. Modern learning based algorithms work exceptionally well for those data distribution sets for which they have been trained on. When machine learning algorithms are presented with data outside this distribution, or when using adversarial examples, accuracy, speed, and other performance measures of these algorithms can suffer. If image sensor noise in images or data sets can be greatly reduced, learning algorithms processing the images or data sets will be subject to a lesser performance degradation.

Still another problem with noise is a result of noise being a high entropy information that is nearly incompressible. This means that compression ratios for a given system or sensor storage media will be greatly reduced for images capturing noisy environments or conditions. Compressed file sizes will typically end as much larger than the equivalent signal captured under normal operating conditions.

To reduce noise, improve image accuracy and detail recovery in normal or low light conditions, and improve compression ratios, a neural network can be used to recover the underlying signal. In effect, media preprocessed with a system and neural network such as disclosed herein can be improved in image quality and compressed to a greater degree, resulting in smaller file sizes, and reduced storage or bandwidth usage. Advantageously, even correctly exposed images benefit from this preprocessing step.

As seen in <FIG>, in one embodiment a system and method <NUM> for improving camera image capture low light capability and reducing noise using neural network processing relies on first determining an ideal exposure or range of exposures (step <NUM>). In a second step <NUM>, at least one purposely underexposed image is captured. In a third step <NUM>, this image is processed using neural network or other machine intelligence system to improve overall system quality. Finally, in step <NUM>, based on actual or estimated image quality, other camera operations can have functional parameters adjusted. For example, low ISO settings can be used instead of high ISO setting traditionally associated with low light conditions. As another example, in video applications, frame rates can be increased.

A wide range of still or video cameras can benefit from use of system and method <NUM>. Camera types can include but are not limited to conventional DSLRs with still or video capability, smartphone, tablet cameras, or laptop cameras, dedicated video cameras, webcams, or security cameras. In some embodiments, specialized cameras such as infrared cameras, thermal imagers, millimeter wave imaging systems, x-ray or other radiology imagers can be used. Embodiments can also include cameras with sensors capable of detecting infrared, ultraviolet, or other wavelengths to allow for hyperspectral image processing.

Cameras can be standalone, portable, or fixed systems. Typically, a camera includes processor, memory, image sensor, communication interfaces, camera optical and actuator system, and memory storage. The processor controls the overall operations of the camera, such as operating camera optical and sensor system, and available communication interfaces. The camera optical and sensor system controls the operations of the camera, such as exposure control for image captured at image sensor. Camera optical and sensor system may include a fixed lens system or an adjustable lens system (e.g., zoom and automatic focusing capabilities). Cameras can support memory storage systems such as removable memory cards, wired USB, or wireless data transfer systems.

In some embodiments, neural network processing can occur after transfer of image data to a remote computational resources, including a dedicated neural network processing system, laptop, PC, server, or cloud. In other embodiments, neural network processing can occur within the camera, using optimized software, neural processing chips, or dedicated FPGA systems.

In some embodiments, results of neural network processing can be used as an input to other machine learning or neural network systems, including those developed for object recognition, pattern recognition, face identification, image stabilization, robot or vehicle odometry and positioning, or tracking or targeting applications. Advantageously, such neural network processed image normalization can, for example, reduce computer vision algorithm failure in high noise environments, enabling these algorithms to work in environments where they would typically fail due to noise related reduction in feature confidence. Typically, this can include but is not limited to low light environments, foggy, dusty, or hazy environments, or environments subject to light flashing or light glare. In effect, image sensor noise is removed by neural network processing so that later learning algorithms have a reduced performance degradation.

In certain embodiments, multiple image sensors can collectively work in combination with the described neural network processing to enable wider operational and detection envelopes, with, for example, sensors having different light sensitivity working together to provide high dynamic range images. In other embodiments, a chain of optical or algorithmic imaging systems with separate neural network processing nodes can be coupled together. In still other embodiments, training of neural network systems can be decoupled from the imaging system as a whole, operating as embedded components associated with particular imagers.

Exposure can be defined as a function of the number of photons collected at a sensor's electron wells and the sensor's quantum efficiency. The number of photons collected is primarily affected by the scene lighting, the aperture of the optical path, and the exposure time. Well-exposedness has been defined as being close to the middle of the range of an analog to digital signal. Determining a particular exposure or range of exposures can be done manually, semiautomatically with manual input, or automatically. Typically, exposure is determined by setting one or more of aperture, ISO, or shutter speed, but other modifications affecting exposure time such as neutral or polarizing filter insertion are also possible. Once an ideal or range of exposure settings is determined or set, the camera inputs (e.g. aperture, ISO, or shutter speed) are purposely adjusted to underexpose the image sensor. Underexposure can involve reducing aperture, decreasing ISO, or increasing shutter speed. Assuming other exposure related settings are maintained constant, an aperture-controlled underexposure would involve increasing the f-stop by two or more, an ISO underexposure would be set to half or less of an ideal setting (e.g. ISO <NUM> set to ISO <NUM>), and shutter speed could be doubled or more (e.g. <NUM>/<NUM> second set to <NUM>/<NUM> of a second). As will be appreciated, various combinations of these settings can be simultaneously adjusted.

Various types of neural networks can be used, including recurrent, generative adversarial, or deep convolutional networks. Convolutional neural networks are particularly useful for image processing applications such as described herein. As seen with respect to <FIG>, a convolutional neural network <NUM> can receive a single underexposed RGB image <NUM> as input. RAW formats are preferred, but compressed JPG images can be used with some loss of quality. Images can be pre-processed with conventional pixel operations or can preferably be fed with minimal modifications into a trained convolutional neural network <NUM>.

Processing proceeds through one or more convolutional layers <NUM>, pooling layer <NUM>, a fully connected layer <NUM>, and ends with RGB output <NUM> of the improved image. In operation, one or more convolutional layers apply a convolution operation to the RGB input, passing the result to the next layer(s). After convolution, local or global pooling layers can combine outputs into a single or small number of nodes in the next layer. Repeated convolutions, or convolution/pooling pairs are possible. Before output, the fully connected layer <NUM> connect every node in one layer to every node in another layer.

One neural network embodiment of particular utility is a fully convolutional neural network. A fully convolutional neural network is composed of convolutional layers without any fully-connected layers usually found at the end of the network. Advantageously, fully convolutional neural networks are image size independent, with any size images being acceptable as input for training. An example of a fully convolutional network <NUM> is illustrated with respect to <FIG>. Data can be processed on a contracting path that includes repeated application of two 3x3 convolutions (unpadded convolutions), each followed by a rectified linear unit (ReLU) and a 2x2 max pooling operation with stride <NUM> for down sampling. At each down sampling step, the number of feature channels is doubled. Every step in the expansive path consists of an up sampling of the feature map followed by a 2x2 convolution (up-convolution) that halves the number of feature channels, provides a concatenation with the correspondingly cropped feature map from the contracting path, and includes two 3x3 convolutions, each followed by a ReLU. The feature map cropping compensates for loss of border pixels in every convolution. At the final layer a 1x1 convolution is used to map each <NUM>-component feature vector to the desired number of classes. While the described network has <NUM> convolutional layers, more or less convolutional layers can be used in other embodiments. Training can include processing input images with corresponding segmentation maps using stochastic gradient descent techniques.

Other embodiments of systems and methods that rely on neural network processing can also be employed. As seen with respect to <FIG>, a procedure <NUM> involves use Bayer image data <NUM> that can be specific to particular sensors or types of sensors. Neural network processing <NUM> is used to denoise the data <NUM> and provide a denoised Bayer image <NUM>. As will be understood, training and operation of the neural network processing can also be specific to particular sensors or types of sensors used to create data <NUM>.

In another embodiment seen with respect to <FIG>, a procedure <NUM> involves use Bayer image data <NUM> that can be specific to particular sensors or types of sensors. Neural network processing <NUM> is used to denoise the data <NUM> and provide a denoised RGB image <NUM>. As will be understood, training and operation of the neural network processing can also be specific to particular sensors or types of sensors used to create data <NUM>.

In another embodiment seen with respect to <FIG>, a procedure <NUM> involves use RGB image data <NUM> that can be specific to particular sensors or types of sensors. Neural network processing <NUM> is used to denoise the data <NUM> and provide a denoised RGB image <NUM>. As will be understood, training and operation of the neural network processing can also be specific to particular sensors or types of sensors used to create data <NUM>.

In another embodiment seen with respect to <FIG>, a procedure <NUM> involves use tensor data <NUM> that can be specific to particular sensors or types of sensors. Neural network processing <NUM> is used to denoise the data <NUM> and provide a denoised tensor data <NUM>. As will be understood, training and operation of the neural network processing can also be specific to particular sensors or types of sensors used to create data <NUM>. In some embodiments, non-optical sensors or systems, including millimeter radar systems, mapping pressure sensors, or other suitable sensors providing tensor data sets can be used.

<FIG> illustrates one embodiment of an imaging pipeline <NUM> for improving image data. Factors that affect analog processing of an image include scene lighting <NUM>, optical path and aperture <NUM>, and features of an image sensor <NUM>. Many of these factors can be automatically adjusted or adjusted to favor factors that will improve efficacy of later neural network processing. For example, flash or other scene lighting can be increased in intensity, duration, or redirected. Filters can be removed from an optical path, apertures opened wider, or shutter speed decreased. Image sensor efficiency or amplification can be adjusted by ISO selection.

In one embodiment, low light images can be captured by increasing one or more of these analog factors prior to analog to digital conversion. Noise or other unwanted artifacts can be removed by later neural network processing <NUM> after analog to digital conversion <NUM> and conversion into a suitable data structure <NUM> such as Bayer derived, RGB, RAW, TIFF, JPG, or the like. For example, a Bayer derived data structure could be defined to stack the color channels depthwise, such that the resulting dimensions are halved spatially and quadrupled depthwise
Image signal processing using an image signal processor <NUM> can include additional digital scaling, tone mapping, pixel correction, demosaicing, dehazing, or the like. In some embodiments, neural network processing can run on the image signal processor <NUM>, while in others a separate processing component can be used. A processed image can be stored, transferred, displayed, classified, encoded, or provided for any other suitable intermediate or end use <NUM>.

<FIG> illustrates a system <NUM> for training neural networks that includes a control and storage module <NUM> able to send respective control signals to an imaging system <NUM> and a display system <NUM>. The imaging system <NUM> can supply processed image data to the control and storage module <NUM>, while also receiving profiling data from the display system <NUM>.

Training neural networks in a supervised or semi-supervised way requires high quality training data. To obtain such data, a system <NUM> provides automated imaging system profiling. The control and storage module <NUM> contains calibration and raw profiling data to be transmitted to the display system <NUM>. Calibration data may contain, but is not limited to, targets for assessing resolution, focus, or dynamic range. Raw profiling data may contain, but is not limited to, natural and manmade scenes captured from a high quality imaging system (a reference system), and procedurally generated scenes (mathematically derived).

An example of a display system <NUM> is a high quality electronic display. The display can have its brightness adjusted or may be augmented with physical filtering elements such as neutral density filters. An alternative display system might comprise high quality reference prints or filtering elements, either to be used with front or back lit light sources. In any case, the purpose of the display system is to produce a variety of images, or sequence of images, to be transmitted to the imaging system.

The imaging system being profiled is integrated into the profiling system such that it can be programmatically controlled by the control and storage computer and can image the output of the display system. Camera parameters, such as aperture, exposure time, and analog gain, are varied and multiple exposures of a single displayed image are taken. The resulting exposures are transmitted to the control and storage computer and retained for training purposes.

The entire system is placed in a controlled lighting environment, such that the photon "noise floor" is known during profiling.

The entire system is setup such that the limiting resolution factor is the imaging system. This is achieved with mathematical models which take into account parameters, including but not limited to: imaging system sensor pixel pitch, display system pixel dimensions, imaging system focal length, imaging system working f-number, number of sensor pixels (horizontal and vertical), number of display system pixels (vertical and horizontal). In effect a particular sensor, sensor make or type, or class of sensors can be profiled to produce high-quality training data precisely tailored to an individual sensors or sensor models.

<FIG> illustrates one embodiment of a neural network system <NUM> whose parameters can be manipulated such that they produce desirable outputs for a set of inputs and are capable of improving imaging quality for noisy or low light image data such as previously described. One such way of manipulating a network's parameters is by "supervised training". In supervised training, the operator provides source/target pairs <NUM> and <NUM> to the network and, when combined with an objective function, can modify some or all the parameters in the network system <NUM> according to some scheme (e.g. backpropagation).

In the described embodiment of <FIG>, high quality training data (source <NUM> and target <NUM> pairs) from various sources such as a profiling system, mathematical models and publicly available datasets, are prepared for input to the network system <NUM>. The method includes data packaging target <NUM> and source <NUM>, and preprocessing lambda target <NUM> and source <NUM>.

Data packaging takes one or many training data sample(s), normalizes it according to a determined scheme, and arranges the data for input to the network in a tensor. Training data sample may comprise sequence or temporal data.

Preprocessing lambda allows the operator to modify the source input or target data prior to input to the neural network or objective function. This could be to augment the data, to reject tensors according to some scheme, to add synthetic noise to the tensor, to perform warps and deformation to the data for alignment purposes or convert from image data to data labels.

The network <NUM> being trained has at least one input and output <NUM>, though in practice it is found that multiple outputs, each with its own objective function, can have synergetic effects. For example, though the overall objective of the system is to reduce image noise, noise reduction performance can be improved through a "classifier head" output whose objective is to classify objects in the tensor. Target output data <NUM>, source output data <NUM>, and objective function <NUM> together define a network's loss to be minimized, the value of which can be improved by additional training or data set processing.

<FIG> is a graph <NUM> illustrating signal to noise ratio versus total system gain for an embodiment. As will be apparent from the graph <NUM>, total system gain or amplification can be increased at the cost of increasing noise and decreasing the signal to noise ratio. Amplification of a signal can occur by analog amplification (or gain), or by digital amplification. In graph <NUM>, three different instances i, ii, and iii of analog amplification followed by digital amplification are indicated. Instance i shows limited analog amplification followed by digital amplification. Instance ii illustrates an increase in analog amplification, while instance iii shows digital amplification only an even greater increase in analog amplification. The best total system gain is provided by maximizing achievable analog gain, and only then providing digital gain. This supports one embodiment of the present disclosure, in which total system gain is improved by first increasing or maximizing analog gain, followed by neural network processing and then digital gain improvements.

The described method and system can provide various benefits for many applications, including:.

As will be understood, the camera system and methods described herein can operate locally or in via connections to either a wired or wireless connect subsystem for interaction with devices such as servers, desktop computers, laptops, tablets, or smart phones. Data and control signals can be received, generated, or transported between varieties of external data sources, including wireless networks, personal area networks, cellular networks, the Internet, or cloud mediated data sources. In addition, sources of local data (e.g. a hard drive, solid state drive, flash memory, or any other suitable memory, including dynamic memory, such as SRAM or DRAM) that can allow for local data storage of user-specified preferences or protocols. In one particular embodiment, multiple communication systems can be provided. For example, a direct Wi-Fi connection (<NUM>. 11b/g/n) can be used as well as a separate <NUM> cellular connection.

Connection to remote server embodiments may also be implemented in cloud computing environments. Cloud computing may be defined as a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned via virtualization and released with minimal management effort or service provider interaction, and then scaled accordingly. A cloud model can be composed of various characteristics (e.g., on-demand self-service, broad network access, resource pooling, rapid elasticity, measured service, etc.), service models (e.g., Software as a Service ("SaaS"), Platform as a Service ("PaaS"), Infrastructure as a Service ("IaaS"), and deployment models (e.g., private cloud, community cloud, public cloud, hybrid cloud, etc.).

Claim 1:
A method (<NUM>) for image capture, comprising the steps of:
determining an exposure range;
setting at least one camera parameter of a camera to capture an underexposed image (<NUM>) outside the exposure range;
processing the underexposed image (<NUM>) using a neural network to recover image details, characterized in that the neural network is a neural network trained using a profile of the camera as an input, the profile obtained using mathematical models with an imaging system including the camera being in a controlled lighting environment such that a photo noise floor is known.