Patent Description:
The present invention provides methods, systems, devices and computer software/program code products that enable the foregoing aspects and others.

Aspects, examples, embodiments and practices of the invention, whether in the form of methods, devices, systems or computer software/program code products will next be described in greater detail in the following Detailed Description of the Invention, in conjunction with the attached drawing figures.

Those skilled in the art will appreciate that while the following detailed description provides sufficient detail to enable one skilled in the art to practice the present invention, the various examples, embodiments and practices of the present invention that are discussed and described below, in conjunction with the attached drawing figures, are provided by way of example, and not by way of limitation. Numerous variations, additions, and other modifications or different implementations of the present invention arc possible, and arc within the scope of the invention. <NPL> proposes a content-aware noise reduction (NR) method for the Digital TV, which preserves image textures and edges. In the proposed method, noise is first reduced by a temporal recursive NR filter and then further suppressed by a spatial NR filter. To preserve the image texture, the saliency map which accurately represents edges in a noisy image is employed in the spatial filter.

The present invention provides methods, systems, devices, and computer software/program code products suitable for a wide range of applications, including, but not limited to: facilitating video communications and presentation of image and video content in telecommunications applications; and facilitating video communications and presentation of image and video content for virtual reality (VR), augmented reality (AR) and head-mounted display (HMD) systems.

Methods, systems, devices, and computer software/program code products in accordance with the invention are suitable for implementation or execution in, or in conjunction with, commercially available computer graphics processor configurations and systems including one or more display screens for displaying images, cameras for capturing images, and graphics processors for rendering images for storage or for display, such as on a display screen, and for processing data values for pixels in an image representation. The cameras, graphics processors and display screens can be of a form provided in commercially available smartphones, tablets and other mobile telecommunications devices, as well as in commercially available laptop and desktop computers, which may communicate using commercially available network architectures including client/server and client/network/cloud architectures.

In the aspects of the invention described below and hereinafter, the algorithmic image processing methods described are executable by digital processors, which can include graphics processor units, including GPGPUs such as those commercially available on cellphones, smartphones, tablets and other commercially available telecommunications and computing devices, as well as in digital display devices and digital cameras. Those skilled in die art to which this invention pertains will understand the structure and operation of digital processors, GPGPUs and similar digital graphics processor units.

One aspect of the present invention relates to methods, systems and computer software/program code products for generating reduced-noise image frames based on image frames received from a digital camera pipeline, as defined in claims <NUM>, <NUM> and <NUM> respectively.

In another aspect of the invention, the noise model comprises a simulation of the inverse behavior of at least selected aspects of the digital camera pipeline.

In another aspect of the invention, the noise model comprises a simulation of the inverse of the behavior of a variable gain amplifier.

In another aspect of the invention, the noise model comprises a simulation of the inverse of the behavior of automatic white-balance logic.

In another aspect of the invention, the noise model comprises a simulation of the inverse of the behavior of automatic exposure control.

In another aspect of the invention, the noise model comprises a simulation of the number of electrons in an electron well associated with a light sensing element of the digital camera pipeline.

In another aspect of the invention, the persistent image buffer comprises a summation of data from prior frames.

In another aspect of the invention, the persistent image buffer comprises an average of data from prior frames.

In another aspect of the invention, at least one pixel in the persistent image buffer is invalidated in response to a selected result of the evaluation of the noise model.

In the re-projection is a selected image warp in response to camera rotation.

An example not covered by the claims relates to methods, systems and computer software/program code products for executing stereo image search between corresponding images generated by at least two cameras.

In one such aspect, a system according to the invention for executing stereo image search between corresponding images generated by at least two cameras comprises:.

In another aspect of the invention, the temporal de-noising pipelines comprise processing modules; the processing modules comprising software modules executing on computational resources.

In another aspect of the invention, the software modules execute on shared computational resources.

In another aspect of the invention, the temporal de-noising pipelines comprise dedicated electronic components.

In an example not covered by the claims, at least two cameras participate in a shutter synchronization protocol.

Another aspect of the invention relates to systems for generating reduced-noise image frames based on image frames received from a digital camera pipeline, as defined in claim <NUM>.

Another aspect of the invention relates to a computer software/program products for use with a digital processing system to enable the digital processing system to generate reduced-noise image frames based on image frames received from a digital camera pipeline as defined in claim <NUM>.

Another aspect of the invention relates to methods for executing stereo image search between corresponding images generated by at least two cameras, comprising:.

An example not covered by the claims relates to a program product for use with a digital processing system to enable the digital processing system to execute stereo image search between corresponding images generated by at least two cameras, the digital processing system comprising a digital processing resource, the digital processing resource comprising at least one digital processor, the program product comprising digital processor-executable program instructions stored on a non-transitory digital processor-readable medium, which when executed in the digital processing resource cause the digital processing resource to:.

Another aspect of the invention further comprises: utilizing a persistent data store to retain information representing the number of valid samples.

Another aspect of the invention further comprises: utilizing the persistent data store to retain information estimating the number of electrons or photons that have been integrated into corresponding data.

Another aspect of the invention further comprises: utilizing information from the noise model to evaluate pixel data to quantify the likelihood that a measured variation in the pixel data is due to noise rather than a change in observed situation.

Another aspect of the invention further comprises: applying an output from the noise model to a video signal, to improve suitability of the signal as an input to a stereo disparity search operation.

In another aspect of the invention, the noise model is a source for blending information useable in executing the blending.

The present invention enables the features described herein to be provided at reasonable computational cost, and in a manner easily accommodated within the digital processing capabilities and form factors of modern mobile devices such as tablets and smartphones, as well as the form factors of laptops, PCs, computer-driven televisions, computer-driven projector devices, and the like, does not dramatically alter the economics of building such devices, and is viable within current or near-current communications network/connectivity architectures.

These and other aspects, examples, embodiments and practices of the invention, whether in the form of methods, devices, systems or computer software/program code products, will be discussed in greater detail below in the following Detailed Description of the Invention and in connection with the attached drawing figures.

Those skilled in the art will appreciate that while the following detailed description provides sufficient detail to enable one skilled in the art to practice the present invention, the various examples, embodiments and practices of the present invention that are discussed and described below, in conjunction with the attached drawing figures, are provided by way of example, and not by way of limitation.

The present invention relates to methods, systems, devices and computer program code products (software) for improving the temporal stability of a sequence of images captured by a camera (or other sensor) by employing a model of the properties of the expected noise. Exemplary embodiments and practices of the invention thus provide temporal de-noising using a computational model for sensor noise. Aspects, embodiments and practices of the invention can be useful for recognizing whether a change in a portion of an image is due to measurement instability, or due to a meaningful change in the observed situation.

The invention can also be employed to improve the suitability of an image sequence for devices, systems, methods and algorithms that are sensitive to variations between frames, such as motion detection for small objects, or for devices, systems, methods and algorithms that perform comparisons between multiple cameras observing the same scene, such as stereo correspondence detection, as may be used in Virtual <NUM>-Dimensional ("V3D") systems.

A camera, light sensor or light sensing element (e.g., a pixel in an image produced by a digital camera), observing a static and unchanging scene, should, in theory, observe an identical value during each equal length time interval of observation. (These time intervals may represent, for example, frames captured by a camera continually observing a scene. ) In practice, however, this does not occur.

Instead, the intensity of the light produced by the aggregation of the observed photons, as reported by the light sensor, may vary greatly across a small time interval. This manifests as random differences observed in some pixels from one frame of video to the subsequent frame. These random fluctuations create temporal instability whereby an identical scene captured with an identical camera produces a slightly different image from one frame to the next. The shorter the time interval, or the lower the sensor's dynamic range, the more problematic the random fluctuations become.

The variation in values produced by the sensor from one time interval to the next can be partially attributed to the discrete or particle nature of light. This effect is called "Shot noise," or "Schott noise," reflecting its elucidation by physicist Walter Schottky. In particular, the photons detected from one image frame to the next will not be identical due to the discrete quantum behavior of individual photons. Shot noise is most visible in photographs taken in low light conditions. Often, this is due to amplification of very limited data by an image processing pipeline, which may include various configurations of light sensors or cameras, signal amplifiers and other signal processing modules and/or algorithms. Such a pipeline might exist in a digital camcorder, cell phone camera, security camera system, or other digital camera device or system.

In some sensor designs, photons interact with light sensitive electronic devices or elements, causing electric charge to build up in a capacitive element known as an electron well. To retrieve the value for the luminance observed by the sensing element, the charge levels are read and then encoded as a digital number using an analog to digital converter. In many instances, the "read" process and the analog to digital conversion process can further introduce error. This source of error is often referred to as "read noise. " Read noise and shot noise combine with other noise sources to produce fluctuating pixel values from a camera pipeline.

See, for example, <FIG>, which is a schematic diagram depicting the effects of quantum variations in light distributions on the measured values of light intensities from identical sensors observing an identical scene across equal-length time intervals. Compare, for example, the differences in measured values of Red and Blue intensities between Frame <NUM> and Frame <NUM>, even with no change in relative positions or orientations of the light source, objects, and sensors.

In particular, <FIG> depicts a configuration <NUM> of light source, objects and sensors, across two imaging frames, Frame <NUM> (left-hand side of <FIG>) and Frame <NUM> (right-hand side of <FIG>). The collective configuration <NUM> includes a blue object <NUM>, a red object <NUM>, a light source <NUM>, a blue sensing element <NUM> and a red sensing element <NUM>. In Frame <NUM>, exemplary light rays <NUM>, <NUM>, <NUM>, and <NUM> emanate from light source <NUM>, with rays <NUM>, <NUM> and <NUM> striking blue object <NUM>. and ray <NUM> striking red object <NUM>. In Frame <NUM>, rays <NUM>, <NUM> and <NUM> reflect from blue object <NUM> and are detected by blue sensor <NUM>, yielding a Blue Intensity of <NUM> for Frame <NUM>; and ray <NUM> reflects from red object <NUM> and is detected by red sensor <NUM>, yielding a Red Intensity of <NUM> for Frame <NUM>.

In Frame <NUM>, rays <NUM>, <NUM> and <NUM> emanate from light source <NUM>, with ray <NUM> striking blue object <NUM>, and rays <NUM> and <NUM> striking red object <NUM>. In Frame <NUM>, ray <NUM> reflects from blue object <NUM> and is detected by blue sensing element <NUM>: and rays <NUM> and <NUM> reflect from red object <NUM> and are detected by red sensing element <NUM>, yielding a Blue Intensity of <NUM> and a Red Intensity of <NUM> for Frame <NUM>.

With no change in the relative positions or orientations of the light source, objects and sensors from Frame <NUM> to Frame <NUM>, <FIG> illustrates that this difference can be attributable to the effects of quantum variations in light distribution.

By way of example, a typical modern cell-phone sensor (camera) may capture as few as several thousand individual electrons in the electron well of a sensing element during the capture of an <NUM> millisecond image frame. In this exemplary sensor, the value is then encoded as an <NUM> bit number ranging from <NUM> to <NUM> levels of intensity. This means only a handful of electrons represent the difference between one digital intensity level value and the adjacent value.

If the sensor or camera pipeline applies a gain function, this can exacerbate the instability by adding additional error, and by magnifying error from other sources. For example, if the gain is applied using analog electronics, then more noise may be introduced. If the gain is applied to the digital data, quantization error may occur before a low precision number is multiplied.

In the sensor of the example above, the frame-to-frame values for a single pixel may fluctuate by as much as <NUM>% of the total measured value, even with absolutely no change to the external situation. This includes a perfectly static camera orientation, unchanged camera setting and configuration, and no changes to external objects and lighting. This means that a measurement may only be accurate to plus or minus <NUM> values from a possible <NUM> representable values. Effectively this means that, for a single pixel from a single frame in the worst case, an accurate and stable value for a pixel can only be known with certainty among <NUM> possible intensity values, each value representing a large range of light intensity.

In a typical instance, the effect is not as extreme as in the worst case. The millions of pixels across a sensor, viewed in aggregate, tend to average out most of the noise effects when performing simple photography operations. This leads to a visually acceptable image in spite of extreme fluctuations in the values from individual sensing elements.

However, the Applicants have observed that beyond simple photography operations intended to produce an image for direct viewing, the fluctuations and noise may be very problematic for devices, methods, processes and algorithms rely on temporal stability from one image frame to the next. This problem is made worse when small parts of an image or individual pixels are examined individually. Such an examination is executed, for example, in connection with stereo correspondence operations.

While the photons, and therefore light intensity readouts, observed from one time interval to the next will have inherent variance, the variance from one time interval to the next will statistically cancel out as the amount of measurement increases. The noise will represent an increasingly less significant proportion of the value as the exposure time increases, provided the sensor has adequate dynamic range to capture all photons during the exposure. Similarly, the average of many time intervals or frames should eventually converge on a true and correct result if the conditions of the observation have indeed not changed. As Schottky observed, the error should diminish in proportion to the square root of the amount of measurement.

Given this, the Applicants have determined that in accordance with the present invention, it is possible to construct a statistical model of the sensor that can be used to quantify the reliability of a measured value, given information about the sensor properties, and additional factors used by the sensor pipeline to construct the image, such as the black level, the sensor gain levels, exposure time and other parameters.

In accordance with the present invention, the model can be designed to emulate the inverse of the behavior of the sensor pipeline's electronics.

In accordance with the present invention, the model can be designed to take a pixel value as input, and estimate the number of electrons in the electron well for a sensing element, and therefore the number of photons that reached that sensing element during the frame capture interval.

Exemplary embodiments and practices of the invention utilize predictions from the above-described statistical model to produce a reduced-noise ("de-noised") image or signal, by selectively incorporating data from earlier measured intervals or frames. By incorporating data from multiple frames, it is possible to arrive at a more accurate solution than could be obtained from any one frame alone.

Exemplary embodiments and practices of the invention utilize a persistent data store to retain and integrate data from previous frames or time intervals.

Exemplary embodiments and practices of the invention can also use this data store to retain additional information, such as the number of valid samples, or the number of electrons or photons or microseconds that have been integrated into the data; and/or can use other meta-data representing the quality of the data.

In addition, in exemplary embodiments and practices of the invention, information from the model can be used to quantify the likelihood that a measured variation is due to noise, versus when the variation is indicative of a change in the observed situation.

The invention can be applied to a video signal to improve the suitability of that signal to be used as an input to a stereo disparity search operation.

The Applicants have recognized that even for frames captured with a comparatively large time interval, the temporal instability in pixel values can be problematic for systems, devices, methods or algorithms that rely on the identification of identical portions of a scene or image, such as stereo image comparison. In many instances, the temporal variance in raw pixel values is not apparently visible when viewing the image directly. This is partially due to bright pixels reaching a saturation threshold or a tone mapping operation effectively reducing the dynamic range for bright pixels. However, the detail present in the raw values representing the bright pixels is very significant to achieving accurate stereo correspondence.

The absolute amount of photon variance due to shot noise is proportional to the absolute number of photons observed. In other words, shot noise occurs irrespective of brightness, so the proportion of noise will have similar properties in bright pixels as well as dark pixels; but algorithms that rely on comparison or subtraction of raw pixel intensity values will detect more variance in bright pixels than in dim pixels with the same amount of noise.

Various embodiments and practices of the present invention may be implemented within a sensor/camera pipeline, including camera pipelines of otherwise convention design, or they may be implemented in the form of a software or hardware module that consumes the output from one or more traditional sensor or camera pipelines in order to produce higher quality and more temporally stable image data and images.

<FIG> depicts an exemplary pipeline, in accordance with embodiments and practices of the present invention, for implementing de-noising to a sequence of images produced by an otherwise conventional camera pipeline.

As shown in <FIG>, in an exemplary practice or embodiment of the invention, frames of image data are produced by a camera connected to an image processing pipeline <NUM>. For each frame, the camera pipeline outputs a <NUM>-dimensional (2D) matrix of digital pixel value information (captured image buffer <NUM>), along with image metadata <NUM>.

Captured image buffer <NUM> is also referred to herein as a frame buffer or camera frame buffer. Note that while <FIG> depicts captured image buffer or camera frame buffer <NUM> as a single block, the functions of the captured image buffer can be divided between two or more elements, which may in some embodiments include elements in camera capture pipeline <NUM>. <FIG> is merely schematic in nature, and many variations thereto are possible, and are within the scope of the present invention.

In some embodiments or practices, the captured image buffer data can represent, can comprise, or be constituted by, raw captured pixel information, prior to any tone mapping operation or other operation that further reduces the sensor's dynamic range.

In some embodiments or practices, the image metadata can include exposure information from an auto-exposure feature in the camera pipeline, and/or white-balance information from an automatic white-balance feature, which can be generated by otherwise conventional camera pipelines.

The image meta-data may also include black level information, variable gain amplification information, exposure time information, and additional information produced by the otherwise conventional camera or imaging pipeline. The information may describe external observations made by sensors including the camera itself, or it may describe the magnitude or presence of actions, such as filter operations, performed by the imaging pipeline.

In accordance with exemplary practices and embodiments of the present invention, Sensor Model <NUM> is operable to consume the captured image buffer <NUM> and the image meta-data <NUM>, and apply selected parameters and logic operations, examples of which are described in greater detail below, to estimate the proportion of noise contained within the value for each pixel in image buffer <NUM>. The sensor model may consume data from the captured image buffer by reading it from a memory.

In an exemplary practice or embodiment of the invention, the sensor model comprises a mathematical model designed to approximate the inverse behavior of the camera capture pipeline <NUM>.

Further in accordance with exemplary practices and embodiments of the present invention, the captured pixel buffer <NUM> and the output of the sensor model <NUM> is made available to de-noising logic <NUM>. The de-noising logic <NUM> produces de-noised output frames <NUM>, in accordance with the present invention, by blending data from the captured image buffer <NUM> and a persistent image buffer <NUM>. The blending function for each pixel in the image may depend on the output of sensor model <NUM>.

In addition, in accordance with exemplary practices and embodiments of the present invention, data from image buffer <NUM> may be integrated into persistent image buffer <NUM>, and metadata in the persistent image metadata storage <NUM> may be updated dynamically. In this manner, as more frames are processed, the data in the persistent image buffer <NUM> and the persistent image metadata <NUM> become increasingly reliable, and therefore the noise and variance in successive output frames will decrease.

The respective formats of the persistent image data in persistent image buffer <NUM> and the frame buffer or captured image buffer <NUM> need not be the same. By way of example, the persistent image data does not need to be a rectangular matrix that resembles a conventional frame buffer, but instead could be a cache indexed by a 2D or 3D spatial position. A <NUM>-dimensional (3D), or multi-layered matrix could also be used, and this would enable persistent information to be retained about areas that are temporarily occluded, while also tracking information about the objects that are occluding them.

Also, there need not be a one-to-one correspondence between frame buffer pixels and an element in the persistent image buffer. A number of types of correspondence, along a continuum of types of correspondence, are possible and are within the scope of the present invention. These types of correspondence can include, but are not limited to, the following:.

<FIG> depicts another exemplary embodiment of the invention, in which de-noised output frames are used to perform stereo correspondence analysis in a multi-camera system. In the embodiment depicted by way of example in <FIG>, the output from two cameras is examined for stereo correspondence by Stereo Correspondence Search module <NUM>. Prior to that, frames are generated by camera pipelines <NUM> and <NUM> in parallel. These frames are made available, along with all appropriate frame metadata, to de-noising pipelines <NUM> and <NUM>. It should be noted that while these pipelines are conceptually separate, they may share some hardware components in their implementation. <FIG> is merely schematic in nature, and many variations thereto are possible, and are within the scope of the present invention.

Referring again to <FIG>, de-noising pipelines <NUM> and <NUM> each output de-noised frame information, <NUM> and <NUM>, respectively. This information may be expressed as raw pixel values without any tone mapping function or saturations thresholds applied. De-noised frame information or images <NUM> and <NUM> are then made available to Stereo Correspondence Search module <NUM> to perform a stereo correspondence search. This module, and the stereo correspondence search operations, can be executed in accordance with the teachings of the Applicants' commonly-owned patent applications listed above and incorporated by reference herein.

In an exemplary practice or embodiment of the present invention, a computational model constructed in accordance with the invention will computationally attempt to estimate the number of electrons in the electron wells for each pixel, based on the digital pixel values from the captured image buffer. To do this, the model may utilize a known electron well capacity, which is an intrinsic property of a camera sensor. This value may be adjusted by a calibration process. The calibration process may have already been performed as part of an initialization routine, or as part of a device manufacturing process.

In an exemplary practice or embodiment of the present invention, the effects of a variable-gain amplifier within the camera image pipeline are also factored into the model. Because any noise that is present in the signal will be amplified by the amplification process, the model should reflect this effect. In many instances, an amplifier in a camera image pipeline applies a logarithmic gain function to match human perception. In accordance with an exemplary practice of the invention, the sensor model computationally applies the inverse of the function applied by the camera pipeline function.

In many instances, a conventional camera pipeline and its modules will provide an indication of the gain from the amplification step applied to the camera, so the computational model of the present invention can utilize or consume this indicated value as an input parameter. (This value is typically provided in decibels.

Similarly, in many instances, a conventional camera pipeline and its modules will provide an indication of the black level. This too, can be used, in accordance with an exemplary practice of the present invention, as an input parameter to the inverse sensor model.

<FIG> depicts an example of a computational noise model in accordance with the present invention, incorporating an inverse sensor model, written in the Visual Basic programming language.

As shown in <FIG>, an example of a computational noise model in accordance with the present invention, incorporating an inverse sensor model, is as follows:
<IMG>.

A larger electron well capacity and a sufficient exposure time will translate to a corresponding reduction in noise. This basic property exists regardless of the physical and electrical technologies employed by the sensor. In any light sensing element, an increase in the sensor's dynamic range combined with an exposure time interval sufficient to make use of the sensor's range will result in a corresponding decrease in noise. The reduction in noise will follow the inverse square root relationship described by Schottky.

Similarly, the noise will also decrease if multiple samples, i.e. frames, of data are integrated. Additional samples will also decrease noise following the inverse square root relationship.

In exemplary practices and embodiments of the present invention, de-noising logic is operable to store persistent data associated with pixels. In many instances, this persistent data may take the form of an average pixel value. Alternatively, this data may be a sum of pixel values over an interval of frames, or it may be another aggregation of pixel values across time.

In an exemplary embodiment, an average pixel value is stored for each pixel using a cumulative average. For each pixel in each captured frame, the captured pixel value is added to the persistent pixel value, and the effective sample count for the persistent pixel is incremented. Then, a pixel value for a de-noised frame can be produced by dividing the sum of pixel values in the persistent pixel by the associated effective sample count for that pixel.

An exemplary embodiment of the present invention can include the persistent tracking of the statistical deviation for each pixel value from one captured frame to the next.

In another embodiment, a running average is used. In this embodiment, the standard deviation for a given pixel can be used to determine the proportion of weight to assign to the newly captured pixel value, and the proportion of weight to assign to the previously captured values, for the purpose of blending the captured pixel value and the persistent pixel value to create a de-noised output pixel value. Alternatively, the noise model may be a source for this blending information.

In yet another embodiment, the data pertaining to a captured pixel is blended with data from a corresponding persistent pixel, where the blending ratios are influenced by metadata from the corresponding captured frame. For example, a frame for which a high gain amplification has been applied would contribute less to a blended value than a frame for which less amplification was applied.

Similarly, various practices or embodiments of the present invention can normalize the data from the pixels with respect to values from the captured frame meta-data, prior to blending.

The Applicants have recognized that in some instances, the captured pixel value will strongly disagree with the associated persistent pixel value. This disagreement may occur if a captured pixel value is outside of an acceptable deviation associated with a persistent pixel value. Alternatively, it may occur if a persistent pixel value is outside of the error range predicted by the noise model. Such disagreement is often indicative of a change in the observed situation. In such an instance, an exemplary embodiment of the present invention may wholly or partially invalidate the associated persistent pixel data and meta-data.

In some embodiments of the invention, the persistent data may be compressed. This compression may be a lossless compression, or a lossy compression. The compression scheme may also utilize the observation that spatially nearby pixels often have similar values.

The Applicants have implemented various embodiments and practices of the present invention, and such implementations have been observed to produce excellent results when the camera is relatively still, as might occur in a number of practical applications and devices, such as a security camera system, or a stationary video conferencing appliance.

However, camera motion, even as minor as small vibrations, may result in constant invalidation of much of the persistent pixel values from frame to frame.

The Applicants have determined, however, that this problem can be largely solved by re-projection or "warping" of the persistent pixel values to compensate for camera movement. If an accurate spatial transformation for the camera, i.e. movement from the previous camera position and orientation, can be determined, many of the values in the persistent pixel buffer may be salvaged and used, through a re-projection transform in accordance with the present invention.

By way of example, see <FIG>, which is a schematic diagram depicting a persistent image buffer that has been re-projected in response to a camera transformation (in this instance, primarily rotation). Compare, for example, the "Original Perspective" and "Re-Projected Perspective" in <FIG>.

In particular, <FIG> depicts a configuration <NUM> of sensor or camera, persistent image buffer, and objects, including blue object <NUM>, green object <NUM>, sensor or camera <NUM>, and persistent image buffer <NUM>. The exemplary persistent image buffer <NUM> schematically depicted in <FIG> has five elements <NUM>, <NUM>, <NUM>, <NUM> and <NUM>.

In the "Original Perspective" (left-hand side of <FIG>), exemplary rays <NUM>, <NUM>, <NUM>, <NUM> and <NUM> radiate to sensor or camera <NUM>. Rays <NUM>, <NUM>, <NUM> and <NUM> trace to the camera <NUM> from the blue object <NUM>; and ray <NUM> radiates to the camera <NUM> from the green object <NUM>, such that the persistent image buffer <NUM> has values, in its respective elements, of B-B-G-B-B (where B represents Blue and G represents Green).

On the right-hand side of <FIG>, the "Re-Projected Perspective" schematically shows that the persistent image buffer has been re-projected in response to a camera transformation (in the example shown in <FIG>. primarily rotation). Given the re-projection, ray <NUM> radiates to the camera <NUM> from the blue object <NUM>; rays <NUM> and <NUM> trace to the camera <NUM> from the green object <NUM>; but rays <NUM> and <NUM>, which trace to the camera <NUM>, do not emanate or originate from any defined object. As a result, for the "Re-Projected Perspective" example shown in <FIG>, the persistent image buffer <NUM> has values of B-G-G-?-?.

In exemplary practices and embodiments of the present invention, if the camera transformation comprises primarily rotation, or if the scene being observed is reasonably uniform in depth, then the re-projection operation is straightforward, and will yield high quality results. If the camera position transformation involves large translation, or if the scene depth varies significantly, the re-projection operation becomes more complex, due to occlusion and disocclusion of portions of the scene.

Nonetheless, the Applicants have recognized that depth information will allow large parts of the scene to be accurately re-projected, leading to higher quality results than would otherwise occur immediately following camera movement.

As noted above, exemplary practices and embodiments of the present invention are well suited to utilization in combination with a stereo image comparison algorithm, method, device or system, which is often used to generate stereo disparity information. This stereo disparity information can be combined with a known device camera configuration to estimate a depth value for each pixel. In accordance with exemplary practices and embodiments of the present invention, this depth value can then be used to perform more accurate re-projection of the persistent pixel information.

In some exemplary embodiments, the camera transformation or tracking can be produced using additional sensors, such as an accelerometer, gyroscope, or another camera or set of cameras. The transformation information may also be a result of various known processes, such as SLAM, PTAM, DTAM, or the like, interpreting data from the same camera or cameras.

In other exemplary embodiments of the present invention, the camera transformation information can be a result of the information from a stereo disparity search on the de-noised output data. In such an embodiment, the initial quality of the de-noised data would be relatively poor, immediately following the camera movement. But across an entire frame, it may be possible to obtain an accurate camera transformation. Then this camera transformation can be utilized to re-project the persistent image buffer stored prior to the camera movement. This approach would sharply reduce the number of frames needed to achieve a low-noise result in many of the pixels.

The computational cost of constructing and utilizing the noise model of the present invention is comparatively low, but evaluating it for every pixel for every frame can be a non-trivial amount of computational cost. Given that many properties of the noise model are constant across an entire frame, including, for example, the black-level, the exposure time, the variable gain, and other parameters, it is possible, in accordance with an exemplary practice of the present invention, to construct a lookup table that can be used for all pixel values in a frame. A lookup table drastically reduces the amount of computation required to evaluate the noise model for each pixel.

Another exemplary practice or embodiment of the present invention is to utilize computational resources within dedicated hardware, such as an imaging pipeline itself. Dedicated hardware may emulate the methods, practices and/or software functions of the present invention as described above, and can also further benefit from accessing data or analog information in the camera pipeline before it has undergone some image processing operations. For example, a hardware implementation may benefit from reading the voltage level from a sensing element before a variable gain amplification has been applied, thus eliminating the need to reverse the effects of the gain when estimating the noise in the data.

in a typical buffer of image data, colors are represented with independent components of an array data structure, for example, red, green, and blue components, typically referred to as RGB. This data is often produced by a sensor implementation in which each light sensing element is occluded by a colored filter that only allows a range of light wavelengths to pass through. In effect, some of the sensing elements are sensitive to certain colors, while other sensing elements are sensitive to other colors. While RGB is the most commonly used independent-color color space, some cameras employ alternate schemes. The exemplary practices and embodiments of the present invention can also function advantageously in color spaces other than RGB.

In one embodiment of the present invention, each color component value is interpreted and modified in isolation. For example, the noise in the red channel of a pixel may be estimated without regard to the values of the green and blue channels of the same pixel. This approach will provide greater accuracy than an approach that interprets the pixel as a single value.

Digital cameras often employ a Bayer Pattern or a similar distribution of color sensing elements. In a Bayer Pattern, there are twice the number of green-sensitive elements, when compared to the number of red sensitive or blue sensitive elements. This is to account for the higher level of sensitivity to green light in the human eye. In effect, this means the green channel receives twice the number of samples as the red or the blue channel. Therefore, there will be a correspondingly lower amount of noise associated with the green channel. In accordance with an exemplary practice of the present invention, the parameters of the statistical model constructed in accordance with the invention may be different for the green channel, to account for this effect. The model may reflect this effect as an additional sample, or it may alternatively reflect this as a larger electron well capacity, or it may apply a similar or corresponding transformation or adjustment elsewhere.

For simplicity or for increased performance, it is also possible to perform one operation for a combined value encoding a compound color value. If an independent-color color space such as RGB is used, the pixel value may be reduced to a scalar quantity. In accordance with the present invention, one method to perform this reduction is to calculate the magnitude of the <NUM>-dimensional color vector using the Pythagorean theorem. Another method in accordance with the present invention is to use only the value of the largest component of the color vector.

There are many known methods to approximate a vector magnitude, with varying computational costs.

If a combined-color color space is used, such as YUV, it is possible to perform the noise estimation with the luminance channel alone.

Those skilled in the art will understand that the above described embodiments, practices and examples of the invention can be implemented using known forms of cameras, sensors, camera pipelines, digital networks, computer processor and telecommunications devices, in which the telecommunications devices can include known forms of cellphones, smartphones, and other known forms of mobile devices, tablet computers, desktop and laptop computers, and known forms of digital network components and server/cloud/network/client architectures that enable communications between such devices.

Those skilled in the art will also understand that method aspects of the present invention can be executed in commercially available digital processing systems, such as servers, PCs, laptop computers, tablet computers, cellphones, smartphones and other forms of mobile devices, as well as known forms of digital networks, including architectures comprising server, cloud, network, and client aspects, for communications between such devices.

The terms "computer software," "computer code product," and "computer program product" as used herein can encompass any set of computer-readable programs instructions encoded on a non-transitory computer readable medium. A computer readable medium can encompass any form of computer readable element, including, but not limited to, a computer hard disk, computer floppy disk, computer-readable flash drive, computer-readable RAM or ROM element or any other known means of encoding, storing or providing digital information, whether local to or remote from the cellphone, smartphone, tablet computer, PC, laptop, computer-driven television, or other digital processing device or system. Various forms of computer readable elements and media are well known in the computing arts, and their selection is left to the implementer.

In addition, those skilled in the art will understand that the invention can be implemented using computer program modules and digital processing hardware elements, including memory units and other data storage units, and including commercially available processing units, memory units, computers, servers, smartphones and other computing and telecommunications devices. The term "modules", "program modules", "components", and the like include computer program instructions, objects, components, data structures, and the like that can be executed to perform selected tasks or achieve selected outcomes. The various modules shown in the drawings and discussed in the description herein refer to computer-based or digital processor-based elements that can be implemented as software, hardware, firmware and/or other suitable components, taken separately or in combination, that provide the functions described herein, and which may be read from computer storage or memory, loaded into the memory of a digital processor or set of digital processors, connected via a bus, a communications network, or other communications pathways, which, taken together, constitute an embodiment of the present invention.

The terms "data storage module", "data storage element", "memory element" and the like, as used herein, can refer to any appropriate memory clement usable for storing program instructions, machine readable files, databases, and other data structures. The various digital processing, memory and storage elements described herein can be implemented to operate on a single computing device or system, such as a server or collection of servers, or they can be implemented and inter-operated on various devices across a network, whether in a server-client arrangement, server-cloud-client arrangement, or other configuration in which client devices can communicate with allocated resources, functions or applications programs, or with a server, via a communications network.

It will also be understood that computer program instructions suitable for a practice of the present invention can be written in any of a wide range of computer programming languages, including Visual Basic, Java, C++, and the like. It will also be understood that method operations shown in the flowcharts can be executed in different orders, and that not all operations shown need be executed, and that many other combinations of method operations are within the scope of the invention as defined by the attached claims. Moreover, the functions provided by the modules and elements shown in the drawings and described in the foregoing description can be combined or sub-divided in various ways, and still be within the scope of the invention as defined by the attached claims.

The Applicants have implemented aspects of the present invention, in prototype form. One implementation comprises a complete device, including four cameras, capable of encoding content and receiving (full-duplex communication). Another is an Apple iPhone-based implementation that can receive and present immersive content (receive-only). The Applicants used the following hardware and software structures and tools, among others, to create the two noted implementations, collectively:.

<FIG> are flowcharts depicting method aspects and exemplary practices of the invention. The methods depicted in these flowcharts are examples only; the organization, order and number of operations in the exemplary practices can be varied; and the exemplary practices and methods can be arranged or ordered differently, and include different or additional functions, whether singly or in combination, while still being within the scope of the present invention.

Elements shown in the flowcharts in parentheses are, among other aspects, optional in a given practice of the invention.

In addition to the discussion below, all the text and respective textual elements of the accompanying flowcharts are incorporated by reference into this Detailed Description of the Invention as if set forth in their entireties in this Detailed Description of the Invention, in the respective order in which they appear in the flowcharts, while noting that the organization, order and number of operations in the exemplary practices can be varied; and the exemplary practices and methods can be arranged or ordered differently, and include different or additional functions.

In particular, <FIG> shows a temporal de-noising method <NUM> according to an exemplary practice of the invention, comprising the following operations:.

<FIG> depicts aspects of a noise model or models in accordance with the invention, comprising the following:.

<FIG> depicts aspects of a persistent image buffer or buffers in accordance with exemplary practices of the invention, comprising the following:.

<FIG> depicts aspects of re-projection in accordance with exemplary practices of the invention, comprising the following:.

<FIG> depicts aspects of stereo search with de-noising in accordance with exemplary practices of the invention, comprising:.

<FIG> depicts aspects of image metadata and other data in accordance with exemplary practices of the invention, comprising the following:.

<FIG> depicts other actions and/or processes in accordance with exemplary practices of the invention, comprising the following:.

<FIG> is a schematic block diagram depicting exemplary devices or systems in which aspects of the present invention may be practiced or embodied.

In particular, <FIG> is a schematic block diagram showing a device or system <NUM> in which the invention may be practiced. The device or system <NUM> may be implemented using known forms of digital processing hardware, such as known forms of smartphones, tablets and/or other forms of digital processing and imaging devices, supplemented as necessary in accordance with the teachings of the present invention. Arrows in <FIG> indicate flow of digital data and/or signals between elements.

By way of example, device of system <NUM> can comprise at least a first digital camera or camera pipeline <NUM>, a second digital camera or camera pipeline <NUM>, and a digital processing resource <NUM> comprising one or more digital processor(s) <NUM>, memory element(s) <NUM>, buffer(s) <NUM>, other storage <NUM>, and interfaces <NUM> to external devices and/or networks.

In accordance with the teachings of the invention, such as discussed above in connection with <FIG>, <FIG>, et seq. , the digital processing resource <NUM> is operable to receive digital data from the cameras of camera pipelines <NUM>, <NUM>, process the data in accordance with the invention, and provide outputs, based on such processing, to internal (i.e., within device or system <NUM>) display element <NUM> or storage <NUM>; and/or to external display, storage or network elements (collectively <NUM>).

The external display, storage or network elements <NUM> may comprise the Internet, devices, processors or other networks connected to or via the Internet, or other network-connected elements or destinations.

In addition, the digital processing resource may receive or consume digital information from such a network or networks <NUM>.

Those skilled in the art will understand that a number of the physical hardware and processing elements discussed in connection with <FIG> can be provided by structures found in commercially available smartphones, tablet computing devices, and other such devices, when configured and operated in accordance with exemplary practices of the present invention, and supplemented by additional hardware elements, such as additional cameras and/or other sensors, as may be required by the present invention. The digital processor(s) of the digital processing resource can be of conventional design, such as those found in commercially available smartphones, tablet computing devices, and other such devices, when configured and operated in accordance with exemplary practices of the present invention.

Claim 1:
A method of generating reduced-noise image frames based on image frames received from a digital camera pipeline (<NUM>, <NUM>), the method comprising:
in a digital processing resource (<NUM>) comprising at least one digital processor (<NUM>):
A. receiving (<NUM>) at least one frame of image pixel data (<NUM>) from a digital camera pipeline;
B. receiving (<NUM>) image metadata (<NUM>) corresponding to the at least one received frame; and
C. executing (<NUM>) temporal de-noising operations for at least selected pixels of the at least one received frame of image pixel data utilizing the image metadata (<NUM>) and a noise model (<NUM>) wherein executing temporal de-noising operations comprises:
(<NUM>) evaluating (<NUM>) the noise model to estimate the noise present in a selected pixel from the at least one received frame, wherein the image metadata is an input to the evaluating of the noise model to estimate the noise present in the selected pixel;
(<NUM>) blending (<NUM>) data from a persistent image buffer (<NUM>) with data from the at least one received frame (<NUM>);
(<NUM>) updating (<NUM>) data in the persistent image buffer (<NUM>); and
(<NUM>) outputting (<NUM>) the blended pixel data to a reduced-noise frame buffer, thereby to generate a reduced-noise image frame, wherein the noise model is a source for information useable in executing the blending (<NUM>).