Motion adaptive stream processing for temporal noise reduction

Techniques related to temporal noise reduction of images are discussed. Such techniques may include generating a noise stream corresponding to an input image and adaptively re-combining the noise stream with a reference image corresponding to the input image and a spatially noise reduced image corresponding to the input image to generate a temporal noise reduced output image.

BACKGROUND

In image processing contexts, particularly in low light conditions, spatial noise reduction may not accurately reduce noise as there is difficulty in distinguishing detail and noise. The use of strong spatial noise reduction (SPNR) may result in either a blurry image with a loss of detail or a noisy image. In such conditions, temporal noise reduction (TNR) may provide higher image and/or video quality.

However, temporal noise reduction techniques may have difficulties reducing noise for fast moving objects and/or for occluded regions (e.g., image regions that were obstructed in a previous image and revealed in a current image). Such fast moving objects and/or occluded regions may not have good matches in the reference image, providing difficulty in applying temporal noise reduction.

It may be advantageous to perform improved temporal noise reduction for images, which may improve image quality by reducing noise in static regions (e.g., those image regions that are not changing in the current image with respect to previous image(s)), moving regions (e.g., those image regions that are moving in the current image with respect to previous image(s)), and occluded regions (e.g., image regions that were obstructed in a previous image and revealed in a current image) without sacrificing detail level. It is with respect to these and other considerations that the present improvements have been needed. Such improvements may become critical as the desire to attain high quality images becomes more widespread.

DETAILED DESCRIPTION

Methods, devices, apparatuses, computing platforms, and articles are described herein related to motion adaptive stream processing for temporal noise reduction and, in particular, to generating and adaptively applying a noise stream for improved temporal noise reduction.

As described above, in imaging processing contexts, particularly in low light conditions, temporal noise reduction may provide improved image quality. As discussed, in some embodiments, the techniques discussed herein may provide for improved temporal noise reduction based on generating and adaptively applying a noise stream. Such techniques may improve image quality by reducing noise in static regions (e.g., those image regions that are not changing in the current image with respect to previous image(s)), moving regions (e.g., those image regions that are moving in the current image with respect to previous image(s)), and occluded regions (e.g., image regions that were obstructed in a previous image and revealed in a current image) without sacrificing detail level.

In some embodiments, an input image may be received and a noise reduced image may be generated based on a noise reduction of an input image. The input image may be any suitable input image (e.g., an image, a video frame, or the like) that has been demosaiced or that is in a color filter array domain (e.g., that has not been demosaiced). For example, the input image may be an RGB image, a YUV image, an image in any YUV variant color space, or the like. The noise reduction of the input image may be performed using any suitable spatial noise reduction technique or techniques. A noise stream corresponding to the input image may be generated based on the input image and the noise reduced image. For example, the noise stream may be the difference between the noise reduced image and the input image. The noise stream may have any suitable data structure. For example, the noise stream may provide noise values for each pixel location corresponding to the input image. The noise stream may be applied to a luma channel or any chroma channel as is discussed herein.

The noise stream may be adaptively combined with an input image, a reference image corresponding to the input image (e.g., a previously temporal noise reduced image), the noise reduced image, and/or a spatially noise reduced image generated based on a spatial noise reduction of the noise reduced image. For example, the noise stream, the reference image, the noise reduced image, and the further noise reduced image may be combined using pixel blending techniques adaptive to motion information such as local motion or the like corresponding to the input image. Such motion information may be generated using any suitable technique or techniques. For example, the noise stream may be highly applied to those regions showing little or no motion and the noise stream may be attenuated (or not applied at all) to those regions showing fast motion or no motion matching. Such techniques may advantageously provide for little noise (e.g., greater spatial noise reduction) for those regions that are fast moving or that were previously occluded as the noise stream is not added back in for those regions and greater noise (e.g., more perceived detail) for those regions that are static. Furthermore, such techniques may provide for applying stronger spatial noise reduction prior to temporal noise reduction (e.g., prior to the temporal noise reduction component) to reduce noise in the moving regions, since the lost details may be added back in for the static regions.

Furthermore, prior to combining the noise stream, the noise stream may be equalized and/or adjusted based on one or more of a local luminance corresponding to the input image, a local chrominance corresponding to the input image, detected content corresponding to the input image, a radial distance from an optical center of the input image, and/or a user preference. For example, the noise stream may be attenuated in lower luminance areas (e.g., noise may be smaller in bright regions than dark regions and noise may be attenuated in dark regions) or lower chrominance areas, flatter areas (e.g., low detected content level areas), and at greater distances from the optical center of the input image and the noise stream may be left unchanged or enhanced in higher luminance areas, higher chrominance areas, texture areas (e.g., high detected content level areas), and at lower distances from the optical center of the input image. Furthermore, the noise stream may be attenuated, left unchanged, or enhanced based on user preference.

The techniques discussed herein may introduce a noise stream (e.g., a difference signal between a spatial noise reduction input and output), which may be added back to the image adaptive to local motion level, detail level, noise characteristics (e.g., noise level dependency on local luminance, chrominance, and/or radial distance from the optical center), and/or user preference. Such techniques may be particularly advantageous in low light or extreme low light conditions. The characteristics of such low light or extreme low light conditions may depend on the optical capabilities of the image capture device and may include light conditions below 0.1 lux, below 1.0 lux, below 20 lux, or the like.

FIG. 1illustrates an example system100for providing temporal noise reduction, arranged in accordance with at least some implementations of the present disclosure. As shown inFIG. 1, system100may include a noise reduction module101, a demosaic module102, and a temporal noise reduction module103, which may include a content detection module131, a local motion estimation module132, a trajectory break module133, a noise equalization module134, a pixel blending module135, and a spatial noise reduction module136. System100may be implemented via any suitable device such as, for example, a personal computer, a laptop computer, a tablet, a phablet, a smart phone, a digital camera, a gaming console, a wearable device, a display device, an all-in-one device, a two-in-one device, a surveillance device, or the like or platform such as a mobile platform or the like. For example, as used herein, a system, device, computer, or computing device may include any such device or platform.

Also as shown, system100(e.g., via noise reduction module101) may receive raw input image111and system100may provide (e.g., via pixel blending module135of temporal noise reduction module103) an output image113, which may be characterized as a temporal noise reduced output image or the like. Noise reduction module101may include a single luma noise reduction module, a luma noise reduction module and one or more chroma reduction modules, or one or more chroma reduction modules depending on the format of raw input image111and the channel or domain in which noise reduction is being implemented (e.g., a luma channel only, a luma channel and one or more chroma channels, or one or more chroma channels). Raw input image111may include any suitable image data, video frame data, or the like in any suitable domain. For example, raw input image111may include data from an image sensor or from an image preprocessor including a particular color value for each pixel location such that raw input image111may be in a color array filter space. For example, raw input image111may include a red, green, or blue value for each pixel location according to a pattern of a color filter space. Although illustrated with respect to raw input image111being an input image in a color array filter space, raw input image111may be any suitable image data such as demosaiced image data, image data in the red, green, blue (RGB) color space, image data in a luma chrominance color space such as the YUV color space, a YUV variant color space, or the like. In an embodiment, demosaic module102may be applied before noise reduction module101. In various examples, raw input image111may be received from a camera, camera array, image preprocessor, or the like.

As shown, a noise reduced image (NRI)141may be generated based on a noise reduction of raw input image111. The noise reduction of raw input image111may be performed using any suitable technique or techniques. For example, the noise reduction of raw input image111may be performed based on filtering techniques (e.g., linear or non-linear filtering), anisotropic diffusion techniques, non-local averaging techniques, or a combination thereof. Also as shown, based on a difference between noise reduced image141and raw input image111, as determined by differencer104, a noise stream (NS)142may be generated. Noise stream142may include any suitable data or data structure. In an embodiment, noise stream142may provide a noise value for each pixel location of raw input image111, for regions of raw input image111, or the like. In an embodiment, noise stream142may include a noise value for each color channel for each pixel location of an input image. Noise stream142may include a high frequency signal containing noise and detail information of raw input image111. By providing noise stream142, noise reduction module101may provide noise reduced image141having less noise, which may provide fewer unwanted artifacts and/or more robust results in later processing such as demosaicing via demosaic module102, local motion tracking via local motion estimation module132, and so on. As is discussed further herein, detail contained in noise stream142may be adaptively added back in later stages to reproduce detail without degrading processing via intervening modules. Also as discussed further herein, noise stream142may be used with temporal noise reduction to reduce noise in moving or occluded regions and to increase detail in static regions by adaptively applying noise stream142based on local motion of raw input image111.

FIG. 2illustrates example images200for the application of temporal noise reduction, arranged in accordance with at least some implementations of the present disclosure. As discussed, images200may include any suitable images, pictures, or frames of video or the like or any suitable data representing images, pictures, or frames of video. For example, images200may include any suitable image or imaging data. Input images200may be in any suitable format and/or color space. In the illustrated example, images200include images of an indoor environment; however, images200include images of any suitable scene including any suitable subjects. As shown, images200may include an image201and an image202such that image202is subsequent to image201. Images201,202may include a moving object211and a static region213(e.g., the regions outside of moving object211). Also as shown with respect to image202, the motion of moving object211(e.g., which has moved up and to the right) may provide an occluded region212such that occluded region212was occluded by moving object211in image201and revealed in image202.

Using the techniques discussed herein, a noise stream (e.g., noise stream142) may be adaptively combined through pixel blending to generate a higher quality output image. For example, with respect to image202, noise may be advantageously applied in higher amounts to static region213(e.g., those regions with minimal or no motion). Providing noise in static region213, which has well established temporal noise reduction over several frames since it is static, may provide detail. For example, since strong temporal noise reduction may be applied in static regions, having more noise will contribute to a detail level improvement in such regions. Furthermore, noise may be advantageously attenuated (or not applied at all) in the region corresponding to moving object211in image202and in occluded region212such that spatial noise reduction (as applied via other modules of the temporal noise reduction) may provide smoothing of the noise in those regions.

Furthermore, the noise stream may be equalized and/or adjusted based on other features of image202. For example, the noise stream may be attenuated in low local luminance areas (as determined by local luminance averaging), in low local chrominance areas (as determined by local chrominance averaging), flat areas (as determined via content detection), and at locations radially distant from a center of image202. Such attenuation may advantageously reduce noise in dark areas (where noise is more often pronounced), flat areas (where noise may provide unwanted artifacts), and in areas radially distant from an image center (where noise may be more pronounced due to lens shading corrections). Similarly, the noise stream may be unchanged (or enhanced) in high local luminance areas (as determined by local luminance averaging where noise may not be as pronounced), high local chrominance areas (as determined by local chrominance averaging), texture areas (as determined via content detection such that more detail may be provided or perceived due to noise), and at locations radially near a center of image202(where noise may not be as influenced by lens shading corrections).

Returning toFIG. 1, as discussed, noise stream142may be in a color array filter space or in any other color space. For example, noise reduction module101may be provided at an early stage of an image processing pipeline (e.g., prior to demosaic module102) and system100may utilize the results of such noise reduction to generate noise stream142. In other examples, demosaicing may be performed prior to noise reduction and differencing to generate noise stream142. As discussed, raw input image may be in any suitable color space or domain. Furthermore, as shown, noise stream142may be determined as the difference of a noise reduction input and output (e.g., a difference between raw input image111and noise reduction image142). Noise stream may be in any suitable color space or domain. Noise stream may be in a luma channel only, in a luma channel and one or more chroma channels, or in one or more chroma channels. Noise stream142may be provided to noise equalization module134, which may equalize or adjust noise stream142to generate an equalized noise stream148as discussed further herein. For example, noise equalization module134may compensate for dependencies in noise stream142. Furthermore, pixel blending module135may adaptively re-combine noise equalized noise stream148to the input signal of raw input image111based on motion information as provided by local motion estimation module132and/or trajectory break detection module133as discussed further herein. Such re-combining of noise equalized noise stream148may be provided to a luma channel of the input signal of input image111and/or to a chroma channel of the input signal of input image111. For example, system100may include a spatial noise reduction module for luma and/or a spatial noise reduction for chroma (e.g., via noise reduction module101). In some embodiments, a chroma noise stream may be taken from an input and the output of the spatial noise reduction for chroma as discussed with respect to noise reduction module101and differencer104. The luma and chroma noise streams may be used separately or they may be combined for application via pixel blending module135after optional processing by noise equalization module134.

As shown, noise reduced image141may be provided to demosaic module102. Demosaic module102may demosaic noise reduced image141to generate input image (II)143using any suitable technique or techniques. For example, demosaic module102may interpolate color values for each pixel missing a particular color value in noise reduced image141. For example, for those pixels having blue values but missing a red and a green value, such red and green values may be determined, for those pixels having red values but missing a blue and a green value, such blue and green values may be determined, and for those pixels having green values but missing a red and a blue value, such red and blue values may be determined. Furthermore, demosaic module102or other modules of system100may provide gamma correction, color correction, image enhancement, or the like to generate input image143. In some embodiments, input image143may be provided in or converted to another color space such as the YUV color space for further processing. For example, input image143may include a luma and chrominance components. As discussed, in some embodiments, demosaicing may be performed before noise reduction such that noise signal142may be generated based on a demosaiced and, optionally, a gamma corrected, color corrected, and/or image enhanced image.

Content detection module131may receive input image143and content detection module131may perform content detection based on input image143to generate detail level144. Content detection module131may perform content detection using any suitable technique or techniques and detail level144may include any suitable data or data structure representative of the content or detail of input image111. For example, detail level144may include a value for each pixel location of input image143indicating a detail level with high values representing texture or edge pixels or regions and low values representing flat regions. As discussed, detail level144may be provided on a pixel-by-pixel basis. In some examples, detail level144may be provided on a region-by-region basis such that regions of input image143are represented by a single detail level value. As discussed, detail level144may include any suitable data or data structure. In some examples, detail level144may be characterized as detail values, content level values, a content level map, or the like. As shown, detail level144may be provided to noise equalization module134for adjusting noise stream142as discussed herein.

Local motion estimation module132may receive input image143and a reference image112(e.g., a previously noise reduced output image as shown). Local motion estimation module132may generate local motion145based on input image143and reference image112using any suitable technique or techniques. For example, local motion145may be generated based on block matching techniques or the like. Local motion145may include any data or data structures representative of local motion in input image143with respect to reference image112. In an embodiment, local motion145may include a motion vector field providing an approximated motion vector (e.g., having a horizontal and vertical component) for each pixel of input image143or for regions of input image143. In an embodiment, local motion145may include motion values for pixels or regions of input image143, which represent an estimation of whether motion is present in input image143with respect to reference image112.

As shown, local motion145(and reference image112and input image143as needed) may be provided to pixel blending module135and trajectory break detection module133, which may generate motion information146. In an embodiment, trajectory break detection module133may track motion across images (e.g., several instances of input images) to determine whether local motion145at particular pixels provides actual motion or mistakenly estimated motion. For example, at local estimation module132, for a particular pixel or region, local estimation module132may provide a motion confidence value or the like representative of confidence level that input image143and reference image112are similar to each other after the application of local motion145. For example, in some cases, a best guess or closest match motion vector of local motion145may be an erroneous match. Trajectory break detection module133may, as discussed, track motion across images and trajectory break detection module133may zero out or adjust any motion that does not have a smooth tracking over time (e.g., broken motion) to eliminate or reduce such mismatches. For example, trajectory break detection module133may support the detection of occluded regions in input image143. In embodiments where local motion145is not available, motion information146may provide values indicating a confidence level that input image and reference image112are similar to each other without application of local motion. In such contexts, motion information146may provide high values for locations where there is no motion and lower values for locations when there is motion.

Motion information146may include any data or data structures representative of local motion in input image143with respect to reference image112. For example, motion information146may include motion confidence values based on input image143and reference image112as discussed. In some examples, local motion145and/or motion information146may be characterized as motion vectors, a motion vector field, motion values, local motion information, a motion map, motion confidence values, local motion confidence maps, or the like. As shown, local motion145and/or motion information146may be provided to pixel blending module135for adaptive application of equalized noise stream148as discussed herein.

Spatial noise reduction module136may also receive input image143and spatial noise reduction module136may provide spatial noise reduction using any suitable technique or techniques to generate noise reduced image147. For example, the noise reduction of input image143may include filtering techniques (e.g., linear or non-linear filtering), anisotropic diffusion techniques, non-local averaging techniques, or the like. Such spatial noise reduction of input image143within temporal noise reduction module103may provide additional smoothing or noise reduction for fast motion regions or occluded regions. Furthermore, noise stream142may be re-applied or combined in static regions as discussed herein.

As discussed, it may be advantageous to equalize and/or adjust noise stream142to generate equalized noise stream148via noise equalization module134of temporal noise reduction module103. Such equalizing and/or adjusting of noise stream142may be based on local luminance, local chrominance, detected content, a radial distance from an optical center, a user preference, or the like.

FIG. 3illustrates an example noise equalization component300, arranged in accordance with at least some implementations of the present disclosure. For example, noise equalization component300may be implemented via noise equalization module134of system100. As shown, noise equalization component300may receive noise stream142, input image143, and detail level144and noise equalization component300may generate equalized noise stream148.

As shown, noise stream142may be received by luma conversion module301, which may convert noise stream142to a luma domain or component to generate luma noise stream311. As discussed, noise stream142may be in a color filter array domain, color domain, or the like and luma conversion module301may convert noise stream142to a luma domain or component. In examples where noise stream142is provided in the luma domain or as a luma component, luma conversion module301may be skipped. Furthermore, in examples where noise stream142is implemented in a chroma domain, luma conversion module301may be skipped. Luma conversion module301may convert noise stream142to luma noise stream311using any suitable technique or techniques. In an embodiment, luma conversion module301may be a low pass filter.

Also as shown, input image143may be received by local averaging module302, which may perform local averaging of a luma component of input image143to generate a local luminance map312. As discussed, input image143may be in any suitable color domain. In some examples, input image143may include a luma component and, in other examples, input image143may be converted to a domain having a luma component or a luma component may be extracted. Local averaging module302may generate local luminance map312using any suitable technique or techniques. For example, local averaging module302may provide a local averaging in a window (e.g., a 2×2 pixel window such that the local average or mean may be characterized as μ2×2) around each pixel of input image143to generate local luminance map312. In an embodiment, local averaging module302may not be included and input image143may be used as local luminance map312. Such embodiments may advantageously save computation complexity and cost. Local luminance map312may include any data or data structure representative of local luminance of input image143such as a local average luminance value for each pixel location of input image143.

Local luminance map312may be provided to local luminance gain module322, which may generate local luminance gain values (GY)342by applying a local luminance to local luminance gain mapping332. For example, for each pixel value of local luminance map312, local luminance gain module322may generate a local luminance dependent gain value of local luminance gain values342by applying local luminance gain mapping332. As shown, local luminance gain mapping332may provide for higher gain values for higher luminance areas (e.g., having lower luminance values indicating higher luminance) and lower gain values for low luminance areas (e.g., having higher luminance values indicating lower luminance) based on a concave upward curve such that noise stream142may be attenuated in lower luminance areas of input image143(e.g., where noise may cause poor image quality). Local luminance gain mapping332may be applied using any suitable technique or techniques such as a look up table a determination step based on a function or the like.

In examples where noise stream142includes a chroma noise stream (e.g., in addition or in alternative to a luma noise stream), input image143may be received by a local chroma averaging module (not shown) which may perform local averaging of a chroma component of input image143to generate a local chrominance map (not shown) or input image143may be used as a local chrominance map. The local chrominance map may include any data or data structure representative of local chrominance of input image143. In analogy to local luminance map312, local luminance gain module322, and local luminance gain mapping322, the local chrominance map may be provided to a local chrominance gain module (not shown), which may generate local chrominance gain values (not shown) by applying a local chrominance mapping (not shown). For example, for each pixel value of the local chrominance map, a local chrominance dependent gain value may be determined. The local luminance gain mapping may provide for lower gain values for lower chrominance areas and higher gain values for higher chrominance areas based on a concave upward curve as shown with respect to local luminance gain mapping332such that noise stream142may be attenuated in lower chrominance areas of input image143(e.g., where noise may cause poor image quality).

Furthermore, detail level144may be received via content detection module131(please refer toFIG. 1). In some examples, noise equalization component300may implement content detection module131or the like, which may generate detail level144or content level map313as discussed herein. As shown, in some examples, detail level144may be characterized as content level map313. In other examples, detail level144may be converted to generate content level map313. Content level map313may include any data or data structure representative of content levels of input image143such as a detail or content level value for each pixel location of input image143such that higher values indicate texture, edges, or the like and lower values indicate flat regions.

Content level map313may be provided to content level gain module323, which may generate content level gain values (GCL)343by applying a content level to content level gain mapping333. For example, for each pixel value of content level map313, content level gain module323may generate a content level dependent gain value of content level gain values343by applying content level gain mapping333. As shown, content level gain mapping333may provide for higher gain values for higher content level values and lower gain values for lower content level values based on a concave downward curve having a flat portion at lower content level values such that noise stream142may be attenuated in lower content level areas of input image143(e.g., where noise can cause artifacts) and left unchanged or enhanced in higher level content areas of input image143(e.g., where noise may provide detail). Content level gain mapping333may be applied using any suitable technique or techniques such as a look up table a determination step based on a function or the like.

Also, radius determination module303may generate a radial distance map314, which may provide for a radial distance from an optical center of input image143. Radius determination module303may generate radial distance map314using any suitable technique or techniques. For example, radius determination module303may generate radial distance map314based on the optics system used to generate input image143or the like. Radial distance map314may include any data or data structure representative of radial distance such as a distance value from optical center for each pixel location of input image143.

Radial distance map314may be provided to radial distance gain module324, which may generate radial distance gain values (GR)344by applying a radial distance to radial distance gain mapping334. For example, for each pixel value of radial distance map314, radial distance gain module324may generate a radial distance dependent gain value of radial distance gain values344by applying radial distance gain mapping334. As shown, radial distance gain mapping334may provide for higher gain values for lower radial distance values and lower gain values for higher radial distance values based on a shallow concave downward curve such that noise stream142may be attenuated at distances further from the optical center of input image143(e.g., where noise can be caused by lens shading correction). Radial distance gain mapping334may be applied using any suitable technique or techniques such as a look up table a determination step based on a function or the like.

As shown, luminance gain values342and/or chrominance gain values, content level gain values343, and radial distance gain values344may be combined by multipliers361and362to generate final gain values (G)351. Luminance gain values342and/or chrominance gain values, content level gain values343, and content level gain values343may be combined using any suitable technique or techniques such as multiplication as shown or other techniques. Final gain values351may include any suitable data or data structure such as a gain value for each pixel of input image143. Furthermore, final gain values351may be applied to luma noise stream311and/or a chroma noise stream by multiplier363. Final gain values351may be applied to luma noise stream311and/or a chroma noise stream using any suitable technique or techniques such as multiplication as shown or other techniques to provide equalized noise stream148. Equalized noise stream148may include a noise value for each pixel of input image143. Furthermore, although not shown inFIG. 3, a user preference (e.g., an application user or an application developer) gain may be applied to noise stream142via a user preference mapping and/or a multiplier. For example, a user preference may be applied to each pixel location to further attenuate noise or the like. Furthermore, user preferences may be provided by adjusting any of local luminance gain mapping332, content level gain mapping333, radial distance gain mapping334, or the like.

FIG. 4illustrates another example noise equalization component400, arranged in accordance with at least some implementations of the present disclosure. For example, noise equalization component400may be implemented via noise equalization module134of system100. As shown, noise equalization component400may receive noise stream142, input image143, and detail level144and noise equalization component400may generate equalized noise stream148. Furthermore, noise equalization component400may implement local averaging module302, local luminance gain module322, a local chrominance averaging module (not shown), a local chrominance gain module (not shown), content level gain module323(and a content detection module or a conversion to content level mapping module as needed), radius determination module303, and radial distance gain module324, which may operate as discussed herein with respect toFIG. 3to generate luminance gain values342, chrominance gain values (not shown), content level gain values343, and radial distance gain values344. The operation of such modules will not be repeated for the sake of brevity.

Furthermore, as discussed, noise stream142may be in a color filter array domain. As shown, clip/coring module401may receive noise stream142and may provide clipping and/or coring of noise stream142and provide the resultant stream to local averaging module402, which may generate a local green channel averages (LAG) and local blue channel averages (LAB)411. Local averaging module402may generate local green channel averages and local blue channel averages411using any suitable technique or techniques. For example, local averaging module302may provide a local averaging in a window (e.g., a 5×5 pixel window such that the local average or mean may be characterized as μ5×5) around each blue pixel location of noise stream142. As shown, local green channel averages and local blue channel averages411may be provided to gain generation module403, which may generate blue channel gain values (GB)412based on local green channel averages and local blue channel averages411using any suitable technique or techniques. In an embodiment, blue channel gain values (GB)412may be generated based on a ratio of the local green channel averages plus a conversion factor to the local blue channel values plus the conversion factor (e.g., GB=(μG+ε)/(μB+ε), where μGand μBmay be the local green and blue channel averages, respectively, and ε may be the conversion factor). For example, blue channel gain values412may equalize or normalize noise stream142to a blue channel of the noise stream.

As shown, blue channel gain values412, luminance gain values342, chrominance gain values (not shown), content level gain values343, and radial distance gain values344may be combined by multipliers461,462, and463to generate final gain values (G)451. Blue channel gain values412, luminance gain values342, content level gain values343, and radial distance gain values344may be combined using any suitable technique or techniques such as multiplication as shown or other techniques. Final gain vales451may include any suitable data or data structure such as a gain value for each pixel of input image143. Furthermore, final gain values451may be applied to noise stream142by multiplier464. Final gain values451may be applied to noise stream142using any suitable technique or techniques such as multiplication as shown or other techniques to provide equalized noise stream148. As discussed above, equalized noise stream148may include a noise value for each pixel of input image143. Furthermore, although not shown inFIG. 4, a user preference (e.g., an application user or an application developer) gain may be applied to noise stream142via a user preference mapping and/or a multiplier. For example, a user preference may be applied to each pixel location to further attenuate noise or the like. Furthermore, user preferences may be provided by adjusting any of local luminance gain mapping332, content level gain mapping333, radial distance gain mapping334, or the like.

Returning toFIG. 1, as shown, pixel blending module135may receive equalized noise stream148, input image143, reference image112, noise reduced image147, local motion145, and motion information146. Pixel blending module135may blend equalized noise stream148, input image143, reference image112, and/or noise reduced image147based on local motion145and/or motion information146to generate output image113, which may be provided as an output of system100and used as a reference image for a subsequent image received via raw input image111(if any). Pixel blending module135may blend equalized noise stream148, input image143, reference image112, and/or noise reduced image147based on local motion145and/or motion information146using any suitable technique or techniques.

For example, as discussed, local motion145and/or motion information146may include motion information on a pixel-by-pixel or region-by-region basis for input image143. For those pixels or regions having low motion (e.g., less than a threshold) or no motion, equalized noise stream148may be combined with a full weighting or a high weighting. For those pixels or regions having high motion (e.g., greater than a threshold), equalized noise stream148may be attenuated or not applied at all. Such techniques may apply a motion threshold and may apply equalized noise stream148based on whether a pixel or region is above or below the threshold. In other examples, equalized noise stream148may be attenuated as motion increases or multiple thresholds may be applied.

Furthermore, pixel blending module135may select weighting among input image143, reference image112, and noise reduced image147in addition to the weighting or selection of equalized noise stream148. For example, pixel blending module135may weigh reference image112more heavily for regions where low or little motion is determined and to attenuate reference image112or not apply it at all for regions where there is motion. Furthermore, the weighting of input image143and noise reduced image147may be provided independent of motion information146or dependent on motion information146such that high motion regions may weigh noise reduced image147more heavily.

As discussed, pixel blending module135may control how much noise (e.g., via equalized noise stream148) is added back to input image143to generate output image113. For example, more noise or all available noise signal may be applied to static regions and no noise or an attenuated amount of the noise signal may be applied to fast motion regions and/or occluded regions.

FIG. 5illustrates example motion information500, arranged in accordance with at least some implementations of the present disclosure. As shown, motion information500may include static regions501(as indicated by gray inFIG. 5) and high motion or occluded regions502,503(as indicated by black inFIG. 5). For example, high motion or occluded regions502,503may include those regions for which no motion vector could be found, those regions for which a large motion vector, those regions for which a high probability of motion was determined, or the like. Such no motion vector regions may be due to an occlusion (e.g., no match could be found because the region was newly revealed) or fast motion where no motion vector could be found within the search limits provided by the local motion search. With reference toFIG. 1, equalized noise stream148may not be applied to high motion or occluded regions502,503(or the application may be attenuated) and equalized noise stream148may be applied to static regions501.

FIG. 6illustrates an example process600for combining a noise stream based on motion information, arranged in accordance with at least some implementations of the present disclosure. Process600may include one or more operations601-609as illustrated inFIG. 6. Process600may be performed by a system (e.g., system100or any other devices or systems discussed herein) or portions of process600may be performed by a system to combine a noise stream with an input image to generate an output image. Process600or portions thereof may be repeated for any number input images, frames of video, noise streams, or the like. For example, process600may provide a temporal noise reduced output image.

As shown, process600may begin at operation601, where process600may begin or continue at a particular pixel or region for blending. Processing may continue at decision operation602, where a determination may be made as to whether local motion for the particular pixel or region is above a threshold. The threshold may be any suitable predetermined threshold or heuristically determined threshold or the like.

If the amount of local motion is greater than the threshold (or equal to the threshold in some examples), processing may continue at operation603, where a noise reduced image may be weighted more heavily in the pixel blending. For example, for pixels or regions of high motion, a spatially noise reduced image generated by the temporal noise reduction processing may be weighed more heavily such that such pixels or regions may have more smoothing and less noise. Furthermore, a reference image (e.g., prior temporal noise reduced image) may not be used at all in such regions as there is no match between the current image and the reference image.

Processing may continue at operation604, where a noise stream may be attenuated or reduced to zero. For example, for pixels or regions of high motion, a generated noise stream or equalized noise stream may be minimally applied or not applied at all such that a smoother image and less noise may be provided for such pixels or regions of high motion.

Returning to decision operation602, if the amount of local motion is less than the threshold (or equal to the threshold in some examples), processing may continue at operation605, where a reference image (e.g., prior temporal noise reduced image) may be weighted more heavily in the pixel blending. For example, for pixels or regions of no motion (e.g., static regions), temporal noise reduction across images may provide higher quality imaging than the image currently being processed.

Processing may continue at operation606, where a noise stream may be fully applied or attenuated only slightly for pixel blending. For example, for pixels or regions of no motion (e.g., static regions), a generated noise stream or equalized noise stream may be maximally applied or such that more detail may be provided (e.g., in such contexts, noise may provide detail or may be perceived as detail by a human viewer) for such pixels or regions of little or no motion.

Processing may continue from operation604or operation606at operation607, where pixel blending may be performed based on the parameters determined at operations603,604or operations605,606as discussed as well as other pixel blending parameters to generate output pixels or regions. For example, pixel blending may be performed based on one or more of a reference image (e.g., reference image112), an input image (e.g., input image143), a noise reduced image (e.g., noise reduced image117), and a noise stream (e.g., noise stream142or equalized noise stream148) responsive, in part, to motion information (e.g., motion information146) to generate output pixels (e.g., of output image113). Such pixel blending may be based on weighting factors applied to each image and the noise stream with those factors determined or adjusted using the techniques discussed with respect to decision operation602and operations603-606and other factors or parameters.

Processing may continue at decision operation608, where a determination may be made as to whether the current pixel or region is the last pixel or region to be processed. If not, processing may continue at operation601as discussed above. If so, processing may continue at end operation609, where the output pixels may be provided as an output image and processing may end.

Returning toFIG. 1, as shown, output image113may be provided by pixel blending135for use as a reference image for a subsequent raw input image (if any). Furthermore, output image113may be provided to another module of system100, to a memory of system100, to a display of system100, or the like. For example, output image113may be used by other modules for further processing, saved to memory for use by a user, displayed to a user, or the like. Output image113may include any suitable data or data format and may be characterized as an output image, an output video frame, a temporal noise reduced output image or video frame, or the like.

As discussed, a noise stream may be leveraged in temporal noise reduction to provide lower noise for moving and/or occluded regions. Using such techniques, strong application of spatial noise reduction may be avoided in such regions, which may maintain detail in such regions. Furthermore, higher detail preservation and higher contrast may be provided by adding back the noise stream (e.g., very high frequency signals) to static regions. Such techniques may provide detail recovery in such regions. Furthermore, by streaming noise (e.g., a difference signal between an input and output of a spatial noise reduction component placed before the temporal noise reduction component), the spatial noise reduction before temporal noise reduction may provide more smoothing, advantageously providing less noise to the temporal noise reduction such that the temporal noise reduction may provide more robust processing.

FIG. 7illustrates an example process700for providing temporal noise reduction, arranged in accordance with at least some implementations of the present disclosure. Process700may include one or more operations701-709as illustrated inFIG. 7. Process700may be performed by a system (e.g., system100or any other devices or systems discussed herein) or portions of process700may be performed by a system to provide temporal noise reduction. Process700or portions thereof may be repeated for any number input images, frames of video, or the like. For example, process700may provide a temporal noise reduced output image.

As shown, process700may begin at operation701, where noise reduction may be performed on an input image such as a raw image to generate a noise reduced image. The input image may be from an image sensor and in a color filter array domain (e.g., not demosaiced) or the input image may be in any suitable domain such as an RGB domain, a YUV domain, or the like.

Processing may continue at operation702, where a noise stream may be generated based on the input image and the noise reduced image. The noise stream may be generated using any suitable technique or techniques. In an embodiment, the noise stream may be generated by differencing the noise reduced image from the input image. For example, the noise stream may be in a color filter array domain or any suitable domain such as an RGB domain, a YUV domain, or the like based on the domain of the input image. The noise stream may include any suitable data or data structure such as a noise value for all or some pixels of the input image. The noise stream may correspond to a luma channel and/or one or more chroma channels.

Processing may continue at operation703, where the noise stream may be converted to the luma domain to generate a luma noise stream or the like. The conversion of the noise stream to the luma domain may include converting from the color filter array domain, RGB domain, or YUV domain or the like. Such conversion may include removing color channel dependencies in the noise stream as discussed with respect toFIG. 4and elsewhere herein. As discussed, in some examples, the noise stream may be implemented in one or more chroma channels or domains. In such examples, operation703may be skipped.

Processing may continue at operation704, where the noise stream may be equalized or adjusted based on local luminance of an input image. The noise stream may be equalized or adjusted based on local luminance using any suitable technique or techniques. In an embodiment, local luminance of an input image (e.g., the input image of operation701or the noise reduced image generated at operation701) may be generated based on local averaging of luminance values around pixel locations of the input image. Any window size may be used for such averaging such as a 2×2 pixel window or the like. Based on the local luminance map (e.g., luminance values for each pixel location), a local luminance mapping may be applied to generate a local luminance gain value for each pixel location. The local luminance mapping may provide for higher gain values for higher luminance areas (e.g., lower values) and lower gain values for lower luminance areas (e.g., higher values) as discussed herein. The noise stream may be equalized or adjusted based on the gain values by multiplying the noise stream value for a pixel location and the gain value for a pixel location to generate a noise stream equalized or adjusted based on local luminance. As discussed, in some examples, the noise stream may, in addition or in the alternative to being provided in a luma domain, be implemented in one or more chroma channels or domains. In such examples, operation704may be skipped or an additional operation may be provided to equalize or adjust the noise stream based on local chrominance as discussed herein.

Processing may continue at operation705, where the noise stream may be equalized or adjusted based on content of an input image. The noise stream may be equalized or adjusted based on content detection of the input image using any suitable technique or techniques. In an embodiment, content levels of an input image (e.g., the input image of operation701or the noise reduced image generated at operation701) may be generated based on content detection performed on the input image. Any suitable content detection techniques such as edge detection, texture detection, or the like may be used. Based on the content level map (e.g., content level values for each pixel location), a content level mapping may be applied to generate a content level gain value for each pixel location. The content level mapping may provide for higher gain values for higher content level values and lower gain values for lower content level values as discussed herein. The noise stream may be equalized or adjusted based on the gain values by multiplying the noise stream value for a pixel location and the gain value for a pixel location to generate a noise stream equalized or adjusted based on local luminance.

Processing may continue at operation706, where the noise stream may be equalized or adjusted based on radial distances from an optical center of an input image. The noise stream may be equalized or adjusted based on radial distances from an optical center of the input image using any suitable technique or techniques. In an embodiment, radial distances of an input image (e.g., the input image of operation701or the noise reduced image generated at operation701) may be generated using any suitable technique or techniques. In an embodiment, the radial distances from an optical center may be known or saved before the input image is generated. Based on the radial distances map (e.g., radial distances values for each pixel location), a radial distances mapping may be applied to generate a radial distances gain value for each pixel location. The radial distances mapping may provide for higher gain values for lower radial distance values and lower gain values for higher radial distance values as discussed herein. The noise stream may be equalized or adjusted based on the gain values by multiplying the noise stream value for a pixel location and the gain value for a pixel location to generate a noise stream equalized or adjusted based on local luminance.

Processing may continue at operation707, where the noise stream may be equalized or adjusted based on user preference. For example, a user or developer or the like may adjust the gain of the noise stream based on a user preference mapping, a gain value applied to the entire noise stream, or the like. Although illustrated with noise stream equalization or adjustment based on local luminance, content detection, radial distance from optical center, and user preference, process700may skip one or more of such noise stream equalizations or adjustments. Such noise stream equalization or adjustment may generate an equalized noise stream as discussed herein.

Processing may continue at operation708, where motion information may be generated based on an input image and a reference image (e.g., a previously processed image). The motion information may be generated using any suitable technique or techniques. In an embodiment, local motion estimation may be performed based on an input image (e.g., the input image of operation701or the noise reduced image generated at operation701) and a reference image and the resultant motion vector field may be analyzed for broken trajectories. Such broken trajectories may be indicated in the motion information using a value dedicated to motion outside of the range of the motion search and broken trajectories and the like. In other examples, local motion estimation may not be provided and motion confidence values may be determined representative of whether motion is detected. For example, motion confidence values may be generated based on comparing an input image to a reference image such that those areas that match are provided low motion values or motion confidence values and those areas that do not match are provided high motion values or motion confidence values. The resultant motion information may include motion vectors, indicators of local motion, motion values, motion confidence values, or the like.

Processing may continue at operation709, where pixel blending may be performed based on the equalized noise stream, the reference image, the input image, and the noise reduced image based on the motion information generated at operation708. The pixel blending may be performed using any suitable technique or techniques. As discussed, the pixel blending may include fully applying the equalized noise stream to image regions corresponding to little or no motion and attenuating or not applying the equalized noise stream to image regions corresponding to fast motion or occluded regions as indicated by the motion information.

Process700may be repeated any number of times either in series or in parallel for any number of images, video frames, or the like. As discussed, process700may provide for temporal noise reduction of the images, video frames, or the like.

FIG. 8is a flow diagram illustrating an example process800for providing temporal noise reduction, arranged in accordance with at least some implementations of the present disclosure. Process800may include one or more operations801-803as illustrated inFIG. 8. Process800may form at least part of a temporal noise reduction process. By way of non-limiting example, process800may form at least part of a temporal noise reduction as performed by system100as discussed herein. Furthermore, process800will be described herein with reference to system900ofFIG. 9.

FIG. 9is an illustrative diagram of an example system900for providing temporal noise reduction, arranged in accordance with at least some implementations of the present disclosure. As shown inFIG. 9, system900may include a central processor901, an image processor902, a memory903, a camera904, and a display905. As discussed, in some embodiments system900may not include camera904and/or display905. Also as shown, central processor901may include or implement noise reduction module101, demosaic module102, content detection module131, local motion estimation module132, trajectory break module133, noise equalization module134, pixel blending module135, and spatial noise reduction module136. In an embodiment, content detection module131, local motion estimation module132, trajectory break module133, noise equalization module134, pixel blending module135, and spatial noise reduction module136may be implemented by a temporal noise reduction module or component. In an embodiment, demosaic module102may not be implemented. In such examples, image processor902may provide a demosaiced image (e.g., in YUV, RGB or any other suitable color space) to noise reduction module101, content detection module131, local motion estimation module132, trajectory break detection module133, noise equalization module134, pixel blending module135, and/or spatial noise reduction module136. In an embodiment, content detection module131, local motion estimation module132, trajectory break detection module133, noise equalization module134, pixel blending module135, and/or spatial noise reduction module136may be characterized as a temporal noise reduction module or component or the like. In an embodiment, the temporal noise reduction module or component may be provided as post processing with respect to an image processor or image processing pipeline. In the example of system900, memory903may store image data, video frame data, noise reduction image data, reference image data, detail level data, content level data, local motion data, motion information data, noise stream data, equalized noise stream data, parameters, thresholds, or any other data discussed herein.

As shown, in some examples, noise reduction module101, demosaic module102, content detection module131, local motion estimation module132, trajectory break module133, noise equalization module134, pixel blending module135, and spatial noise reduction module136may be implemented via central processor901. In other examples, one or more or portions of noise reduction module101, demosaic module102, content detection module131, local motion estimation module132, trajectory break module133, noise equalization module134, pixel blending module135, and spatial noise reduction module136may be implemented via image processor902, an image processing unit, an image processing pipeline, or the like. In some examples, noise reduction module101, demosaic module102, content detection module131, local motion estimation module132, trajectory break module133, noise equalization module134, pixel blending module135, and spatial noise reduction module136may be implemented in hardware as a system-on-a-chip (SoC).

Image processor902may include any number and type of image or graphics processing units that may provide the operations as discussed herein. Such operations may be implemented via software or hardware or a combination thereof. For example, image processor902may include circuitry dedicated to manipulate and/or analyze images obtained from memory903. Central processor901may include any number and type of processing units or modules that may provide control and other high level functions for system900and/or provide any operations as discussed herein. Memory903may be any type of memory such as volatile memory (e.g., Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), etc.) or non-volatile memory (e.g., flash memory, etc.), and so forth. In a non-limiting example, memory903may be implemented by cache memory. In an embodiment, one or more or portions of noise reduction module101, demosaic module102, content detection module131, local motion estimation module132, trajectory break module133, noise equalization module134, pixel blending module135, and spatial noise reduction module136may be implemented via an execution unit (EU) of image processor902. The EU may include, for example, programmable logic or circuitry such as a logic core or cores that may provide a wide array of programmable logic functions. In an embodiment, one or more or portions of noise reduction module101, demosaic module102, content detection module131, local motion estimation module132, trajectory break module133, noise equalization module134, pixel blending module135, and spatial noise reduction module136may be implemented via dedicated hardware such as fixed function circuitry or the like. Fixed function circuitry may include dedicated logic or circuitry and may provide a set of fixed function entry points that may map to the dedicated logic for a fixed purpose or function.

Returning to discussion ofFIG. 8, process800may begin at operation801, where a noise reduced image may be generated based on a noise reduction of an input image. The noise reduced image may be generated using any suitable technique or techniques. In an embodiment, noise reduction module101as implemented via central processor901may generate the noise reduced image based on spatial noise reduction of the input image. The input image may be any suitable image in any suitable color space. In an embodiment, the input image may be a demosaiced input image. In an embodiment, the input image may be in a color filter array domain. In such an embodiment, the noise reduced image may be demosaiced to generate an input image for temporal noise reduction processing.

Processing may continue at operation802, where a noise stream corresponding to the input image may be generated based on noise reduced image and the noise reduced image. The noise stream may be generated using any suitable technique or techniques. In an embodiment, a differencer as implemented via central processor901may generate the noise stream as a difference between the noise reduced image and the noise reduced image. The noise stream may be in any suitable color space. In an embodiment, the noise stream may include a luma and chroma components. In an embodiment, the noise stream may include color channels. In an embodiment, the noise stream may be in a color filter array domain.

Processing may continue at operation803, where at least a portion of the noise stream may be adaptively combined with a reference image corresponding to the input image and a second noise reduced image corresponding to the input image to generate a temporal noise reduced output image. The noise stream may be adaptively combined with the images using any suitable technique or techniques. In an embodiment, noise equalization module134and/or pixel blending module135as implemented via central processor901may adaptively combine the noise stream with the images. In an embodiment, the noise reduced image generated at operation801may be the second noise reduced image. In an embodiment, the noise reduced image generated at operation801may undergo further spatial reduction to generate the second noise reduced image. For example, spatial noise reduction module136as implemented via central processor901may generate the second noise reduced image based on the noise reduced image. The reference image may be generated using any suitable technique or techniques. In an embodiment, the reference image may be a prior processed (e.g., prior temporal noise reduced output image generated via process800).

As discussed, the noise stream may be adaptively combined with a reference image and a noise reduced image using any suitable technique or techniques such as pixel blending techniques. In an embodiment, the noise stream may be adaptively combined with the reference image, the noise reduced image generated at operation801, and the second noise reduced image based on motion information corresponding to the input image. For example, adaptively combining the portion of the noise stream may include applying the noise stream based on motion information corresponding to the input image. In an embodiment, the motion information may include a first location having a first motion value and a second location having a second motion value greater than the first motion value and applying the noise stream may include applying the noise stream at a greater level at the first location than the second location. In an embodiment, the local motion value information may include a first location having a first motion value less than a threshold and a second location having a second motion value greater than the threshold and applying the noise stream may include applying the noise stream at the first location and not applying the noise stream at the second location.

In an embodiment, adaptively combining the portion of the noise stream may include adjusting the noise stream based on one or more of local luminance corresponding to the input image, detected content corresponding to the input image, or a radial distance from an optical center of the input image.

In an embodiment, adaptively combining the portion of the noise stream may include adjusting the noise stream based on a local luminance map corresponding to the input image by applying a local luminance dependent gain to the noise stream. For example, the noise stream may be attenuated for areas of low luminance (e.g., dark areas) and not attenuated for areas of high luminance (e.g., bright areas). For example, the local luminance map may have a first local luminance value at a first location and a second local luminance value less than the first local luminance value at a second location and the local luminance dependent gain may provide a first local luminance gain value for the first location and a second local luminance gain value greater than the first local luminance gain value for the second location responsive to the local luminance map.

In an embodiment, adaptively combining the portion of the noise stream may include adjusting the noise stream based on a local chrominance map corresponding to the input image by applying a local chrominance dependent gain to the noise stream. For example, the noise stream may be attenuated for areas of low chrominance and not attenuated for areas of high chrominance. For example, the local chrominance map may have a first local chrominance value at a first location and a second local chrominance value less than the first local chrominance value at a second location and the local chrominance dependent gain may provide a first local chrominance gain value for the first location and a second local chrominance gain value greater than the first local chrominance gain value for the second location responsive to the local chrominance map.

In an embodiment, adaptively combining the portion of the noise stream may include adjusting the noise stream based on a content level map corresponding to the input image by applying a content level dependent gain to the noise stream. For example, the content level map may have a first content detection value at a first location and a second content detection value less than the first content detection value at a second location and the content level dependent gain may provide a first content level gain value for the first location and a second content level gain value less than the first level dependent gain value for the second location responsive to the content level map.

In an embodiment, adaptively combining the portion of the noise stream may include adjusting the noise stream based on a radial distance from an optical center by applying a radial distance dependent gain to the noise stream. For example, a first radial distance adaptive gain value for a first location may be less than a second radial distance adaptive gain value for a second location responsive to the first location being a greater distance from the optical center than the second location.

As discussed, the noise stream may be adjusted or equalized prior to or as a part o adaptively combining the noise stream with the discussed images. In an embodiment, the noise stream may be converted to a luma noise stream prior to such adjusting, equalizing, or adaptively combining the noise stream with the images.

Furthermore, as discussed, process800may include adaptively combining the noise stream with the reference image and the second noise reduced image comprises pixel blending the noise stream, the reference image, the second noise reduced image, and the noise reduced image based on motion information corresponding to the input image. In an embodiment, content detection may be performed based on the noise reduced image to generate a content level map, local motion estimation may be performed based on the noise reduced image and the reference image to generate a local motion map, trajectory break detection may be performed on the local motion map to generate a local motion confidence map, a local luminance map may be generated based on the noise reduced image, and the noise stream may be adjusted based on the content level map and the local luminance map. For example, adaptively combining the portion of the noise stream may include applying the noise stream based on the final local motion map and the local motion confidence map.

Process800may be repeated any number of times either in series or in parallel for any number of images, video frames, or the like. As discussed, process800may provide for temporal noise reduction of the images, video frames, or the like.

Various components of the systems described herein may be implemented in software, firmware, and/or hardware and/or any combination thereof. For example, various components of the devices or systems discussed herein may be provided, at least in part, by hardware of a computing System-on-a-Chip (SoC) such as may be found in a computing system such as, for example, a smart phone. Those skilled in the art may recognize that systems described herein may include additional components that have not been depicted in the corresponding figures. For example, the systems discussed herein may include additional components that have not been depicted in the interest of clarity.

FIG. 10is an illustrative diagram of an example system1000, arranged in accordance with at least some implementations of the present disclosure. In various implementations, system1000may be a mobile device system although system1000is not limited to this context. For example, system1000may be incorporated into a personal computer (PC), laptop computer, ultra-laptop computer, tablet, touch pad, portable computer, handheld computer, palmtop computer, personal digital assistant (PDA), cellular telephone, combination cellular telephone/PDA, television, smart device (e.g., smart phone, smart tablet or smart television), mobile interne device (MID), messaging device, data communication device, cameras (e.g. point-and-shoot cameras, super-zoom cameras, digital single-lens reflex (DSLR) cameras), a surveillance camera, a surveillance system including a camera, and so forth.

In various implementations, system1000includes a platform1002coupled to a display1020. Platform1002may receive content from a content device such as content services device(s)1030or content delivery device(s)1040or other content sources such as image sensors1019. For example, platform1002may receive image data as discussed herein from image sensors1019or any other content source. A navigation controller1050including one or more navigation features may be used to interact with, for example, platform1002and/or display1020. Each of these components is described in greater detail below.

Processor1010may be implemented as a Complex Instruction Set Computer (CISC) or Reduced Instruction Set Computer (RISC) processors, x86 instruction set compatible processors, multi-core, or any other microprocessor or central processing unit (CPU). In various implementations, processor1010may be dual-core processor(s), dual-core mobile processor(s), and so forth.

Image signal processor1017may be implemented as a specialized digital signal processor or the like used for image processing. In some examples, image signal processor1017may be implemented based on a single instruction multiple data or multiple instruction multiple data architecture or the like. In some examples, image signal processor1017may be characterized as a media processor. As discussed herein, image signal processor1017may be implemented based on a system on a chip architecture and/or based on a multi-core architecture.

Graphics subsystem1015may perform processing of images such as still or video for display. Graphics subsystem1015may be a graphics processing unit (GPU) or a visual processing unit (VPU), for example. An analog or digital interface may be used to communicatively couple graphics subsystem1015and display1020. For example, the interface may be any of a High-Definition Multimedia Interface, DisplayPort, wireless HDMI, and/or wireless HD compliant techniques. Graphics subsystem1015may be integrated into processor1010or chipset1005. In some implementations, graphics subsystem1015may be a stand-alone device communicatively coupled to chipset1005.

In various implementations, display1020may include any television type monitor or display. Display1020may include, for example, a computer display screen, touch screen display, video monitor, television-like device, and/or a television. Display1020may be digital and/or analog. In various implementations, display1020may be a holographic display. Also, display1020may be a transparent surface that may receive a visual projection. Such projections may convey various forms of information, images, and/or objects. For example, such projections may be a visual overlay for a mobile augmented reality (MAR) application. Under the control of one or more software applications1016, platform1002may display user interface1022on display1020.

In various implementations, content services device(s)1030may be hosted by any national, international and/or independent service and thus accessible to platform1002via the Internet, for example. Content services device(s)1030may be coupled to platform1002and/or to display1020. Platform1002and/or content services device(s)1030may be coupled to a network1060to communicate (e.g., send and/or receive) media information to and from network1060. Content delivery device(s)1040also may be coupled to platform1002and/or to display1020.

Image sensors1019may include any suitable image sensors that may provide image data based on a scene. For example, image sensors1019may include a semiconductor charge coupled device (CCD) based sensor, a complimentary metal-oxide-semiconductor (CMOS) based sensor, an N-type metal-oxide-semiconductor (NMOS) based sensor, or the like. For example, image sensors1019may include any device that may detect information of a scene to generate image data.

Movements of the navigation features of navigation controller1050may be replicated on a display (e.g., display1020) by movements of a pointer, cursor, focus ring, or other visual indicators displayed on the display. For example, under the control of software applications1016, the navigation features located on navigation controller1050may be mapped to virtual navigation features displayed on user interface1022, for example. In various embodiments, navigation controller1050may not be a separate component but may be integrated into platform1002and/or display1020. The present disclosure, however, is not limited to the elements or in the context shown or described herein.

In various implementations, drivers (not shown) may include technology to enable users to instantly turn on and off platform1002like a television with the touch of a button after initial boot-up, when enabled, for example. Program logic may allow platform1002to stream content to media adaptors or other content services device(s)1030or content delivery device(s)1040even when the platform is turned “off.” In addition, chipset1005may include hardware and/or software support for 5.1 surround sound audio and/or high definition 7.1 surround sound audio, for example. Drivers may include a graphics driver for integrated graphics platforms. In various embodiments, the graphics driver may comprise a peripheral component interconnect (PCI) Express graphics card.

In various implementations, any one or more of the components shown in system1000may be integrated. For example, platform1002and content services device(s)1030may be integrated, or platform1002and content delivery device(s)1040may be integrated, or platform1002, content services device(s)1030, and content delivery device(s)1040may be integrated, for example. In various embodiments, platform1002and display1020may be an integrated unit. Display1020and content service device(s)1030may be integrated, or display1020and content delivery device(s)1040may be integrated, for example. These examples are not meant to limit the present disclosure.

As described above, system1000may be embodied in varying physical styles or form factors.FIG. 11illustrates an example small form factor device1100, arranged in accordance with at least some implementations of the present disclosure. In some examples, system1000may be implemented via device1100. In other examples, device100or portions thereof may be implemented via device1100. In various embodiments, for example, device1100may be implemented as a mobile computing device having wireless capabilities. A mobile computing device may refer to any device having a processing system and a mobile power source or supply, such as one or more batteries, for example.

Examples of a mobile computing device may include a personal computer (PC), laptop computer, ultra-laptop computer, tablet, touch pad, portable computer, handheld computer, palmtop computer, personal digital assistant (PDA), cellular telephone, combination cellular telephone/PDA, smart device (e.g., smart phone, smart tablet or smart mobile television), mobile internet device (MID), messaging device, data communication device, cameras, and so forth.

Examples of a mobile computing device also may include computers that are arranged to be worn by a person, such as wrist computers, finger computers, ring computers, eyeglass computers, belt-clip computers, arm-band computers, shoe computers, clothing computers, and other wearable computers. In various embodiments, for example, a mobile computing device may be implemented as a smart phone capable of executing computer applications, as well as voice communications and/or data communications. Although some embodiments may be described with a mobile computing device implemented as a smart phone by way of example, it may be appreciated that other embodiments may be implemented using other wireless mobile computing devices as well. The embodiments are not limited in this context.

As shown inFIG. 11, device1100may include a housing with a front1101and a back1102. Device1100includes a display1104, an input/output (I/O) device1106, and an integrated antenna1108. Device1100also may include navigation features1111. I/O device1106may include any suitable I/O device for entering information into a mobile computing device. Examples for I/O device1106may include an alphanumeric keyboard, a numeric keypad, a touch pad, input keys, buttons, switches, microphones, speakers, voice recognition device and software, and so forth. Information also may be entered into device1100by way of microphone (not shown), or may be digitized by a voice recognition device. As shown, device1100may include a camera1105(e.g., including a lens, an aperture, and an imaging sensor) and a flash1110integrated into back1102(or elsewhere) of device1100. In other examples, camera1105and/or flash1110may be integrated into front1101of device1100and/or additional cameras (e.g., such that device1100has front and back cameras) may be provided.

The following examples pertain to further embodiments.

In one or more first embodiments, a method for providing temporal noise reduction comprises generating a noise reduced image based on a noise reduction of an input image, generating a noise stream corresponding to the input image based on the input image and the noise reduced image, and adaptively combining at least a portion of the noise stream with a reference image corresponding to the input image and a second noise reduced image corresponding to the input image to generate a temporal noise reduced output image.

Further to the first embodiments, adaptively combining the portion of the noise stream comprises applying the noise stream based on motion information corresponding to the input image.

Further to the first embodiments, adaptively combining the portion of the noise stream comprises applying the noise stream based on motion information corresponding to the input image, wherein the motion information comprises a first location having a first motion value and a second location having a second motion value greater than the first motion value, and wherein applying the noise stream comprises applying the noise stream at a greater level at the first location than the second location.

Further to the first embodiments, adaptively combining the portion of the noise stream comprises applying the noise stream based on motion information corresponding to the input image, wherein the motion information comprises a first location having a first motion value less than a threshold and a second location having a second motion value greater than the threshold, and wherein applying the noise stream comprises applying the noise stream at the first location and not applying the noise stream at the second location.

Further to the first embodiments, adaptively combining the noise stream with the reference image and the second noise reduced image comprises pixel blending the noise stream, the reference image, the second noise reduced image, and the noise reduced image based on motion information corresponding to the input image.

Further to the first embodiments, adaptively combining the portion of the noise stream comprises adjusting the noise stream based on at least one of local luminance corresponding to the input image, local chrominance corresponding to the input image, detected content corresponding to the input image, or a radial distance from an optical center of the input image.

Further to the first embodiments, adaptively combining the portion of the noise stream comprises adjusting the noise stream based on a local luminance map corresponding to the input image by applying a local luminance dependent gain to the noise stream, wherein the local luminance map has a first local luminance value at a first location and a second local luminance value less than the first local luminance value at a second location, and wherein the local luminance dependent gain provides a first local luminance gain value for the first location and a second local luminance gain value greater than the first local luminance gain value for the second location responsive to the local luminance map.

Further to the first embodiments, adaptively combining the portion of the noise stream comprises adjusting the noise stream based on a content level map corresponding to the input image by applying a content level dependent gain to the noise stream, wherein the content level map has a first content detection value at a first location and a second content detection value less than the first content detection value at a second location, and wherein the content level dependent gain provides a first content level gain value for the first location and a second content level gain value less than the first level dependent gain value for the second location responsive to the content level map.

Further to the first embodiments, adaptively combining the portion of the noise stream comprises adjusting the noise stream based on a radial distance from an optical center by applying a radial distance dependent gain to the noise stream, wherein a first radial distance adaptive gain value for a first location is less than a second radial distance adaptive gain value for a second location responsive to the first location being a greater distance from the optical center than the second location.

Further to the first embodiments, the input image and the noise stream are in a color filter array domain and the method further comprises demosaicing the noise reduced image to generate a demosaiced image, applying spatial noise reduction to the demosaiced image to generate the second noise reduced image, and converting the noise stream to a luma noise stream prior to adaptively combining the noise stream with the reference image and the second noise reduced image.

Further to the first embodiments, the input image comprises a demosaiced input image and the method further comprises applying a second spatial noise reduction to the input image to generate the second noise reduced image.

Further to the first embodiments, the method further comprises performing content detection based on the noise reduced image to generate a content level map, performing local motion estimation based on the noise reduced image and the reference image to generate a local motion map, performing trajectory break detection on the local motion map to generate a local motion confidence map, generating a local luminance map based on the noise reduced image, and adjusting the noise stream based on the content level map and the local luminance map, wherein adaptively combining the portion of the noise stream comprises applying the noise stream based on the local motion map and the local motion confidence map.

In one or more second embodiments, a system for providing provide temporal noise reduction comprises a memory configured to store an input image and an image processor coupled to the memory, the image processor to generate a noise reduced image based on a noise reduction of the input image, to generate a noise stream corresponding to the input image based on the input image and the noise reduced image, and to adaptively combine at least a portion of the noise stream with a reference image corresponding to the input image and a second noise reduced image corresponding to the input image to generate a temporal noise reduced output image.

Further to the second embodiments, the image processor to adaptively combine the portion of the noise stream comprises the image processor to apply the noise stream based on motion information corresponding to the input image.

Further to the second embodiments, the image processor to adaptively combine the portion of the noise stream comprises the image processor to apply the noise stream based on motion information corresponding to the input image, wherein the motion information comprises a first location having a first motion value and a second location having a second motion value greater than the first motion value, and wherein the image processor to apply the noise stream comprises the image processor to apply the noise stream at a greater level at the first location than the second location.

Further to the second embodiments, the image processor to adaptively combine the portion of the noise stream comprises the image processor to apply the noise stream based on motion information corresponding to the input image, wherein the motion information comprises a first location having a first motion value less than a threshold and a second location having a second motion value greater than the threshold, and wherein the image processor to apply the noise stream comprises the image processor to apply the noise stream at the first location and to not apply the noise stream at the second location.

Further to the second embodiments, the image processor to adaptively combine the portion of the noise stream comprises the image processor to apply the noise stream based on motion information corresponding to the input image, wherein the motion information comprises a first location having a first motion value and a second location having a second motion value greater than the first motion value and the image processor to apply the noise stream comprises the image processor to apply the noise stream at a greater level at the first location than the second location and/or wherein the motion information comprises a first location having a first motion value less than a threshold and a second location having a second motion value greater than the threshold and the image processor to apply the noise stream comprises the image processor to apply the noise stream at the first location and to not apply the noise stream at the second location.

Further to the second embodiments, the image processor to adaptively combine the noise stream with the reference image and the second noise reduced image comprises the image processor to pixel blend the noise stream, the reference image, the second noise reduced image, and the noise reduced image based on motion information corresponding to the input image.

Further to the second embodiments, the image processor to adaptively combine the portion of the noise stream comprises the image processor to adjust the noise stream based on at least one of local luminance corresponding to the input image, local chrominance corresponding to the input image, detected content corresponding to the input image, or a radial distance from an optical center of the input image.

Further to the second embodiments, the image processor to adaptively combine the portion of the noise stream comprises the image processor to adjust the noise stream based on a local luminance map corresponding to the input image by applying a local luminance dependent gain to the noise stream, wherein the local luminance map has a first local luminance value at a first location and a second local luminance value less than the first local luminance value at a second location, and wherein the local luminance dependent gain provides a first local luminance gain value for the first location and a second local luminance gain value greater than the first local luminance gain value for the second location responsive to the local luminance map.

Further to the second embodiments, the image processor to adaptively combine the portion of the noise stream comprises the image processor to adjust the noise stream based on a content level map corresponding to the input image by applying a content level dependent gain to the noise stream, wherein the content level map has a first content detection value at a first location and a second content detection value less than the first content detection value at a second location, and wherein the content level dependent gain provides a first content level gain value for the first location and a second content level gain value less than the first level dependent gain value for the second location responsive to the content level map.

Further to the second embodiments, the image processor to adaptively combine the portion of the noise stream comprises the image processor to adjust the noise stream based on a radial distance from an optical center by applying a radial distance dependent gain to the noise stream, wherein a first radial distance adaptive gain value for a first location is less than a second radial distance adaptive gain value for a second location responsive to the first location being a greater distance from the optical center than the second location.

Further to the second embodiments, the input image and the noise stream are in a color filter array domain and the image processor is further to demosaic the noise reduced image to generate a demosaiced image, to apply spatial noise reduction to the demosaiced image to generate the second noise reduced image, and to convert the noise stream to a luma noise stream prior to adaptively combining the noise stream with the reference image and the second noise reduced image.

Further to the second embodiments, the input image comprises a demosaiced input image and the image processor is further to apply a second spatial noise reduction to the input image to generate the second noise reduced image.

In one or more third embodiments, a system comprises means for generating a noise reduced image based on a noise reduction of an input image, means for generating a noise stream corresponding to the input image based on the input image and the noise reduced image, and means for adaptively combining at least a portion of the noise stream with a reference image corresponding to the input image and a second noise reduced image corresponding to the input image to generate a temporal noise reduced output image.

Further to the third embodiments, the means for adaptively combining the portion of the noise stream comprise means for applying the noise stream based on motion information corresponding to the input image.

Further to the third embodiments, the means for adaptively combining the portion of the noise stream comprise means for applying the noise stream based on motion information corresponding to the input image, wherein the motion information comprises a first location having a first motion value and a second location having a second motion value greater than the first motion value, and wherein the means for applying the noise stream comprise means for applying the noise stream at a greater level at the first location than the second location.

Further to the third embodiments, the means for adaptively combining the portion of the noise stream comprise means for applying the noise stream based on motion information corresponding to the input image, wherein the motion information comprises a first location having a first motion value less than a threshold and a second location having a second motion value greater than the threshold, and wherein the means for applying the noise stream comprise means for applying the noise stream at the first location and not applying the noise stream at the second location.

Further to the third embodiments, the means for adaptively combining the portion of the noise stream comprise means for adjusting the noise stream based on a local luminance map corresponding to the input image by applying a local luminance dependent gain to the noise stream, wherein the local luminance map has a first local luminance value at a first location and a second local luminance value less than the first local luminance value at a second location, and wherein the local luminance dependent gain provides a first local luminance gain value for the first location and a second local luminance gain value greater than the first local luminance gain value for the second location responsive to the local luminance map.

Further to the third embodiments, the means for adaptively combining the portion of the noise stream comprise means for adjusting the noise stream based on a content level map corresponding to the input image by applying a content level dependent gain to the noise stream, wherein the content level map has a first content detection value at a first location and a second content detection value less than the first content detection value at a second location, and wherein the content level dependent gain provides a first content level gain value for the first location and a second content level gain value less than the first level dependent gain value for the second location responsive to the content level map.

Further to the third embodiments, the means for adaptively combining the portion of the noise stream comprises means for adjusting the noise stream based on a radial distance from an optical center by applying a radial distance dependent gain to the noise stream, wherein a first radial distance adaptive gain value for a first location is less than a second radial distance adaptive gain value for a second location responsive to the first location being a greater distance from the optical center than the second location.

In one or more fourth embodiments, at least one machine readable medium comprises a plurality of instructions that, in response to being executed on a device, cause the device to provide temporal noise reduction by generating a noise reduced image based on a noise reduction of an input image, generating a noise stream corresponding to the input image based on the input image and the noise reduced image, and adaptively combining at least a portion of the noise stream with a reference image corresponding to the input image and a second noise reduced image corresponding to the input image to generate a temporal noise reduced output image.

Further to the fourth embodiments, adaptively combining the portion of the noise stream comprises applying the noise stream based on motion information corresponding to the input image.

Further to the fourth embodiments, adaptively combining the portion of the noise stream comprises adjusting the noise stream based on a local luminance map corresponding to the input image by applying a local luminance dependent gain to the noise stream, wherein the local luminance map has a first local luminance value at a first location and a second local luminance value less than the first local luminance value at a second location, and wherein the local luminance dependent gain provides a first local luminance gain value for the first location and a second local luminance gain value greater than the first local luminance gain value for the second location responsive to the local luminance map.

Further to the fourth embodiments, adaptively combining the portion of the noise stream comprises adjusting the noise stream based on a content level map corresponding to the input image by applying a content level dependent gain to the noise stream, wherein the content level map has a first content detection value at a first location and a second content detection value less than the first content detection value at a second location, and wherein the content level dependent gain provides a first content level gain value for the first location and a second content level gain value less than the first level dependent gain value for the second location responsive to the content level map.

Further to the fourth embodiments, adaptively combining the portion of the noise stream comprises adjusting the noise stream based on a radial distance from an optical center by applying a radial distance dependent gain to the noise stream, wherein a first radial distance adaptive gain value for a first location is less than a second radial distance adaptive gain value for a second location responsive to the first location being a greater distance from the optical center than the second location.

Further to the fourth embodiments, the machine readable medium comprises further instructions that, in response to being executed on the device, cause the device to provide temporal noise reduction by performing content detection based on the noise reduced image to generate a content level map, performing local motion estimation based on the noise reduced image and the reference image to generate a local motion map, performing trajectory break detection on the local motion map to generate a final local motion map, generating a local luminance map based on the noise reduced image, and adjusting the noise stream based on the content level map and the local luminance map, wherein adaptively combining the portion of the noise stream comprises applying the noise stream based on the final local motion map.

In one or more fifth embodiments, at least one machine readable medium may include a plurality of instructions that in response to being executed on a computing device, causes the computing device to perform a method according to any one of the above embodiments.

In one or more sixth embodiments, an apparatus may include means for performing a method according to any one of the above embodiments.