Patent Description:
Visual media, such as images and video, can be captured with a camera or video camera. Often, if the camera or video camera includes sufficient features and includes a proper lens, the captured media reflects the color, contrast, sharpness, and/or the like desired by the end-user. However, full-featured cameras or video cameras and/or powerful lenses can be expensive. Furthermore, full-featured cameras or videos cameras and/or powerful lenses can be bulky and difficult to carry in certain situations.

Today, portable devices, such as cell phones or tablets, include a built-in camera. Because portable devices include a wide array of functions in addition to the ability to capture visual media, many people forgo carrying the bulky, expensive cameras and lenses in favor of the portable devices. However, the cameras built into the portable devices have a restricted number of features. In addition, the built-in cameras generally have basic lenses. Thus, the visual media captured by the built-in cameras are generally of a poorer quality and lack the desired color, contrast, sharpness, and/or the like.

Applications available on the portable device or another device can be used to process and enhance the captured visual media. For example, applications can be used by the end-user to adjust color, enhance contrast, sharpen edges, and/or the like. However, many of the applications can be processor-intensive and/or require additional hardware. This can be especially problematic on portable devices, as the applications can increase battery consumption and/or the size of the device.

<CIT> discloses a method which includes receiving stereoscopic image data, where the stereoscopic image data includes a first image obtained from a first camera and a second image obtained from a second camera. The method further includes identifying an object in the first image by analyzing the first image and the second image, and displaying the first image. The identified object in the first image is selectable.

<CIT> discloses an apparatus for enhancing an image. The image comprises a first imaging device for producing images at a first resolution, and a second imaging device for producing images at a second resolution. An image processor aligns the images from the first and second imaging devices to form a plurality of aligned images, computes a flow estimation for each of the aligned images, and uses the flow estimations to enhance regions in a first aligned image with information from at least one other aligned image.

<CIT> discloses blurring simulated in post-processing for captured images. A 3D image is received from a 3D camera, and depth information in the 3D image is used to determine the relative distances of objects in the image. One object is chosen as the subject of the image, and an additional object in the image is identified.

A layer generator divides the 3D image into a series of depth steps and generates a layer that contains the objects in each depth step. The layer generator uses object distances generated by the distance calculator to determine the contents of each depth step. After the layers are generated, the blur factor calculator and blur filter operate in conjunction to apply an image blur to each layer. The layer generator also identifies the layer corresponding to the subject as the subject layer, and the blur factor calculator and blur filter may be configured so that they do not apply image blur to the subject layer.

<CIT> discloses a method for detecting an object in a depth image. The method includes determining a detection window covering a region in the depth image, wherein a location of the detection window is based on a location of a candidate pixel in the depth image, and wherein a size of the detection window is based on a depth value of the candidate pixel and a size of the object. A foreground region in the detection window is segmented based on the depth value of the candidate pixel and the size of the object. A feature vector is determined based on depth values of the pixels in the foreground region and the feature vector is classified to detect the object.

<CIT> discloses methods for emphasizing objects in an image, such as a panoramic image. For example, a method includes receiving a depthmap generated from an optical distancing system, wherein the depthmap includes position data and depth data for each of a plurality of points. The optical distancing system measures physical data. The depthmap is overlaid on the panoramic image according to the position data. Data is received that indicates a location on the panoramic image and, accordingly, a first point of the plurality of points that is associated with the location. The depth data of the first point is compared to depth data of surrounding points to identify an area on the panoramic image corresponding to a subset of the surrounding points. The panoramic image is altered with a graphical effect that indicates the location.

The systems, methods, and devices of the invention each have several aspects, no single one of which is solely responsible for its desirable attributes. Without limiting the scope of this invention as expressed by the claims which follow, some features will now be discussed briefly. After considering this discussion, and particularly after reading the section entitled "Detailed Description" one will understand how the features of this invention provide advantages that include improved communications between access points and stations in a wireless network.

As described above, full-featured cameras or video cameras that have powerful lenses and/or other components can be expensive. Applications available on a device, such as a portable device, can be used to process and enhance captured visual media (e.g., images, video, etc.) to mimic the effects produced by the powerful lenses and/or other components. For example, applications can be used by an end-user to adjust color, enhance contrast, sharpen edges, change geometric shapes, and/or the like. However, many of the applications are processor-intensive and/or require additional hardware to implement some or all of the enhancement features. As an example, the process or determining a portion of the media to enhance can be processor-intensive. In addition, special hardware, such as a depth map engine, could be needed to determine the portion of the media to enhance and/or a portion of the media not to enhance. The processor-intensive nature of such applications and/or the necessity to include additional hardware can be especially problematic on portable devices because the applications can increase battery consumption and/or the size of the device.

Accordingly, the systems and methods described herein use the existing hardware on a device to enable applications to perform enhancement features. In addition, the systems and methods described herein allow the device to perform certain operations, such as those operations that are processor-intensive, while the device is idle, being charged, or otherwise in a state in which battery consumption is not an issue. In this way, the systems and methods described herein allow devices to offer various enhancement features while minimizing battery consumption and maintaining or even reducing the size of the device.

While the techniques described herein can be applied to any type and any number of images or sequences of images (e.g., video), the techniques as disclosed herein are described with respect to stereo images and/or stereo video. Stereo images (or a stereo pair of images) generally include a first image and a second image of a common scene. The first and second images may throughout this disclosure also be referred to as left and right images, left and right views, or left (L) and right (R) frames. The first and second images of the stereo pair can be displayed simultaneously or in rapid succession to create a scene with 3D objects. Stereo images or video can be captured using two cameras and are transmitted as a stream of left frames and right frames. Sometimes, although not common, stereo images or video are transmitted as an L frame and are accompanied by a depth map (D) for that frame. However, in other cases as described herein, stereo images or video are transmitted without a depth map for the frames. The depth map, however, can be computed using the L and R frames. Computing the depth map given the two views (L and R) is usually an involved and computationally intensive process. Real-time implementation usually entails a hardware module exclusively for the purpose of computing disparity (e.g., which can be used to determine the depth map). Often, the best disparity calculation algorithms are so complex that they cannot be implemented in real-time or can only be implemented in real-time by devices with significant processing resources. Thus, computing the depth map generally takes extra design and implementation effort if done from scratch. However, as described herein, the existing system can be reused for computing depth, saving on that effort. When the depth map of the video is available, depth aware enhancements can be applied to the stereo image or video.

The term "disparity" as used in this disclosure generally describes the horizontal offset of a pixel in one image relative to a corresponding pixel in the other image (e.g., a difference in spatial orientation between two images). Corresponding pixels, as used in this disclosure, generally refers to pixels (one in a left image and one in a right image) that are associated with the same point in the 3D object when the left image and right image are synthesized to render a 2D or 3D image.

A plurality of disparity values for a stereo pair of images can be stored in a data structure referred to as a disparity map. The disparity map associated with the stereo pair of images represents a two-dimensional (2D) function, d(x, y), that maps pixel coordinates (x, y) in the first image to disparity values (d), such that the value of d at any given (x, y) coordinate in a first image corresponds to the shift in the x-coordinate that needs to be applied to a pixel at coordinate (x, y) in the second image to find the corresponding pixel in the second image. For example, as a specific illustration, a disparity map may store a d value of <NUM> for a pixel at coordinates (<NUM>, <NUM>) in the first image. In this illustration, given the d value of <NUM>, data describing pixel (<NUM>, <NUM>), such as chroma and luminance values, in the first image, occurs at pixel (<NUM>, <NUM>) in the second image.

The elemental information available in a 2D image is the color value of each pixel. Therefore, one technique for identifying corresponding pixels is to look for the best match of the color value of every pixel in a first image (also called a reference image) among the pixels of a second image (also called a target image), where the target image refers to the companion image in the stereo pair of images. The target image can be one of a left or right image, and the reference image can be the other of the left or right image. As a rectified stereo pair is being considered, the search space in the target image can be limited to the pixels in the same epipolar line as in the reference image. This technique, however, assumes that the color values of the pixels associated with the same point in the object are the same, which is not always a correct assumption. For example, object surfaces generally do not reflect light uniformly in all directions and the two image capture-sensors used to acquire the stereo set may have different colorimetric calibrations. Additionally, the same color value may be found in several pixels along the line, providing several potential matches. Further, a particular pixel or group of pixels might be occluded in the target image, meaning that they are behind an object in the reference image and thus not visible in the 3D image. Thus, disparity values may not be available for pixels in which no corresponding pixel is identified. Interpolation or similar techniques can be used to determine disparity values when such values are otherwise not available.

A depth map can be formed based on the disparity map. For example, the disparity map can be scaled such that the disparity values range from <NUM> to a set number (e.g., <NUM>). The scaled disparity map is referred to herein as a depth map. Generally, the scaled values in the depth map correspond with a grayscale (e.g., an <NUM>-bit grayscale if the set number is <NUM>). Thus, as described and illustrated below, the depth map can be graphically represented via a grayscale image.

Most portable devices include an encoder and/or a decoder to transmit data efficiently to other devices. In particular, the encoder and/or the decoder can be used to compress and/or decompress visual media such that the visual media can more quickly be transmitted across a medium. As described herein, an existing encoder and/or decoder can be leveraged to perform additional functions. For example, the encoder and/or decoder can be used to generate in real-time a depth map and determine depth information in an image or sequence of images. Such depth information can be useful in identifying portions of the image that are related (e.g., pixels that, when combined, comprise an object) and portions of the image that are unrelated (e.g., neighboring pixels that are part of different objects). An end-user can the select a portion of the image, and based on the depth information, portions of the image that are related to the selected portion can be enhanced by a processor of the device and/or portions of the image that are unrelated to the selected portion can be degraded by the processor. The systems and methods as disclosed herein are described in greater detail with respect to <FIG>.

<FIG> is a block diagram illustrating an example system in which a source device enhances a stereo image. As illustrated in <FIG>, system <NUM> may include a source device <NUM> with image source <NUM>, encoder <NUM>, processor <NUM>, and image display <NUM>. Source device <NUM> may comprise a wireless communication device, such as a wireless handset, a so-called cellular or satellite radiotelephone, or any wireless device that can communicate picture and/or video information over a communication channel, in which case the communication channel may comprise a wireless communication channel.

Image source <NUM> provides a stereo pair of images, including first view <NUM> and second view <NUM>, to encoder <NUM> and processor <NUM>. Image source <NUM> can provide first view <NUM> and second view <NUM> to encoder <NUM> and processor <NUM> at the same time or at a different time. For example, first view <NUM> and second view <NUM> can be provided to encoder <NUM> immediately after or soon after first view <NUM> and second view <NUM> are stored in image source <NUM>. First view <NUM> and second view <NUM> can then be provided to processor <NUM> when an end-user indicates a desire to enhance the first view <NUM> and/or the second view <NUM>.

Image source <NUM> may comprise an image sensor array (e.g., a digital still picture camera or digital video camera), a computer-readable storage medium comprising one or more stored images, an interface for receiving digital images from an external source, a processing unit that generates digital images such as by executing a video game or other interactive multimedia source, or other sources of image data. Image source <NUM> may generally correspond to a source of any one or more of captured, pre-captured, and/or computer-generated images. In some examples, image source <NUM> may correspond to a camera of a cellular (i.e., mobile) telephone. In general, references to images in this disclosure include both still pictures as well as frames of video data. Thus, aspects of this disclosure may apply both to still digital pictures as well as frames of captured digital video data or computer-generated digital video data.

For example, image source <NUM> may capture two views of a scene at different perspectives. In various examples, image source <NUM> may comprise a standard two-dimensional camera, a two camera system that provides a stereoscopic view of a scene, a camera array that captures multiple views of the scene, a camera that captures one view plus depth information, or a first camera on a first device and a second camera on a second device that together provide a stereoscopic view of the scene.

Encoder <NUM> uses first view <NUM> and second view <NUM> to generate a depth map <NUM>. Encoder <NUM> transfers the depth map <NUM> to processor <NUM>. Processor <NUM> uses first view <NUM>, second view <NUM>, and the depth map <NUM> to generate enhanced first view <NUM>, which is an enhanced version of first view <NUM>, and/or to generate enhanced second view <NUM>, which is an enhanced version of second view <NUM>. Processor <NUM> transmits enhanced first view <NUM> and/or enhanced second view <NUM> to image display <NUM>.

Based on enhanced first view <NUM> and/or enhanced second view <NUM>, image display <NUM> can render an enhanced two-dimensional or three-dimensional image. For example, image display <NUM> can synthesize enhanced first view <NUM> and enhanced second view <NUM> to form and display a single two-dimensional image. As another example, image display <NUM> can simultaneously or alternately display two-dimensional versions of enhanced first view <NUM> and enhanced second view <NUM> (e.g., to create a three-dimensional effect). As another example, image display <NUM> can synthesize enhanced first view <NUM> and enhanced second view <NUM> to form and display a single three-dimensional image.

In general, the human vision system (HVS) perceives depth based on an angle of convergence to an object. Objects relatively nearer to the viewer are perceived as closer to the viewer due to the viewer's eyes converging on the object at a greater angle than objects that are relatively further from the viewer. To simulate three dimensions in multimedia such as pictures and video, image display <NUM> can display two images to a viewer, one image (left and right) for each of the viewer's eyes. Objects that are located at the same spatial location within the image will be generally perceived as being at the same depth as the screen on which the images are being displayed.

To create the illusion of depth, objects may be shown at slightly different positions in each of the images along the horizontal axis. The difference between the locations of the objects in the two images is referred to as disparity. In general, to make an object appear closer to the viewer, relative to the screen, a negative disparity value may be used, whereas to make an object appear further from the user relative to the screen, a positive disparity value may be used. Pixels with positive or negative disparity may, in some examples, be displayed with more or less resolution to increase or decrease sharpness or blurriness to further create the effect of positive or negative depth from a focal point.

Image display <NUM> may comprise a stereoscopic display or an autostereoscopic display. In general, stereoscopic displays simulate three-dimensions by displaying two images while a viewer wears a head mounted unit, such as goggles or glasses, that direct one image into one eye and a second image into the other eye. In some examples, each image is displayed simultaneously (e.g., with the use of polarized glasses or color-filtering glasses). In some examples, the images are alternated rapidly, and the glasses or goggles rapidly alternate shuttering, in synchronization with the display, to cause the correct image to be shown to only the corresponding eye. Auto-stereoscopic displays do not use glasses but instead may direct the correct images into the viewer's corresponding eyes. For example, auto-stereoscopic displays may be equipped with cameras to determine where a viewer's eyes are located and mechanical and/or electronic means for directing the images to the viewer's eyes.

<FIG> is a block diagram illustrating an example system in which a source device transmits an enhanced stereo image to a destination device. As illustrated in <FIG>, system <NUM> may include source device <NUM> with image source <NUM>, encoder <NUM>, processor <NUM>, an output interface <NUM>, and may further include a destination device <NUM> with image display <NUM>, decoder <NUM>, and input interface <NUM>. In the example of <FIG>, destination device <NUM> receives encoded image data <NUM> from source device <NUM>. Source device <NUM> and/or destination device <NUM> may comprise wireless communication devices, such as wireless handsets, so-called cellular or satellite radiotelephones, or any wireless devices that can communicate picture and/or video information over a communication channel, in which case the communication channel may comprise a wireless communication channel. Destination device <NUM> may be referred to as a three-dimensional (3D) display device or a 3D rendering device.

As described above with respect to <FIG>, image source <NUM> provides a stereo pair of images, including first view <NUM> and second view <NUM>, to encoder <NUM> and processor <NUM>. Image source <NUM> can provide first view <NUM> and second view <NUM> to encoder <NUM> and processor <NUM> at the same time or at a different time. For example, first view <NUM> and second view <NUM> can be provided to encoder <NUM> immediately after or soon after first view <NUM> and second view <NUM> are stored in image source <NUM>. First view <NUM> and second view <NUM> can then be provided to processor <NUM> when an end-user indicates a desire to enhance the first view <NUM> and/or the second view <NUM>.

Encoder <NUM> uses first view <NUM> and second view <NUM> to generate a depth map <NUM>. Encoder <NUM> transfers the depth map <NUM> to processor <NUM>. Processor <NUM> uses first view <NUM>, second view <NUM>, and the depth map <NUM> to generate enhanced first view <NUM>, which is an enhanced version of first view <NUM>, and/or to generate enhanced second view <NUM>, which is an enhanced version of second view <NUM>. Processor <NUM> transmits enhanced first view <NUM> and/or enhanced second view <NUM> to encoder <NUM>.

Encoder <NUM> forms encoded image data <NUM>, which includes encoded image data for enhanced first view <NUM> and/or for enhanced second view <NUM>. In some examples, encoder <NUM> may apply various lossless or lossy coding techniques to reduce the number of bits needed to transmit encoded image data <NUM> from source device <NUM> to destination device <NUM>. Encoder <NUM> passes encoded image data <NUM> to output interface <NUM>.

When enhanced first view <NUM> and/or enhanced second view <NUM> is a digital still picture, encoder <NUM> may be configured to encode the enhanced first view <NUM> and/or enhanced second view <NUM> as, for example, a Joint Photographic Experts Group (JPEG) image. When enhanced first view <NUM> and/or enhanced second view <NUM> is a frame of video data, encoder <NUM> may be configured to encode enhanced first view <NUM> and/or enhanced second view <NUM> according to a video coding standard such as, for example Motion Picture Experts Group (MPEG), MPEG-<NUM>, International Telecommunication Union (ITU) H. <NUM>, ISO/IEC MPEG-<NUM> Visual, ITU-T H. <NUM> or ISO/IEC MPEG-<NUM> Visual, ITU H. <NUM>, ISO/IEC MPEG-<NUM> Visual and ITU-T H. <NUM> (also known as ISO/IEC MPEG-<NUM> Advanced Video Coding (AVC)), including its SVC and Multiview Video Coding (MVC) extensions, ITU-T H. <NUM>, or other video encoding standards. The ITU-T H. <NUM>/MPEG-<NUM> (AVC) standard, for example, was formulated by the ITU-T Video Coding Experts Group (VCEG) together with the ISO/IEC Moving Picture Experts Group (MPEG) as the product of a collective partnership known as the Joint Video Team (JVT). In some aspects, the techniques described in this disclosure may be applied to devices that generally conform to the H. <NUM> standard. <NUM> standard is described in ITU-T Recommendation H. <NUM>, Advanced Video Coding for generic audiovisual services, by the ITU-T Study Group, and dated March, <NUM>, which may be referred to herein as the H. <NUM> standard or H. <NUM> specification, or the H. <NUM>/AVC standard or specification. The Joint Video Team (JVT) continues to work on extensions to H. <NUM>/MPEG-<NUM> AVC. A draft of MVC is described in "Advanced video coding for generic audiovisual services," ITU-T Recommendation H. <NUM>, Mar <NUM>. In addition, High Efficiency Video Coding (HEVC) is currently being developed by the Joint Collaboration Team on Video Coding (JCT-VC) of ITU-T Video Coding Experts Group (VCEG) and ISO/IEC Motion Picture Experts Group (MPEG). A draft of the HEVC standard, referred to as "<NPL> et al.

In some embodiments, not shown, encoder <NUM> is configured to encode the depth map <NUM>, which is then transmitted in a bitstream as part of encoded image data <NUM>. This process can produce one depth map for the one captured view or depth maps for several transmitted views. Encoder <NUM> may receive one or more views and the depth maps, and code them with video coding standards like H. <NUM>/AVC, MVC, which can jointly code multiple views, or scalable video coding (SVC), which can jointly code depth and texture.

When enhanced first view <NUM> and/or enhanced second view <NUM> corresponds to a frame of video data, encoder <NUM> may encode enhanced first view <NUM> and/or enhanced second view <NUM> in an intra-prediction mode or an inter-prediction mode. As an example, the ITU-T H. <NUM> standard supports intra-prediction in various block sizes, such as 16x16, 8x8, or 4x4 for luma components, and 8x8 for chroma components, as well as inter-prediction in various block sizes, such as 16x16, 16x8, 8x16, 8x8, 8x4, 4x8 and 4x4 for luma components and corresponding scaled sizes for chroma components. In this disclosure, "NxN" and "N by N" may be used interchangeably to refer to the pixel dimensions of the block in terms of vertical and horizontal dimensions (e.g., 16x16 pixels or <NUM> by <NUM> pixels). In general, a 16x16 block will have <NUM> pixels in a vertical direction and <NUM> pixels in a horizontal direction. Likewise, an NxN block generally has N pixels in a vertical direction and N pixels in a horizontal direction, where N represents a positive integer value that may be greater than <NUM>. The pixels in a block may be arranged in rows and columns. Blocks may also be NxM, where N and M are integers that are not necessarily equal.

Block sizes that are less than <NUM> by <NUM> may be referred to as partitions of a <NUM> by <NUM> macroblock. Likewise, for an NxN block, block sizes less than NxN may be referred to as partitions of the NxN block. Video blocks may comprise blocks of pixel data in the pixel domain, or blocks of transform coefficients in the transform domain (e.g., following application of a transform such as a discrete cosine transform (DCT), an integer transform, a wavelet transform, or a conceptually similar transform to the residual video block data representing pixel differences between coded video blocks and predictive video blocks). In some cases, a video block may comprise blocks of quantized transform coefficients in the transform domain.

Smaller video blocks can provide better resolution, and may be used for locations of a video frame that include high levels of detail. In general, macroblocks and the various partitions, sometimes referred to as sub-blocks, may be considered to be video blocks. In addition, a slice may be considered to be a plurality of video blocks, such as macroblocks and/or sub-blocks. Each slice may be an independently decodable unit of a video frame. Alternatively, frames themselves may be decodable units, or other portions of a frame may be defined as decodable units. The term "coded unit" or "coding unit" may refer to any independently decodable unit of a video frame such as an entire frame, a slice of a frame, a group of pictures (GOP) also referred to as a sequence or superframe, or another independently decodable unit defined according to applicable coding techniques.

In general, macroblocks and the various sub-blocks or partitions may all be considered to be video blocks. In addition, a slice may be considered to be a series of video blocks, such as macroblocks and/or sub-blocks or partitions. In general a macroblock may refer to a set of chrominance and luminance values that define a <NUM> by <NUM> area of pixels. A luminance block may comprise a <NUM> by <NUM> set of values, but may be further partitioned into smaller video blocks, such as <NUM> by <NUM> blocks, <NUM> by <NUM> blocks, <NUM> by <NUM> blocks, <NUM> by <NUM> blocks or other sizes. Two different chrominance blocks may define color for the macroblock, and may each comprise <NUM> by <NUM> sub-sampled blocks of the color values associated with the <NUM> by <NUM> area of pixels. Macroblocks may include syntax information to define the coding modes and/or coding techniques applied to the macroblocks. Macroblocks or other video blocks may be grouped into decodable units such as slices, frames or other independent units. Each slice may be an independently decodable unit of a video frame. Alternatively, frames themselves may be decodable units, or other portions of a frame may be defined as decodable units.

Output interface <NUM> transmits encoded image data <NUM> to destination device <NUM>. Input interface <NUM> receives encoded image data <NUM> from output interface <NUM>. Aspects of this disclosure are not necessarily limited to wireless applications or settings. For example, aspects of this disclosure may be applied to over-the-air television broadcasts, cable television transmissions, satellite television transmissions, Internet video transmissions, encoded digital video that is encoded onto a storage medium, or other scenarios. Accordingly, the communication channel may comprise any combination of wireless or wired media suitable for transmission of encoded video and/or picture data.

Output interface <NUM> may send a bitstream including encoded image data <NUM> to input interface <NUM> of destination device <NUM>. For example, output interface <NUM> may encapsulate encoded image data <NUM> in a bitstream using transport level encapsulation techniques (e.g., MPEG-<NUM> Systems techniques). Output interface <NUM> may comprise, for example, a network interface, a wireless network interface, a radio frequency transmitter, a transmitter/receiver (transceiver), or other transmission unit. In other examples, source device <NUM> may be configured to store the bitstream including encoded image data <NUM> to a physical medium such as, for example, an optical storage medium such as a compact disc, a digital video disc, a Blu-Ray disc, flash memory, magnetic media, or other storage media. In such examples, the storage media may be physically transported to the location of destination device <NUM> and read by an appropriate interface unit for retrieving the data. In some examples, the bitstream including encoded image data <NUM> may be modulated by a modulator/demodulator (MODEM) before being transmitted by output interface <NUM>.

Although image source <NUM> may provide multiple views (i.e. enhanced first view <NUM> and enhanced second view <NUM>), source device <NUM> may transmit only the enhanced first view <NUM> or the enhanced second view <NUM>. For example, image source <NUM> may comprise an eight camera array, intended to produce four pairs of views of a scene to be viewed from different angles. Source device <NUM> may transmit only one image of each pair to destination device <NUM>. In some embodiments, source device <NUM> may transmit additional information along with the single image, such as the depth map <NUM>. Thus, rather than transmitting eight views, source device <NUM> may transmit four views and/or plus depth/disparity information (e.g., depth map <NUM>) for each of the four views in the form of a bitstream including encoded image data <NUM>, in this example. In some examples, processor <NUM> may receive disparity information for an image (e.g., the depth map <NUM>) from a user or from another external device.

After receiving the bitstream with encoded image data <NUM> and decapsulating the data, in some examples, input interface <NUM> may provide encoded image data <NUM> to decoder <NUM> (or to a MODEM that demodulates the bitstream, in some examples).

Decoder <NUM> receives the encoded image data <NUM> from input interface <NUM>. Decoder <NUM> decodes the encoded image data <NUM> to extract enhanced first view <NUM> and/or enhanced second view <NUM>. Based on the enhanced first view <NUM> and/or the enhanced second view <NUM>, image display <NUM>, which is the same as or similar to image display <NUM>, can render a two-dimensional or three-dimensional image. Although not shown in <FIG>, enhanced first view <NUM> and/or enhanced second view <NUM> may undergo additional processing at either source device <NUM> or destination device <NUM>.

<FIG> is a block diagram illustrating an example of an encoder that may implement techniques in accordance with aspects described in this disclosure. Encoder <NUM> may be configured to perform any or all of the techniques of this disclosure. For example, mode select unit <NUM> (e.g., motion estimation unit <NUM>) may be configured to determine a depth map based on a sequence of images (e.g., a left image followed by a right image) received by the encoder <NUM>. However, aspects of this disclosure are not so limited. In some examples, the techniques described in this disclosure may be shared among the various components of encoder <NUM>. In some examples, in addition to or instead of, a processor (not shown) may be configured to perform any or all of the techniques described in this disclosure.

Encoder <NUM> may perform intra- and inter-coding of video blocks within video slices. Intra-coding relies on spatial prediction to reduce or remove spatial redundancy in video within a given video frame or picture. Inter-coding relies on temporal prediction to reduce or remove temporal redundancy in video within adjacent frames or pictures of a video sequence. Intra-mode (I mode) may refer to any of several spatial based coding modes. Inter-modes, such as uni-directional prediction (P mode) or bi-prediction (B mode), may refer to any of several temporal-based coding modes. Encoder <NUM> may also generate a disparity map and/or a depth map using the inter-coding techniques described herein.

As shown in <FIG>, encoder <NUM> receives a current image block within an image or a video frame to be encoded. In the example of <FIG>, encoder <NUM> includes mode select unit <NUM>, reference frame memory <NUM>, summer <NUM>, transform processing unit <NUM>, quantization unit <NUM>, and entropy encoding unit <NUM>. Mode select unit <NUM>, in turn, includes motion estimation unit <NUM>, motion compensation unit <NUM>, intra-prediction unit <NUM>, and partition unit <NUM>. For video block reconstruction, encoder <NUM> also includes inverse quantization unit <NUM>, inverse transform unit <NUM>, and summer <NUM>. A deblocking filter (not shown in <FIG>) may also be included to filter block boundaries to remove blockiness artifacts from reconstructed video. If desired, the deblocking filter would typically filter the output of summer <NUM>. Additional filters (in loop or post loop) may also be used in addition to the deblocking filter. Such filters are not shown for brevity, but if desired, may filter the output of summer <NUM> (as an in-loop filter).

During the encoding process, encoder <NUM> receives an image, a video frame, or slice to be coded. The image, frame, or slice may be divided into multiple blocks. Motion estimation unit <NUM> and motion compensation unit <NUM> perform inter-predictive coding of the received block relative to one or more blocks in one or more reference frames to provide temporal prediction. Intra-prediction unit <NUM> may alternatively perform intra-predictive coding of the received block relative to one or more neighboring blocks in the same frame or slice as the block to be coded to provide spatial prediction. Encoder <NUM> may perform multiple coding passes (e.g., to select an appropriate coding mode for each block of video data).

Moreover, partition unit <NUM> may partition blocks of image or video data into sub-blocks, based on an evaluation of previous partitioning schemes in previous coding passes. For example, partition unit <NUM> may initially partition an image, frame, or slice into largest coding units (LCUs), and partition each of the LCUs into sub-coding units (CUs) based on a rate-distortion analysis (e.g., rate-distortion optimization). Mode select unit <NUM> (e.g., partition unit <NUM>) may further produce a quadtree data structure indicative of partitioning of an LCU into sub-CUs. Leaf-CUs of the quadtree may include one or more PUs and one or more TUs.

Mode select unit <NUM> may select one of the coding modes (e.g., intra or inter) based on error results, and provide the resulting intra- or inter-coded block to summer <NUM> to generate residual block data and to summer <NUM> to reconstruct the encoded block for use as a reference frame. Mode select unit <NUM> also provides syntax elements, such as motion vectors, intra-mode indicators, partition information, and other such syntax information, to entropy encoding unit <NUM>.

Motion estimation unit <NUM> and motion compensation unit <NUM> can be highly integrated, but are illustrated separately for conceptual purposes. Motion estimation or the prediction of motion information, performed by motion estimation unit <NUM>, is the process of generating motion vectors, which estimate motion for video blocks. A motion vector, for example, may indicate the displacement of a PU of a video block within a current video frame or picture relative to a predictive block within a reference frame (or other coded unit) relative to the current block being coded within the current frame (or other coded unit). A predictive block is a block that is found to closely match the block to be coded, in terms of pixel difference, which may be determined by sum of absolute difference (SAD), sum of square difference (SSD), or other difference metrics. In some examples, encoder <NUM> may calculate values for sub-integer pixel positions of reference pictures stored in reference frame memory <NUM>. For example, encoder <NUM> may interpolate values of one-quarter pixel positions, one-eighth pixel positions, or other fractional pixel positions of the reference picture. Therefore, motion estimation unit <NUM> may perform a motion search relative to the full pixel positions and fractional pixel positions and output a motion vector with fractional pixel precision.

Motion estimation unit <NUM> calculates a motion vector for a PU of a video block in an inter-coded slice by comparing the position of the PU to the position of a predictive block of a reference picture. The reference picture may be selected from a first reference picture list (e.g., List <NUM>), a second reference picture list (e.g., List <NUM>), or a third reference picture list (e.g., List C), each of which identify one or more reference pictures stored in reference frame memory <NUM>. The reference picture may be selected based on the motion information of blocks that spatially and/or temporally neighbor the PU. The selected reference picture may be identified by a reference index. Motion estimation unit <NUM> sends the calculated motion vector and/or the reference index to entropy encoding unit <NUM> and/or motion compensation unit <NUM>.

In addition, the motion estimation unit <NUM> generates a disparity map by comparing two images that are received in sequence (e.g., a left image and a right image) and determining disparity values for each portion (e.g., each pixel, each group of pixels, etc.) in the received images in a manner as described herein. For example, the disparity values can be determined by analyzing the motion between two images. The motion estimation unit <NUM> can upscale the disparity map to generate a depth map.

Motion compensation, performed by motion compensation unit <NUM>, may involve fetching or generating the predictive block based on the motion vector determined by motion estimation unit <NUM>. Upon receiving the motion vector for the PU of the current video block, motion compensation unit <NUM> may locate the predictive block to which the motion vector points in one of the reference picture lists. Summer <NUM> forms a residual video block by subtracting pixel values of the predictive block from the pixel values of the current video block being coded, forming pixel difference values, as discussed below. In some embodiments, motion estimation unit <NUM> can perform motion estimation relative to luma components, and motion compensation unit <NUM> can use motion vectors calculated based on the luma components for both chroma components and luma components. Mode select unit <NUM> may generate syntax elements associated with the video blocks and the video slice for use by decoder <NUM> in decoding the video blocks of the video slice.

Intra-prediction unit <NUM> may intra-predict a current block, as an alternative to the inter-prediction performed by motion estimation unit <NUM> and motion compensation unit <NUM>, in some embodiments. In particular, intra-prediction unit <NUM> may determine an intra-prediction mode to use to encode a current block. In some examples, intra-prediction unit <NUM> may encode a current block using various intra-prediction modes (e.g., during separate encoding passes) and intra-prediction unit <NUM> (or mode select unit <NUM>, in some examples) may select an appropriate intra-prediction mode to use from the tested modes.

For example, intra-prediction unit <NUM> may calculate rate-distortion values using a rate-distortion analysis for the various tested intra-prediction modes, and select the intra-prediction mode having the best rate-distortion characteristics among the tested modes. Rate-distortion analysis generally determines an amount of distortion (or error) between an encoded block and an original, unencoded block that was encoded to produce the encoded block, as well as a bitrate (that is, a number of bits) used to produce the encoded block. Intra-prediction unit <NUM> may calculate ratios from the distortions and rates for the various encoded blocks to determine which intra-prediction mode exhibits the best rate-distortion value for the block.

After selecting an intra-prediction mode for a block, intra-prediction unit <NUM> may provide information indicative of the selected intra-prediction mode for the block to entropy encoding unit <NUM>. Entropy encoding unit <NUM> may encode the information indicating the selected intra-prediction mode. Encoder <NUM> may include in the transmitted bitstream configuration data, which may include a plurality of intra-prediction mode index tables and a plurality of modified intra-prediction mode index tables (also referred to as codeword mapping tables), definitions of encoding contexts for various blocks, and indications of a most probable intra-prediction mode, an intra-prediction mode index table, and a modified intra-prediction mode index table to use for each of the contexts.

As described above, encoder <NUM> forms a residual video block by subtracting the prediction data provided by mode select unit <NUM> from the original video block being coded. Summer <NUM> represents the component or components that perform this subtraction operation. Transform processing unit <NUM> applies a transform, such as a DCT or a conceptually similar transform (e.g., wavelet transforms, integer transforms, sub-band transforms, etc.), to the residual block, producing a video block comprising residual transform coefficient values. The transform may convert the residual information from a pixel value domain to a transform domain, such as a frequency domain. Transform processing unit <NUM> may send the resulting transform coefficients to quantization unit <NUM>. Quantization unit <NUM> quantizes the transform coefficients to further reduce bit rate. The quantization process may reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, quantization unit <NUM> may then perform a scan of the matrix including the quantized transform coefficients. Alternatively, entropy encoding unit <NUM> may perform the scan.

Following quantization, entropy encoding unit <NUM> entropy codes the quantized transform coefficients. For example, entropy encoding unit <NUM> may perform CAVLC, CABAC, SBAC, PIPE coding, or another entropy coding technique. In the case of context-based entropy coding, context may be based on neighboring blocks. Following the entropy coding by entropy encoding unit <NUM>, the encoded bitstream may be transmitted to another device (e.g., decoder <NUM>) or archived for later transmission or retrieval.

Inverse quantization unit <NUM> and inverse transform unit <NUM> apply inverse quantization and inverse transformation, respectively, to reconstruct the residual block in the pixel domain (e.g., for later use as a reference block). Motion compensation unit <NUM> may calculate a reference block by adding the residual block to a predictive block of one of the frames stored in reference frame memory <NUM>. Motion compensation unit <NUM> may also apply one or more interpolation filters to the reconstructed residual block to calculate sub-integer pixel values for use in motion estimation. Summer <NUM> adds the reconstructed residual block to the motion compensated prediction block produced by motion compensation unit <NUM> to produce a reconstructed video block for storage in reference frame memory <NUM>. The reconstructed video block may be used by motion estimation unit <NUM> and motion compensation unit <NUM> as a reference block to inter-code a block in a subsequent video frame.

<FIG> is a block diagram illustrating an example of a decoder that may implement techniques in accordance with aspects described in this disclosure. Decoder <NUM> may be configured to perform any or all of the techniques of this disclosure. For example, the motion compensation unit <NUM> may be configured to generate a disparity map and/or a depth map. However, aspects of this disclosure are not so limited. In some examples, the techniques described in this disclosure may be shared among the various components of decoder <NUM>. In some examples, in addition to or instead of, a processor (not shown) may be configured to perform any or all of the techniques described in this disclosure.

In the example of <FIG>, decoder <NUM> includes an entropy decoding unit <NUM>, motion compensation unit <NUM>, intra prediction unit <NUM>, inverse quantization unit <NUM>, inverse transformation unit <NUM>, reference frame memory <NUM>, and summer <NUM>. Decoder <NUM> may, in some examples, perform a decoding pass generally reciprocal to the encoding pass described with respect to encoder <NUM> (<FIG>). Motion compensation unit <NUM> may generate prediction data based on motion vectors received from entropy decoding unit <NUM>, while intra-prediction unit <NUM> may generate prediction data based on intra-prediction mode indicators received from entropy decoding unit <NUM>.

During the decoding process, decoder <NUM> receives an encoded image bitstream that represents blocks of an encoded image or video slice and associated syntax elements from encoder <NUM>. Entropy decoding unit <NUM> of decoder <NUM> entropy decodes the bitstream to generate quantized coefficients, motion vectors or intra-prediction mode indicators, and/or other syntax elements. Entropy decoding unit <NUM> forwards the motion vectors and other syntax elements to motion compensation unit <NUM>. Decoder <NUM> may receive the syntax elements at the image, video slice level, and/or the block level.

When the image or video slice is coded as an intra-coded (I) slice, intra prediction unit <NUM> may generate prediction data for a block of the current image or video slice based on a signaled intra prediction mode and data from previously decoded blocks of the current frame or picture. When the image or frame is coded as an inter-coded (e.g., B, P or GPB) slice, motion compensation unit <NUM> produces predictive blocks for a video block of the current video slice based on the motion vectors and other syntax elements received from entropy decoding unit <NUM>. The predictive blocks may be produced from one of the reference pictures within one of the reference picture lists. Decoder <NUM> (e.g., the motion compensation unit <NUM>) may construct the reference frame lists, List <NUM>, List <NUM>, and/or List C, using default construction techniques based on reference pictures stored in reference frame memory <NUM>. Motion compensation unit <NUM> determines prediction information for a video block of the current video slice by parsing the motion vectors and other syntax elements, and uses the prediction information to produce the predictive blocks for the current video block being decoded. For example, motion compensation unit <NUM> uses some of the received syntax elements to determine a prediction mode (e.g., intra- or inter-prediction) used to code the blocks of the video slice, an inter-prediction slice type (e.g., B slice, P slice, or GPB slice), construction information for one or more of the reference picture lists for the slice, motion vectors for each inter-encoded block of the image or slice, inter-prediction status for each inter-coded block of the image or slice, and/or other information to decode the blocks in the current image or video slice.

In addition, the motion compensation unit <NUM> generates a disparity map by comparing two images that are received in sequence (e.g., a left image and a right image) and determining disparity values for each portion (e.g., each pixel, each group of pixels, etc.) in the received images in a manner as described herein. For example, the disparity values can be determined by analyzing the motion between two images. The motion compensation unit <NUM> can upscale the disparity map to generate a depth map.

Motion compensation unit <NUM> may use interpolation filters as used by encoder <NUM> during encoding of the video blocks to calculate interpolated values for sub-integer pixels of reference blocks. In this case, motion compensation unit <NUM> may determine the interpolation filters used by encoder <NUM> from the received syntax elements and use the interpolation filters to produce predictive blocks.

Inverse quantization unit <NUM> inverse quantizes (e.g., de-quantizes) the quantized transform coefficients provided in the bitstream and decoded by entropy decoding unit <NUM>. The inverse quantization process may include use of a quantization parameter QPY calculated by encoder <NUM> for each video block in the video slice to determine a degree of quantization and, likewise, a degree of inverse quantization that should be applied.

Inverse transform unit <NUM> applies an inverse transform (e.g., an inverse DCT), an inverse integer transform, or a conceptually similar inverse transform process, to the transform coefficients in order to produce residual blocks in the pixel domain.

In some cases, inverse transform unit <NUM> may apply a <NUM>-dimensional (<NUM>-D) inverse transform (in both the horizontal and vertical direction) to the coefficients. According to the techniques of this disclosure, inverse transform unit <NUM> may instead apply a horizontal <NUM>-D inverse transform, a vertical <NUM>-D inverse transform, or no transform to the residual data in each of the TUs. The type of transform applied to the residual data at encoder <NUM> may be signaled to decoder <NUM> to apply an appropriate type of inverse transform to the transform coefficients.

After motion compensation unit <NUM> generates the predictive block for the current block based on the motion vectors and other syntax elements, decoder <NUM> forms a decoded video block by summing the residual blocks from inverse transform unit <NUM> with the corresponding predictive blocks generated by motion compensation unit <NUM>. Summer <NUM> represents the component or components that perform this summation operation. If desired, a deblocking filter may also be applied to filter the decoded blocks in order to remove blockiness artifacts. Other loop filters (either in the coding loop or after the coding loop) may also be used to smooth pixel transitions, or otherwise improve the video quality. The decoded video blocks in a given frame or picture are then stored in reference picture memory <NUM>, which stores reference pictures used for subsequent motion compensation. Reference frame memory <NUM> also stores decoded video for later presentation on a display device, such as image device <NUM> of <FIG> or image display <NUM> of <FIG>.

The techniques disclosed herein are described with respect to encoders, such as encoder <NUM>. However, this is not meant to be limiting. The same techniques can be used by decoders, such as decoder <NUM>, to accomplish similar results.

Encoders, such as encoder <NUM>, convert raw video frames into standard compressed formats. The compression process is aided by a motion detection algorithm that computes pixelwise/blockwise motion for inter-coded frames. As discussed above, inter-coded frames can be of two types: P-frames and B-frames. P-frames contain forward motion values referring to a previous intra-coded frame (I-frame). B-frames contain bi-directional motion values, referring to previous and next intra-coded frames. An intra-coded frame is encoded spatially. An intra-coded frame is not associated with any motion values or any other frame in the stream. The motion value in an inter-coded frame is the disparity or difference between the two video frames. In an embodiment, if a normal 2D video stream (e.g., the images captured using a single camera) is replaced with a video stream that includes interleaved left (L) and right (R) views of a 3D video stream (e.g., the images captured using two cameras), the motion values represent the disparity between each pair of L and R frames (e.g., the difference in the spatial orientation between the L and R frames).

The encoder may be configured to encode alternate frames of the interleaved stereo video as I-frames and P-frames. <FIG> illustrates a sequencing of frames. As illustrated in <FIG>, the sequence includes four frames: L<NUM> <NUM>, R<NUM> <NUM>, L<NUM> <NUM>, and R<NUM> <NUM>. For the sake of simplicity, only four frames are illustrated; however, the techniques described herein apply to any number of frames. In an embodiment, L<NUM> <NUM> is the first frame received by encoder <NUM> and R<NUM> <NUM> is the last frame received by encoder <NUM>. The L frames (e.g., L<NUM> <NUM> and L<NUM> <NUM>) are encoded as I-frames and the R frames (e.g., R<NUM> <NUM> and R<NUM> <NUM>) are encoded as P-frames. Thus, the sequence of frames as received by the encoder <NUM> alternates between I-frames and P-frames. This helps ensure that disparity values can be calculated.

<FIG> illustrates a depth computation flow. As illustrated in <FIG>, encoder <NUM> receives the sequence of frames L<NUM> <NUM>, R<NUM> <NUM>, L<NUM> <NUM>, and R<NUM> <NUM> as data <NUM>. In an embodiment, encoder <NUM> is set to a high-bitrate setting so that the P-frame has mostly inter-coded macroblocks (MBs). As an example, the minimum MB size may be 8x8. It may be desirable to have as many inter-coded macroblocks as possible in the R frame because the inter-coded macroblocks contain the motion vector information referring back to the corresponding L frame.

Encoder <NUM> can extract motion vector information from the MBs (represented by box <NUM>) as data <NUM>, and the blocks that are inter-coded and intra-coded can be graphically represented in downsampled depth map <NUM>. Black blocks, not shown, in downsampled depth map <NUM> may represent the intra-coded blocks. Intra-coded blocks may be common in the occluded or the revealed regions where no match can be found between the L and R frames.

In an embodiment, the MB motion in the X direction can approximately represent the blockwise disparity between the two stereo frames (which can be converted into a depth map). The smaller the MB size, the higher the depth map resolution. The MB frame can be further cleared of intra-coded patches and noisy depth values by refinements, such as median filtering (e.g., a 5x5 median filter) (represented by box <NUM>). In an example, because the minimum MB size is 8x8 pixels, the MB motion image is <NUM> times downsampled as compared to the original input frame. To apply enhancement algorithms, the image may have to be upsampled by a factor of <NUM> (or another similar value).

The upscaling is represented by box <NUM>. In an embodiment, the encoder <NUM> upscales the depth map <NUM> by pixel-repeating the depth map <NUM>. The encoder <NUM> may determine a depth filter for each pixel by using a size of a filter kernel, a spatial distance of the pixel being filtered from the center of the kernel (e.g., sd), and/or a gradient of the disparity computed at the pixel being filtered and a gradient of the color components of the image computed at the pixel being filtered (e.g., id). The depth filter may be a product of two Gaussian components: a spatial component (e.g., -e-(sd*sd)/λs) and an image color and depth value gradient component (e.g., e-(id*id)/λc), where λs and λc may decide a spread factor of the corresponding Gaussian kernel. The encoder <NUM> may carry out a plurality of filtering iterations on a previously filtered depth map to improve the depth map. In another embodiment, the encoder <NUM> upscales the depth map <NUM> using sensor inputs. For example, the encoder <NUM> may create an initial estimate using a simple, bilinear upscale. The encoder <NUM> may identify depth transitions in the initial estimate, fill in depth values at the transitions using extrapolation, and smooth the filled depth values (e.g., reduce the difference in depth values at the depth transitions). The result of upscaling may be graphically represented by depth map <NUM>, which includes fewer black blocks and is an upsampled version of downsampled depth map <NUM>.

The upscaling represented by box <NUM> and performed by the encoder <NUM> may have several benefits. For instance, depth map computation from stereo images can be a computationally intensive process. Using the process described herein, the encoder <NUM> can then reduce computational resource usage and increase processing speed if the encoder <NUM> processes smaller images or generates sparse (e.g., downsampled) depth maps. In addition, if inaccurate, an upscaled depth map can affect depth map segmentation and final enhancement quality. Thus, using the process described herein, the encoder <NUM> can improve depth map segmentation and/or final enhancement quality as compared to encoders that use a generic upscaling process.

In an embodiment, a depth map generated by the encoder <NUM> has a higher enhancement quality than a depth map generated using a generic upscaling process. While a ground truth depth map (e.g., an ideal depth map) may have a higher enhancement quality than the depth map generated by the encoder <NUM>, the encoder <NUM> may use fewer samples (e.g., <NUM>/<NUM> the number of samples) as would be used when generating the ground truth depth map.

The depth map, such as depth map <NUM>, is used to enhance a stereo image or video. For example, the depth map <NUM> can be used to demonstrate edge enhancement in a scene (e.g., the sharpening of an object in a scene). As another example, the depth map <NUM> can be used to adjust contrast in a scene, change colors in the scene, change geometric shapes in the scene, and/or the like. The edge enhancement can vary in strength based on depth of the object in the scene. In an embodiment, the idea is to select an object in a picture using a cursor. An algorithm separates out the object or the region from the rest of the image based on the depth map and applies different degrees of enhancement or different filters based on the depth.

<FIG> illustrates a flow for enhancing an image. As illustrated in <FIG>, an image <NUM> is enhanced by processor <NUM> into enhanced image <NUM>. A portion of image <NUM> is selected at point <NUM>. The processor <NUM> can retrieve depth map <NUM> from encoder <NUM> in response to the selection or in response to generation of the depth map <NUM> at an earlier time. As an example, the depth value of the image <NUM> at point <NUM> is D. A threshold of ±Δ around D can be set by processor <NUM>. The threshold ±Δ can alternatively be computed by processor <NUM> based on the maximum depth range as well.

Using a brushstroke (or similar shape) of size NxN, the processor <NUM> can grow a region R surrounding the point <NUM> by identifying a continuous portion of the image <NUM> starting from the point <NUM> that has a depth of D±Δ (e.g., D plus or minus the threshold). In an embodiment, the region R can be in any form or shape.

The processor <NUM> stops growing the region R in a direction once a portion of the image <NUM> is reached that has a depth other than a depth of D±Δ.

The processor <NUM> can then enhance the portion of the image <NUM> within the region R once the processor <NUM> stops growing region R. For example, the portion within the region R can be enhanced by using a high pass filter. The portion of the image outside of the region R can be smoothed (or blurred) by using a gaussian blur that is proportional to the difference between the depth D and the depth of the respective portion of the image <NUM>. As another example, the portion within the region R can be enhanced by modifying the color of the pixels within the region R. As another example, the portion within the region R can be cut and/or moved to another portion of the image <NUM>. As illustrated in <FIG>, the white portion of depth map <NUM> represents the shape and size of region R when point <NUM> is selected. Likewise, region <NUM> in image <NUM> roughly represents the shape and size of region R when point <NUM> is selected. Region <NUM> roughly represents the area of image <NUM> outside of region R when point <NUM> is selected.

<FIG> illustrates another flow for enhancing an image. As illustrated in <FIG>, different non-contiguous portions of image <NUM> can have the same depth value (within a threshold value). For example, the white portions of depth map <NUM> all have the same depth value within a threshold value. However, because there are gaps between the white portions in depth map <NUM>, not all of the white portions will be enhanced by processor <NUM>. As discussed above with respect to <FIG>, point <NUM> is selected in image <NUM>. Thus, while several portions of image <NUM> include the same depth value within a threshold value, only the portions of image <NUM> that include pixels that are contiguous with point <NUM> and that have a depth value within a threshold value of the depth at point <NUM> are enhanced (e.g., the portions of image <NUM> surrounding point <NUM> are enhanced up to the point that an edge is reached in the depth map <NUM>). Depth map <NUM> illustrates those portions of image <NUM> that satisfy such criteria. Accordingly, region <NUM> of image <NUM> is enhanced, and the other portions of image <NUM> are either left alone or degraded.

<FIG> illustrates an example original image <NUM>. <FIG> illustrates a depth map <NUM> that can be computed by the encoder <NUM> based on the original image <NUM> in <FIG>. <FIG> illustrates the depth map <NUM> after it is refined and upscaled (e.g., to a <NUM>-<NUM> scale).

<FIG> illustrate various images that are enhanced using the scaled depth map <NUM> as described herein. As illustrated in <FIG>, point <NUM> is selected, and pixels that are contiguous with point <NUM> and that share a depth value within a threshold value are enhanced. Point <NUM> and other portions of image <NUM> are degraded.

As illustrated in <FIG>, point <NUM> is selected, and pixels that are contiguous with point <NUM> and that share a depth value within a threshold value are enhanced. Point <NUM> and other portions of image <NUM> are degraded.

As illustrated in <FIG>, point <NUM> is selected, and pixels that are contiguous with point <NUM> and that share a depth value within a threshold value are enhanced. Points <NUM> and <NUM> and other portions of image <NUM> are degraded.

<FIG> illustrates an example method <NUM> for enhancing an image. The method <NUM> can be performed by one or more components of a portable electronic device. For example, the method <NUM> can be performed by encoder <NUM> (e.g., motion estimation unit <NUM>), decoder <NUM> (e.g., motion compensation unit <NUM>), and/or processor <NUM>, for example. In some embodiments, other components may be used to implement one or more of the steps described herein.

At block <NUM>, a left image and a right image stored in a memory unit are retrieved. In an embodiment, the left image and the right image each depict a same scene. At block <NUM>, a depth map based on a difference in spatial orientation between the left image and the right image is determined.

At block <NUM>, a portion of the left image or the right image selected by a user is identified. At block <NUM>, an enhancement region surrounding the portion selected by the user is determined based on the determined depth map. At block <NUM>, the enhancement region is enhanced.

Claim 1:
A method for enhancing an image, the method comprising:
retrieving (<NUM>) a left image (<NUM>) and a right image (<NUM>) from an image source (<NUM>), the left image and the right image each depicting a same scene from a different viewpoint;
determining a depth map based on a difference in spatial orientation between the left image and the right image;
identifying a point of the left image or the right image selected by a user, wherein the selected point is associated with a first depth;
determining (<NUM>) one enhancement region surrounding the selected point based on the determined depth map, wherein determining the enhancement region comprises starting from the selected point of the left image or the right image and identifying, as the enhancement region, one contiguous portion of the left image or the right image that includes pixels that are contiguous with the selected point and have a depth within a threshold of the first depth; and
sharpening (<NUM>) only the enhancement region.