Implementations generally relate to providing depth-of-field renderings. In some implementations, a method includes linearizing an image. The method further includes partitioning a depth map of the image into a plurality of depth intervals. The method further includes blurring pixels associated with each depth interval, where the pixels of each depth interval are blurred separately from the pixels of the other depth intervals. The method further includes applying at least one camera response function to the image after the pixels of the plurality of depth intervals are blurred.

BACKGROUND

Depth-of-field rendering algorithms are used to simulate the blur created by a single-lens reflex (SLR) lens. The typical input to a depth-of-field rendering algorithm is an all-in-focus image and a depth map. Given new lens settings (e.g., focal depth and aperture), a depth-of-field rendering algorithm blurs the input image, where the amount of blur is guided by the depth values. Depth-of-field renderings are used in many graphics applications such as video games or image editing applications in order to add realistic blur to an original all-in-focus image, thus creating a more pleasing image. Such algorithms are very expensive to run, as they involve variable-size convolutions.

SUMMARY

Implementations generally relate to providing depth-of-field renderings. In some implementations, a method includes linearizing an image. The method further includes partitioning a depth map of the image into a plurality of depth intervals. The method further includes blurring pixels associated with each depth interval, where the pixels of each depth interval are blurred separately from the pixels of the other depth intervals. The method further includes applying at least one camera response function to the image after the pixels of the plurality of depth intervals are blurred.

With further regard to the method, in some implementations, the linearizing of the image includes removing at least one camera response function that has been already applied to the image. In some implementations, each depth interval corresponds to a predetermined power-of-N blur radius. In some implementations, the depth intervals are disjointed. In some implementations, the blurring is a disk blur. In some implementations, the method further includes masking pixels of each depth interval that are not being blurred while pixels of a current depth interval are being blurred. In some implementations, the method further includes: masking pixels of each depth interval that are not being blurred while pixels of a current depth interval are being blurred; and in-painting masked pixels. In some implementations, the method further includes blending blurred pixels of at least one depth interval with previously blurred pixels of at least one other depth interval. In some implementations, the at least one camera response function is a gamma correction.

In some implementations, a method includes linearizing an image, where the linearizing includes removing at least one camera response function that has been already applied to the image. The method further includes partitioning a depth map of the image into a plurality of depth intervals. The method further includes blurring pixels associated with each depth interval, where the pixels of each depth interval are blurred separately from the pixels of the other depth intervals. The method further includes blending blurred pixels of at least one depth interval with previously blurred pixels of at least one other depth interval. The method further includes applying at least one camera response function to the image after the pixels of the plurality of depth intervals are blurred.

With further regard to the method, in some implementations, the at least one camera response function is a gamma correction. In some implementations, each depth interval corresponds to a predetermined power-of-N blur radius. In some implementations, the method further includes masking pixels of each depth interval that are not being blurred while pixels of a current depth interval are being blurred. In some implementations, the blurring is applied to each depth interval using a disk kernel, and wherein a blur radius changes per pixel.

In some implementations, a system includes one or more processors, and logic encoded in one or more tangible media for execution by the one or more processors. When executed, the logic is operable to perform operations including: linearizing an image; partitioning a depth map of the image into a plurality of depth intervals; blurring pixels associated with each depth interval, where the pixels of each depth interval are blurred separately from the pixels of the other depth intervals; and applying at least one camera response function to the image after the pixels of the plurality of depth intervals are blurred.

With further regard to the system, in some implementations, to linearize the image, the logic when executed is further operable to perform operations including removing at least one camera response function that has been already applied to the image. In some implementations, each depth interval corresponds to a predetermined power-of-N blur radius. In some implementations, the depth intervals are disjointed. In some implementations, the blurring is a disk blur. In some implementations, the logic when executed is further operable to perform operations including masking pixels of each depth interval that are not being blurred while pixels of a current depth interval are being blurred. In some implementations, the logic when executed is further operable to perform operations including: masking pixels of each depth interval that are not being blurred while pixels of a current depth interval are being blurred; and in-painting masked pixels.

DETAILED DESCRIPTION

Implementations provide depth-of-field renderings using a layer-based approach that significantly reduces rendering time. Using layers reduces computation time and improves quality by using the layers to deal with depth boundaries. In various implementations, a system linearizes an image by removing at least one camera response function such as gamma correction that has been already applied to the image. As described in more detail below, in various implementations, the camera response function is removed once, before any other processing. The system then partitions a depth map of the image into separate depth intervals. In some implementations, each depth interval corresponds to a predetermined power-of-N blur radius.

The system then blurs pixels associated with each depth interval, where the pixels of each depth interval are blurred separately from the pixels of the other depth intervals. In some implementations, the system masks pixels of each depth interval that are not being blurred while pixels of a current depth interval are being blurred, and where the masking prevents pixels from being blurred. In some implementations, the system blends blurred pixels of a depth interval with previously blurred pixels of at least one other depth interval. The system then applies at least one camera response function such as a gamma correction to the image after the pixels of the depth intervals are blurred. As described in more detail below, the camera response function is re-applied once, after all of the layers have been merged together.

As a result, portions of the image that are in the focal plane are clear, and portions of the image that are out of the focal plane are blurred by different amounts according to the depth of the pixels and the focal plane. Implementations render an initially all-in-focus image such that it appears to have been captured with a single-lens reflex (SLR) camera.

FIG. 1illustrates a block diagram of an example network environment100, which may be used to implement the implementations described herein. In some implementations, network environment100includes a system102, which includes a server device104and a social network database106. In various implementations, the term system102and phrase “social network system” may be used interchangeably. Network environment100also includes client devices110,120,130, and140, which may communicate with each other via system102. Network environment100also includes a network150.

For ease of illustration,FIG. 1shows one block for each of system102, server device104, and social network database106, and shows four blocks for client devices110,120,130, and140. Blocks102,104, and106may represent multiple systems, server devices, and social network databases. Also, there may be any number of client devices. In other implementations, network environment100may not have all of the components shown and/or may have other elements including other types of elements instead of, or in addition to, those shown herein.

In various implementations, users U1, U2, U3, and U4may communicate with each other using respective client devices110,120,130, and140. Users U1, U2, U3, and U4may also interact with system102to provide images for depth-of-field renderings.

In the various implementations described herein, processor of system102causes the elements described herein (e.g., original all-in-focus images, depth-of-field renderings, etc.) to be displayed in a user interface on one or more display screens.

While some implementations are described herein in the context of a social network system, these implementations may apply in contexts other than a social network. For example, implementations may apply locally for an individual user. For example, system102may perform the implementations described herein on a stand-alone computer, tablet computer, smartphone, etc.

FIG. 2illustrates an example simplified flow diagram for providing depth-of-field renderings, according to some implementations. Referring to bothFIGS. 1 and 2, a method is initiated in block202, where system102linearizes an image. In various implementations, system102obtains the image, where the entire image is in focus.

In some implementations, to linearize the image, system102removes at least one camera response function that has been already applied to the image. In some implementations, to linearize the image, system102removes at least one camera response function that has been already applied to the image, where the camera response function is a gamma correction that has been already applied to the image.

FIG. 3illustrates an example image300that has not been linearized, according to some implementations. As shown, the entire image300is in focus. Also, a camera response function has been applied to image300. More specifically, in this example implementation, a gamma correction has been applied to image300.

FIG. 4illustrates example image300after being linearized, according to some implementations. As shown, the camera response function has been removed from image300. More specifically, in this example implementation, system102has removed the gamma correction from image300.

In various implementations, system102also obtains a depth map of the image, where the depth map provides depth-of-field values for the pixels in the image.

In block204, system102partitions a depth map of the image into multiple depth intervals. The number depth intervals is a predetermined number, which will depend on the particular implementation. In various implementations, system102partitions the depth map of the image into disjointed depth intervals. Each pixel of the image is associated with one of the layers of the depth map. Accordingly, the image is partitioned into different layers or depth intervals such that each pixel of the image is associated with one of the layers/intervals.

In some implementations, each depth interval corresponds to a predetermined power-of-N blur radius. The actual value of N may vary and will depend on the particular implementation. For example, each interval may substantially correspond to a power-of-2 blur radius.

In block206, system102blurs pixels associated with each depth interval, where the pixels of each depth interval are blurred separately from the pixels of the other depth intervals. In various implementations, system102processes each depth interval (or depth layer) separately. For example, system102may starting from the back interval and work toward the front interval. In various implementations, the pixels of the image are grouped by degrees of blur. Blurring groups of pixels by interval is computationally much faster than blurring pixels individually.

In some implementations, system102computes a blur mask to select which pixels in the current layer need to be blurred. As indicated above, system102may select each interval one at a time, from back to front.

FIGS. 5A-5Fillustrate a series of example masks for image300, from back to front of the image.FIGS. 5A-5Fshow 6 masks corresponding to 6 depth layers or depth intervals. In various implementations described herein, the terms depth layers, layers, depth intervals, and intervals are used interchangeably. The number of masks and corresponding intervals may vary depending on the particular implementation.

FIG. 5Aillustrates an example mask502corresponding to the backmost layer or interval, according to some implementations. As shown, mask502is indicated with gray. In some implementations, mask502includes a mask border504, which is indicated with white. A function of mask border504is described in more detail below. Referring to bothFIGS. 4 and 5A, mask502masks the backmost interval (e.g., backmost portion or most distant background) ofFIG. 4.

FIG. 5Billustrates an example mask512corresponding to an intermediary interval immediately in front of the interval corresponding to mask502ofFIG. 5A, according to some implementations.FIG. 5Bshows mask512and a border514. Referring to bothFIGS. 4 and 5B, mask512masks the interval immediately in front of the backmost interval ofFIG. 4.

FIG. 5Cillustrates an example mask522corresponding to an intermediary interval immediately in front of the interval corresponding to mask512ofFIG. 5B, according to some implementations.FIG. 5Cshows mask522and a border524.

FIG. 5Dillustrates an example mask532corresponding to an intermediary interval immediately in front of the interval corresponding to mask522ofFIG. 5C, according to some implementations.FIG. 5Dshows mask532and a border534.

FIG. 5Eillustrates an example mask542corresponding to an intermediary interval immediately in front of the interval corresponding to mask532ofFIG. 5D, according to some implementations.FIG. 5Eshows mask542and a border544.

FIG. 5Fillustrates an example mask552corresponding to an intermediary interval immediately in front of the interval corresponding to mask542ofFIG. 5E, according to some implementations.FIG. 5Fshows mask552and a border554. Referring to bothFIGS. 4 and 5F, mask552masks the frontmost interval ofFIG. 4.

Referring toFIGS. 5A-5F, the gray portions of masks502,512,522,532,542, and552do not overlap. In some implementations, portions of border514overlap with the gray portion of mask502, portions of border524overlap with the gray portion of mask512, portions of border534overlap with the gray portion of mask522, portions of border544overlap with the gray portion of mask532, and portions of border554overlap with the gray portion of mask542.

FIGS. 6A-6Fillustrate a series of example results from a culling process, where intervals for an image in front of a given masked interval are culled or removed from the image.

FIG. 6Aillustrates an example culled image600, where the backmost interval602had been masked (by mask502ofFIG. 5A) and the group of intervals604in front of interval602have been removed, according to some implementations.

In some implementations, system102masks the red-green-blue (RGB) color of the input image in order to remove all the pixels in front of the current layer (e.g., current interval602). As shown, mask502has been removed after the culling step.

In some implementations, when system102processes intervals in front of the focal plane, system102does not mask the RGB color of the input image. This prevents color bleeding. Also, in some implementations, when system102processes intervals behind or in back of the focal plane, system102does mask the RGB color of the input image. This removes artifacts.

In some implementations, system102may in-paint the removed pixels in front of the current interval (e.g., interval602). In various implementations, system102in-paints both the color image and the depth map (e.g., both masked pixels and masked depth values). As shown, in the example implementation, the pixels of the group of intervals604are in-painted. In some implementations, to in-paint the layers, system102in-paints masked pixels in front of the current layer, but not the ones behind the current layer, which remain black. Also, in some implementations, system102dilates the masks so that there is an overlap of size (e.g., max_blur) so that the blend between layers is seamless. In some implementations, system102does not in-paint the removed pixels. This is because, in some implementations, the removed pixels are later occluded by overlaid intervals.

FIG. 6Billustrates an example culled image610, where the interval612immediately in front of backmost interval602(blacked out) had been masked (by mask512ofFIG. 5B), and the group of intervals614in front of interval612have been removed, according to some implementations.

Also, as shown, in some implementations, system102may mask the color image to set the color pixels behind the current layer (e.g., interval612or any intervals behind the current layer) to 0. As a result, the pixels of interval602are black. This is one way to enable system102to process each interval one at a time. As shown, mask512has been removed after the culling step. In some implementations, system102does not mask the color image to set the color pixels behind the current layer (e.g., interval612) to 0. System102may still process each interval one at a time.

In some implementations, system102may mask the original depth map to remove the pixels in front and behind the current layer. As shown, in some implementations, system102may in-paint the removed depth map pixels in front of the current layer (e.g., interval612). In various implementations, system102in-paints both the color image and the depth map (e.g., both masked pixels and masked depth values). In some implementations, system102does not mask the original depth map to remove the pixels in the front and behind the current layer. In some implementations, system102does not in-paint the removed depth map pixels in front of the current layer. In various implementations, omitting some steps increases computation speed.

FIG. 6Cillustrates an example culled image620, where the interval622immediately in front of interval612(not shown/blacked out) had been masked (by mask522ofFIG. 5C) and the group of intervals624in front of interval622have been removed, according to some implementations. As shown, in some implementations, system102may in-paint the removed depth map pixels in front of the current layer (e.g., interval622).

FIG. 6Dillustrates an example culled image630, where the interval632immediately in front of interval622had been masked (by mask532ofFIG. 5D) and the group of intervals634in front of interval622(blacked out) have been removed, according to some implementations. As shown, in some implementations, system102may in-paint the removed depth map pixels in front of the current layer (e.g., interval632).

FIG. 6Eillustrates an example culled image640, where the interval642immediately in front of interval632(not shown/blacked out) had been masked (by mask542ofFIG. 5E) and the group of intervals644in front of interval632have been removed, according to some implementations.

FIG. 6Fillustrates an example culled image650, where the interval652immediately in front of interval642(not shown/blacked out) had been masked (by mask552ofFIG. 5F) and the group of intervals in front of interval642have been removed, according to some implementations.

FIGS. 7A-7Fillustrate a series of example results from the blurring process, where intervals are individually blurred. In some implementations, system102masks pixels of each depth interval that are not being blurred while pixels of a current depth interval are being blurred.

In some implementations, the blurring is a disk blur, which provides renderings with bokeh effects. More specifically, in some implementations, system102applies the blur to each depth interval/layer using a disk kernel, instead of a gaussian kernel. In various implementations, system102changes the blur radius per pixel, so that two neighboring pixels with similar depth and that belong to different layers have comparable blur radii, thus avoiding quantization errors.

In some implementations, system102may apply the following equation to convert from depth to blur radius. Given the focal plane d_f in depth units, and the blur at infinity in pixels b_inf (related to the aperture), the blur of a pixel with depth d is:
b=b_inf*abs(d−d_f)/d.

Such disk blurring is advantageous over Gaussian blurring, because Gaussian blurring cannot produce bokeh effects.

FIG. 7Billustrates an example blurred image710, according to some implementations. As shown, interval612is blurred. In various implementations, interval612is blurred to a lesser degree than interval602(FIG. 7A).

FIG. 7Cillustrates an example blurred image720, according to some implementations. As shown, interval622is blurred. In various implementations, interval622is blurred to a lesser degree than interval612(FIG. 7B).

FIG. 7Dillustrates an example blurred image730, according to some implementations. As shown, interval632is blurred. In various implementations, interval632is blurred to a lesser degree than interval622(FIG. 7C).

FIG. 7Eillustrates an example blurred image740, according to some implementations. As shown, interval642is blurred. In various implementations, interval642is blurred to a lesser degree than interval632(FIG. 7D).

FIG. 7Fillustrates an example blurred image750, according to some implementations. As shown, interval652is blurred. In various implementations, interval652is blurred to a lesser degree than interval642(FIG. 7E).

FIGS. 8A-8Fillustrate a series of example results from the blending process, where intervals are blended with adjacent intervals. In some implementations, the blurring is a disk blur. In some implementations, system102blends blurred pixels of a depth interval with previously blurred pixels of at least one other depth interval. In various implementations, this step may be multi-threaded.

FIG. 8Aillustrates an example image800that includes blurred interval602, according to some implementations.FIG. 8Ais the same asFIG. 7A, because blurred interval602is not yet blended.

FIG. 8Billustrates an example blended image810, where intervals602and612are blended, according to some implementations. Interval612is added to interval602. As shown, edge of interval602that corresponds to the gray edge of mask502ofFIG. 5Ais blended with the edge of interval612that corresponds to white border514of mask512ofFIG. 5B. The resulting blended image810includes intervals602and612, where their adjoining edges are blended. In various implementations, system102applies a blending algorithm to the white borders504,514,524,534,544, and554. In various implementations, system102blends the pixels in the white borders linearly.

FIG. 8Cillustrates an example blended image820, where intervals612and622are blended, according to some implementations. Interval622is added to interval612. As shown, edge of interval612that corresponds to the gray edge of mask512ofFIG. 5Bis blended with the edge of interval622that corresponds to white border524of mask522ofFIG. 5C. The resulting blended image820includes intervals612and622, where their adjoining edges are blended.

FIG. 8Dillustrates an example blended image830, where intervals622and632are blended, according to some implementations. Interval632is added to interval622. As shown, the edge of interval622that corresponds to the gray edge of mask522ofFIG. 5Cis blended with the edge of interval632that corresponds to white border534of mask532ofFIG. 5D. The resulting blended image830includes intervals622and632, where their adjoining edges are blended.

FIG. 8Eillustrates an example blended image840, where intervals632and642are blended, according to some implementations. Interval642is added to interval632. As shown, edge of interval642that corresponds to the gray edge of mask532ofFIG. 5Dis blended with the edge of interval642that corresponds to white border544of mask542ofFIG. 5E. The resulting blended image840includes intervals632and642, where their adjoining edges are blended.

FIG. 8Fillustrates an example blended image850, where intervals642and652are blended, according to some implementations. Interval652is added to interval642. As shown, the edge of interval642that corresponds to the gray edge of mask542ofFIG. 5Eis blended with the edge of interval652that corresponds to white border554of mask552ofFIG. 5F. The resulting blended image850includes intervals642and652, where their adjoining edges are blended.

In block208, system102applies at least one camera response function to the image after the pixels of the depth intervals are blurred. In other words, the blurred layers are merged together before the camera response function is applied. In various implementations, the camera response function is a gamma correction. As such, system102applies a gamma correction to the image after the pixels of the depth intervals are blurred.

FIG. 9illustrates an example final depth-of-field rendering900, where system102has applied a gamma correction, according to some implementations. As a result, portions of the image that are in the focal plane are clear, and portions of the image that are out of the focal plane are blurred by different amounts according to the depth of the pixels and the focal plane. Implementations render an initially all-in-focus image such that it appears to have been captured with a single-lens reflex (SLR) camera or digital SLR (DSLR) camera. Furthermore, implementations enable a realistic blur with any aperture shape, and enables any kind of blur, e.g., disk blur, which is an ideal lens blur generated by a circular aperture. Other types of blurs can be used to simulate different aperture shapes.

Implementations described herein provide various benefits. For example, implementations provide depth-of-field renderings at a speed that is significantly faster that conventional solutions without sacrificing quality. Implementations are compatible with graphics processing unit (GPU) hardware, while producing high quality blur with various types of blur.

While system102is described as performing the steps as described in the implementations herein, any suitable component or combination of components of system102or any suitable processor or processors associated with system102may perform the steps described.

FIG. 10illustrates a block diagram of an example server device1000, which may be used to implement the implementations described herein. For example, server device1000may be used to implement server device104ofFIG. 1, as well as to perform the method implementations described herein. In some implementations, server device1000includes a processor1002, an operating system1004, a memory1006, and an input/output (I/O) interface1008. Server device1000also includes a social network engine1010and a media application1012, which may be stored in memory1006or on any other suitable storage location or computer-readable medium. Media application1012provides instructions that enable processor1002to perform the functions described herein and other functions.

For ease of illustration,FIG. 10shows one block for each of processor1002, operating system1004, memory1006, I/O interface1008, social network engine1010, and media application1012. These blocks1002,1004,1006,1008,1010, and1012may represent multiple processors, operating systems, memories, I/O interfaces, social network engines, and media applications. In other implementations, server device1000may not have all of the components shown and/or may have other elements including other types of elements instead of, or in addition to, those shown herein.

Although the description has been described with respect to particular embodiments thereof, these particular embodiments are merely illustrative, and not restrictive. Concepts illustrated in the examples may be applied to other examples and implementations. For example, some implementations are described herein in the context of a social network system. However, the implementations described herein may apply in contexts other than a social network. For example, implementations may apply locally for an individual user.

Note that the functional blocks, methods, devices, and systems described in the present disclosure may be integrated or divided into different combinations of systems, devices, and functional blocks as would be known to those skilled in the art.

Any suitable programming languages and programming techniques may be used to implement the routines of particular embodiments. Different programming techniques may be employed such as procedural or object-oriented. The routines may execute on a single processing device or multiple processors. Although the steps, operations, or computations may be presented in a specific order, the order may be changed in different particular embodiments. In some particular embodiments, multiple steps shown as sequential in this specification may be performed at the same time.

A “processor” includes any suitable hardware and/or software system, mechanism or component that processes data, signals or other information. A processor may include a system with a general-purpose central processing unit, multiple processing units, dedicated circuitry for achieving functionality, or other systems. Processing need not be limited to a geographic location, or have temporal limitations. For example, a processor may perform its functions in “real-time,” “offline,” in a “batch mode,” etc. Portions of processing may be performed at different times and at different locations, by different (or the same) processing systems. A computer may be any processor in communication with a memory. The memory may be any suitable data storage, memory and/or non-transitory computer-readable storage medium, including electronic storage devices such as random-access memory (RAM), read-only memory (ROM), magnetic storage device (hard disk drive or the like), flash, optical storage device (CD, DVD or the like), magnetic or optical disk, or other tangible media suitable for storing instructions for execution by the processor. The software instructions can also be contained in, and provided as, an electronic signal, for example in the form of software as a service (SaaS) delivered from a server (e.g., a distributed system and/or a cloud computing system).