Image processing apparatus and method utilizing a lookup table and spherical harmonic coefficients

An image processing apparatus may include a storage unit to store a lookup table (LUT) including information on corresponding relations between an occlusion vector related to at least one point of a 3-dimensional (3D) object and a spherical harmonics (SH) coefficient; and a rendering unit to determine a first SH coefficient corresponding to a first occlusion vector related to a first point of the 3D object using the LUT and to determine a pixel value of the first point using the first SH coefficient.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the priority benefit of Korean Patent Application No. 10-2013-0011761, filed on Feb. 1, 2013, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference.

BACKGROUND

Example embodiments of the following disclosure relate to an image processing apparatus and method, and more particularly, to an apparatus and method for rendering an image in a spherical harmonics (SH)-based lighting environment.

2. Description of the Related Art

Generally, a spherical harmonics (SH) basis refers to a small number of parameters per frequency band, capable of expressing intensity of all directions. Further, SH may refer to functions that are defined on a sphere. As such, the SH basis is widely applied in various fields including physics, for example. In the field of image processing, the SH basis is applicable even to rendering based on an environment map defined in all directions and may also be used for rendering of a diffuse surface.

Different from lighting or material which are globally defined, visibility information is defined for every surface point. Therefore, since data quantity of the visibility information is great, it is more efficient to read out a stored value through pre-computation in real time rather than computing the information in real time. Accordingly, this method is difficult to be directly applied to dynamic geometry incapable of pre-computation.

Conventionally, the SH based accesses have been already suggested. However, lighting environments have not been considered properly. Also, it is difficult to consider a difference in intensity of every direction with respect to a lighting environment map.

SUMMARY

The foregoing and/or other aspects are achieved by providing an image processing apparatus including a storage unit to store a lookup table (LUT) including information on corresponding relations between an occlusion vector related to at least one point of a 3-dimensional (3D) object and a spherical harmonics (SH) coefficient, and a rendering unit to determine a first SH coefficient using the LUT and to determine a pixel value of the first point using the first SH coefficient.

The first SH coefficient may correspond to a first occlusion vector related to a first point of the 3D object.

The rendering unit may compute the first occlusion vector by sampling an occlusion map related to the first point.

The rendering unit may compute the first occlusion vector by computing occlusion angles related to direction in a number related to the sampling with respect to the first point, using a depth image and a normal image related to the first point.

The image processing apparatus may further include a computation unit to generate the LUT by performing pre-computation that projects the occlusion vector of the at least one point to the at least one predetermined SH basis.

The computation unit may include an occlusion vector computation unit to compute an occlusion vector of the at least one point using a Z-buffer value and a normal vector of the at least one point, an occlusion map generation unit to generate an occlusion map of the at least one point by reflecting a bidirectional reflectance distribution function (BRDF) related to the at least one point, an SH coefficient computation unit to compute an SH coefficient corresponding to the occlusion map by projecting the occlusion map to the at least one SH basis, and an LUT update unit to update the LUT by reflecting information on corresponding relations between the occlusion vector and the SH coefficient to the LUT.

The rendering unit may compute a second SH coefficient by projecting a lighting environment related to the first point to at least one predetermined SH basis, and determines the pixel value of the first point by performing dot product of the first SH coefficient and the second SH coefficient.

The foregoing and/or other aspects are also achieved by providing an image processing apparatus including an occlusion vector computation unit to compute an occlusion vector of at least one point included in a 3-dimensional (3D) object, using a Z-buffer value and a normal vector of the at least one point, an occlusion map generation unit to generate an occlusion map of the at least one point by reflecting a bidirectional reflectance distribution function (BRDF) related to the at least one point, a spherical harmonics (SH) coefficient computation unit to compute an SH coefficient corresponding to the at least one occlusion map by projecting the occlusion map to at least one predetermined SH basis, and a lookup table (LUT) update unit to update the LUT by reflecting information on corresponding relations between the at least one occlusion vector and the at least one SH coefficient to the LUT.

The foregoing and/or other aspects are achieved by providing an image processing method including computing a first occlusion vector related to a first point to be rendered, determining a first spherical harmonics (SH) coefficient corresponding to the first occlusion vector using a lookup table (LUT), and rendering a pixel vale of the first point using the first SH coefficient.

The computing of the first occlusion vector may include generating an occlusion map using a depth image and a normal image related to the first point, and computing the first occlusion vector by sampling the occlusion map.

The computing of the first occlusion vector may include computing occlusion angles related to directions in a number related to the sampling with respect to the first point, and determining of the occlusion angles as elements of the first occlusion vector.

The rendering may include computing a second SH coefficient by projecting a lighting environment related to the first point to at least one predetermined SH basis, and determining the pixel value of the first point by performing dot product of the first SH coefficient and the second SH coefficient.

The image processing method may further include generating the LUT by performing pre-computation that projects the occlusion vector of the at least one point including the first point to the at least one SH basis.

The generating of the LUT may include computing an occlusion vector of at least one point using a Z-buffer value and a normal vector of the at least one point, generating an occlusion map of the at least one point by reflecting a bidirectional reflectance distribution function (BRDF) related to the at least one point, computing a spherical harmonics (SH) coefficient corresponding to the at least one occlusion map by projecting the occlusion map to at least one predetermined SH basis, and updating the LUT by reflecting information on corresponding relations between the occlusion vector and the SH coefficient to the LUT.

The foregoing and/or other aspects are also achieved by providing an image processing method computing an occlusion vector of at least one point included in a 3-dimensional (3D) object, using a Z-buffer value and a normal vector of the at least one point, generating an occlusion map of the at least one point by reflecting a bidirectional reflectance distribution function (BRDF) related to the at least one point, computing a spherical harmonics (SH) coefficient corresponding to the at least one occlusion map by projecting the occlusion map to at least one predetermined SH basis, and updating the LUT by reflecting information on corresponding relations between the at least one occlusion vector and the at least one SH coefficient to the LUT.

The foregoing and/or other aspects are also achieved by providing an image processing apparatus, including: a computation unit to generate an LUT by computing at least one occlusion vector and SH coefficients; a storage unit to store the LUT including information relating to the at least one occlusion vector and the SH coefficients; and a rendering unit to derive a first SH coefficient using the LUT, thereby performing real-time rendering.

The computation unit may compute the at least one occlusion vector using a Z-buffer value and a normal vector of at least one point of a 3D object, and may compute the SH coefficients by projecting an occlusion map generated by reflecting a BRDF related to the at least one point, to at least one SH basis.

The rendering unit may determine a first SH coefficient corresponding to a first occlusion vector related to a first point of the 3D object using the LUT, and may determine a pixel value of the first point using the first SH coefficient.

According to example embodiments, the rendering unit of the image processing apparatus may quickly derive the SH coefficient using the LUT, thereby performing real-time rendering.

DETAILED DESCRIPTION

Reference will now be made in detail to example embodiments, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout.

Terms to be used below are selected as general terms based on their functions in the present disclosure and may vary according to users, user's intentions, or practices.

But, in specific cases, terms arbitrarily selected by the applicant are also used for easy understanding and explanation and, in this case, meanings of the terms will be mentioned in corresponding detailed description section, so the present disclosure should be understood not by lexical meanings of the terms but by given meanings of the terms.

FIG. 1illustrates a block diagram of an image processing apparatus100, according to example embodiments.

The image processing apparatus100may express factors considered in rendering, such as lighting, material, and visibility information, using a spherical harmonics (SH) basis. In addition, visibility may be applied to dynamic geometry through computation based on screen space ambient occlusion, instead of pre-computation of the visibility information.

The image processing apparatus100may include a storage unit110and a rendering unit120. Each of the above-described units may include at least one processing device. Further, the storage unit110may be a hard drive capable of storing data, however, the present disclosure is not limited thereto.

The storage unit110may store a lookup table (LUT) including information regarding an occlusion vector related to at least one point of a 3-dimensional (3D) object and SH coefficients.

The rendering unit120may determine a first SH coefficient corresponding to a first occlusion vector related to a first point of the 3D object using the LUT, and determine a pixel value of the first point using the first SH coefficient.

The rendering unit120may quickly derive the SH coefficient using the LUT, thereby performing real time rendering.

For example, the rendering unit120may compute the first occlusion vector by sampling an occlusion map related to the first point.

In this case, the rendering unit120may compute the first occlusion vector by computing occlusion angles related to directions in a number related to the sampling, with respect to the first point, using a depth image and a normal image related to the first point.

In addition, the rendering unit120may compute a second SH coefficient by projecting a lighting environment related to the first point to at least one SH basis, and determine a pixel value of the first point by performing dot product of the first SH coefficient and the second SH coefficient.

In another example embodiment, the image processing apparatus100may further include a computation unit200(as shown inFIG. 2, for example). The computation unit200may include at least one processing device.

The computation unit200may generate the LUT by performing pre-computation that projects the occlusion vector of the at least one point to the at least one SH basis.

A process of generating the LUT through the pre-computation will be described with reference toFIGS. 2 to 7.

FIG. 2illustrates a block diagram of the computation unit200of the image processing apparatus100, according to other example embodiments.

The computation unit200may include an occlusion vector computation unit210, an occlusion map generation unit220, an SH coefficient computation unit230, and an LUT update unit240.

The occlusion vector computation unit210may compute an occlusion vector of at least one point included in the 3D object, using a Z-buffer value (also referred to as a depth buffer) and a normal vector of the at least one point. For purpose of simplicity, a description of a z-buffer and a normal vector will be omitted.

The occlusion map generation unit220may generate an occlusion map of the at least one point, by reflecting a bidirectional reflectance distribution function (BRDF) related to at least one point.

The SH coefficient computation unit230may compute at least one SH coefficient corresponding to the at least one occlusion map, by projecting the occlusion map to at least one predetermined SH basis.

The LUT update unit240may update the LUT by reflecting information on corresponding relations between the at least one occlusion vector and the at least one SH coefficient to the LUT.

FIG. 3illustrates an SH basis according to example embodiments.

The SH may refer to a Fourier transform in a 1D circle and may constitute an orthonormal basis.

The SH may be expressed by the following Equation 1 to Equation 3.

A Lambertian surface used for rendering may function as a low pass filter. That is, frequency-space analysis is applicable to rendering.

In diffuse surface rendering, for example, when 9 SH bases301,311,312,313,321,322,323,324, and325are used as shown inFIG. 3, an error rate of about 3% or less with respect to a ground truth may be achieved.

Considering this matter, quantity of computation and a bandwidth may be reduced by as much as thousands or tens of thousand times.

FIGS. 4 and 5illustrate a computation process of an occlusion vector with respect to a particular pixel, according to example embodiments.

The image processing apparatus100may perform pre-computation with respect to all possible cases related to visibility, while considering ambient objects on a screen space.

Results of the pre-computation may be generated as the occlusion vectors corresponding to the respective cases, and stored as keys. Further, the pre-computation may be performed by the computation unit.

The occlusion vector may be computed using a Z-buffer value and a normal vector of at least one point. The occlusion vector may be defined as a direction vector with respect to a plurality of directions, such that the direction vector originates from the at least one point P401, as shown inFIG. 4.

Referring toFIG. 4, a direction vector410with respect to 8 directions may be generated with respect to at least one point401of the object. The 8 directions described above are exemplary, and thus, the present disclosure is not limited to 8 directions.

An angle of the direction vector410, that is, the occlusion angle, may be computed using a position of an occluder on the screen space.

For example, an angle between a center of the at least one point401of the object with respect to the Z-buffer, that is, a depth buffer, and an XY-plane of a horizon vector passing over an occluder located around may be defined as the occlusion angle (seeFIG. 5, for example).

The occlusion vector may be determined to be an orderly arrangement of the occlusion angles with respect to the eight directions.

The occlusion angles computed in the process ofFIG. 4may be generated as the occlusion vector with respect to the various cases as shown inFIG. 5through discretization.

That is, the occlusion angles may be discretized at every 10 degrees, thereby being determined as the occlusion vectors with respect to the respective cases. However, the present disclosure is not limited to the occlusion angles being discretized at every 10 degrees. That is, the occlusion angles may be discretized at other angle intervals.

FIG. 6illustrates an occlusion map generation process according to example embodiments.

The occlusion vectors computed in the process ofFIGS. 4 and 5may not be directly converted into the SH coefficients.

Therefore, to compute the SH coefficients with respect to the occlusion vectors, an occlusion map600may be generated first using the occlusion vectors.

The occlusion map600may be understood as a map obtained by interpolating angles between the occlusion vectors and rendering occlusion directions with respect to all directions.

In this case, the occlusion map600may be generated by reflecting a BRDF related to at least one point601.

FIG. 7illustrates a process of updating an LUT using the occlusion map600ofFIG. 6, according to example embodiments. That is, the LUT stored in the storage unit110may be updated.

The occlusion map600generated in the process ofFIG. 6may be used to update the LUT that stores information on corresponding relationships between the occlusion vector related to at least one point of the 3D object and the SH coefficient.

The SH coefficient may be computed by projecting the occlusion map to at least one SH basis.

The LUT may be updated by reflecting the information on the corresponding relations between the occlusion vector and the SH coefficient related to the occlusion vector.

FIG. 8illustrates a block diagram of an image processing apparatus800according to other example embodiments.

The image processing apparatus800may include a computation unit810, a storage unit820, and a rendering unit830. The computation unit810may or may not be the same as the computation unit200ofFIG. 2. Further, the storage unit820and the rendering unit830may or may not be the same as the storage unit110and the rendering unit120ofFIG. 1, respectively.

The computation unit810may generate an LUT through pre-computation that projects an occlusion vector of at least one point of a 3D object to at least one predetermined SH basis.

In this case, the computation unit810may compute the at least one occlusion vector using a Z-buffer value and a normal vector of the at least one point, and may compute SH coefficients by projecting an occlusion map generated by reflecting a BRDF related to the at least one point, to the at least one SH basis.

The storage unit820may store an LUT storing information on corresponding relationships between the occlusion vector related to the at least one point and the SH coefficients.

The rendering unit830may determine a first SH coefficient corresponding to a first occlusion vector related to a first point of the 3D object using the LUT and determine a pixel value of the first point using the first SH coefficient.

As such, the rendering unit830may quickly derive the SH coefficient using the LUT, thereby performing real time rendering.

For example, the rendering unit830may sample an occlusion map related to the first point, thereby computing the first occlusion vector.

In this case, the rendering unit830may compute the first occlusion vector by computing occlusion angles related to directions in a number related to the sampling, with respect to the first point, using a depth image and a normal image related to the first point.

In addition, the rendering unit830may compute a second SH coefficient by projecting a lighting environment related to the first point to at least one SH basis, and determine a pixel value of the first point by performing dot product of the first SH coefficient and the second SH coefficient.

When a Z-buffer and a normal vector related to the 3D object are input, an occlusion vector related to the 3D object and a corresponding SH coefficient may be pre-computed by the computation unit.

The computation unit may compute the occlusion vector with respect to the at least one point910, using the Z-buffer and the normal vector of the at least one point910of the 3D object.

In addition, the computation unit may generate an occlusion map by reflecting a BRDF related to the at least one point910, and compute SH coefficients related to the occlusion vector by projecting the occlusion map to the at least one SH basis.

Information on corresponding relations between the occlusion vector the SH coefficients computed by the computation unit may be stored in the LUT of the storage unit.

FIGS. 10 and 11illustrate a real time rendering process, according to example embodiments.

The rendering unit may perform rendering using an occlusion map1010related to the first point910of the 3D object.

A color value of each pixel may be determined through dot product of a second SH coefficient considering a lighting environment and a first SH coefficient related to visibility information and a BRDF of a corresponding pixel.

Since SH with respect to the lighting environment is globally computed only once with respect to an entire scene, real time computation may be achieved efficiently.

However, since real time computation of the first SH coefficient related to the visibility and the BRDF is impossible, a result of pre-computation by the computation unit may be used.

The rendering unit may sample the occlusion map1010related to the first point910as shown in1020ofFIG. 10, thereby computing a first occlusion vector with respect to the 3D object.

For example, the rendering unit may compute the first occlusion vector by generating and randomly sampling direction vectors1022with respect to eight directions with respect to a first point1021, using a depth image and a normal image related to the first point1021.

Occlusion angles related to the direction vectors1022may be computed as shown inFIG. 11.

Referring toFIG. 11, a tangent line of a provided surface S at point P may be a tangent vector. A vector tangential to a highest portion of the surface with respect to the tangent vector Vt may be a horizon vector Vh.

The occlusion angles may be obtained through dot product of the tangent vector and the horizontal vector.

The rendering unit may compute occlusion vectors from the occlusion angles and obtain a first SH coefficient related to the first points910and1021by referencing the LUT.

The rendering unit may compute a second SH coefficient by projecting a lighting environment related to the first points910and1021to at least one predetermined SH basis, and determine pixel values of the first points910and1021by performing dot product of the first SH coefficient and the second SH coefficient.

FIG. 12illustrates a flowchart of an image processing method, according to example embodiments.

In operation1210, a depth image and a normal image related to a first point to be rendered may be input.

In operation1220, the rendering unit may compute a first occlusion vector related to the first point.

That is, in operation1220, the rendering unit may generate an occlusion map using the depth image and the normal image related to the first point, and may compute the first occlusion vector by sampling the occlusion map.

In addition, in operation1220, the sampling may refer to the computing of occlusion angles related to different directions in a number related to the sampling with respect to the first point, and determining of the occlusion angles as elements of the first occlusion vector.

In operation1230, the rendering unit may determine a first SH coefficient corresponding to the first occlusion vector using an LUT.

In operation1240, the rendering unit may render a pixel value of the first point using the first SH coefficient.

In another embodiment, the rendering unit may further compute a second SH coefficient by projecting a lighting environment related to the first point to at least one predetermined SH basis, and determine the pixel value of the first point by performing dot product of the first SH coefficient and the second SH coefficient.

According to some example embodiments, the image processing method may generate the LUT through pre-computation of a computation unit. Generation and update of the LUT may be performed as shown inFIG. 13.

FIG. 13illustrates a flowchart of an image processing method according to other example embodiments.

In operation1310, the occlusion vector computation unit may compute an occlusion vector of the at least one point included in the 3D object, using a Z-buffer value and a normal vector of the at least one point.

In operation1320, the occlusion map generation unit may generate an occlusion map of the at least one point by reflecting a BRDF related to the at least one point.

In operation1330, the SH coefficient computation unit may compute at least one SH coefficient corresponding to the at least one occlusion map, by projecting the occlusion map to at least one predetermined SH basis.

In operation1340, the LUT update unit may update the LUT by reflecting information on corresponding relations between the at least one occlusion vector and the at least one SH coefficient to the LUT.

Further, according to an aspect of the embodiments, any combinations of the described features, functions and/or operations can be provided.

Moreover, the various embodiments of the image processing apparatus discussed above may include at least one processor to execute at least one of the above-described units and methods.