Patent ID: 12205260

DETAILED DESCRIPTION OF THE EMBODIMENTS

The following description is written by referring to terms of this technical field. If any term is defined in this specification, such term should be interpreted accordingly. In addition, the connection between objects or events in the below-described embodiments can be direct or indirect provided that these embodiments are practicable under such connection. Said “indirect” means that an intermediate object or a physical space exists between the objects, or an intermediate event or a time interval exists between the events.

The disclosure herein includes image processing circuits and image processing methods. On account of that some or all elements of the image processing circuits could be known, the detail of such elements is omitted provided that such detail has little to do with the features of this disclosure, and that this omission nowhere dissatisfies the specification and enablement requirements. The image processing methods can be performed by the image processing circuits or their equivalents. A person having ordinary skill in the art can choose components or steps equivalent to those described in this specification to carry out the present invention, which means that the scope of this invention is not limited to the embodiments in the specification.

FIG.1is a functional block diagram of an image processing circuit according to an embodiment of the present invention, andFIG.2is a functional block diagram of an image processing circuit according to another embodiment of the present invention.FIG.1andFIG.2are similar, with the difference in the input data of the base layer mapping circuit140, which will be discussed in detail below.

The image processing circuit100and the image processing circuit200include a pre-processing module110, a logarithmic conversion module120, a detail extraction module130, a base layer mapping circuit140, a detail layer enhancement module150, an adder circuit160, an inverse logarithmic conversion module170, and a post-processing module180.

The pre-processing module110converts the input data Dininto the luminance data Y(y, x) (e.g., grayscale data). The input data Dincan be a raw image file or an image file in the RGB format (hereinafter referred to as an RGB image file). The pre-processing module110converts the raw image file into the luminance data Y(y, x) according to equation (1), or converts the RGB image file into the luminance data Y(y, x) according to equation (2).
Y(y,x)=Σi=−22Σj=−22w(i,j)Raw(y+i,x+j)  (1)
Y(y,x)=0.2R(x,y)+0.7G(x,y)+0.1B(x,y)  (2)

In equation (1), w(i,j) is the filter coefficient, which may be a coefficient that resembles Gaussian filtering. For example:

w=[0121012421248421242101210]orw=[0000001210024200121000000].

The logarithmic conversion module120, which maps the luminance data Y(y, x) to the logarithmic domain, aims at mapping the luminance data Y(y, x) to a domain that is consistent with the perception of the human eyes (i.e., to generate the target data Log Y(y, x)). The mapping of the luminance data Y(y, x) to the logarithmic domain can be embodied using equation (3).
LogY(y,x)=LUT(Y(y,x))  (3)

where LUT( ) is an operation using the lookup table (LUT) (i.e., a table lookup operation). In some embodiments, the table lookup operation of equation (3) may be a logarithmic curve-based table lookup operation, as shown in equation (4).
LogY(y,x)=log2(Y(y,x)+)  (4)

where ∈ is a small constant that prevents log2(Y(y, x)) from being a meaningless value (e.g., −∞) when the luminance data Y(y, x) is 0.

The detail extraction module130processes the target data Log Y(y, x) to generate at least one base layer and multiple detail layers. The detail extraction module130utilizes multi-level guided image filters (GIFs) to divide the target data Log Y(y, x) into multiple detail layers of different sizes. Therefore, the detail layer Dtinoutputted by the detail extraction module130includes multiple detail layer data, and the base layer Binincludes at least one base layer data. The detail extraction module130will be discussed in detail below with reference toFIG.5andFIG.6.

The base layer mapping circuit140converts the base layer Bin(FIG.1) or the target data LogY(y, x) (FIG.2) to generate the converted base layer data. In the embodiment ofFIG.2, the base layer mapping circuit140uses the target data LogY(y, x) as the base layer. In some embodiments, the base layer mapping circuit140performs mapping according to a gamma curve or a gamma-like curve shown inFIG.3. One purpose of the base layer mapping circuit140is to improve the brightness of the base layer.

The detail layer enhancement module150converts the detail layer data Dtinto generate the converted detail layer data Dtout. In some embodiments, the detail layer enhancement module150maps the detail layer data Dtinby looking up a table, as in equation (5).
Dtout=LUTd1(Dtin)  (5)

where the table lookup operation LUTd1( ) can be a linear enhancement operation (FIG.4A) or a nonlinear enhancement operation (FIG.4B).

The adder circuit160sums the converted base layer data and the converted detail layer data to generate the output data Log YTM.

The inverse logarithmic conversion module170performs the inverse logarithmic conversion on the output data Log YTMto generate the output luminance data YTM. In some embodiments, the inverse logarithmic conversion module170operates based on equation (6).
YTM=2Log YTM−E(6)

The post-processing module180generates the image data Doutaccording to the input data Din, the luminance data Y(y, x), and the output luminance data YTM. The image data Doutis the outcome of the contrast enhancement of the input data Din. In some embodiments, the post-processing module180operates based on equation (7) (corresponding to the input data Dinbeing the raw image file) or equation (8) (corresponding to the input data Dinbeing the RGB image file).

RawTM=YTMY*Raw(7){RTM=YTMY*RGTM=YTMY*GBTM=YTMY*B(8)

In some embodiments, each component inFIG.1is embodied by hardware (circuit).

FIG.5is a functional block diagram of one embodiment of the detail extraction module130. The detail extraction module130includes a first-level detail extraction module205_aand a second-level detail extraction module205_b. The first-level detail extraction module205_aincludes a detail layer extraction circuit205_a0, and the second-level detail extraction module205_bincludes a detail layer extraction circuit205_b0. The detail layer extraction circuit205_a0processes the target data Log Y(y, x) to generate the base layer data B0 and the detail layer data Dt0. The detail layer extraction circuit205_a0includes a guided filter factor calculation circuit210_a0, a buffer circuit220_a0, an interpolation circuit230_a0, a filter circuit240_a0, and an adder circuit250_a0. The detail layer extraction circuit205_b0processes the base layer data B0 to generate the base layer data B1 and the detail layer data Dt1. The detail layer extraction circuit205_b0includes a guided filter factor calculation circuit210_b0, a buffer circuit220_b0, an interpolation circuit230_b0, a filter circuit240_b0, and an adder circuit250_b0.

The guided filter factor calculation circuit210_a0and the guided filter factor calculation circuit210_b0perform calculations based on the following equations (9) to (17) to generate the down-sampled guided filter factor adspand the down-sampled guided filter factor bdsp. For the guided filter factor calculation circuit210_a0, “Y” in equation (9) is the target data Log Y(y, x). For the guided filter factor calculation circuit210_b0, “Y” in equation (9) is the base layer data B0.

Ydsp(x,y)=1N2⁢∑i=0N-1∑j=0N-1Y⁡(Nx+i,Ny+j)(9)Ymean(x,y)=∑i=-M-12M-12∑j=-M-12M-12w⁢(i,j)⁢Ydsp⁢(x+i,y+j)(10)Ymean⁢2(x,y)=∑i=-M-12M-12∑j=-M-12M-12w⁡(i,j)⁢Ydsp(x+i,y+j)2(11)Ymean(x,y)=1M2⁢∑i=-M-12M-12∑j=-M-12M-12Ydsp(x+i,y+j)(12)Ymean⁢2(x,y)=1M2⁢∑i=-M-12M-12∑j=-M-12M-12Ydsp(x+i,y+j)(13)adsp(x,y)=Ymean⁢2(x,y)-Ymean(x,y)2Ymean⁢2(x,y)-Ymean(x,y)2+ϵ(14)bdsp(x,y)=Ymean(x,y)-a⁢Ymean(x,y)(15)amdsp(x,y)=∑i=-M-12M-12∑j=-M-12M-12w⁡(i,j)⁢adsp(x+i,y+j)(16)bmdsp(x,y)=∑i=-M-12M-12∑j=-M-12M-12w⁡(i,j)⁢bdsp(x+i,y+j)(17)

Equation (9) is to perform down-sampling (down-sampling rate being N) on “Y” by averaging. Equation (10) and equation (11) respectively perform M*M filtering on Ydsp(x,y) (i.e., the down-sampled gi) and Ydsp(x,y)2, the value of M being proportional to the size (i.e., the data amount) of w(i,j) in equation (10) and equation (11) is the filter coefficient, which can be the filter coefficient of mean filtering or Gaussian filtering. When w(i,j) is the filter coefficient of the mean filtering

(i.e.,w⁡(i,j)=1M2),
equation (10) and equation (11) become equation (12) and equation (13), respectively. Equation (14) and equation (15) are used to calculate the factors of the GIFs (adspand bdsp), while equation (16) and equation (17) are used to filter the factors adspand bdsp, respectively (for mean filtering,

w⁡(i,j)=1M2).

The buffer circuit220_a0and the buffer circuit220_b0store the down-sampled guided filter factor amdspand the down-sampled guided filter factor bmdsp. In some embodiments, if the computation speed of the guided filter factor calculation circuit210_a0and the guided filter factor calculation circuit210_b0is fast enough, the buffer circuit220_a0and the buffer circuit220_b0may be omitted.

The interpolation circuit230_a0and the interpolation circuit230_b0perform calculations based on the following equations (18) to (27) to generate the average guided filter factor ameanand the average guided filter factor bmean.

xd=floor⁢(xN)(18)yd=floor(yN)(19)xoff=xN-xd(20)yoff=yN-yd(21)ay⁢0(x,yd)=(1-xoff)⁢amdsp(xd,yd)+xoff⁢amdsp(xd+1,yd)(22)ay⁢1(x,yd)=(1-xoff)⁢amdsp(xd,yd+1)+xoff⁢amdsp(xd+1,yd+1)(23)by⁢0(x,yd)=(1-xoff)⁢bmdsp(xd,yd)+xoff⁢bmdsp(xd+1,yd)(24)by⁢1(x,yd)=(1-xoff)⁢bmdsp(xd,yd+1)+xoff⁢bmdsp(xd+1,yd+1)(25)amean(x,y)=(1-yoff)⁢ay⁢0(x,yd)+yoff⁢ay⁢1(x,yd+1)(26)bmean(x,y)=(1-yoff)⁢by⁢0(x,yd)+yoff⁢by⁢1(x,yd+1)(27)

Equations (18) to (21) are provided for the up-sampling operation, equations (22) to (25) are provided for the interpolation operation in the X direction, and equations (26) to (27) are provided for the interpolation operation in the Y direction. Therefore, the interpolation circuit230_a0and the interpolation circuit230_b0can also be referred to as the bilinear interpolation circuits.

The filter circuit240_a0and the filter circuit240_b0perform guided filtering based on the following equation (28) to generate the base layer data B0 and the base layer data B1, respectively. The base layer data B0 and the base layer data B1 are respectively the result (Ygif(x,y)) of guided filtering of the target data Log Y(y, x) and guided filtering of the base layer data B0.
Ygif(x,y)=amean(x,y)Y(x,y)+bmean(x,Y)  (28)

The output of the adder circuit250_a0is Dt0=Log Y(y, x)−B0, and the output of the adder circuit250_b0is D t1=B0−B1.

In some embodiments, the first-level detail extraction module205_auses a filter with a smaller size (i.e., the down-sampling rate N of equation (9) being larger), and the second-level detail extraction module205_buses a filter with a larger size (i.e., the down-sampling rate N of equation (9) being smaller). In such a design, the base layer data B0 is the low-medium frequency detail of the target data Log Y(y, x), the detail layer data Dt0 is the high-frequency detail of the target data Log Y(y, x), the base layer data B1 is the low-frequency detail of the target data Log Y(y, x), and the detail layer data Dt1 is the intermediate-frequency (IF) detail of the target data Log Y(y, x). In this way, the detail extraction module130can extract the details of the different frequency components of the target data Log Y(y, x).

For the embodiment ofFIG.1, the output of the detail extraction module130includes the base layer Binand the detail layer Dtin. The base layer Binincludes at least one of the base layer data B0 and the base layer data B1 (which is/are used as the input of the base layer mapping circuit140), and the detail layer Dtinincludes the detail layer data Dt0 and the detail layer data Dt1. For the embodiment ofFIG.2, the output of the detail extraction module130is the detail layer Dtin, which includes the detail layer data Dt0 and the detail layer data Dt1.

FIG.6is a functional block diagram of the detail extraction module130according to another embodiment. The detail extraction module130includes a first-level detail extraction module205_aand a second-level detail extraction module205_b. The first-level detail extraction module205_aincludes a detail layer extraction circuit205_a0and a detail layer extraction circuit205_a1. The second-level detail extraction module205_bincludes a detail layer extraction circuit205_b0and a detail layer extraction circuit205_b1. The detail layer extraction circuit205_a0processes the target data Log Y(y, x) to generate the base layer data B01 and the detail layer data Dt01. The detail layer extraction circuit205_a1processes the target data Log Y(y, x) to generate the base layer data B02 and the detail layer data Dt02. The detail layer extraction circuit205_b0processes the base layer data B01 or the base layer data B02 to generate the base layer data B11 and the detail layer data Dt11. The detail layer extraction circuit205_b1processes the base layer data B01 or the base layer data B02 to generate the base layer data B12 and the detail layer data Dt12.

The structure and operating principle of the detail layer extraction circuit205_a0and the detail layer extraction circuit205_b0inFIG.6are respectively the same as those of the detail layer extraction circuit205_a0and the detail layer extraction circuit205_b0inFIG.5. The detail layer extraction circuit205_a1includes a guided filter factor calculation circuit210_a1, a buffer circuit220_a1, an interpolation circuit230_a1, a filter circuit240_a1, and an adder circuit250_a1. The detail layer extraction circuit205_b1includes a guided filter factor calculation circuit210_b1, a buffer circuit220_b1, an interpolation circuit230_b1, a filter circuit240_b1, and an adder circuit250_b1. The operating principles of the detail layer extraction circuit205_a1and the detail layer extraction circuit205_b1are respectively the same as those of the detail layer extraction circuit205_a0and the detail layer extraction circuit205_b0and thus omitted herein for brevity. Likewise, in some embodiments, the buffer circuit220_a1and the buffer circuit220_b1may be omitted.

Similarly, in some embodiments, the sizes (or the down-sampling rate N) of the filters of both the detail layer extraction circuit205_a0and the detail layer extraction circuit205_a1are smaller (or greater) than the sizes (or the down-sampling rate N) of the filters of both detail layer extraction circuit205_b0and detail layer extraction circuit205_b1. In such a design, the base layer data B01 and the base layer data B02 are the IF and low-frequency details of the target data Log Y(y, x), the detail layer data Dt01 and the detail layer data Dt02 are the high-frequency details of the target data Log Y(y, x), the base layer data B11 and the base layer data B12 are the low-frequency details of the target data Log Y(y, x), and the detail layer data Dt11 and the detail layer data Dt12 are the IF details of the target data Log Y(y, x). In this way, the detail extraction module130can extract the details of the different frequency components of the target data Log Y(y, x).

In some embodiments, the size (or the down-sampling rate N) of the filter of the detail layer extraction circuit205_a0is different from the size of the filter of the detail layer extraction circuit205_a1; the size (or the down-sampling rate N) of the filter of the detail layer extraction circuit205_b0is different from the size of the filter of the detail layer extraction circuit205_b1.

For the embodiment ofFIG.1, the detail extraction module130outputs the base layer Binand the detail layer Dtin. The base layer Binincludes at least one of the base layer data B01, the base layer data B02, the base layer data B11, and the base layer data B12 (which is/are used as the input of the base layer mapping circuit140), and the detail layer Dtinincludes the detail layer data Dt01, the detail layer data Dt02, the detail layer data Dt11, and the detail layer data Dt12. For the embodiment ofFIG.2, the output of the detail extraction module130is the detail layer Dtin, which includes the detail layer data Dt01, the detail layer data Dt02, the detail layer data Dt11, and the detail layer data Dt12.

In some embodiments, the detail layer extraction circuit205_a1ofFIG.6can be omitted.

In other embodiments, the detail layer extraction circuit205_b1ofFIG.6can be omitted.

Based on the embodiments ofFIG.5andFIG.6, people having ordinary skill in the art can scale up the levels of the detail extraction module130to more than three, and each level includes at least one detail layer extraction circuit.

The base layer mapping circuit140generates the converted base layer data (i.e., LogYglobal) based on equation (29).
LogYglobal=LUTglobal(X)  (29)

For the embodiment ofFIG.1, the input “X” is one of the base layer data B0 and the base layer data B1 (corresponding to the embodiment ofFIG.5), or one of the base layer data B01, the base layer data B02, the base layer data B11, and the base layer data B12 (corresponding to the embodiment ofFIG.6). For the embodiment ofFIG.2, the input “X” is the target data Log Y(y, x). The table lookup operation LUTglobal( ) may correspond to the curve inFIG.3.

FIG.7andFIG.8each show the functional block diagram of the detail layer enhancement module150according to different embodiments.FIG.7corresponds to the embodiment ofFIG.5, andFIG.8corresponds to the embodiment ofFIG.6. In the embodiment ofFIG.7, the detail layer enhancement module150includes a detail layer enhancement circuit150_aand a detail layer enhancement circuit150_b. In the embodiment ofFIG.8, the detail layer enhancement module150further includes a detail layer enhancement circuit150_cand a detail layer enhancement circuit150_d. The detail layer enhancement circuits150a,150_b,150c, and150_doutput the converted detail layer data based on equation (5). As shown inFIG.7andFIG.8, LUTd1(Dt0), LUTd1(Dt1), LUTd1(Dt01), LUTd1(Dt02), LUTd1(Dt11), and LUTd1(Dt12) correspond to Dt0, Dt1, Dt01, Dt02, Dt11, and Dt12, respectively.

For the embodiments ofFIG.5andFIG.7, the output data Log YTMinFIG.1andFIG.2is Log Yglobal+LUTd1(Dt0)+LUTd1(Dt1). For the embodiments ofFIG.6andFIG.8, the output data Log YTMinFIG.1andFIG.2is Log Yglobal+LUTd1(Dt01)+LUTd1(Dt02)+LUTd1(Dt11)+LUTd1(Dt12).

To sum up, the image processing circuit100of the present invention enhances the detail layers of different frequencies of the image separately, so that a better image contrast enhancement effect can be achieved with lower hardware complexity.

In addition to the aforementioned image processing circuit, the present invention also discloses a corresponding image processing method, which can be applied to an image processing device. The method is performed by the above-discussed image processing circuit100, image processing circuit200, or their equivalents.FIG.9andFIG.10are the flowcharts of one embodiment of the method, including the following steps.

Step S810: Converting the input data Dininto the luminance data Y(y, x). This step corresponds to the pre-processing module110and equation (1) or equation (2).

Step S820: Performing logarithmic conversion on the luminance data Y(y, x) to obtain the target data Log Y(y, x). This step corresponds to the logarithmic conversion module120and equation (3) or (4).

Step S830: Generating multiple base layer data (e.g., the base layer data B0 and base layer data B1 inFIG.5, or the base layer data B01, base layer data B02, base layer data B11, and base layer data B12 inFIG.6) and multiple detail layer data (e.g., the detail layer data Dt0 and detail layer data Dt1 inFIG.5, or the detail layer data Dt01, detail layer data Dt02, detail layer data Dt11, and detail layer data Dt12 inFIG.6) according to the target data Log Y(y, x) and multi-level guided filtering operations. This step corresponds to the detail extraction module130and equations (9) to (28).

Step S840: Converting the target data LogY(y, x) or one of the base layer data to obtain a converted base layer data. This step corresponds to the base layer mapping circuit140and equation (29).

Step S850: Converting the detail layer data to generate multiple converted detail layer data. This step corresponds to the detail layer enhancement module150and equation (5).

Step S860: Summing up the converted base layer data and the converted detail layer data to obtain the output data Log YTM. This step corresponds to the adder circuit160.

Step S870: Performing inverse logarithmic conversion on the output data Log YTMto obtain the output luminance data YTM. This step corresponds to the inverse logarithmic conversion module170and equation (6).

Step S880: Generating the image data Doutaccording to the luminance data Y(y, x), the output luminance data YTM, and the input data Din. This step corresponds to the post-processing module180and equation (7) or (8).

FIG.10shows the sub-steps of step S830, including steps S910to S940.

Step S910: Performing first guided filtering on the target data Log Y(y, x) to generate the first base layer data (the base layer data B0 or base layer data B01). This step may correspond to the guided filter factor calculation circuit210_a0, the interpolation circuit230_a0, and the filter circuit240_a0of the detail layer extraction circuit205_a0inFIG.5orFIG.6.

Step S920: Generating the first detail layer data (the detail layer data Dt0 or detail layer data Dt01) according to the target data Log Y(y, x) and the first base layer data. This step may correspond to the adder circuit250_a0of the detail layer extraction circuit205_a0inFIG.5orFIG.6.

Step S930: Performing second guided filtering on the first base layer data to generate the second base layer data (the base layer data B1 or base layer data B11). This step may correspond to the guided filter factor calculation circuit210_b0, the interpolation circuit230_b0, and the filter circuit240_b0of the detail layer extraction circuit205_b0inFIG.5orFIG.6.

Step S940: Generating the second detail layer data (the detail layer data Dt1 or detail layer data Dt11) according to the first base layer data and the second base layer data. This step may correspond to the adder circuit250_b0of the detail layer extraction circuit205_b0inFIG.5orFIG.6.

Note that people having ordinary skill in the art can perform operations corresponding toFIG.6according to the steps ofFIG.10. For example, people having ordinary skill in the art can selectively (1) perform steps S910and S920again to implement the operations of the detail layer extraction circuit205_a1to generate the third base layer data (the base layer data B02) and the third detail layer data (the detail layer data Dt02), or (2) perform step S930and step S940again to implement the operations of the detail layer extraction circuit205_b1to generate the third base layer data (the base layer data B12) and the third detail layer data (the detail layer data Dt12), or (3) perform steps S910to S940again to implement the operations of the detail layer extraction circuit205_a1and the detail layer extraction circuit205_b1to generate the third base layer data (the base layer data B02), the fourth base layer data (the base layer data B12), the third detail layer data (the detail layer data Dt02), and the fourth detail layer data (the detail layer data Dt12).

Various functional components or blocks have been described herein. As appreciated by persons skilled in the art, in some embodiments, the functional blocks can preferably be implemented through circuits (either dedicated circuits, or general purpose circuits, which operate under the control of one or more processors and coded instructions), which typically comprise transistors or other circuit elements that are configured in such a way as to control the operation of the circuitry in accordance with the functions and operations described herein. As further appreciated by persons skilled in the art, the specific structure or interconnections of the circuit elements can typically be determined by a compiler, such as a register transfer language (RTL) compiler. RTL compilers operate upon scripts that closely resemble assembly language code, to compile the script into a form that is used for the layout or fabrication of the ultimate circuitry. Indeed, RTL is well known for its role and use in the facilitation of the design process of electronic and digital systems.

The aforementioned descriptions represent merely the preferred embodiments of the present invention, without any intention to limit the scope of the present invention thereto. Various equivalent changes, alterations, or modifications based on the claims of the present invention are all consequently viewed as being embraced by the scope of the present invention.