Patent ID: 12192645

DETAILED DESCRIPTION OF THE EXEMPLARY EMBODIMENTS

The present disclosure is more particularly described in the following examples that are intended as illustrative only since numerous modifications and variations therein will be apparent to those skilled in the art. Like numbers in the drawings indicate like components throughout the views. As used in the description herein and throughout the claims that follow, unless the context clearly dictates otherwise, the meaning of “a,” “an” and “the” includes plural reference, and the meaning of “in” includes “in” and “on.” Titles or subtitles can be used herein for the convenience of a reader, which shall have no influence on the scope of the present disclosure.

The terms used herein generally have their ordinary meanings in the art. In the case of conflict, the present document, including any definitions given herein, will prevail. The same thing can be expressed in more than one way. Alternative language and synonyms can be used for any term(s) discussed herein, and no special significance is to be placed upon whether a term is elaborated or discussed herein. A recital of one or more synonyms does not exclude the use of other synonyms. The use of examples anywhere in this specification including examples of any terms is illustrative only, and in no way limits the scope and meaning of the present disclosure or of any exemplified term. Likewise, the present disclosure is not limited to various embodiments given herein. Numbering terms such as “first,” “second” or “third” can be used to describe various components, signals or the like, which are for distinguishing one component/signal from another one only, and are not intended to, nor should be construed to impose any substantive limitations on the components, signals or the like.

The present disclosure is related to a method for exposure compensation applied to a high dynamic range (HDR) video and a circuit. When an HDR video is produced, an exposure compensation process is performed according to an exposure value ratio that is detected in real time. In the exposure compensation process, an image sensor and an image signal processor do not rely on a queue to conduct parameter synchronization for avoiding brightness discontinuity and abnormal composition that occur in an intersection region between frames.

Continuous images are converted to an HDR video through a high dynamic range process. The high dynamic range process uses at least two frames, in which each of the two frames is an image with a height “m” and a width “n”, to render an HDR image. The pixel value (P) of each pixels of the image can be expressed by Pi,j,kbased on a weight, a height, and a number of exposed frames, in which “i” indicates a position at an x-direction of the image, “j” indicates a position at a y-direction of the image, and “k” indicates the number of the exposed frames. For implementing the HDR video, two frames including a long exposure frame and a short exposure frame are incorporated. With an image captured by a static camera as an example, in addition to a moving region of the image, the remaining region is static. Accordingly, the pixel values of a previous frame and a next frame have a relationship of Pi,j,1=Pi,j,2*R, in which “Pi,j,1” indicates a pixel value of a first frame, “Pi,j,2” indicates a pixel value of a second frame, and “R” indicates an exposure value ratio between the first frame and the second frame. Further, according to one embodiment of the present disclosure, the exposure compensation process is performed in rows. For a first row of the image (i=1 to n), the exposure value ratio R can be obtained by comparing the sums of pixel values of rows of the two exposed frames. The exposure value ratio R can be calculated by equation 1.

R=∑i=1n⁢Pi,1,1∑j=1n⁢Pi,1,2.Equation⁢1

However, using the two frame exposure values to implement high dynamic range would cause two problems. One of which is that the content in a long exposure frame is different from the content in a short exposure frame due to the pixel pair in the previous frame and the next frame changes in a moving region. Therefore, the above-mentioned method does not apply to the video having the moving region. The other one of which is that an error may occur since the pixel values of the overly-dark or overly-bright region are too small and too close to an overexposure region. Accordingly, the method for exposure compensation applied to the HDR video is provided for eliminating the abovementioned problems.

A pair of pixel values formed by the previous and next frames can be expressed by “{Pi,j,1, Pi,j,2}.” A processor exposure value ratio RISPis configured to be inputted to the image signal processor in advance. Further, in a beginning of the method, a brightness range threshold is set for excluding overly-dark and overly-bright pixels, and a light-dark ratio threshold is set for excluding one or more potential moving regions from the video. An exemplary example of selecting a pixel range being used to render the HDR video is as follows.

A first frame (Pi,j,1) is configured to be a bright frame. An appropriate brightness range without overexposed region and/or overly-dark region is chosen according to a brightness distribution. For example, a brightness range [1024, 3072] is chosen from a 12-bit image.

A second frame (Pi,j,2) is configured to be a dark frame. A reasonable range [Pi,j,1/(RISP+ε), Pi,j,1/(RISP−ε)] of the second frame (Pi,j,2) can be obtained according to the pixel values of the first frame (Pi,j,1), the exposure value ratio (RISP) of the image signal processor, and a potential error ratio (ε) Next, in the previous frame and the next frame, only the pixel pair in the bright frame (i.e., first frame Pi,j,1) and the dark frame (i.e., second frame Pi,j,2) that are within the reasonable range are taken. The reasonable range indicates a range where the overly-bright and/or overly-dark pixels are excluded according to a brightness range threshold. Further, a light-dark ratio threshold can also be used for excluding one or more potential moving regions from the video.

Reference is made toFIG.4, which is a schematic diagram depicting a range having a valid pixel pair. In the diagram, pixel values of a long exposure frame and a short exposure frame form an x-y region. The pixel values of the long exposure frame and the short exposure frame are within a range enclosed by four restriction lines. In the exemplary example, the reasonable range is enclosed by a pixel value a, a pixel value b, an upper limit ratio RISP+ε, and a lower limit ratio RISP−ε, such that the reference valid pixel pair in the previous frame and the next frame can be obtained. The relevant description is as follows, and reference is made toFIG.5, which is a flowchart describing a process of obtaining the valid pixel pair in one embodiment of the present disclosure.

In the process, the pixel values (Pi,j,k) of the sequential frames in the video are obtained (step S501). The processor exposure value ratio (RISP) of the image signal processor is also obtained (step S503). Next, an instant exposure value ratio (R) of a current frame is calculated (step S505). The brightness range threshold is used to exclude the overly-dark and the overly-bright pixels in each of the frames (step S507). The light-dark ratio threshold is used to exclude the moving region in each of the frames (step S509).

According to one embodiment of the present disclosure, reference is made toFIG.4, in which a horizontal axis denotes a long exposure pixel value, and a vertical axis denotes a short exposure pixel value. An upper limit ratio401and a lower limit ratio403are set according to the processor exposure value ratio RISPof the image signal processor. The upper limit ratio401is such as “RISP+ε”, and the lower limit ratio403is such as “RISP−ε” of the present example. The long exposure pixel value, the short exposure pixel value, the upper limit ratio401, and the lower limit ratio403enclose a ratio range covering a reasonable ratio405. Reference can be further made to the above-mentioned brightness range indicative of an appropriate brightness range. The brightness range is a range enclosed by the pixel values (a) and the pixel values (b) after overexposed pixels and the overly-dark pixels are excluded. Therefore, the valid pixel pair within the reasonable ratio is chosen after the overexposed pixels or the overly-dark pixels are excluded (step S511).

Further, in the step S505of acquiring an instant exposure value ratio R, a calculation for acquiring the instant exposure value ratio for valid pixel pair in each of the frames is performed. Reference is made toFIG.6, which is a flowchart describing a process of calculating the exposure value ratio in one embodiment of the present disclosure.

In the beginning, a predetermined value of an exposure value ratio can be a processor exposure value ratio (RISP) of an image signal processor (step S601). Similarly, an instant exposure value ratio (R) of the current frame is obtained (step S603). In one embodiment of the present disclosure, in order to maintain continuity of the picture in the video, the exposure compensation can be performed in units of a row. According to a range having the valid pixel pair shown inFIG.4, rather than a sum of pixel values in a single row, a sum of pixel values of one or more rows until a current row in each of the frames is used for enhancing stability when multiple rows are present. The equation for calculating the instant exposure value ratio R is as follows.

In an exemplary example, when the exposure value ratio for each of the frames is calculated in units of a row of the frames, a previous row exposure value ratio (Rin) is firstly obtained (step S605). In addition to setting a first row exposure value ratio to be an initial processor exposure value ratio (RISP) inputted to the image signal processor, an instant exposure value ratio for each of the rows equals to a previous row exposure value ratio plus an adjustment value for each of the rows, that can be expressed by an equation “Rout=Rin+Rm”, in which “Rout” indicates the instant exposure value ratio, “Rin” indicates the previous row exposure value ratio, and “Rm” is the adjustment value for each of the rows. Accordingly, the instant exposure value ratio Routcan be obtained from the previous row exposure value ratio Rin. Therefore, the method can determine the adjustment value (Rm) in advance (step S607).

The adjustment value Rmcan be exemplarily determined by an equation 2.
Rm=0, if −E≤R−Rin≤E, in which “E” is a threshold for adjusting the ratio;  Equation 2:

Rm=Rm, if R−Rin>E;

Rm=−Rm, if R−Rin<−E.

Therefore, the above steps can be used to calculate an instant exposure value ratio (Rout) of a current row (step S609). The exposure value ratio for every row can be obtained by repeating the above steps (step S611).

In the process of determining the instant exposure value ratio Rout, if a frame is initially configured to be that the processor exposure value ratio RISPof the image signal processor is lower than a practical exposure value ratio Rsof the image sensor, the exposure value ratio (R1) of a first row of the frame is R1=RISP, and the exposure value ratio (R2) of a second row equals to the processor exposure value ratio (RISP) of the image signal processor plus an adjustment value (Rm), namely R2=RISP+Rm. The exposure value ratios of each of the rows in each frame can also be calculated. According to one embodiment of the present disclosure, a threshold for adjusting the exposure value ratio can be incorporated to converge the exposure value ratio of each of the frames, that is, to calculate the exposure value ratio of each of the rows until the exposure value ratio of a row j satisfies the relationship: Rj=RISP+j*Rm>R−E, in which “R” is an instant exposure value ratio, “E” is the threshold for adjusting the exposure value ratio. Afterwards, the exposure value ratio enters a stable region that can be a ratio range covering the reasonable ratio405shown inFIG.4, at this time, the exposure value ratio for each of the frames of the video is converged. Therefore, the continuity can be guaranteed when change of the exposure value ratio for each of the rows is fixed before the row j, and no abnormality is caused by incorrect exposure value ratio in the following frames because the exposure value ratio is converged.

Afterwards, the multiple converged exposure value ratios of multiple rows are summed up for obtaining the stable exposure value ratio for each frame (step S613).

According to the embodiments relating to the abovementioned chosen valid pixel pair and the exposure value ratio between the frames, the method for exposure compensation applied to a high dynamic range video can be implemented, and a circuitry applying the method can be referred to in a circuit block diagram shown inFIG.1. The circuit as shown in the diagram is configured to be disposed in an image-acquisition device. The circuitry includes a control circuit (such as a controller18ofFIG.1) that is used to control operations of an image signal processor and an image sensor of the image-acquisition device. In particular, the method adapted to the circuitry does not use queue control when processing calculation of exposure values in the image signal processor.FIG.7further shows an exemplary example of parameter control applied to the image signal processor for rendering the HDR video.

According to the embodiment of the present disclosure, the control circuit controls operations of the circuit components of the image-acquisition device. When the image-acquisition device is powered on, the lens is driven to capture continuous images sensed by the image sensor so as to generate image signals. The image signals are firstly stored in a memory. The image signal processor then processes the image signals. The circuitry embodies an HDR video exposure compensation circuit. The control circuit performs the method for exposure compensation applied to the HDR video upon the images.FIG.7schematically shows the continuous images represented by a zeroth frame (frame 0), a first frame (frame 1), a second frame (frame 2), a third frame (frame 3), a fourth frame (frame 4), and a fifth frame (frame 5) that are generated when the image-acquisition device captures images of a specific object. Since the image signal processor does not adapt the manner of queue control, and at least two frames are required to render the HDR video, the image sensor accordingly needs two frames to generate a new exposure value. Therefore, the image sensor parameters that are used to process the sequential frames will take effect after delaying two frames. For example, the zeroth frame (frame 0) image shown in top portion ofFIG.7is used to generate sensor parameters S0 that take effect at the third frame after delaying two frames. Similarly, the first frame image is used to generate sensor parameters S1 that take effect at the fourth frame, the second frame image is used to generate sensor parameters S2 that take effect at the fifth frame, the third frame is used to generate sensor parameters S3 that take effect at a sixth frame (not shown in this diagram), and the fourth frame is used to generate sensor parameters S4 that take effect at a seventh frame (not shown in this diagram).

According one further embodiment of the method for exposure compensation applied to a high dynamic range video, rather than the conventional method that uses a queue to synchronize the image sensor and the image signal processor, the image signal processor of the present disclosure conducts exposure compensation according to the instant exposure value ratio between a previous frame and a next frame. The relevant embodiment can be referred to in a flowchart shown inFIG.9.

In the method, an initial exposure value ratio is set as the processor exposure value ratio. A fixed adjustment ratio can be used to control both the image sensor and the image signal processor at the same time (step S901). The image data of continuous images can be retrieved from a video. The abovementioned embodiments are referred to so as to obtain a valid pixel pair with respect to each of the frames, and calculate an exposure value ratio in real time. Therefore, the valid pixel pair of each of the frames is acquired so as to calculate an exposure value ratio (step S903). In one of the embodiments of the present disclosure, a predetermined processor exposure value ratio (RISP) of the image signal processor is adapted as the exposure value ratio for the zeroth frame (frame 0). Next, the exposure value ratio between the previous frame and the next frame is referred to for obtaining the fixed adjustment ratio (Radj) for each of the frames. The fixed adjustment ratio Radjcan be used as a fixed ratio that is used to fine-tune the ratio of each of the frames. Therefore, every frame can be controlled to have the same error for acquiring the exposure value ratio for every frame. Since the continuity of the frames of the video is considered, an in-frame exposure value ratio can be fine-tuned in units of a row (step S905).

With using two frame exposure values to perform a high dynamic range process as an example, when the sequential frames are retrieved from the video a first frame exposure value ratio is used as the processor exposure value ratio (RISP) that multiplies the fixed adjustment ratio (Radj), and a second frame exposure value ratio is used as the processor exposure that multiplies twice the fixed adjustment ratio; correspondingly, an initial exposure value ratio that corresponds to the first frame exposure value ratio is set in the image sensor. Next, an error is adjusted to be the same as the fixed adjustment ratio (Radj). Therefore, the exposure value ratio corresponding to the second frame exposure value ratio set in the image sensor is a result of the initial exposure value ratio multiplying the fixed adjustment ratio.

For example, starting with a predetermined processor exposure value ratio (RISP), the fixed adjustment ratio Radjis used to adjust the exposure value ratio for each of the frames. The exposure value ratio of a first frame is RISP*Radj. The image sensor has an initial exposure value ratio Rs. An error can be adjusted to the fixed adjustment ratio Radj. The exposure value ratio of a second frame is RISP*Radj*Radj. The exposure value ratio applied to the image sensor is Rs*Radj, and the error is still a multiple of Radj. Afterwards, the error sustains until the convergence ends and then the error disappears since the number of error frames of the image signal processor and the image sensor are the same. If the ratio adjustment value for each frame is fixed, then the error for each frame can be fixed without using the processor queue, and the discontinuity in the intersection region during the convergence can be avoided.

Furthermore, a combination of the process of detecting the exposure value ratio and the process of high dynamic range compensation can compensate an error between the compensated picture and an expected exposure value ratio. In the embodiment of exposure compensation that the in-frame exposure value ratio is fine-tuned in units of a row, the speed of convergence and the rows to be affected are determined based on the adjustment value Rmfor each of the rows and the fixed adjustment ratio Radj.

Table 1 exemplarily shows outputs of the image sensor. The initial exposure value ratio (i.e., the processor exposure value ratio (RISP)) is 4 times. The adjustment value (Rm) is 3.125% (1/32). The fixed adjustment ratio (Radj) controls the frame requiring 4 rows to be converged progressively if the exposure value ratio of each of the frames raises 12.5% (1/8). According to Table 1, the exposure value ratio error (Ei) for each row in each frame changes until E5through the convergence process.

TABLE 1frameRISPRmRsE1E2E3E4E5. . .04.00.1254.01.01.01.01.01.014.50.1414.01.1251.0901.0541.0191.025.06250.1584.51.1251.0901.0541.0191.035.6950.1785.06251.1251.0901.0541.0191.0. . .

Table 1 shows a whole process of convergence of the exposure value ratio, in which the exposure value ratio error (Ei) for each of the rows is fixed, and the brightness of the pixels in each of the rows can be ensured to change stably during the convergence process when the given fixed adjustment ratio Radjis small enough and the adjustment value Rmfor each row is large enough. Therefore, the number of rows to be affected can be limited to a certain number.

Further, in one further embodiment of the present disclosure, reference is made toFIG.8, which is a schematic diagram depicting outputting an image that has a removable region. The image sensor outputs an output image80. In the method for exposure compensation applied to an HDR video, for using a larger output image having a height k+m, the image sensor can discard the data of several rows. The process in the method starts by being performed on the output image80having a removable region801with a height k from an initial processor exposure value ratio. The exposure value ratio is converged by an adjustment value Rmof each of the rows and a fixed adjustment ratio Radj. The removable region801with the height k in every frame can be removed after the convergence process is completed, and the output image region803with a height m is taken.

Thus, when the image outputted by the image sensor has a height m+k, the rows with erroneous exposure value ratio (i.e., the removable region801) are removed so as to form the output image with the height m after the convergence process. Therefore, the rows with the erroneous exposure value ratio, which are configured to be discarded, can be excluded from the process of convergence (step S907). In one further embodiment, the removable region801may not need to be removed but can be made less visible to the human eyes via other manners. Finally, the image being compensated with correct exposure value ratio can be obtained (step S909). After that, a series of sequential frames undergone the high dynamic range compensation are obtained, and an HDR video is outputted via an output circuit of the image-acquisition device (step S911).

It should be noted that, in the method for exposure compensation applied to a high dynamic range video, a stability of changes in the whole convergence process can be guaranteed when the fixed adjustment ratio Radjis controlled to be small enough so that the problem of continuity and abnormal composition can be prevented.

In summation, according to the abovementioned embodiments of the method and the circuitry for exposure compensation applied to a high dynamic range video, the exposure compensation method uses an exposure value ratio of a pixel pair in the frames of a video to perform compensation. Therefore, the image sensor and the image signal processor can take effect synchronously. The method does not use the queue that may cause discontinuity to the intersection region of the images in the conventional technology. In one embodiment (but not limited thereto), the exposure value ratio of the image sensor can be updated in units of a row in real time. The exposure compensation can be performed with an instant detected exposure value ratio for guaranteeing the continuity of the compensated exposure between the rows or the frames. Thus, rather than the queue being used for processing the HDR video, the method uses the proposed exposure value ratio to continuously process the HDR video for ensuring the stable convergence process regardless of the difference in the number of the effective frames between the image sensor and the image signal processor. The problems of abnormal composition and flickering phenomenon on the screen can be prevented when implementing the HDR video.

The foregoing description of the exemplary embodiments of the disclosure has been presented only for the purposes of illustration and description and is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Many modifications and variations are possible in light of the above teaching.

The embodiments were chosen and described in order to explain the principles of the disclosure and their practical application so as to enable others skilled in the art to utilize the disclosure and various embodiments and with various modifications as are suited to the particular use contemplated. Alternative embodiments will become apparent to those skilled in the art to which the present disclosure pertains without departing from its spirit and scope.