Patent Publication Number: US-10771785-B2

Title: Image encoding apparatus, method of controlling the same, and storage medium

Description:
BACKGROUND OF THE INVENTION 
     Field of the Invention 
     The present invention relates to an encoding technique for video. 
     Description of the Related Art 
     In an image capturing apparatus as typified by a digital video camera or the like, image data (RAW image) obtained by an image sensor is DeBayer processed (alternatively called demosaicing processing), is transformed into a signal comprising one piece of luminance component data and two pieces of color difference component data, and so-called development processing such as noise elimination, optical distortion correction, and image correction is performed for each signal. It is typical for the development-processed luminance signals and color difference signals to then be compressed and encoded and recorded on a recording medium. 
     On the other hand, there exist apparatuses capable of recording RAW images. Although the amount of data recorded is large, the RAW images are substantially unprocessed image data prior to development processing and are a form of recorded image data used preferably by high level users because of the advantage that high level editing can be performed later. 
     However, it is preferred that the RAW images described above be compressed in order to be recorded onto a limited recording medium because the data amount is large. However, depending on image capturing conditions, this compression may lead to a degradation of image quality. 
     In Japanese Patent Laid-Open No. 2007-243515, a configuration for encoding by adaptive quantization in which a quantization coefficient is changed according to a characteristic of visual perception is described. However, in a case where the quantization coefficient is changed with a characteristic of visual perception, problems such as the correction level being weakened or the code amount becoming uncontrollable in an image with many points of correction occur. 
     SUMMARY OF THE INVENTION 
     The present invention is made in view of the corresponding problems and provides a technique for generating encoded data for which degradation of image quality is suppressed even when performing a control of a code amount to a target code amount. 
     According to an aspect of the invention, there is provided an image encoding apparatus for encoding a video, the apparatus comprising: a separation unit configured to separate a plurality of planes respectively configured in a single component for a frame of an inputted video; a wavelet transform unit configured to perform a wavelet transformation of a plane of interest in the plurality of planes obtained by the separation; an extraction unit configured to, from each sub-band obtained by the wavelet transform unit, extract in order blocks representing the same region of an image; a quantizing unit configured to, using a quantization parameter, quantize wavelet transformation coefficients for each of the blocks; and an encoding unit configured to encode the wavelet transformation coefficients after the quantization by the quantizing unit, wherein the quantizing unit includes a determination unit configured to, for each of the blocks, determine a correction parameter for correcting the quantization parameter based on a direct current value and an alternating current value of the block, and quantizes wavelet transformation coefficients in accordance with the quantization parameter resulting from correction by the correction parameter determined by the determination unit. 
     By virtue of the present invention, it becomes possible to generate encoded data for which degradation of image quality is suppressed even when performing a control of a code amount to a target code amount. 
     Further features of the present invention will become apparent from the following description of exemplary embodiments (with reference to the attached drawings). 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a block configuration diagram of an image capturing apparatus to which an embodiment is applied. 
         FIG. 2  is a view illustrating an example of a Bayer array. 
         FIG. 3  is a block configuration diagram of a RAW compression unit in an embodiment. 
         FIG. 4  is a view illustrating plane-separation processing of a plane-separation unit. 
         FIG. 5  is a view showing a relationship between a sub-band relationship by wavelet transformation and a block to be encoded. 
         FIGS. 6A and 6B  are views illustrating an example of a transition of a code amount in a case where there is no correction parameter and in a case where there is a correction parameter. 
         FIGS. 7A and 7B  are views illustrating one example of a correction parameter table in an embodiment. 
         FIG. 8  is a view illustrating an example of a counter in an embodiment. 
         FIG. 9  is a flowchart illustrating compression encoding processing of a RAW image in an embodiment. 
         FIG. 10  is a flowchart illustrating details of encoding processing of a G 1  plane. 
     
    
    
     DESCRIPTION OF THE EMBODIMENTS 
     Hereinafter, embodiments according to the present invention are described in detail in accordance with the accompanying drawings. 
     First Embodiment 
       FIG. 1  is a block configuration diagram of an image capturing apparatus  100  that an image encoding apparatus of the first embodiment applies. 
     The image capturing apparatus  100  has a control unit  150  that controls the whole apparatus and an operation unit  151  that accepts instruction operations of a user. The control unit  150  is configured by a ROM that stores programs that a CPU executes and a RAM used as a work area, for example. Also, the image capturing apparatus  100  has an image capturing optical unit  101 , an image sensor unit  102 , a sensor signal processing unit  103 , a developing unit  104 , a display processing unit  105 , a display unit  106 , an external output terminal  107 , a RAW compression unit  109 , a RAW decompression unit  110 , a buffer  111 , and a recording unit  112 . 
     Hereinafter, description is given of processing at a time of recording in the above configuration. This processing is performed under the control by the control unit  150  in a case where recording is instructed from the operation unit  151 . 
     An optical image of a subject targeted for image capture forms on the image sensor unit  102  via the image capturing optical unit  101 . The image sensor unit  102  is configured by a large number of image capturing elements in a two-dimensional array on which red, green, and blue color filters are arranged on the front surface and generates an electrical signal in accordance with the intensity of light obtained by each element. Also, the image sensor unit  102  outputs the electrical signal of the image obtained by capturing at 30 frames/second. The array of color filters of the image sensor unit  102  in the embodiment is illustrated in  FIG. 2 . 2×2 pixels in this array are configured by two green (G 1  and G 2 ) pixels (reference numerals  1101   a  and  1101   b ), one blue (B) pixel (reference numeral  1102 ), and one red (R) pixel (reference numeral  1103 ). Also, these 2×2 pixels are arranged in regular intervals. Such an array is generally called a Bayer array. 
     The electrical signal obtained by the image sensor unit  102  is transformed into digital data by the sensor signal processing unit  103  and pixel restoration processing is performed. In the restoration processing, processing for interpolating a pixel to be restored using a value of surrounding pixels and subtracting a predetermined offset value with respect to a value of missing pixels or pixels with low reliability in the image sensor unit  102  is included. In the present embodiment, image information of the Bayer array outputted from the sensor signal processing unit  103  is referred to as a RAW image or RAW image data meaning an image that is not yet developed. 
     The developing unit  104  performs development processing on a RAW image from the sensor signal processing unit  103  or the RAW decompression unit  110  described later. Specifically, the developing unit  104  executes DeBayer processing (demosaicing processing) on the RAW image, performs a white balance adjustment, transforms one pixel to a signal comprising one luminance component and two chrominance components, eliminates noise and corrects optical distortion included in each signal, and performs so-called development processing such as optimizing the image. Also, the developing unit  104  supplies developed image data to the display processing unit  105 . 
     The display processing unit  105  resizes the image data from the developing unit  104  in accordance with the resolution of the display unit  106 , outputs it to the display unit  106 , then displays the image. The display processing unit  105  may output the development-processed image data to a display device connected externally by a video output terminal  107 . The video output terminal  107  is a general purpose interface such as HDMI (registered trademark) or SDI, for example. 
     The RAW compression unit  109  receives the RAW image from the sensor signal processing unit  103 , generates encoded data by high-efficiency encoding using a technique such as wavelet transformation or differential encoding, and stores this as encoded data of the RAW image data to the buffer  111 . The recording unit  112  stores (saves) the encoded data stored in the buffer as a RAW image file to a recording medium  113 . The recording medium  113  is a built-in large capacity memory or hard disk, or a recording region on an attachable/detachable type memory card or network, and the type does not particularly matter. 
     Hereinafter, description is given of processing at a time of playback in the above configuration. This processing is performed under control by the control unit  150  in a case where playback is instructed from the operation unit  151 . 
     When a playback operation is started, the recording unit  112  reads the file, which is encoded data of a RAW image that the user designated, from the recording medium  113  to the buffer  111 . Also, the RAW decompression unit  110  decompresses (decodes) the encoded data of the RAW image read to the buffer  111 , and supplies the RAW image, which is the result thereof, to the developing unit  104 . After that, similarly to during recording, the image that the RAW image obtained by decompressing represents is displayed on the display unit  106  or is displayed on a display apparatus (not shown) that is connected to the video output terminal  107 . 
     Next, a detailed description regarding configuration and processing of the RAW compression unit  109 , which is a feature in the embodiment, is given. 
       FIG. 3  is a block configuration diagram of the RAW compression unit  109  in an embodiment. The RAW compression unit  109  includes a plane-separation unit  301 , a DWT (wavelet transformation) unit  302 , a buffer  303 , a block-extraction unit  304 , a quantizing unit  305 , an entropy encoding unit  306 , a code-amount calculating unit  307 , a parameter processing unit  308 , and a code-amount control unit  309 . 
     The plane-separation unit  301  receives the RAW image from the sensor signal processing unit  103  and separates the RAW image into a plurality of planes. Specifically, as illustrated in  FIG. 4 , the plane-separation unit  301  separates the RAW image into a G 1  plane configured by only a G 1  component, a G 2  plane configured by only a G 2  component, an R plane configured by only an R component, and a B plane configured by only a B component. When the number of pixels in the horizontal direction and the number of pixels in the vertical direction of the RAW image are made to be W and H respectively, each of these four planes are the size of W/2 horizontal pixels and H/2 vertical pixels, and it can be said that each plane represents a monochrome multi-value image with one pixel/one component. Also, the plane-separation unit  301  supplies the G 1  plane, the G 2  plane, the R plane, and the B plane obtained by the separation processing to the DWT unit  302  in that order. Also, the processing from this point onward is encoding processing of a single plane, in other words a monochrome multi-value image, irrespective the type of the G 1  plane, the G 2  plane, the R plane, and the B plane. Also, a plane targeted for encoding is referred to as a plane of interest hereinafter. 
     The DWT unit  302  performs wavelet transformation on the plane of interest supplied from the plane-separation unit  301  (one of the G 1  plane, the G 2  plane, the R plane, or the B plane) as a monochrome multi-value image. Generally, although a wavelet transformation can recursively be applied to a sub-band LL obtained by a previous transformation, the wavelet transformation is described as being performed only once in order to simplify the description in the embodiment.  FIG. 5  illustrates a positional relationship of sub-bands LL, HL, LH, and HH obtained by one wavelet transformation. The DWT unit  302  stores in the buffer  303  the transformation coefficients within each sub-band obtained by performing the wavelet transformation. 
     The block-extraction unit  304  extracts from the sub-bands LL, HL, LH, and HH stored in the buffer  303  the blocks  501  through  504  configured by transformation coefficients of a number m×n (a plurality of) which is set in advance in a raster scan order that the arrow symbols of  FIG. 5  illustrate. Also, the block-extraction unit  304  supplies the extracted blocks  501  through  504  to the quantizing unit  305  and the parameter processing unit  308 . Note, as will be understood if one is skilled in the art, the blocks  501  through  504  represent a set of wavelet transform coefficients of an identical area of an image targeted for encoding but the types of sub-bands that they respectively belong to differ. Also, the blocks  501  through  504  to be encoded are referred to as blocks of interest hereinafter. 
     The quantizing unit  305  quantizes the transformation coefficients of a block of interest supplied from the block-extraction unit  304  by using a quantization step specified by a quantization parameter Q set from the code-amount control unit  309  and supplies the result to the entropy encoding unit  306 . 
     The entropy encoding unit  306  entropy encodes the quantization coefficients for the block of interest and outputs generated encoded data to the buffer  111  (refer to  FIG. 1 ). Also, the entropy encoding unit  306  supplies information indicating a code amount of the encoded data obtained from block of interest to the code-amount calculating unit  307 . Note, although the entropy encoding is assumed to be Golomb coding, the type of coding does not matter. 
     The code-amount calculating unit  307  calculates a difference between an accumulated code amount of encoded data generated until a block immediately preceding the block of interest and a target code amount for the immediately preceding block, and supplies the result to the code-amount control unit  309 . 
     The number of blocks included in one sub-band is N. Also, the target code amount of one plane set from the control unit  150  is represented as T. As a result, the target code amount per one block can be represented as “T/N”. Also, an i-th block of the plane of interest is represented as B i  and the code amount of the encoded data obtained by encoding the block B i  is represented as C(B i ). The difference E(i) of the code amount of the code generated from the block B i  and the target code amount T/N per one block can be represented as follows.
 
 E ( i )= C ( B   i )− T/N  
 
     A block of interest to be encoded currently is an i-th block B i . In other words, the encoding is assumed to be completed from a first block B i  to an immediately preceding block B i-1 . In such a case, the code-amount calculating unit  307  calculates value “ΣE(i−1)” (accumulated code amount difference hereinafter) accumulated (total) from the first block B i  difference E(1) to the immediately preceding block B i-1  difference E(i−1), and supplies this to the code-amount control unit  309 .
 
Σ E ( i− 1)=Σ{ C ( B   k )− T/N} 
 
or
 
Σ E ( i− 1)=Σ C ( B   k )−( i− 1)× T/N  
 
     (Where, k=1, 2, . . . , i−1) 
     The parameter processing unit  308 , based on the wavelet transformation coefficients of the block of interest Bi, calculates a correction parameter Px for correcting the quantization parameter for the block of interest and sets the parameter to the code-amount control unit  309  (detailed description is given later). 
     The code-amount control unit  309 , according to the following formula, determines the quantization parameter Q of the block of interest B i  and sets this to the quantizing unit  305 .
 
 Q=Q   ini   +r×ΣE ( i− 1)+ Px   (1)
 
     Here, Q ini  is a coefficient indicating an initial quantization parameter and r is a coefficient indicating a control sensitivity, and these are set together by the control unit  150 . A variation range of the quantization parameter Q increases as the sensitivity r increases, and control of the code amount becomes easy. However, because a difference in the quantization parameters between blocks tends to become large, this appears as a difference in image quality between blocks. 
     Here, cases in which there is no correction parameter Px of Formula (1) or in which Px=0 are considered. 
       FIG. 6A  is a graph illustrating a transition of a code amount in encoding processing of a plane of interest in such a case. The horizontal axis represents the number of blocks and the vertical axis represents a code amount. A straight line  601  represents y=i×T/N. The straight line  601  can also be said to be a straight line connecting the target code amount T of the plane at the position of the last block and the origin. On the other hand, a curved line  602  represents the accumulated code amount in a process of actually performing encoding processing. Specifically, it represents y=ΣC(B i ). 
     As shown in the figure, in a case where the accumulated code amount at a point in time immediately prior to block B i-1  exceeds the straight line  601 , the quantization parameter Q for the block of interest B i  is updated to a larger value according to Formula (1). As a result, the value of the transformation coefficients post-quantization in the block of interest B i  tends to become smaller, and as a result it is possible to control the code amount for the block of interest B i  to tend smaller. 
     Meanwhile, in a case where the accumulated code amount at a point in time immediately preceding block Bi−1 falls below the straight line  601 , the quantization parameter Q for the block of interest Bi is updated to a smaller value according to Formula (1). As a result, the value of the transformation coefficient post-quantization in the block of interest Bi tends to become larger, and as a result it is possible to control the code amount for the block of interest Bi to tend to increase. 
     By continuing to perform the above processing up to the final block, the total code amount of the plane of interest can be controlled so as to approximate the target code amount T. 
     However, the code amount control processing described above focuses only on the code amount obtained by encoding for each block, and the extent of degradation of image quality is not taken into consideration. In a case where a transformation coefficient obtained by the wavelet transformation is a relatively small value, the post-quantization value becomes 0 and information that the transformation coefficient originally had is lost. In particular, there is a tendency for a wavelet transformation coefficient obtained from a dark portion within an image to generally become a small value in any sub-band. As a result, the dark portion of the image obtained by decoding is expressed with less gradations than the original number of gradations, and the image quality ends up being bad. Accordingly, in the embodiment, the quantization parameter tends to become a small value in the dark portions in the image. 
     As previously described, whether the block of interest B i  belongs to a dark portion can be estimated from pre-quantized transformation coefficients obtained by the wavelet transformation. Accordingly, the parameter processing unit  308  in the embodiment calculates the correction parameter Px based on the blocks  501  through  504  representing the transformation coefficients of the block of interest B i  obtained by the wavelet transformation. In other words, a correction parameter Px is determined such that the stronger the extent to which the block of interest B i  belongs to a dark portion, the smaller the encoding parameter that the code-amount control unit  309  generates tend to become. 
     Specifically, the parameter processing unit  308  calculates average values LL ave , HL ave , LH ave , and HH ave  of the wavelet transformation coefficients in the blocks  501  through  504  of the block of interest B i . Out of these, the maximum values of HL ave , LH ave , and HH ave  are values representing a degree to which the frequency of the block of interest is high or low. Hereinafter, the value is called an AC value (or an alternating current value). Also, LL ave  is a value representing the extent of the brightness which is a direct current component of the block of interest. Hereinafter, this value is called the DC value (a direct current value).
 
 AC =max{ HL   ave   , LH   ave   , HH   ave }
 
 DC=LL   ave  
 
     (Here, max{ . . . } is a function that returns a maximum value within the parentheses) 
     Also, as in  FIG. 7A , the parameter processing unit  308  references the correction parameter table indicated by a two-dimensional coordinate space, selects one value stored in a coordinate {DC, AC} position obtained by the above processing, and supplies this to the code-amount control unit  309  as the correction parameter Px. 
     In the correction parameter table of  FIG. 7A , the horizontally rightward direction corresponds to brightness and the vertical downward direction corresponds to frequency. In the table shown in the figure, thresholds Th_ 01  and Th_ 02  are set for the brightness axis and thresholds Th_ 10  and Th_ 20  are set for the frequency axis. Also, the coordinate space of brightness and frequency is divided into 3×3 regions by these thresholds. Symbols within the parentheses in each region represent codes indicating positive/negative for the value of the correction parameter that the region represents, and the relationship of the correction parameters for the regions is as follows.
 
 P 00&lt; P 10&lt; P 01= P 20&lt; P 11=0&lt; P 21&lt; P 02&lt; P 12&lt; P 22
 
     In other words, the darker and lower frequency it is, the more the resulting value is less than zero; and the brighter and higher frequency it is, the more the resulting value is greater than zero (positive value). 
     Currently, a case in which the block of interest B i  belongs to a dark portion of a plane targeted for encoding is considered. In such a case, the DC value and the AC value of a block of interest will probably both be small for the reason previously described. As a result, the parameter processing unit  308  selects a negative value correction parameter from the table and sets it to the code-amount control unit  309  as the correction parameter Px. Accordingly, the quantization parameter Q used when encoding the block of interest B i  is corrected to a smaller value compared to a case where there is no correction parameter Px described previously. Also, the code-amount control unit  309  sets the post-correction quantization parameter in the quantizing unit  305 . As a result, it becomes possible to control such that information loss due to quantization becomes lower and so that it becomes possible to suppress the degradation of image quality of dark portions in particular. 
     Here, attention should be given to the fact that in previously described Formula (1), the correction parameter Px which is for not only code amount control but also suppressing degradation of image quality is newly added. It is possible to cause the total code amount of the plane of interest to be approximately the target code amount T as illustrated in  FIG. 6A  when simply performing only code amount control. On the other hand, when dominance of image degradation suppression of the correction parameter Px which is based on the above-described DC and AC components becomes larger than necessary, as illustrated in  FIG. 6B , it results in excessive correction, and there is the possibility that planes whose code amount is larger than the target code amount T will continue to be generated. 
     Accordingly, the parameter processing unit  308  of the embodiment determines the correction parameter Px so not to fall into such a continuous excessive correction. Hereinafter, an example in which this specific problem is solved is shown. 
     The parameter processing unit  308  has 3×3 counters C 00  through C 22  as illustrated in  FIG. 8 . The counters C 00  through C 22  correspond to the 3×3 correction parameters P 00  through P 22  in the correction parameter table illustrated in  FIG. 7A . Also, in a case where a target for encoding is the G 1  plane from among the four planes, the parameter processing unit  308  performs processing for incrementing the corresponding counter by “1” whenever one of the correction parameters P 00  through P 22  in the correction parameter table illustrated in  FIG. 7A  is selected. For example, the parameter processing unit  308  selects the correction parameter P 00 , and increases the counter C 00  by “1” in a case where the correction parameter P 00  is set in the code-amount control unit  309  as the correction parameter Px. As a result of the above, in a case where the encoding of the G 1  plane is completed, the usage frequency of the correction parameters P 00  through P 22  is known from the counters C 00  through C 22  after the G 1  plane is encoded. 
     Note that here, description regarding the G plane is continued and the G 2  plane, the R plane, and the B plane will be described later. 
     As can be understood from  FIG. 7A , because the negative value correction parameters are P 00 , P 10 , P 01 , and P 20 , the number of times C in which a negative value correction parameter Px is used in the encoding processing for the G 1  plane is the total of the counters C 00 , C 10 , C 01 , and C 20 .
 
 C=C 00+ C 10+ C 01+ C 20
 
     The parameter processing unit  308  has a predetermined storing and holding unit, and stores in the storing and holding unit a value that the above number of times C represents in a case where the encoding of the G 1  plane has completed. 
     Also, when the encoding of the G 1  plane has completed, the parameter processing unit  308  stores in the storing and holding unit a difference code amount ΣE(N) in relation to the target code amount T of the plane. The case where the difference code amount ΣE(N) is 0 indicates that the code amount of the G 1  plane is the same as the target code amount T. Also, the case where the difference code amount ΣE(N) is positive indicates that the code amount of the G 1  plane exceeds the target code amount T. The case where the difference code amount ΣE(N) is negative indicates that the code amount of the G 1  plane is less than the target code amount T. 
     Here, when encoding of the initial G 1  plane from among the four planes obtained from the frame of interest (RAW image) is started, the parameter processing unit  308  reads from the storing and holding unit the difference code amount ΣE(N) and the value C at a time of completion of the encoding of the G 1  plane of an immediately preceding frame (called the preceding G 1  plane or simply the preceding plane hereinafter). 
     Also, the parameter processing unit  308 , based on the difference code amount ΣE(N) and the value C, determines whether or not there was excessive correction in the encoding of the preceding G 1  plane, in other words, whether or not an increase in the code amount was introduced due to excessive use of negative correction parameter Px. 
     Specifically, the parameter processing unit  308 , by using positive thresholds Th_a and Th_b which are set in advance, determines that there was excessive correction in the encoding processing of the preceding G 1  plane in a case where the following condition is satisfied.
 
Condition: Σ E ( N )&gt;Th_ a  AND  C &gt;Th_ b  
 
     In a case where there was excessive correction for the preceding G plane, prior to the encoding of the G plane of interest, the parameter processing unit  308  updates the thresholds Th_ 01 , Th_ 02 , Th_ 10 , and Th_ 20  which define the correction parameter table of  FIG. 7A  in accordance with the following Formula (2).
 
Th_01←Th_01−Δ T 0
 
Th_02←Th_02−Δ T 0
 
Th_10←Th_10−Δ T 1
 
Th_20←Th_20−Δ T 1  (2)
 
     Here, ΔT 0  and ΔT 1  are positive values set in advance. 
     As a result of the above, the correction parameter table of  FIG. 7A  is updated to be the correction parameter table of  FIG. 7B , and in proportion to the logical area representing a negative correction parameter becoming smaller, a negative correction parameter becomes less likely to be selected. 
     Also, in a case where the G 1  plane of interest is to be encoded, the parameter processing unit  308 , based on the wavelet transformation coefficient received from the block-extraction unit  304 , calculates the AC value and the DC value, references the correction parameter table of  FIG. 7B  after the update, determines the correction parameter Px, and supplies it to the code-amount control unit  309 . 
     As a result of the above, even if the code amount exceeds the target code amount T due to excessive correction in the encoding of the preceding G 1  plane, it is possible to suppress the occurrence of the continuous excessive correction. 
     Note, in the above explanation, the thresholds for determining the correction parameter table are subtracted by ΔT 0  and ΔT 1  but as shown in the following Formulas (2′), the coefficients R 0  and R 1  (0&lt;R 0 , R 1 &lt;1) may be updated by multiplying the previous threshold.
 
Th_01←Th_01× R 0
 
Th_02←Th_02× R 0
 
Th_10←Th_10× R 1
 
Th_20←Th_20× R 1  (2′)
 
     Note that in a case where the difference code amount ΣE(N) of the preceding G 1  plane is negative and the thresholds of the table of the correction parameters are smaller than the initial values, configuration is such that the thresholds are increased so as to restore the original thresholds. The increase processing may be executed in accordance with the following calculations.
 
Th_01←Th_01+Δ T 0
 
Th_02←Th_02+Δ T 0
 
Th_10←Th_10+Δ T 1
 
Th_20←Th_20+Δ T 1  (3)
 
     Although the above is in regards to the G 1  plane generated from the plane of interest, the following is in regards to the correction parameters at a time of encoding processing for the G 2  plane, the R plane, and the B plane. 
     First, although it is the G 2  plane, since this G 2  plane is for the same color component as the G 1  plane, encoding is performed using the correction parameter table determined by the G 1  plane as it is. Regarding the R plane, the threshold values of the correction parameter table for the R plane are determined by multiplying the thresholds of the correction parameter table determined when encoding the G 1  plane of interest by a predetermined value. Regarding the B plane, similarly to the R plane, the thresholds of the correction parameter table for the B plane are determined by multiplying the thresholds of the correction parameter table determined when encoding the G 1  plane of interest by a predetermined value. 
     Based on the above, encoding processing of the RAW compression unit  109  in the embodiment is described in accordance with the flowchart of  FIG. 9 . 
     Firstly, in step S 101 , the control unit  150 , prior to recording a RAW image video, initializes (sets initial values of) the four thresholds Th_ 01 , Th_ 02 , Th_ 10 , and Th_ 20  of the correction parameter table for the G 1  plane to be used when encoding the initial frame for recording of a video. Also, the control unit  150  initializes the four thresholds of the correction parameter tables for the R plane and the B plane. Also, even though there is no frame preceding the initial frame, the difference code amount ΣE(N) for the preceding G 1  plane is set to “0” for convenience. 
     In step S 102 , the parameter processing unit  308  determines whether or not the thresholds Th_ 01 , Th_ 02 , Th_ 10 , and Th_ 20  for the correction parameter table for the G plane should be updated. In the case where at least one of Conditions 1 and 2 indicated below are satisfied, it is determined that there should be an update. 
     Condition 1: there was excessive correction for the preceding G 1  plane. 
     Condition 2: the total code amount for the preceding G 1  plane is less than the target code amount T (in other words, E(N)&lt;0), the thresholds of the correction parameter table for the G 1  plane are smaller than the initial setting values. 
     As described previously, since, in step S 102 , ΣE(N)=0, and neither of Conditions 1 and 2 are satisfied. As a result, upon encoding of the initial frame, the correction parameter table for which the initial setting was performed is used, and the processing branches to step S 105 . 
     Also, in the case where either of the above Condition 1 and Condition 2 is satisfied in the process of performing encoding of a frame after the first frame, the parameter processing unit  308 , in step S 103 , updates the thresholds of the correction parameter table for the G plane. 
     Specifically, in the case where Condition 1 is satisfied, the parameter processing unit  308 , in accordance with Formula (1) described previously, updates the threshold Th_ 01 , Th_ 02 , Th_ 10 , and Th_ 20  of the correction parameter table for the G 1  plane. 
     Also, in the case where Condition 2 is satisfied, the parameter processing unit  308 , in accordance with Formula (2) described previously, updates the threshold Th_ 01 , Th_ 02 , Th_ 10 , and Th_ 20  of the correction parameter table for the G 1  plane. 
     When the processing for updating the thresholds of the correction parameter table for the G 1  plane is finished, the parameter processing unit  308  advances the processing to step S 104 . In step S 104 , based on the thresholds of the correction parameter table for the G 1  plane after update, the threshold of the correction parameter tables for the R plane and the B plane are determined using a function or a table that is set in advance. Note that, as described previously, when the G 2  plane is encoded, update processing is unnecessary since the correction parameter table for the G 1  plane is used as is. 
     In step S 105 , the plane-separation unit  301  inputs the RAW image data which is the frame of interest from the sensor signal processing unit  103 , generates four planes—the G 1  plane, the G 2  plane, the R plane, and the B plane—from the RAW image data, and supplies them in that order to the DWT unit  302 . The result of this is that the encoding processing for the G 1  plane in step S 106  and the encoding processing for the G 2  plane, the R plane, and the B plane in step S 107  are performed. 
     Then, in step S 108 , the control unit  150  determines whether or not an instruction to end recording of the video is received via the operation unit  151  from the user. In the case where there is an end instruction, the control unit  150  ends the encoding processing by the RAW compression unit  109 . Also, in the case where there is no end instruction, the control unit  150  returns the processing to step S 102 , and continues the encoding processing of the next frame. 
     Here, description of the encoding processing for the G 1  plane in step S 106  is given in accordance with the flowchart of  FIG. 10 . 
     In step S 11 , the DWT unit  302  receives the G 1  plane supplied from the plane-separation unit  301 , and executes a wavelet transformation. Then, the DWT unit  302  stores in the buffer  303  the wavelet transformation coefficients for each sub-band obtained by the wavelet transformation. 
     In step S 112 , the block-extraction unit  304  extracts from the buffer  303  the block of interest B i ( 501  to  504  in  FIG. 5 ) that is the target of encoding, and supplies it to the parameter processing unit  308  and the quantizing unit  305 . 
     In step S 113 , the parameter processing unit  308  calculates the DC value and the AC value for the block of interest B i  supplied by the block-extraction unit  304 . Then, the parameter processing unit  308 , based on the calculated DC value and AC value, selects one of the correction parameters P 00  through P 22  in the correction parameter table (refer to  FIGS. 7A and 7B ), and supplies that to the code-amount control unit  309  as the correction parameter Px. At that time, the parameter processing unit  308  increases the corresponding one in the counters C 00  through C 22  indicated in  FIG. 8  by “1”. 
     In step S 114 , the code-amount control unit  309 , based on the correction parameter Px from the parameter processing unit  308  and the difference code amount ΣE(i−1) from the code-amount calculating unit  307  to the block immediately preceding the block of interest B i , determines the quantization parameter Q for the block of interest B i  (refer to Formula (1)). Then, the code-amount control unit  309  sets determined quantization parameter Q in the quantizing unit  305 . 
     In step S 115 , the quantizing unit  305  quantizes the wavelet transformation coefficients for the block of interest B i  supplied from the block-extraction unit  304 . Then, the quantizing unit  305  supplies to the entropy encoding unit  306  the wavelet transformation coefficients after quantization. 
     In step S 116 , the entropy encoding unit  306  encodes the transformation coefficients after quantization from the quantizing unit  305 , generates encoded data, and outputs it to the buffer  111 . At that time, the entropy encoding unit  306  supplies to the code-amount calculating unit  307  the code amount C(B i ) for the block of interest B i . 
     In step S 117 , the code-amount calculating unit  307  receives the code amount C(B i ) from the code-amount calculating unit  307 , and updates the difference code amount ΣE(i) from the first block to the block of interest B i  for the G 1  plane of interest. 
     Then, the control unit  150 , in step S 118 , determines whether or not the block of interest B i  is the last block in the G 1  plane of interest. In the case where the block of interest is not the last block, the control unit  150  returns the processing to step S 112  in order to encode the next block. On the other hand, in the case where the block of interest is the last block, the control unit  150 , in relation to the code-amount control unit  309 , saves in the storage unit the plane of interest difference code amount E(N) and the value C representing the number of times a negative correction parameter was selected, and finishes this processing. 
     The above is the encoding processing for the G 1  plane. The encoding for the G 2  plane, the R plane, and the B plane are basically the same as  FIG. 9  and so description thereof is omitted. 
     As described above, by virtue of the present embodiment, even if while suppressing degradation of gradation properties of dark areas in an image, the target code amount for a frame is locally exceeded, it becomes possible to cause the average code amount per frame to approximate the target code amount. 
     Note that in the foregoing embodiment, it was assumed that it is determined whether or not to perform an update of the thresholds for the correction parameter table for the G 1  plane, and the determination for the G 1  plane is followed for the R and B planes. However, while the G 2  plane may be the same as the G 1  plane, the R and B planes may also be determined to be the same as the G 1  plane. 
     Also, the correction parameter table in the foregoing embodiment was assumed to be of a configuration in which there are 3×3 regions where both the axial direction representing the direct current value and the axial direction representing the frequency are divided into three. However, there is no limitation to this number, and generally, it is possible to use a table having n×m correction parameters. 
     Note that in the foregoing embodiment, the wavelet transformation was described as something that is performed one time, but the number of times the wavelet transformation is performed does not particularly matter. For example, the wavelet transformation is assumed to be performed n times, for example. In such a case, the size of the block in the sub-band obtained by the n-th wavelet transformation is a size that is ½ n-1  both horizontally and vertically with respect to the first time. For the DC value, the transformation coefficients in the n-th sub-band LL may be obtained from the average value. Also, for the AC value, the sum of the sub-bands at each stage for which an appropriate weighting coefficient α is multiplied with the maximum value of the average value of HL, LH, and HH may be obtained. 
     Specifically, the average value of the transformation coefficients for the sub-bands HL, LH, and HH obtained in the j-th wavelet transformation are expressed as HL ave (j), LH ave (j), and HH ave (j), and when the multiplier coefficient for the j-th time is made to be α j , the AC value for the block of interest B i  is represented as in the following formula.
 
 AC=α   n ×max{ HL   ave ( n ), LH   ave ( n ), HH   ave ( n )}+α n-1 ×max{ HL   ave ( n− 1), LH   ave ( n− 1), HH   ave ( n− 1)}+ . . . +α 1 ×max{ HL   ave (1), LH   ave (1), HH   ave (1)}
 
     Also, in the case where wavelet transformations are performed n times, the parameter processing unit  308  may select the correction parameter based on the DC value and the AC value calculated as described above. 
     Second Embodiment 
     In above-described first embodiment, it was described that G 1 , G 2 , R, and B planes are separated from the RAW image, and each is encoded. Since it is sufficient that the RAW image can be reconfigured from the encoded data, there is no limitation to the above-described four planes. 
     For example, the plane-separation unit  301  may, in accordance with the following Formula (4), calculate one brightness component Y and three chrominance components C 1 , C 2 , C 3 .
 
 Y =( R+B+G 1+ G 2)/4
 
 C 1= R−G 1
 
 C 2= B−G 2
 
 C 3=( R+G 1)/2−( B+G 2)/2  (4)
 
     Also, the plane-separation unit  301  may generate a Y plane, a C 1  plane, a C 2  plane, and a C 3  plane which are configured as a single component. In such a case, a determination as to whether or not to perform processing to update the correction parameter table for the Y plane which represents brightness may be performed, and the C 1 , C 2 , and C 3  color difference planes may be updated so as to follow the brightness Y plane. The RAW decompression unit  110  may back calculate G 1 , G 2 , R, and B values from the Y, C 1 , C 2 , and C 3  planes which are the decoding results of the encoded data, and generate a RAW image in a Bayer array. 
     Third Embodiment 
     In the first embodiment, in the case where there is excessive correction in the encoding processing for the preceding G 1  plane, configuration is such that by making the thresholds of the correction parameter table smaller, negative correction parameter selection in the encoding processing for the G 1  plane of interest tends not to happen. However, the possibility that excessive correction will occur across a plurality of frames cannot be denied. When excessive correction occurs continuously, frames that exceed the target code amount T continue, and it means that there will be the possibility that an upper limit on the code amount that can be tolerated will be exceeded. 
     Accordingly, in the third embodiment, description is given of an example in which in the case where the number of frames over which excessive correction continues is M 0 , for example, from there on the thresholds Th_ 01 , Th_ 02 , Th_ 10 , and Th_ 20  of the correction parameter table are not updated, and the correction parameters P 00  through P 22  for the correction parameter table themselves are corrected. 
     As illustrated in  FIG. 7A , the relationship between the correction parameters P 00  through P 22  in the initial value settings is the relationship indicated below.
 
 P 00&lt; P 10&lt; P 01= P 20&lt; P 11=0&lt; P 21&lt; P 02&lt; P 12&lt; P 22
 
     In the case where excessive correction occurs and the number of times it continues is less than M 0  times, processing according to the first embodiment is performed. Then, in the case where excessive correction is continued for M 0  times, the control unit  150 , in relation to the parameter processing unit  308 , performs the following setting.
 
 P 00&lt; P 10&lt; P 01= P 20=0&lt; P 11&lt; P 21&lt; P 02&lt; P 12&lt; P 22
 
     Then, in the case where excessive correction is continued for another M 1  times, the control unit  150 , in relation to the parameter processing unit  308 , performs the following setting.
 
 P 00&lt; P 10=0&lt; P 01= P 20&lt; P 11&lt; P 21&lt; P 02&lt; P 12&lt; P 22
 
     In other words, the more that the number of times that excessive correction increases, the more the number of positive correction parameters increases from bright and high-frequency towards dark and low-frequency. In other words, the more that the number of times that excessive correction increases, the more the number of negative correction parameters is reduced from dark and low-frequency towards bright and high-frequency. 
     Also, in the case where the difference code amount ΣE(N) for the preceding G 1  plane is negative (less than the target code amount T), the control unit  150  performs processing to return to the initial relationship:
 
 P 00&lt; P 10&lt; P 01= P 20&lt; P 11=0&lt; P 21&lt; P 02&lt; P 12&lt; P 22.
 
     The result of the above is that by virtue of the third embodiment, it is possible to suppress an increase in the code amount caused by excessive correction. Note that the third embodiment is premised upon the first embodiment, but it may also be applied to the second embodiment. 
     OTHER EMBODIMENTS 
     Embodiment(s) of the present invention can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like. 
     While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions. 
     This application claims the benefit of Japanese Patent Application No. 2017-236294, filed Dec. 8, 2017, which is hereby incorporated by reference herein in its entirety.