Image data processing method and recording apparatus allowing original image of broad dynamic range to be reproduced

An image data processing method transforms a picture signal having a broad dynamic range to another picture signal having a narrow dynamic range. The picture signal having the narrow dynamic range is subjected to inverse transform to thereby output a recovered signal having the broad dynamic range. A difference between the original picture signal and the recovered signal, both having the broad dynamic range, is produced and then stored in a recording medium as difference data. A parameter for software processing to be executed for reproduction of the original picture signal later is also stored in the recording medium. In this way, the method can transform the original data with the broad dynamic range to the data of a standard file format with the narrow dynamic range, record the transformed data, and allow the original data to be recovered and effectively used later, as desired.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a method of processing image data and an apparatus for recording image data advantageously applicable to a digital still camera or the like for forming image data and recording the image data.

2. Description of the Background Art

Conventional solid-state image pickup devices, e.g., CCD (Charge Coupled Device) image sensors had a dynamic range far narrower than a latitude particular to silver halide photo-sensitive type of films and other traditional photosensitive materials. Today, a solid-state image pickup device with a broad dynamic range is available because of the rapid progress of semiconductor fabrication technology and that of broad dynamic range shooting technology, e.g., multiple exposure shooting.

Various schemes have been proposed for promoting the effective use of raw data having a broad dynamic range. Japanese patent laid-open publication No. 90380/1994, for example, discloses a procedure that compresses image data with a knee curve by analog processing and then expands the compressed image data with an inverse knee curve by digital processing to thereby obtain image data having a broad dynamic range.

To record image data representative of a scene picked up and having a broad dynamic range in the form of a digital picture signal, use is made of a defacto, standard file format, e.g., BMP (Bit MaP)), TIFF (Tag Image File Format) or JPEG (Joint Photographic Experts Group) format. Today's defacto, standard file format has a dynamic range whose unit data (pixel data in the case of a picture) consists of eight bits because an eight-bit dynamic range is a defacto, standard dynamic range adopted by the above typical formats. In the future, the standard file format may be replaced with another format having a different standard dynamic range.

At the present stage of development, for conversion to the standard file format, pixel data represented by an RGB (Red, Green and Blue) model or similar data model must be rendered by a preselected number of quantizing levels, i.e., eight-bit levels at the present stage. As for an RGB model, for example, an R, a G and a B channel constituting pixel data each are represented by eight-bit levels, e.g., 255 quantizing levels.

However, the problem with the conversion using a preselected number of quantizing levels is that the original image data with a broad dynamic range looses essential part thereof having high accuracy, i.e., represented with a great number of quantizing levels. The high-accuracy data lost cannot be recovered in the event of reproduction to be effected by software processing later. Particularly, software processing for enhancing the gain of the dark portion of a picture cannot be executed with accuracy above eight bits.

Assume that data with a great number of quantizing levels, e.g., twelve-bit data is recorded in order to effectively use the entire data having a broad dynamic range. Then, such data do not adapt to the standard file format described previously and cannot be transformed to the JPEG or similar compressed format. Moreover, the data with the great number of quantizing levels occupy a broad space in a memory.

SUMMARY OF THE INVENTION

It is an object of the present invention to provide an image data processing method and recording apparatus allowing original data of a broad dynamic range to be effectively used later, as desired.

In accordance with the present invention, a method of processing image data begins with a step of transforming broad-range image data having a broad dynamic range to narrow-range image data narrower in dynamic range than the broad-range image data. The narrow-range image data are inversely transformed to output inversely transformed image data having the same dynamic range as the broad-range image data. Difference data is then calculated representative of a difference between the broad-range image data and the inversely transformed image data. A file is generated that relates at least the difference data, information relating the difference data to the step of transforming and the narrow-range image data to one another.

Also, in accordance with the present invention, an image data recording apparatus includes at least one image processing circuitry and a storage. The image processing circuitry includes a transforming circuit for transforming input image data to output image data having a smaller number of quantizing levels than the input image data and feeding the output image data to another image processing circuitry. An inverse transforming circuit inversely transforms the output image data to thereby produce inversely transformed image data having the same dynamic range as the input image data. A calculating circuit produces difference data representative of a difference between the input image data and the output image data by calculation. The image processing circuitry therefore transforms broad-range image data having a broad dynamic range to narrow-range image data narrower in dynamic range than the broad-range image data. At least the narrow-range image data, difference data and information relating the difference data to the transforming circuit are recorded in the storage while being related to one another.

DESCRIPTION OF THE PREFERRED EMBODIMENT

Referring toFIG. 1of the drawings, an image data recording apparatus embodying the present invention is implemented as a digital still camera by way of example. InFIG. 1, part of the digital still camera not directly relevant to the understanding of the present invention is not shown. Signals are designated by reference numerals attached to signal lines on which they appear. As shown, the digital still camera, generally10, includes a CCD image sensor or solid-state image pickup device12having a broad dynamic range. The CCD image sensor12has an output15connected to the input of a CDS (Correlated Double Sampling) circuit14via a capacitor16. The CDS circuit14plays the role of a noise reducer for reducing reset noise output from the CCD image sensor12.

The CDS circuit14has an output19connected to a GCA (Gain Controlled Amplifier)18for gain control. The GCA18has an output21connected to an ADC (Analog-to-Digital Converter)20. The ADC20converts an analog image signal21, which is output from the GCA18, to a digital RGB image signal22. The digital RGB image signal22has a relatively great number of quantizing levels, i.e., twelve bits. The digital image signal22is input to a digital signal processor26.

The digital signal processor26executes various kinds of digital signal processing in order to transform the digital image signal22to a standard file format, as will be described specifically later. As a result, the digital image signal22is transformed to image data23having a smaller number of quantizing levels than the image signal22. In the illustrative embodiment, the number of quantizing levels of the image data23is assumed to be eight bits. Such transform executed by the digital signal processor26is generally classified into linear processing and nonlinear processing, as will be described later. Although the image data23mentioned above has an eight-bit dynamic range predominant in the state-of-the-art imaging technologies, the number of quantizing levels thereof can be freely selected in accordance with the possible future standard file format.

The digital signal processor26outputs difference data25in addition to the image data23. The difference data25is representative of a difference between the original or non-processed image signal and the processed image signal and has heretofore been lost due to transform of the processor26, as will be described more specifically later. The image data23and difference data25both are input to recording medium, e.g., an optical disk or a magnetic disk.

The camera10shown inFIG. 1may use any suitable solid-state image pickup device other than the CCD image sensor12, if desired. The illustrative embodiment is not limited to a digital still camera, but may be implemented as any other image data recording apparatus dealing with an image signal. The illustrative embodiment is therefore applicable to any system so long as it includes an image signal inputting means and a recording means and has a standard file format. For example, the illustrative embodiment is applicable even to a system not including a CCD image sensor or similar image pickup device. In that case, the system will record an original or non-processed image signal input thereto in an image data recording apparatus via a removable recording medium or some communicating means.

FIG. 2shows the digital signal processor26and recording medium24included in the illustrative embodiment more specifically. As shown, the digital signal processor26processes the input image signal22to thereby generate a complete image file30and a subfile32. The complete image file30and subfile32are written to the recording medium24independently of each other. The complete image file30is representative of the picture signal having the standard format and provided with eight-bit accuracy as conventional. The subfile32includes the difference between the complete image file30and the original image signal22.

It has been customary with digital signal processing to round off data having a great number of quantizing levels to data having a small number of quantizing levels by omitting the details of the data. By contrast, the illustrative embodiment records the difference data32heretofore discarded in the recording medium24together with the complete image file30, so that the original data22can be reproduced by software, not shown, later. This makes it possible to effectively use the broad dynamic range originally prepared with the CCD image sensor12.

Reference will be made toFIGS. 3A and 3Bfor describing the digital signal processor26in detail. As shown, the processor26includes N (1 or greater natural number) processing circuits, i.e., a first, a second, an (N−1)th and an Nth processing circuit40,42,44and46. The processor26further includes N inverse processing circuits, i.e., a first, a second, an (N−1)th and an Nth inverse processing circuit50,52,54and56associated with the processing circuits40,42,44and46, respectively. The processor26additionally includes N difference calculators, i.e., difference calculators60,62,64and66.

Let the configuration and operation of the processor26be described by taking the first processing circuit40, first inverse processing circuit50and difference calculator60by way of example. The picture signal22with the broad dynamic range is input to the first processing circuit40. The first processing circuit40processes the picture signal22by linear or nonlinear processing. The first inverse processing circuit50inversely transforms a processed picture signal70output from the first processing circuit40. The resulting output or inversely transformed signal72of the inverse processing circuit50and the original picture signal22are input to the difference calculator60. The difference calculator60produces a difference between the original picture signal22and the recovered picture72.

More specifically, the first processing circuit40discards information contained in the picture signal22and attained by taking advantage of the broad dynamic range thereof, typically details represented by the lower bits of the picture signal22. The picture signal70output from the first processing circuit40is input to both of the second processing circuit42and first inverse processing circuit50. Even though the inverse processing circuit50executes inverse transform with the picture signal70, it cannot recover part of the information discarded by the first processing circuit40. In the illustrative embodiment, the difference calculator60produces a difference between the original picture signal22and the recovered picture signal72restored by inverse transform and records the difference in the recording medium24as first difference data74. That is, the discarded data is recovered and stored in the recording medium24.

The difference calculator60subtracts the recovered signal72from the original picture signal22. This is because it is known beforehand that the recovered signal72output from the inverse processing circuit50contains a smaller amount of information than the original picture signal22.

Further, the difference calculator60attaches to the first difference data74a first parameter76representative of the source of the difference data74, whereby it is recognized that the difference data74is derived from the first processing circuit40. For example, the first parameter76may be “1”, which is an ordinal number assigned to the processing circuit40, or the parameter of processing executed by the processing circuit40. The parameter76therefore shows correspondence between the difference data74and the first processing circuit40. The parameter76is written to the recording medium24via a signal line25A together with the difference data74.

As stated above, the illustrative embodiment records not only the difference data74but also the parameter76meant for software processing that may be executed to reproduce the original picture signal later. The difference data74and parameter76stored in the recording medium24allow the original picture signal22with the broad dynamic range to be fully reproduced by software processing later, although not shown or described specifically. More specifically, the parameter76shows that the difference data74has been derived from the processing of the first processing circuit40. Therefore, it is understood that the picture signal22input to the processing circuit40can be fully restored by inversely transforming the picture signal70input to the second processing circuit42with the inverse processing circuit50and further by adding to the result of inverse transform the difference data74.

The second to Nth processing circuits42,44and46, like the first processing circuit40, respectively execute digital processing with input picture signals70,80,90and100. The difference calculators62,64and66output difference data84,94and104, respectively. The difference data84,94and104are also written to the recording medium24together with a second parameter to an Nth parameter86,96and106, respectively. In this manner, the difference data74,84,94and104and parameters76,86,96and106are recorded in the recording medium24as a subfile32independent of the complete image file30. The parameters86,96and106may also be ordinal numbers assigned to the second to Nth processing circuits42,44and46, respectively.

As stated above, the parameters76,86,96and106show correspondence between the processing circuits40,42,44and46and the difference data74,84,94and104, respectively. All the data stored in the subfile32may be sequentially applied to the image data, which are included in the complete image file30and having the standard format, in an order opposite to the order in which they are generated. This allows the original picture data22input to the first processing40to be fully reproduced, as desired.

The processing to be executed by each of the first to Nth processing circuits40,42,44and46will be described hereinafter. While the processing is either linear or nonlinear, linear processing will be described first with reference toFIG. 4.FIG. 4shows processing circuitry110a, corresponding to part110ofFIGS. 3A and 3Bsurrounded by a dotted line. As shown, the processing circuitry110aincludes a linear processing circuit40athat is assumed to be the first processing circuit40,FIG. 3A, by way of example. The linear processing circuit40areceives a twelve-bit picture signal22aand feeds a ten-bit data70ato a difference calculator60a. The original picture signal22ais input to the difference calculator60aas well.

Typical linear processing transforms a picture signal represented by a great number of quantizing levels to a picture signal represented by a small number of quantizing levels. The linear processing circuit40aexecutes such linear processing. For example, assume that the picture signal22ainput to the processing circuit40ahas twelve bits and should be transformed to the picture signal70ahaving ten bits. Then, twelve bits can render 4,095 levels while ten bits can render 1,023 levels, meaning that the number of levels is compressed to one-fourth. More specifically, as shown inFIG. 5, the picture signal22arepresented by fine levels (solid line) is transformed to the picture signal70ahaving rough levels (dotted line). That is, four consecutive levels of the original picture signal22aare rounded off to a single level of the transformed signal70a. As a result, the original picture signal22alooses part of its information representative of details that are previously presented in the lower bits of the broad dynamic range.

A specific operation of the linear processing circuit40a,FIG. 4, will be described hereinafter. Let binary notation be represented by bracket expression “[ ]2” for convenience. Assume that the original picture signal22ainput to the processing circuit40ais twelve-bit data [101000101101]2=2,605. Then, when the twelve-bit data is divided by 4 and rounded off, the resulting ten-bit data70ais [1010001011]2=651. Therefore, even if the ten-bit data70ais multiplied by 4 by inverse processing (not shown inFIG. 4), only data [101000101100]2=2,604 short of the original data 2,605 is recovered. However, difference data25a, which is [01]2, representative of a difference between the original data22aand the inversely processed data is available in the recording medium24. The original data22acan therefore be fully restored if [01]2is added to [101000101100]2.

As stated above, linear processing can produce a difference with an extremely simple configuration. The input22ato the processing circuit40ais the original twelve-bit data [101000101101]2. On the other hand, the output70afrom the processing circuit40ais ten-bit data [1010001011]2in the form without the lower two bits of the input22a. The difference calculator60asubtracts the data70afrom the data22awhile aligning the uppermost bits of the data22aand70a. In a strict sense, before the subtraction, the data70amust be subjected to processing corresponding to the inverse processing described previously, i.e., adding (logical) ZEROs to two lower bits to thereby output [101000101100]2. However, this extra step is not necessary because the difference calculator60aalways aligns the uppermost bits of the data22aand70aat the time of subtraction.FIG. 4shows such simplified circuitry. The difference25acan be obtained also by simply dividing the input data22ainto upper bits and lower bits, although not shown or described specifically.

Nonlinear processing will be described with reference toFIG. 6, which shows a more specific configuration of the circuitry shown inFIG. 1. InFIG. 6, an input picture signal120is assumed to have already been digitized by the ADC circuit20,FIG. 1. As shown, the image signal120is input to an OB (Optical Black) processor122. The OB processor122produces reference black data from the optical black portion of the image signal120. The resulting reference black data123is input to a WB (White Balance) processor124. The WB processor124controls white balance such that white included in a picture appears to be white to eye with natural color balance. The OB processor122and WB processor124do not reduce the number of bits of the picture signal120at all.

The output22aof the WB processor124is input to a 12/10 processor110athat executes the linear processing described with reference toFIG. 4. The output70aof the 12/10 processor110ais input to a γ corrector126that executes nonlinear processing. The 12/10 processor110adirectly writes difference data125in the recording medium24. The difference data125derived from the linear processing is apparently the lower bits simply omitted from 12-bit data22aas described with reference toFIG. 4. In this sense, the difference data125will be referred to as lower-bit data125hereinafter.

As for the nonlinear processing, the γ corrector126produces a 1/γ-power of the input picture signal70ain accordance with a value γ particular to a following display or similar receiver, not shown, that projects a picture. This successfully provides a picture appearing relatively dark on the receiver with sufficient lightness.

As shown inFIG. 8A, in the illustrative embodiment, while the signal70ainput to the γ corrector126has ten bits (1,023 levels), the signal128output from the gamma corrector126has eight bits (255 levels). A γ−1processor130is connected to the output128of the γ corrector126for inversely transforming the picture signal128corrected to the eight-bit data. The γ corrector126and γ−1processor130therefore cooperate to execute nonlinear processing in the same manner as linear processing as a whole.

The ten-bit data70aand data132output from the γ−1processor130are input to a difference calculator134. The difference calculator134subtracts the data132from the data70ato thereby output difference data136. In addition, the difference calculator134attaches a parameter138including the ordinal number of the γ corrector126to the difference data136.

A YC processor140is also connected to the output128of the γ corrector126. The YC processor140transforms the picture data128, which is in the form of RGB model, to the data23represented by a luminance component (Y) and color components (C). The YC image data23is recorded in the recording medium24in the form of the complete image file30having the standard format, e.g., JPEG format. As for the JPEG format, use is made of a reversible compression system that allows the original data to be restored by expansion.

The operation of the circuitry shown inFIG. 6will be described hereinafter. The WB processor124following the OB processor122outputs twelve-bit picture data22a. The 12/10 processor or linear processor110atransforms the picture data22ato ten-bit data70a. At the same time, the 12/10 processor110arecords the lower-bit data125representative of the lower two bits of the data22ain the recording medium24. The γ corrector126further reduces the number of quantizing levels of the ten-bit data70anonlinearly and outputs eight-bit data128. The γ−1processor130inversely transforms the eight-bit data128to thereby output data132, which is short of the original ten-bit data70aas in the case of linear processing. The difference calculator134therefore produces a difference between the data70ainput to the γ corrector126and the data132and records the difference in the recording medium24as difference data136.

The difference calculator134adds a parameter138including the ordinal number of the γ corrector126to the difference data136. The difference calculator134delivers the parameter138to the recording medium24via a signal line25b. Consequently, the lower-bit data125, difference data136and parameter138are stored in the recording medium24as a subfile32a.

The picture signal128subjected to γ correction is input to the YC processor140as well. The YC processor140records its output23in the recording medium24as a complete image file30having the standard format. Software processing, not shown or described, expands the complete image file30later, executes inverse YC processing with the expanded image file30, and sequentially applies the information of the subfile32ain the inverse way. This allows the picture signal70ainput to the γ correction126and therefore the original picture signal22ato be fully reproduced.

Another more specific configuration of the circuitry shown inFIGS. 3A and 3Bwill be described with reference toFIG. 7. InFIG. 7, structural elements identical with those shown inFIG. 6are designated by identical reference numerals; the WB processor124and blocks preceding it are not shown. The circuitry ofFIG. 7is identical with the circuitry ofFIG. 6up to the 12/10 processor110athat outputs the lower-bit data125. The circuitry ofFIG. 7does not include the γ−1processor130, but directly inputs the eight-bit data128output from the γ corrector126to the difference calculator134. Also, the ten-bit picture signal70aoutput from the 12/10 processor110ais input to a 10/8 processor150. The 10/8 processor150further reduces the ten-bit picture data70ato eight-bit picture data152and feeds the data152to the difference calculator134.

The significance of the circuitry shown inFIG. 7will be described with reference toFIGS. 8A and 8B. As shown inFIG. 8A, the corrected picture signal128has eight bits. Therefore, when a difference between the picture signal128and the ten-bit data not subjected to γ correction is produced, difference data occupies an area154at least of ten bits as shown inFIG. 8Aand therefore needs a substantial memory space for storage. In light of this, inFIG. 8B, the 10/8 processor150reduces the ten-bit data70ato the eight-bit data152. In this case, as shown inFIG. 8B, the difference data represented by an area156has eight bits or less and therefore reduces the space for storage.

In operation, the 12/10 processor110aoutputs ten-bit data70awhile recording lower-bit data125in the recording medium24. The γ corrector126nonlinearly transforms the ten-bit data70ato thereby output eight-bit data128, which is represented by a curve128inFIG. 8A. The 10/8 processor150linearly transforms the ten-bit data70ato thereby output eight-bit data152while recording the two lower bits cut off in the recording medium24as lower-bit data160. Parameters162include the ordinal numbers of the linear processor110aand γ processor126in order to show correspondence between the difference data125,160and156and the sources110a,150and126. Parameters162are also recorded in the recording medium24via signal lines25a,25cand25d.

By applying the subfile32bconsisting of the difference data and parameters stated above to software processing later, it is possible to fully reproduce the data70aand therefore the original data22a.

The present invention may, of course, execute any suitable nonlinear processing other than γ correction shown and described. While the illustrative embodiment records the files in the recording medium24mounted on the camera, they may be transferred to a recording medium remote from the camera by a wired or a wireless communication channel. In this sense, the recording medium24is not essential with the present invention.

In summary, an image data recording apparatus of the present invention is adaptive to a standard file format available for image data. Also, the apparatus of the present invention records a difference file cut off at a processing stage and a parameter representative of a source from which the difference file is derived and thereby allows data having a broad dynamic range to be fully reproduced by software later by using the parameter. That is, original image data can be effectively used.

Further, the difference data derived from linear processing makes data processing circuitry simple and has a small size feasible for a system. Moreover, when the number of bits is reduced by nonlinear processing, a difference between the processed data and data identical in the number of bits with the processed data, but derived from original data, is produced. This difference data has small size and saves a memory space.

The entire disclosure of Japanese patent application No. 2000-343456 filed on Nov. 10, 2000, including the specification, claims, accompanying drawings and abstract of the disclosure is incorporated herein by reference in its entirety.

While the present invention has been described with reference to the particular illustrative embodiment, it is not to be restricted by the embodiment. It is to be appreciated that those skilled in the art can change or modify the embodiment without departing from the scope and spirit of the present invention.