Patent Description:
A raw moving image pickup apparatus that records a moving image called a raw moving image and a raw moving image reproduction apparatus that reproduces and edits a captured raw moving image are known. The raw moving image is data obtained from light received by a complementary metal-oxide semiconductor (CMOS) sensor or the like, and is a moving image obtained by recording data called raw data, which is yet to be subjected to image processing, as data of each frame. Unlike moving image data such as Moving Picture Experts Group (MPEG) data that has been subjected to image processing in advance, performing image processing on each frame of the raw moving image at the time of reproduction or editing enables editing with less degradation or image processing adapted to reproduction output, thereby enabling acquisition of higher image quality data. Meanwhile, since the raw moving image requires the image processing to be performed on each frame at the time of reproduction, the load due to setting the image processing and the load due to the image processing are heavy. Furthermore, since the raw moving image is composed of high image quality data, the file size thereof is large.

As the image processing with respect to a moving image, processing to reduce a change (e.g., brightness) between frames is commonly known. In a raw moving image, an optimum image processing parameter for each frame can be determined. In a case where an image capturing scene changes suddenly during capturing of the moving image and the optimum image processing parameter also changes suddenly, if the moving image is reproduced with a significant change between frames, the moving image may be displayed in an unnatural manner, for example, with flicker. In this regard, in a case where an amount of change in an image processing parameter between frames is too large, applying an image processing parameter obtained by recalculating the amount of change to make the change gentler can provide a display in which a sudden change is prevented. <CIT> proposes, in a case where moving images have a steep change at a combining section thereof at the time of combining the moving images, generating a parameter to make a smooth change, applying the parameter, and then combining the moving images.

US patent application publication no. <CIT> describes an image pick-up apparatus that has an image pick-up unit that picks up an image and outputs RAW still image data; a creation unit that creates, when the RAW still image data is picked up, a development parameter that is a parameter used in development processing of the RAW still image data; and a saving unit that saves, in a storage unit, the development parameter created when the RAW still image data is picked up as a first development parameter which can be used in development processing of the RAW still image data, and further saves, in the storage unit, a development parameter created at other time as a second development parameter which can be used in development processing of the RAW still image data, when saving the RAW still image data outputted from the image pick-up unit in the storage unit.

JP patent application publication no. <CIT> describes an imaging device which is provided with a parameter calculating part for calculating a parameter required for image processing in each frame when reproducing RAW image data on the basis of the RAW image data sequentially outputted from an imaging element in a frame unit, and a medium recording part for recording the calculated parameter corresponding to each frame as a parameter file different from the RAW image data as well as each RAW image data in a recording medium.

US patent application publication no. <CIT> describes an image capturing apparatus which is provided with an image capturing unit and a generating unit. The image capturing unit includes an imaging sensor and captures a subject image to generate RAW data. The generating unit generates a moving image file including moving image data made up of a plurality of sets of RAW data which is generated by the image capturing unit, and incidental information data corresponding to each of the sets of RAW data and including at least one of image capturing information regarding the RAW data and property information of the RAW data.

At the time of reproducing a raw moving image captured by a moving image pickup apparatus, image processing can be performed on each frame of the raw moving image using an image processing parameter obtained through change amount reduction processing. However, a process load is heavy at the time of reproducing the raw moving image, and calculating the image processing parameter further increases the processing load, thereby lowering responsivity. In the case of starting the reproduction from the middle of the raw moving image, if the change amount reduction processing is performed on a midway frame from which the reproduction is started and subsequent frames, the image processing parameter may be different between when the raw moving image is reproduced from the middle and when the raw moving image is reproduced from the beginning. Thus, there is an issue that the image processing parameter has to be calculated from an initial frame through the change amount reduction processing even in a case where the reproduction is started from the middle of the raw moving image, and thus an increased processing load further lowers responsivity. In addition, between the image processing parameter calculated through the change amount reduction processing and an image processing parameter that has not been subjected to the change amount reduction processing, a necessary image processing parameter is different depending on a type of reproduction or extraction of the raw moving images.

In view of the above-mentioned issue, the present invention provides a moving image pickup apparatus and an image processing apparatus to which an image processing parameter can be applied without losing responsivity and convenience.

In order to solve the aforementioned problems, one aspect of the present disclosure provides an image pickup apparatus as specified in claims <NUM> to <NUM>.

The present invention in its second aspect provides a control method as specified in claim <NUM>.

The present invention in its third and fourth aspects provides a computer program and a computer-readable storage medium as specified in claims <NUM> and <NUM>.

An exemplary embodiment of the present invention will be described in detail below with reference to the accompanying drawings. <FIG> is a block diagram illustrating an entire image processing system according to an exemplary embodiment of the present invention.

A typical exemplary embodiment of the present invention will be described.

<FIG> is a block diagram illustrating the entire image processing system according to the exemplary embodiment of the present invention. The image processing system includes a moving image pickup apparatus <NUM> and a moving image reproduction apparatus <NUM>. The moving image pickup apparatus <NUM> includes a correction amount calculation unit <NUM>, a change amount reduction processing unit <NUM>, a recording unit <NUM>, an output unit <NUM>, a control unit <NUM>, an image pickup unit <NUM>, and an operation unit <NUM>.

The control unit <NUM> is a control unit that controls the entire moving image pickup apparatus <NUM> and includes one or more processors. The control unit <NUM> reads a program from a program memory (not illustrated) and performs control of each unit and various types of arithmetic processing based on the program. The image pickup unit <NUM> includes, for example, an image pickup element that picks up an image. The operation unit <NUM> includes a button and a touch panel that accept operations from a user. The control unit <NUM> is notified of a user operation input to the operation unit <NUM>, and performs control to execute processing corresponding to the input to the operation unit <NUM>. The recording unit <NUM> records various types of data in a storage medium (not illustrated) such as a memory card. The output unit <NUM> outputs data recorded in the storage medium by the recording unit <NUM> or image data captured by the image pickup unit <NUM> to an external apparatus in a wired or wireless manner. The storage medium in which the recording unit <NUM> records data may be a storage medium built in the moving image pickup apparatus <NUM> instead of the memory card.

In the moving image pickup apparatus <NUM>, the control unit <NUM> starts raw moving image capturing processing in response to input of an instruction for starting capturing of the raw moving image to the operation unit <NUM> by the user. In the raw moving image capturing processing, an image processing unit (not illustrated) performs compression processing on a raw moving image picked up and acquired by the image pickup unit <NUM>, and the recording unit <NUM> records the raw moving image as a raw moving image file in the storage medium. The raw moving image capturing processing is performed on the raw moving image picked up by the image pickup unit <NUM> until an instruction for ending capturing of the raw moving image is input to the operation unit <NUM>. In other words, in the raw moving image capturing processing, the raw moving image captured by the image pickup unit <NUM> from the input of the instruction for starting the capturing of the raw moving image until the input of the instruction for ending the capturing of the raw moving image is recorded as one raw moving image file in the storage medium.

In the raw moving image capturing processing of the moving image pickup apparatus <NUM> according to the present embodiment, the recording unit <NUM> records, together with the raw moving image, an image processing parameter to be used in image processing at the time of reproducing the raw moving image in the storage medium. Thus, the moving image pickup apparatus <NUM> according to the present embodiment further includes the correction amount calculation unit <NUM> that calculates the image processing parameter and the change amount reduction processing unit <NUM>. Instead of providing the correction amount calculation unit <NUM> and the change amount reduction processing unit <NUM> aside from the control unit <NUM>, the control unit <NUM> may have functions of the correction amount calculation unit <NUM> and the change amount reduction processing unit <NUM>.

The correction amount calculation unit <NUM> calculates an ideal correction amount for each frame of the raw moving image. The change amount reduction processing unit <NUM> calculates a moving image correction amount using the ideal correction amount calculated by the correction amount calculation unit <NUM> to make a change amount between frames gentle. Here, the ideal correction amount is an optimum correction amount for a frame of the raw moving image. The moving image correction amount is a correction amount appropriate for reproducing a moving image in which a sudden change between frames is reduced. The ideal correction amount and the moving image correction amount calculated for each frame of the raw moving image are stored in the raw moving image file as information about each frame of the raw moving image and recorded in the storage medium by the recording unit <NUM>. Processing of calculating the ideal correction amount performed by the correction amount calculation unit <NUM> and processing of calculating the moving image correction amount performed by the change amount reduction processing unit <NUM> will be described in detail below.

While the image processing parameter (ideal correction amount and moving image correction amount) is recorded in the raw moving image file in the description above, the image processing parameter may be recorded in a file different from the raw moving image file. In such a case, desirably, association information indicating a correspondence relationship between the raw moving image file and the file of the image processing parameter is recorded, and in the case of reading the raw moving image file, the raw moving image file is read together with the file of the image processing parameter.

The moving image reproduction apparatus <NUM> includes an input unit <NUM>, a correction amount selection unit <NUM>, a reproduction output unit <NUM>, a control unit <NUM>, a storage unit <NUM>, an image processing unit <NUM>, and an operation unit <NUM>.

The input unit <NUM> communicates with an external apparatus in a wired or wireless manner and acquires data from the external apparatus. The control unit <NUM> is a control unit that controls the entire moving image reproduction apparatus <NUM>, and includes one or more processors. The control unit <NUM> reads a program from a program memory (not illustrated) and performs control of each unit and various types of arithmetic processing based on the program. The storage unit <NUM> is a storage unit that stores data acquired from the external apparatus via the input unit <NUM>. The image processing unit <NUM> performs various types of image processing on image data (raw moving image) stored in the storage unit <NUM>. The image processing performed by the image processing unit <NUM> includes development processing of the raw moving image and image correction processing. The operation unit <NUM> is an operation unit that accepts an operation by the user, and accepts a user operation via a button, a keyboard, a touch panel, and a mouse. The control unit <NUM> is notified of the user operation input to the operation unit <NUM>, and performs control to execute processing corresponding to the input to the operation unit <NUM>. The reproduction output unit <NUM> outputs the raw moving image to a display apparatus to reproduce the raw moving image on the display apparatus.

Basic processing executed by the moving image reproduction apparatus <NUM> will be described. First, the moving image reproduction apparatus <NUM> acquires the raw moving image file recorded by the moving image pickup apparatus <NUM> via the input unit <NUM> and stores the raw moving image file in the storage unit <NUM>. Alternatively, the moving image reproduction apparatus <NUM> may acquire the raw moving image file not by communicating with the moving image pickup apparatus <NUM> but by reading the raw moving image file from the storage medium in which the raw moving image file is recorded and store the raw moving image file in the storage unit <NUM>. Here, the ideal correction amount and the moving image correction amount for each frame are recorded as the image processing parameter in the raw moving image file acquired from the moving image pickup apparatus <NUM>, as described above. In a case where reproduction of the raw moving image stored in the storage unit <NUM> is instructed by the user via the operation unit <NUM>, the control unit <NUM> reads the raw moving image file as a process target from the storage unit <NUM>. The image processing unit <NUM> performs image processing on each frame of the raw moving image in the raw moving image file based on the image processing parameter included in the raw moving image file that is a reproduction target. Then, the reproduction output unit <NUM> reproduces and outputs the raw moving image subjected to the image processing performed by the image processing unit <NUM> to be displayed on the display apparatus. At the time of reproduction of the raw moving image, the user can instruct the start of the reproduction not only from the initial frame of the raw moving image but also from any frame of the raw moving image selected by the user through a user operation on the operation unit <NUM>. The user can also instruct a pause during the reproduction of the moving images, or instruct the reproduction of only the selected frame.

In the present exemplary embodiment, the image processing parameter includes two parameters, i.e., the ideal correction amount and the moving image correction amount. Thus, the moving image reproduction apparatus <NUM> according to the present exemplary embodiment further includes the correction amount selection unit <NUM> that determines which correction amount to use, the ideal correction amount or the moving image correction amount, based on a user operation or an output mode. Instead of providing the correction amount selection unit <NUM> aside from the control unit <NUM>, the control unit <NUM> may have a function of the correction amount selection unit <NUM>. The image processing unit <NUM> performs image processing using the correction amount selected by the correction amount selection unit <NUM>, and the reproduction output unit <NUM> reproduces and outputs an image subjected to the image processing. In the case of performing extraction processing on the raw moving image, the correction amount selection unit <NUM> selects which correction amount to use from among the ideal correction amount, the moving image correction amount, and a correction amount obtained by recalculating the moving image correction amount. Then, the image processing unit <NUM> performs image processing on an image based on the selected correction amount, and the storage unit <NUM> stores the image subjected to the image processing or stores the extracted raw image to which the selected correction amount is added.

The image processing parameter calculated for each frame may include various types of parameters used in the image processing that change during image capturing. Examples of such parameters include a correction amount of marginal illumination correction for correcting an optical aberration. The marginal illumination correction refers to a correction intended to maintain a light amount to be uniform by applying a reverse characteristic of a characteristic in which a light amount decreases from the optical center toward a periphery due to an optical factor of a lens or a sensor.

Subsequently, processing of calculating the ideal correction amount performed by the correction amount calculation unit <NUM> will be described with reference to <FIG> is a flowchart illustrating the processing of calculating the ideal correction amount performed by the correction amount calculation unit <NUM>. The processing of calculating the ideal correction amount is implemented by the correction amount calculation unit <NUM> reading a program from a program memory (not illustrated) and executing the program. The processing of calculating the ideal correction amount is performed in parallel with the raw moving image capturing processing and is performed with respect to all frames of the raw moving image picked up by the image pickup unit <NUM> and recorded in the raw moving image capturing processing. More specifically, the ideal correction amount is calculated with respect to each of all the frames of the raw moving image in the raw moving image file.

In step S201, the correction amount calculation unit <NUM> selects an initial frame captured in the raw moving image capturing processing as a process target frame. In the present exemplary embodiment, since the ideal correction amount is calculated in parallel with the capturing of the raw moving image, the ideal correction amount is sequentially calculated starting from the initial frame. In step S203, the correction amount calculation unit <NUM> acquires image capturing setting information regarding an image capturing condition at the time of capturing the processing target frame, i.e., a calculation target frame of which the ideal correction amount is calculated. Subsequently, in step S204, the correction amount calculation unit <NUM> acquires an image capturing parameter of the moving image pickup apparatus <NUM>. Then, in step S205, the correction amount calculation unit <NUM> calculates the ideal correction amount based on the image capturing setting information acquired in step S203 and the image capturing parameter acquired in step S204, and determines the calculated ideal correction amount as an ideal correction amount of the processing target frame. In step S206, upon completion of the calculation of the ideal correction amount of the processing target frame, the correction amount calculation unit <NUM> determines whether the calculation of the ideal correction amount has been completed with respect to all the frames of the raw moving image that is a recording target. In a case where the calculation of the ideal correction amount with respect to all the frames has been completed (YES in step S206), the processing ends. In a case where the calculation of the ideal correction amount with respect to all the frames has not been completed (NO in step S206), the processing proceeds to step S202. In step S202, the correction amount calculation unit <NUM> sets a next frame as the processing target frame. Then, the processing returns to step S203. The processing in steps S203 to S206 is repeated until the processing with respect to all the frames is completed. The ideal correction amount calculated and set to each frame by the correction amount calculation unit <NUM> is temporarily stored in a memory (not illustrated), and is used in subsequent processing of calculating the moving image correction amount and recording of the image processing parameter by the recording unit <NUM>.

In the calculation of the ideal correction amount, the correction amount calculation unit <NUM> acquires the following information. In step S203, in a case where calculation of the ideal correction amount for the marginal illumination correction is performed, the correction amount calculation unit <NUM> acquires image capturing setting information such as International Standards Organization (ISO) sensitivity, an object distance, a focal length, and an aperture value at the time of image capturing. In step S204, the correction amount calculation unit <NUM> acquires information regarding optical characteristics of the moving image pickup apparatus <NUM> such as characteristics of a complementary metal-oxide semiconductor (CMOS) sensor and characteristics of a lens in use. In step S205, the correction amount calculation unit <NUM> calculates the ideal correction amount for the marginal illumination correction from the acquired image capturing setting information and the information regarding the optical characteristics.

Subsequently, the processing of calculating the moving image correction amount performed by the change amount reduction processing unit <NUM> will be described with reference to <FIG> is a flowchart illustrating the processing of calculating the moving image correction amount performed by the change amount reduction processing unit <NUM>. The processing of calculating the moving image correction amount performed by the change amount reduction processing unit <NUM> is processing to calculate a correction amount with which a change amount between frames is reduced with respect to the ideal correction amount and to set the correction amount as the moving image correction amount. The processing of calculating the moving image correction amount is implemented by the change amount reduction processing unit <NUM> reading a program from the program memory (not illustrated) and executing the program. The processing of calculating the moving image correction amount is performed in parallel with the raw moving image capturing processing and is performed with respect to all the frames of the raw moving image picked up by the image pickup unit <NUM> and recorded in the raw moving image capturing processing. More specifically, the moving image correction amount is calculated with respect to each of all the frames of the raw moving image in the raw moving image file. In the present exemplary embodiment, since the ideal correction amount is calculated in parallel with the capturing of the raw moving image, the moving image correction amount is sequentially calculated starting from the initial frame. Since the ideal correction amount calculated in the processing of calculating the ideal correction amount is used in the processing of calculating the moving image correction amount, the processing of calculating the moving image correction amount is performed to determine the moving image correction amount after the processing of calculating the ideal correction amount with respect to the processing target frame is completed and the ideal correction amount is determined.

In step S301, the change amount reduction processing unit <NUM> selects the initial frame captured in the raw moving image capturing processing as the processing target (calculation target) frame. In a case where the initial frame is the processing target frame, there is no previous frame. Thus, in step S302, the change amount reduction processing unit <NUM> reads the ideal correction amount of the initial frame calculated by the correction amount calculation unit <NUM> from the memory (not illustrated), and sets the ideal correction amount of the initial frame (head frame) as the moving image correction amount of the initial frame. The change amount reduction processing unit <NUM> holds the moving image correction amount set at this time to be used in determining the moving image correction amount of the next frame. In step S303, upon completion of the processing of the initial frame, the change amount reduction processing unit <NUM> selects the next frame as the processing target frame. In step S304, with respect to the second or subsequent frame, the change amount reduction processing unit <NUM> reads and acquires the ideal correction amount set to the processing target frame by the correction amount calculation unit <NUM> from the memory. Subsequently, in step S305, the change amount reduction processing unit <NUM> calculates a difference between the moving image correction amount of the previous frame that has been held and the ideal correction amount of the processing target frame acquired in step S304, and determines whether the difference is a threshold (change amount reduction threshold) or more. More specifically, the change amount reduction processing unit <NUM> temporarily sets the ideal correction amount of the processing target frame as the moving image correction amount of the processing target frame, and calculates a difference in the moving image correction amount between the processing target frame and the previous frame as a difference in correction amount between frames. If the previous frame is the head frame, the ideal correction amount of the head frame is used as the moving image correction amount of the previous frame. In other cases, the moving image correction amount calculated with respect to the previous frame is used as the moving image correction amount of the previous frame. Since the moving image correction amount calculated with respect to the previous frame is used in the processing of calculating the moving image correction amount with respect to the next frame, the change amount reduction processing unit <NUM> holds the calculated moving image correction amount until completion of the processing of calculating the moving image correction amount with respect to the next frame.

In a case where the change amount reduction processing unit <NUM> determines that the difference in correction amount between the frames is the threshold (change amount reduction threshold) or more (YES in step S305), the processing proceeds to step S306. In step S306, the change amount reduction processing unit <NUM> calculates the moving image correction amount so as to decrease the difference in correction amount. Here, the change amount reduction processing unit <NUM> calculates a value by adding a half of the difference between the moving image correction amount of the previous frame and the ideal correction amount of the processing target frame to the moving image correction amount of the previous frame. While the change amount reduction processing unit <NUM> adds the half of the difference between the moving image correction amount of the previous frame and the ideal correction amount of the processing target frame to the moving image correction amount of the previous frame in the present exemplary embodiment, a value obtained by dividing the difference by a predetermined number (one or more), instead of two, may be used. In other words, the change amount reduction processing unit <NUM> is to decrease the difference to a predetermined ratio and add the decreased difference to the moving image correction amount of the previous frame, and a ratio to which the difference is decreased is not limited thereto. In the description below, the ratio is referred to as a gain coefficient, and the moving image correction amount of the processing target frame is calculated by the following formula.

In the present exemplary embodiment, the change amount reduction processing unit <NUM> performs the processing of comparing an amount of change between the correction amounts of the processing target frame and the previous frame, and recalculating the correction amount to decrease the difference. However, the processing is not limited thereto and only needs to be processing of obtaining the moving image correction amount based on inter-frame information. Subsequently, in step S307, the change amount reduction processing unit <NUM> sets the moving image correction amount calculated in step S306 as the moving image correction amount of the currently-selected processing target frame.

In a case where the change amount reduction processing unit <NUM> determines that the difference in correction amount is less than the threshold (change amount reduction threshold) (NO in step S305), the processing proceeds to step S308. In step S308, the change amount reduction processing unit <NUM> sets the ideal correction amount of the processing target frame acquired in step S304 as the moving image correction amount.

In step S309, after setting the moving image correction amount of the processing target frame, the change amount reduction processing unit <NUM> determines whether the processing of calculating the moving image correction amount (setting the moving image correction amount) has been completed with respect to all the frame of the raw moving image that is the recording target. In a case where the processing of calculating the moving image correction amount of all the frames has been completed (YES in step S309), the processing ends. In a case where the processing of calculating the moving image correction amount of all the frames has not been completed (NO in step S309), the processing returns to step S303. Then, the change amount reduction processing unit <NUM> sets the next frame as the processing target frame, and repeats the processing in steps S303 to S309 until completion of the processing with respect to all the frames. The moving image correction amount calculated and set to each frame by the change amount reduction processing unit <NUM> is temporarily stored in the memory (not illustrated), and is later used to record the image processing parameter by the recording unit <NUM>.

Here, examples of results of calculating moving image correction amounts in the change amount reduction processing unit <NUM> will be described with reference to <FIG>. In the example, there are five frames of the raw moving image from a frame <NUM>, serving as the head frame, to a frame <NUM>. The ideal correction amount of each frame is zero for the frame <NUM> and the frame <NUM> and is <NUM> for the frame <NUM> to the frame <NUM>. In this case, it is considered that the image capturing condition has been changed so that the ideal correction amount suddenly changes during a transition from the frame <NUM> to the frame <NUM>. Furthermore, in step S305, the change amount reduction threshold for determining whether to perform the change amount reduction processing is <NUM>, and the gain coefficient is <NUM>/<NUM>. First, in steps S301 and S302, the change amount reduction processing unit <NUM> reads the head frame <NUM> and sets the ideal correction amount of zero as it is as the moving image correction amount. Subsequently, the change amount reduction processing unit <NUM> reads the frame <NUM>. Since the moving image correction amount of the previous frame is zero, the ideal correction amount of the frame <NUM> is also zero, and the difference in correction amount between the frames is less than <NUM> (change amount reduction threshold), the moving image correction amount of the frame <NUM> is zero. Subsequently, the change amount reduction processing unit <NUM> reads the frame <NUM>. The moving image correction amount of the previous frame is zero, and the ideal correction amount of the frame <NUM> is <NUM>. Since the difference in correction amount between the frames is <NUM>, which is more than the change amount reduction threshold of <NUM>, the change amount reduction processing unit <NUM> calculates the moving image correction amount with which a change amount is reduced. Since the gain coefficient is <NUM>/<NUM>, adding <NUM>, which is obtained by multiplying the difference of <NUM> by <NUM>/<NUM>, to zero, which is the moving image correction amount of the previous frame, results in <NUM>. The amount of <NUM> is the moving image correction amount of the frame <NUM>. Similarly, the difference between the ideal correction amount (<NUM>) of the frame <NUM> and the moving image correction amount (<NUM>) of the frame <NUM> is <NUM>, which is more than the change amount reduction threshold of <NUM>. Thus, adding <NUM>, which is obtained by multiplying the difference of <NUM> between the frames by <NUM>/<NUM>, to <NUM>, which is the moving image correction amount of the frame <NUM>, results in <NUM>. The amount of <NUM> is the moving image correction amount of the frame <NUM>. The ideal correction amount of the frame <NUM> is <NUM>, and the difference between the ideal correction amount and the moving image correction amount of the previous frame <NUM> is <NUM>, which is less than the change amount reduction threshold of <NUM>. Thus, <NUM>, which is the same as the ideal correction amount of the frame <NUM>, is applied to the moving image correction amount of the frame <NUM>. As described above, in a case where the ideal correction amount of each frame has been determined, in the processing of calculating the moving image correction amount, the change amount reduction processing unit <NUM> can sequentially calculate the moving image correction amount from the head frame and calculate the moving image correction amount to be applied to each frame.

Subsequently, recording processing performed by the recording unit <NUM> will be described with reference to <FIG> is a flowchart illustrating the recording processing performed by the recording unit <NUM>. The recording processing is performed by the control unit <NUM> reading a program from the program memory and controlling the recording unit <NUM> based on the read program. The recording processing is part of the raw moving image capturing processing and starts to be performed in response to the start of the raw moving image capturing processing. In the recording processing, the recording unit <NUM> performs processing of recording the raw moving image captured in the raw moving image capturing processing as the raw moving image file in the storage medium.

First, in step S401, the recording unit <NUM> creates a new raw moving image file on the storage medium. Subsequently, the control unit <NUM> performs processing in steps S402 and S403 with respect to each frame of the raw moving image captured in the raw moving image capturing processing. In step S402, the control unit <NUM> acquires frames of the raw moving image captured by the image pickup unit <NUM>, and controls the recording unit <NUM> to record the frames of the raw moving image in the raw moving image file created in step S401. The frames of the raw moving image may have been subjected to compression processing or the like. Next, in step S403, the control unit <NUM> acquires the ideal correction amount set by the correction amount calculation unit <NUM> and the moving image correction amount set by the change amount reduction processing unit <NUM> with respect to each of the frames of the raw moving image recorded in step S402, and controls the recording unit <NUM> to record the ideal correction amount and the moving image correction amount in the raw moving image file as information regarding image processing parameter of each of the frames of the raw moving image recorded in step S402. In the present exemplary embodiment, the image processing parameter is recorded in a form of being included in metadata in a format of the raw moving image file. However, the image processing parameter (correction amount) may be recorded in a different file from the raw moving image file.

In step S404, the control unit <NUM> determines whether the processing in steps S402 and S403 has been completed with respect to all the frames of the raw moving image captured in the raw moving image capturing processing. In a case where the processing has been completed with respect to all the frames (YES in step S404), the processing proceeds to step S405. In a case where the processing has not been completed with respect to all the frames (NO in step S404), the processing returns to step S402. The control unit <NUM> performs the processing in steps S402 and S403 on the next frame, and repeats the processing in steps S402 to S404 until completion of the processing with respect to all the frames.

In a case where the recording processing has been completed with respect to all the frames (YES in step S404), the processing proceeds to step S405. In step S405, the recording unit <NUM> performs recording completion processing of the raw moving image file created in S401 and in which data of the frames is recorded in steps S402 to S404 and ends the processing of recording the raw moving image.

In this manner, the moving image pickup apparatus <NUM> according to the present exemplary embodiment adds the image processing parameter to each of the frames of the raw moving image at the time of capturing the raw moving image, and records the image processing parameter as the raw moving image file. Then, the moving image pickup apparatus <NUM> according to the present exemplary embodiment records, as the image processing parameter, not only the ideal correction amount based on the image capturing condition and the image capturing parameter, but also the moving image correction amount with which a change amount with respect to the ideal correction amount between frames is reduced. In other words, the moving image pickup apparatus <NUM> records two kinds of parameters in the same image processing. Calculating and recording the image processing parameter in advance at the time of capturing the raw moving image, which requires image processing imposing a heavy load at the time of reproduction, can reduce a load of calculating the image processing parameter at the time of reproduction. Furthermore, the image processing achieving higher image quality based on the ideal correction amount and the image processing in which a moving image can be smoothly reproduced based on the moving image correction amount can be selectively performed without performing recalculation of the image processing parameter at the time of reproduction of the raw moving image.

The description has been given of the exemplary embodiment in which the moving image pickup apparatus <NUM> individually performs the processing of calculating the ideal correction amount, the processing of calculating the moving image correction amount, and the recording processing. However, the moving image pickup apparatus <NUM> may collectively perform the processing of calculating the ideal correction amount, the processing of calculating the moving image correction amount, and the recording processing for each frame.

Subsequently, reproduction processing of the raw moving image file performed by the moving image reproduction apparatus <NUM> will be described with reference to <FIG>. In the moving image reproduction apparatus <NUM>, the raw moving image file is stored in the storage unit <NUM>, as described above. The reproduction processing is performed in response to acceptance of an instruction for reproducing the raw moving image stored in the storage unit <NUM> from the user via the operation unit <NUM>. The reproduction processing is implemented by the control unit <NUM> reading a program from the program memory and performing control of each unit and arithmetic processing based on the read program. First, in step S501, the control unit <NUM> determines whether a reproduction instruction provided by the user is not an instruction for continuously reproducing the moving image but an instruction for reproducing one frame. In a case where a pause instruction is provided during reproduction of the moving image, an image of one frame is reproduced by the pause instruction. Thus, the control unit <NUM> determines that the reproduction instruction is the instruction for reproducing one frame, and sets the frame to be reproduced at the time of a pause as a reproduction target frame. In a case where the control unit <NUM> determines that the reproduction instruction is the instruction for reproducing one frame (YES in step S501), the processing proceeds to step S502. In a case where the control unit <NUM> determines that the reproduction instruction is not the instruction for reproducing one frame (NO in step S501), the processing proceeds to step S504.

In step S502, the correction amount selection unit <NUM> selects the ideal correction amount as a correction amount to be used. In a case where the moving image reproduction apparatus <NUM> reproduces a raw image of one frame instead of continuously reproducing a plurality of frames of the raw moving image (i.e., reproducing the raw moving image), there is no need to reduce the change amount. Thus, the correction amount selection unit <NUM> selects the ideal correction amount to perform optimum image processing for the one frame that is the reproduction target. Then, in step S503, the image processing unit <NUM> performs development processing and image processing using the selected ideal correction amount on the frame of the raw moving image selected as the reproduction target, and causes the reproduction output unit <NUM> to reproduce and output the processed image. In other words, the frame selected as the reproduction target is reproduced and output after being subjected to the image processing using the ideal correction amount, and is displayed on the display apparatus.

In a case where the reproduction instruction is not the instruction for reproducing one frame (NO in step S501), the processing proceeds to step S504. In step S504, the control unit <NUM> determines whether the reproduction instruction is an instruction for reproducing the raw moving image. In a case where the control unit <NUM> determines that the reproduction instruction is the instruction for reproducing the moving image (YES in step S504), the processing proceeds to step S505. In a case where the control unit <NUM> determines that the reproduction instruction is not the instruction for reproducing the moving image (NO in step S504), the processing proceeds to step S507.

In step S505, the correction amount selection unit <NUM> selects the moving image correction amount as the correction amount to be used. In the case of continuously reproducing the plurality of frames (i.e., reproducing the raw moving image), a large amount of change in correction amount between the frames leads to unnatural display of the moving image. Thus, the correction amount selection unit <NUM> selects the moving image correction amount with which the amount of change in correction amount between the frames is reduced. In step S506, the image processing unit <NUM> sequentially performs the development processing and the image processing using the selected moving image correction amount on the frames of the raw moving image that is the reproduction target, and causes the reproduction output unit <NUM> to reproduce and output the processed moving image. The image processing unit <NUM> sequentially performs the image processing on each of the frames and causes the reproduction output unit <NUM> to reproduce and output the processed moving image until a final frame of the moving image is reproduced and output or until the instruction for pausing the reproduction is input by the user, and the image-processed moving image is reproduced and displayed on the display apparatus.

In a case where the reproduction instruction is neither the instruction for reproducing one frame nor the instruction for reproducing the moving image (NO in step S504), the processing proceeds to step S507. In step S507, the control unit <NUM> performs another reproduction processing in response to a user instruction.

In this manner, in the case of reproducing the raw moving image file, the moving image reproduction apparatus <NUM> according to the present exemplary embodiment automatically determines whether to use the ideal correction amount or the moving image correction amount based on a type of the reproduction instruction from the user. Thus, even if the user does not perform an operation to select which one to use from among a plurality of image processing parameters, the moving image reproduction apparatus <NUM> can automatically select a parameter appropriate for a reproduction method, perform the image processing using the parameter appropriate for the reproduction method, and then perform the reproduction.

Subsequently, a description will be given of extraction editing processing of the moving image reproduction apparatus <NUM> with reference to <FIG>. In the extraction editing processing, a frame is extracted from the moving image in the raw moving image file stored in the storage unit <NUM>, and the frame is recorded as a new file. <FIG> is a flowchart illustrating the extraction editing processing performed by the moving image reproduction apparatus <NUM>. The extraction editing processing is performed in response to acceptance of an instruction for extraction editing of the raw moving image stored in the storage unit <NUM> provided by the user via the operation unit <NUM>. The instruction for extraction editing includes an instruction for a setting regarding whether to extract a still image or moving image, a setting regarding whether to record the frame as a raw file or as a developed image, and a setting regarding a frame or frame range that is an extraction target frame. The extraction editing processing is performed by the control unit <NUM> reading a program from the program memory and performing control of each unit or arithmetic processing based on the read program.

First, in step S601, the control unit <NUM> determines whether the instruction for extraction editing is directed to still image extraction or moving image extraction. In a case where the control unit <NUM> determines that the instruction for extraction editing is directed to the still image extraction (YES in step S601), the processing proceeds to step S602. In a case where the control unit <NUM> determines that the instruction for extraction editing is directed to the moving image extraction (NO in step S601), the processing proceeds to step S608.

In step S602, in the case of the still image extraction, the control unit <NUM> extracts a frame set as an extraction target by the user from the frames of the raw moving image in the raw moving image file. Subsequently, in step S603, the control unit <NUM> determines if a setting is to record the extracted frame as a raw file (raw still image file) or to develop the extracted frame and record the frame as a Joint Photographic Experts Group (JPEG). In a case where the setting is to record the extracted file as the raw file (YES in step S603), the processing proceeds to step S604. In a case where the setting is to develop the extracted frame and record the frame as the JPEG file (NO in step S603), the processing proceeds to step S606.

In step S604, the control unit <NUM> adds the ideal correction amount, included in the image processing parameter recorded in the raw moving image file as frame information regarding the frame that is the extraction target, to the frame extracted in step S602 as the image processing parameter thereof. In the case where the extracted frame is to be recorded as the raw still image file, the control unit <NUM> adds the ideal correction amount to the extracted frame as the image processing parameter, but does not add the moving image correction amount thereto. In step S605, the control unit <NUM> stores the frame to which the ideal correction amount is added as image processing information about the frame in step S604 to the storage unit <NUM> as the raw still image file. At this time, processing of converting the extracted frame or metadata from a format for the raw moving image file to a format for the raw still image file is also performed.

In step S606, the control unit <NUM> controls the image processing unit <NUM> to perform image processing on the frame extracted in step S602 using the ideal correction amount, included in the image processing parameter recorded in the raw moving image file as frame information regarding the frame. The control unit <NUM> controls the image processing unit <NUM> to perform not only the image processing using the ideal correction amount, but also the development processing of the frames of the raw moving image. Furthermore, the control unit <NUM> controls the image processing unit <NUM> to perform JPEG compression processing. In step S607, the control unit <NUM> stores the JPEG image subjected to the image processing and the JPEG compression processing in step S606 as a JPEG file in the storage unit <NUM>.

In step S608, in the case of the moving image extraction, the control unit <NUM> recalculates the moving image correction amount of each of frames of the raw moving image in an extraction range set by the instruction for extraction editing. The control unit <NUM> performs processing of recalculating the moving image correction amount in step S608 in a similar manner to the processing of calculating the moving image correction amount illustrated in <FIG>, but not with respect to all the frames of the raw moving image, but with respect to frames of the raw moving image in the extraction range. More specifically, a head frame in the extraction range serves as the initial frame in the processing of recalculating the moving image correction amount. In the head frame in the extraction range, the ideal correction amount set to the head frame is set as the recalculated moving image correction amount. The control unit <NUM> recalculates the moving image correction amount with respect to the next or subsequent frame so as to reduce a difference in correction amount between frames. Thus, in a case where the ideal correction amount and the moving image correction amount set to the head frame in the extraction range are the same in correction amount, a result of recalculating the moving image correction amount in the extraction range is the same in correction amount as the moving image correction amount set (recorded) in the raw moving image file. Thus, in a case where the ideal correction amount and the moving image correction amount set to the head frame in the extraction range are the same in correction amount, the control unit <NUM> does not recalculate the moving image correction amount and may use the moving image correction amount recorded in the raw moving image file in steps S611 and S613 instead of the moving image correction amount obtained as the result of the recalculation.

Subsequently, in step S610, the control unit <NUM> determines whether a setting is to record the extracted frames of the raw moving image as a raw file (raw moving image file) or as a Moving Picture Expert Group-<NUM> (MP4) file after performing the developing processing on extracted frames. In a case where the setting is to record the extracted frames as the raw file (YES in step S610), the processing proceeds to step S611. In a case where the setting is to record the extracted frames as the MP4 file after being subject to the development processing (NO in step S610), the processing proceeds to step S613.

In step S611, the control unit <NUM> adds, as the image processing parameter, the ideal correction amount recorded in the raw moving image file and the moving image correction amount recalculated in step S608 to each of the frames, which are extracted in step S609, of the raw moving image in the extraction range. In this manner, in a case where frames of the raw moving image is extracted and recorded as the raw moving image file, the ideal correction amount and the moving image correction amount are added. However, the moving image correction amount to be added is not the one recorded in the raw moving image file, but the one obtained by recalculating the moving image correction amount in the extraction range. In step S612, the control unit <NUM> records the frames of the raw moving image in the extraction range, to which the ideal correction amount and the moving image correction amount are added as the image processing parameters for each frame in step S611, as the raw moving image file to the storage unit <NUM>.

In step S613, the control unit <NUM> controls the image processing unit <NUM> to perform image processing on each of the frames, which are extracted in step S609, of the raw moving image in the extraction range using the ideal correction amount included in the image processing parameter recorded in the raw moving image file. The control unit <NUM> controls the image processing unit <NUM> to perform not only the image processing using the ideal correction amount, but also the development processing of the frames of the raw moving image, and the compression processing for the moving image (e.g., MPEG data and High Efficiency Video Coding (HEVC) data). Then, in step S614, the control unit <NUM> stores the moving image subjected to the development processing, the image processing, and the compression processing for the moving images in step S613 as an MP4 file to the storage unit <NUM>.

In this manner, in the extraction editing processing according to the present exemplary embodiment, the control unit <NUM> selects a correction amount to be recorded together with the image or a correction amount to be used in the image processing from the ideal correction amount and the moving image correction amount (recalculated moving image correction amount) depending on whether the extraction is the still image extraction. Thus, the control unit <NUM> is configured to select a correction amount appropriate for each of a still image and a moving image. While the control unit <NUM> automatically selects a correction amount in the present exemplary embodiment, a correction amount selected by the user may be applied in a case where the user selects the correction amount via the operation unit <NUM>.

In this manner, the image processing apparatus according to the present exemplary embodiment is configured to automatically select a correction amount to be used in the image processing or a correction amount to be recorded from the ideal correction amount, the moving image correction amount, and the recalculated moving image correction amount based on a reproduction method and a recording method of the raw moving image. With this configuration, since the image processing apparatus automatically selects a correction amount appropriate for a usage situation of the raw moving image, the user can apply the correction amount appropriate for the usage situation of the raw moving image without selecting the correction amount to be used (recorded).

Claim 1:
An image pickup apparatus (<NUM>), comprising:
image pickup means (<NUM>);
recording means (<NUM>) for recording a raw moving image picked up and acquired by the image pickup means, in a storage medium; and
calculation means (<NUM>) for calculating an image processing parameter of the raw moving image,
wherein the calculation means (<NUM>) is arranged to calculate a first parameter calculated based on an image capturing setting of the raw moving image and a second parameter calculated based on the first parameter as the image processing parameter of the raw moving image,
wherein the first parameter and the second parameter are parameters indicating a correction amount for each frame for the same image processing, the first parameter being an ideal correction amount, and the second parameter being a moving image correction amount calculated such that an amount of change in the second parameter between frames of the raw moving image is decreased with respect to the ideal correction amount, and
wherein the recording means (<NUM>) is arranged to record the first parameter and the second parameter together with the raw moving image in the storage medium at a time of picking up of the raw moving image.