Patent Publication Number: US-8526777-B2

Title: Moving image recording method and information processing device

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
     This application is a U.S. continuation application filed under 35 U.S.C. 111(a) claiming benefit under 35 U.S.C. 120 and 365(c) of PCT application JP2007/057259, filed on Mar. 30, 2007, the entire contents of which are incorporated herein by reference. 
    
    
     FIELD 
     The embodiment discussed herein is related to a moving image recording method and an information processing device. 
     BACKGROUND 
     Up until now, there have been discussed reproducing systems capable of separately recording an audio data file and a moving image data file and reproducing audio and moving image in synchronization with each other even without any dedicated processor having high processing performance.
     Patent Document 1: Japanese Laid-Open Patent Publication No. 7-95522   

     SUMMARY 
     According to an aspect of the present invention, there is provided a moving image recording method including digitizing a video information item obtained by photographing and an audio information item obtained by recording to obtain a digitized video data item and a digitized audio data item each accompanied by a time information item, respectively; temporarily storing the digitized video data item accompanied by the time information item in such a manner as to separate the digitized video data item accompanied by the time information item into the time information item and the digitized video data item; temporarily storing the digitized audio data item accompanied by the time information item in such a manner as to separate the digitized audio data item accompanied by the time information item into the time information item and the digitized audio data item; and combining the time information item and the digitized video data item that are separately stored in the temporary storing the digitized video data item with the time information item and the digitized audio data item that are separately stored in the temporarily storing the digitized audio data item in synchronization with each other so as store the time information items, the digitized video data item, and the digitized audio data item in a predetermined format. 
     The object and advantages of the present invention will be realized and attained by means of the elements and combinations particularly pointed out in the claims. 
     It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the present invention, as claimed. 
    
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
         FIG. 1  is a function block diagram of an information processing device that performs a moving image recording method according to an embodiment of the present invention; 
         FIG. 2A  is a diagram for illustrating the flow of processing by the moving image recording method according to the embodiment of the present invention; 
         FIG. 2B  is a diagram for illustrating the flow of processing by the moving image recording method according to the embodiment of the present invention; 
         FIG. 2C  is a diagram for illustrating the flow of processing by the moving image recording method according to the embodiment of the present invention; 
         FIG. 3  is a diagram illustrating a state in which digitized video data items, time information items corresponding to the digitized video data items, digitized audio data items, and time information items corresponding to the digitized audio data items are combined with each other through time synchronization processing and format conversion processing and recorded in a video/audio/time file in the moving image recording method according to the embodiment of the present invention; 
         FIG. 4A  is a diagram illustrating a state in which the digitized video data items and the time information items corresponding to the digitized video data items are separately recorded in the moving image recording method according to the embodiment of the present invention; 
         FIG. 4B  is a diagram illustrating a state in which the digitized video data items and the time information items corresponding to the digitized video data items are separately recorded in the moving image recording method according to the embodiment of the present invention; 
         FIG. 5  is an operations flowchart for illustrating the flow of the time synchronization processing in the moving image recording method according to the embodiment of the present invention; 
         FIG. 6  is an operations flowchart for illustrating operations related to a mode change in the information processing device illustrated in  FIG. 1 ; 
         FIG. 7  is a diagram for illustrating the mode change in the information processing device illustrated in  FIG. 1 ; 
         FIG. 8  is a diagram for illustrating the operations in a low load mode in the information processing device illustrated in  FIG. 1 ; 
         FIG. 9  is a diagram for illustrating the operations in a high load mode in the information processing device illustrated in  FIG. 1 ; 
         FIG. 10  is a diagram for illustrating the operations in a restoration mode in the information processing device illustrated in  FIG. 1 ; 
         FIG. 11  is a diagram for illustrating the mode change along with a change in a processing load in the information processing device illustrated in  FIG. 1 ; and 
         FIG. 12  is a hardware block diagram of a computer in a case where the information processing device illustrated in  FIG. 1  is implemented by the computer. 
     
    
    
     DESCRIPTION OF EMBODIMENTS 
     When moving image information items photographed by cameras and recorded by microphones are digitized and recorded, MP4 is, for example, used as a recording format. 
     According to MP4, digitized video data items and digitized audio data items are recorded in order of time, and time information items corresponding to the digitized video data items and the digitized audio data items are also recorded in order of time. 
     Here, the time information items are composed of information items indicating times at which the video data items and the audio data items were recorded and composed of information items on the data sizes of the data items recorded at the corresponding times. 
     When moving images are digitized and recorded in such a format, video inputs from cameras are sorted at every predetermined time interval to provide video data items accompanied by corresponding time information items. In addition, audio inputs from microphones are sorted at every predetermined time interval to provide audio data items accompanied by corresponding time information items. 
     The video data items and the audio data items are temporarily stored in a video buffer unit and an audio buffer unit, respectively. Then, these data items are read and recorded in a video/audio/time file in MP4 format through time synchronization processing and format conversion processing. 
     When the moving images are reproduced from the video/audio/time file, information items on the data sizes of the video data items and the audio data items included in the time information items corresponding to times are read as control information items to reproduce the moving images with time. In accordance with the control information items thus read, the video data items and the audio data items are successively read and reproduced by amounts equivalent to the data sizes of specified data items. As a result, the video data items and the audio data items of the moving images initially taken and recorded at the same time are successively reproduced in synchronization with each other. 
     However, the time synchronization processing and the format conversion processing require a certain processing time. Therefore, when the video inputs or the audio inputs are processed at high speed and with high quality, the synchronization processing and the format conversion processing are not properly processed in time, which may result in the abandonment of input data items exceeding a tolerable processing amount. 
     Particularly, since information processing devices such as mobile phones required to be downsized and reduced in weight have limitation on enhancing data processing performance, there would be a high likelihood of such an abandonment of input data items. 
     The present invention has been made in view of the above circumstances and has an object of providing a moving image recording method capable of reliably recording moving images without abandoning input data items even if data items to be processed are large in amount. In addition, the present invention has an object of providing an information processing device for executing the moving image recording method. 
     An embodiment of the present invention is described in detail below with reference to the accompanying drawings. 
       FIG. 1  is a function block diagram of an information processing device, e.g., a cell phone having the configuration of executing a moving image recording method according to the embodiment of the present invention. 
     The information processing device has a camera  11  serving as a video input unit; a microphone  21  serving as an audio input unit; a video digitization unit  12  that digitizes video information items output from the camera  11 ; an audio digitization unit  22  that digitizes audio information items output from the microphone  21 ; a video buffer unit  13 ; an audio buffer unit  23 ; a temporary video buffer unit  14  that stores a video time information file  14   a  and a video data file  14   b ; a temporary audio buffer unit  24  that stores an audio time information file  24   a  and an audio data file  24   b ; a combining unit  25  that includes a time synchronization processing unit  15  and a format conversion processing unit  16  (see  FIGS. 2A and 2B ); and a video/audio/time file storage unit  26 A that stores a video/audio/time file  26 . 
     The video digitization unit  12  and the audio digitization unit  22  output, for example, digitized moving image information items photographed by the camera  11  and recorded by the microphone  21 . 
     The video buffer unit  13  and the audio buffer unit  23  store digitized video data items and digitized audio data items output from the video digitization unit  12  and the audio digitization unit  22 , respectively. 
     The combining unit  25  reads the digitized video data items and the digitized audio data items from the video buffer unit  13  and the audio buffer unit  23 , respectively. In the combining unit  25 , the time synchronization processing unit  15  applies time synchronization processing to these digitized video data items and the digitized audio data items, and the data format processing unit  16  applies data format processing to the digitized video data items and the digitized audio data items thus subjected to the time synchronization processing, thereby generating the video/audio/time file  26 . The combining unit  25  stores the generated video/audio/time file  26  in the video/audio/time file storage unit  26 A serving as a recording medium. 
     The temporary video buffer unit  14  and the temporary audio buffer unit  24  have the digitized video data items and the digitized audio data items temporarily stored therein in a high load mode described below. 
       FIG. 2A  is a diagram for illustrating operations of the information processing device in a low load mode described below along with  FIG. 1 . 
     According to the embodiment of the present invention, as illustrated in  FIG. 2A , video inputs from the camera  11  are sorted at every predetermined time interval to provide video data items accompanied by corresponding time information items. In addition, audio inputs from the microphone  21  are sorted at every predetermined time interval to provide audio data items accompanied by corresponding time information items. Then, the video data items and the audio data items are temporarily stored in the video buffer unit  13  and the audio buffer unit  23 , respectively. The video data items and the audio data items are read and recorded in the video/audio/time file  26  in MP4 format through the time synchronization processing unit  15  and the format conversion processing unit  16  of the combining unit  25 . 
     The video/audio/time file  26  has a data structure as illustrated on the right side of  FIG. 3 . When the video/audio/time file  26  reproduces moving images, the time information items for respective times stored in a time information section  26   a  at the upper stage of the video/audio/time file  26 , i.e., the time information items on the video data items and the audio data items and information items on data sizes are read as control information items to reproduce the moving images with time. In accordance with the control information items thus read, the video data items and the audio data items of a video/audio data section  26   b  at the lower stage of the video/audio/time file  26  are successively read and reproduced by amounts equivalent to the data sizes of specified data items. As a result, the video data items and the audio data items of the moving images initially taken and recorded at the same time are successively reproduced in synchronization with each other. 
     However, the time synchronization processing unit  15  and the format conversion processing unit  16  require a certain processing time for the time synchronization processing and the format conversion processing, respectively. Therefore, when video inputs or audio inputs are processed at high speed and with high quality, the synchronization processing unit  15  and the format conversion processing unit  16  are not properly processed in time, which may result in the abandonment of input data items exceeding a tolerable processing amount. 
     Particularly, since information processing devices such as mobile phones required to be downsized and reduced in weight have limitations on enhancing data processing performance, there would be a high likelihood of such an abandonment of input data items. 
     According to the embodiment of the present invention, as illustrated in  FIG. 2B , the video information items and the audio information items obtained by photographing and recording are first digitized in the high load mode. Then, without being subjected to the time synchronization processing by the time synchronization processing unit  15  and the format conversion processing by the format conversion processing unit  16 , the video information items are recorded in the temporary video buffer unit  14  in such a manner as to be separated into video time information items  14   a  and digitized video data items  14   b , and the audio information items are recorded in the temporary audio buffer unit  24  in such a manner as to be separated into audio time information items  24   a  and digitized audio data items  24   b.    
     The processing in the high load mode does not include the time synchronization processing and the format conversion processing, but it includes only the process of recording the digitized data items in such a manner as to be separated into the time information items and the video data items and be separated into the time information items and the audio data items. Therefore, a required data processing amount can be reduced. Accordingly, input data items can be reliably recorded without being abandoned. 
     As illustrated in  FIG. 2C , in a restoration mode described below, the digitized video data items  14   b  and the corresponding time information items  14   a , and the digitized audio data items  24   b  and the corresponding time information items  24   a , which are separately recorded, are successively read in order of time and subjected to the time synchronization processing by the time synchronization processing unit  15  and the format conversion processing by the format conversion processing unit  16 . As a result, the video/audio/time file  26  of the data structure illustrated on the right side of  FIG. 3  is obtained. 
     As described above, according to the embodiment of the present invention, when moving images are recorded in the high load mode, the digitized video data items are temporarily stored in such a manner as to be separated into the digitized video data items and the corresponding time information items, while the digitized audio data items are temporarily stored in such a manner as to be separated into the digitized audio data items and the corresponding time information items. Then, the data items thus temporarily stored are read and rearranged in chronological order and converted into a predetermined format. Unlike a related art in which input data items exceeding a tolerable processing amount are abandoned, the configuration according to the embodiment of the present invention enables the reliable storage of input data items without abandoning the input data items. 
     In other words, according to the related art, when moving images are recorded by, e.g., a cell phone and a microphone, digitized video data items, digitized audio data items, and time information items corresponding to the digitized video data items and the digitized audio data items are converted into a predetermined format (e.g., MP4 format) as a single file and recorded so that videos and audios are reproduced in synchronization with each other. 
     In this related art, when a large amount of high quality videos are processed by an information processing device such as a cell phone having relatively low processing performance, the time synchronization processing and the format conversion processing require a long processing time. As a result, input data items are not properly processed in time, which may result in the abandonment of the input data exceeding a tolerable processing amount. 
     In view of the above problem in the related art, the embodiment of the present invention is configured to record moving images in the low load mode in the following manner. At first, as illustrated in  FIG. 2A , video information items output from the camera  11  and audio information items output from the microphone  21  are digitized by the video digitization unit  12  and the audio digitization unit  22 , respectively, as in the case of the related art. Then, if it is determined that the storage rate of the video buffer unit  13  or the audio buffer unit  23  does not exceed 80% of its capacity and that the storage destinations of the present data items are not the temporary video buffer unit  14  and the temporary audio buffer unit  24 , the digitized video data items and the digitized audio data items are buffered by the video buffer unit  13  that temporarily stores video data items and by the audio buffer unit  23  that temporarily stores audio data items, respectively. After that, the digitized video data items, the digitized audio data items, which are stored in the video buffer unit  13  and the audio buffer unit  23 , respectively, and the time information items corresponding the digitized video data items and the digitized audio data items are converted into a predetermined format (such as MP4 format) and stored in the video/audio/time file storage unit  26 A as the video/audio/time file  26 . 
     Here, the low load mode described above refers to a mode in which the data amounts of the video information items input through the camera  11  and the video digitization unit  12  and the data amounts of the audio information items input through the microphone  21  and the audio digitization unit  22  do not exceed the performances of the time synchronization processing by the time synchronization processing unit  15  and the format conversion processing by the format conversion processing unit  16  of the combining unit  25 . In other words, in the low load mode, input data items are not required to be abandoned. 
     If the data amounts per unit time of the digitized video data items output from the camera  11  or the data amounts per unit time of the digitized audio data items output from the microphone  21 , i.e., frame rates, bit rates, etc., exceed a certain value, the information processing device changes to the high load mode. 
     Here, as a method for determining whether the data amounts per unit time of the digitized video data items output from the camera  11  or the data amounts per unit time of the digitized audio data items output from the microphone  21  exceed the certain value, data amounts stored in the video buffer unit  13  or the audio buffer unit  23  are taken into consideration. For example, if the storage data amount of a buffer memory constituting the video buffer unit  13  or the audio buffer unit  23  reaches 80% of the storage capacity of the buffer memory, it can be determined that the data amounts per unit time of the digitized video data items output from the camera  11  or the digitized audio data items output from the microphone  21  exceed the certain value. 
     As illustrated in  FIG. 2B , when moving images are recorded in the high load mode, the video information items output from the camera  11  and the audio information items output from the microphone  21  are digitized by the video digitization unit  12  and the audio digitization unit  22 , respectively, and then separately stored in the temporary video buffer unit  14  and the temporary audio buffer unit  24 , respectively. In other words, the digitized video information items are recorded in such a manner as to be separated into a video time information file  14   a  and a video data file  14   b , while the digitized audio information items are recorded in such a manner as to be separated into an audio time information file  24   a  and an audio data file  24   b.    
     The video time information file  14   a , the video data file  14   b , the audio time information file  24   a , and the audio data file  24   b  are described below. 
     In the video time information file  14   a , times at which the video data items were recorded and the data sizes of the video data items recorded at the corresponding times are stored. In the video data file  14   b , the video data items per se are stored. Furthermore, in the audio time information file  24   a , times at which the audio data items were recorded and the data sizes of the audio data items recorded at the corresponding times are stored. In the audio data file  24   b , the audio data items per se are stored. 
     Here, the digitized video data items obtained by digitizing the video information items output from the camera  11  and the digitized audio data items obtained by digitizing the audio information items output from the microphone  21  are accompanied by time information items, i.e., information items on times at which the information items were recorded and by information items on the data sizes of the information items in units of data items at a predetermined time interval. 
     According to the embodiment of the present invention, the time information items are separated from the digitized video information items in the high load mode. As a result, as illustrated in  FIG. 4A , the digitized video data items and the time information items corresponding to the digitized video data items are separately recorded in the temporary video buffer unit  14  as the video data file  14   b  and the video time information file  14   a  in order of time (i.e., times  0 ,  1 ,  2 , . . . , N). 
     Similarly, the time information items are separated from the digitized audio information items corresponding to the predetermined time interval. As a result, as illustrated in  FIG. 4B , the digitized audio data items and the time information items corresponding to the digitized audio data items are separately recorded in the temporary audio buffer  24  as the audio data file  24   b  and the audio time information file  24   a  in order of time (i.e., times  0 ,  1 ,  2 , . . . , N). 
     In the high load mode, while the input data items are recorded in the temporary video buffer unit  14  and the temporary audio buffer unit  24 , storage data items are successively read from the video buffer unit  13  and the audio buffer unit  23 . Then, as illustrated in  FIG. 2A , the read storage data items are subjected to the time synchronization processing by the time synchronization processing unit  15  and the format conversion processing by the format conversion processing unit  16  and recorded in the video/audio/time file storage unit  26 A as the video/audio/time file  26 . 
     If it is determined that the storage data amounts of the buffer memories whose storage capacity has temporarily exceeded 80% are reduced to less than a predetermined value as a result of the time synchronization processing and the format conversion processing subjected to the storage data items read from the video buffer unit  13  and the audio buffer unit  23 , the information processing device changes to the restoration mode. 
     Here, the restoration mode refers to a mode in which the digitized video data items stored in the temporary video buffer unit  14  and the digitized audio data items stored in the temporary audio buffer unit  24  in the high load mode are subjected to the time synchronization processing by the time synchronization processing unit  15  and the format conversion processing by the format conversion processing  16  and stored in the video/audio/time file storage unit  26 A as the video/audio/time file  26 , thereby processing the storage data items of the temporary video buffer unit  14  and the temporary audio buffer unit  24 . 
     Here, a case where the storage data amounts of the video buffer unit  13  and the audio buffer unit  23  are reduced to less than the predetermined value can be recognized, for example, as a case where the storage data amounts of the respective buffer memories constituting the video buffer unit  13  and the audio buffer unit  23  become less than 80% of their storage capacities or to a case where the storage data items of the video buffer unit  13  and the audio buffer unit  23  are completely processed. 
     In the restoration mode, as illustrated in  FIG. 2C , the time synchronization processing unit  15  and the format conversion processing unit  16  start applying the time synchronization processing and the format conversion processing, respectively, to the digitized video data items recorded in the video data file  14   b , the time information items corresponding to the digitized video data items recorded in the video time information file  14   a , the digitized audio data items recorded in the audio data file  24   b , and the time information items corresponding to the digitized audio data items recorded in the audio time information file  24   a.    
     In the time synchronization processing by the time synchronization processing unit  15  and the format conversion processing by the format conversion processing unit  16 , the video data file  14   b  in which the digitized video data items and the time information items corresponding to the video data items are separately recorded in order of time, the video time information file  14   a  in which the video time information items corresponding to the digitized video data items are separately recorded in order of time, the audio data file  24   b  in which the digitized audio data items are separately recorded in order of time, and the audio time information file  24   a  in which the audio time information items are separately recorded are rerecorded in chronological order to generate the video/audio/time file  26 . 
     The time synchronization processing by the time synchronization processing unit  15  and the format conversion processing by the format conversion processing unit  16  are described below with reference to  FIG. 3 . 
       FIG. 3  is a diagram for illustrating the recorded information items of the video time information file  14   a , the video data file  14   b , the audio time information file  24   a , and the audio data file  24   b  and illustrating the recorded information items of the video/audio/time file  26  obtained by applying the time synchronization processing and the format conversion processing to the video time information file  14   a , the video data file  14   b , the audio time information file  24   a , and the audio data file  24   b.    
     As illustrated in  FIG. 3 , in the video time information file  14   a , the time information items indicating the times at which the video data items were recorded and the data sizes of the video data items recorded at the corresponding times are stored in order of time. In the video data file  14   b , the corresponding video data items per se are stored in order of time. 
     Similarly, in the audio time information file  24   a , the time information items indicating the times at which the audio data items were recorded and the data sizes of the audio data items recorded at the corresponding times are stored in order of time. In the audio data file  24   b , the corresponding audio data items per se are stored in order of time. 
     In the time information section  26   a  at the upper stage of the video/audio/time information file  26 , the time information items indicating the times at which the video data items and the audio data items were recorded and the data sizes of the video data items and the audio data items recorded at the corresponding times are stored in order of time. In the video/audio data section  26   b  at the lower stage of the video/audio/time information file  26 , the corresponding video data items and the audio data items per se are alternately stored in order of time. 
     From among the video data file  14   b  in which the video data items per se are separately recorded in order of time (i.e., times  0 ,  1 ,  2 , . . . , N) by the time synchronization processing and the format conversion processing and video time information file  14   a  in which the time information items and the data sizes are separately recorded in order of time (i.e., times  0 ,  1 ,  2 , . . . , N) by the time synchronization processing and the format conversion processing, the video data items and the audio data items at the same time stored in the video data file  14   b  and the audio data file  24   b , respectively, are alternately stored in the video/audio data section  26   b  of the video/audio/time file  26  in order of time. In other words, in the video/audio data section  26   b  of the video/audio/time file  26 , the digitized video data item at time  0 , the digitized audio data item at time  0 , the digitized video data item at time  1 , the digitized audio data item at time  1 , . . . , the digitized video data item at time N, and the digitized audio data item at time N are recorded in this order. 
     Similarly, in the time information section  26   a  of the video/audio/time file  26 , the time information items at the same time stored in the video time information file  14   a  and the audio time information file  24   a , respectively, are alternately recorded in order of time. In other words, the time information item representing time  0 , the data size of the video data item, and the data size of the audio data item each serving as the time information item at time  0 , the time information item representing time  1 , the data size of the audio data item, and the data size of the audio data item each serving as the time information item at time  1 , . . . , and the time information item representing time N, the data size of the video data item, and the data size of the audio data item each serving as the time information item at time N are recorded in this order in the time information section  26   a  of the video/audio/time file  26 . 
     When moving images are actually reproduced from the video/audio/time file  26  thus generated, the following processing is performed. 
     In other words, the time information items, the data sizes of the video data items at the corresponding times, and the data sizes of the audio data items at the corresponding times are successively read from the beginning of the time information section  26   a  of the video/audio/time file  26  in order of time. Then, the video data items and the audio data items at the corresponding times are read from the video/audio data section  26   b  of the video/audio/time file  26  by amounts equivalent to the data sizes of the specified video data items and the data sizes of the specified audio data items. Since the video data items and the audio data items are alternately stored for each time in the video/audio data section  26   b  in order of time, the data items of corresponding data amounts are successively read from the video/audio data section  26   b  in accordance with the data sizes of the video data items and the data sizes of the audio data items at the corresponding times stored in the time information section  26   a . With this operation, the video data items and the audio data items at the corresponding times are successively alternately read and reproduced. 
     As described above, according to the embodiment of the present invention, in the high load mode, when respective data items are recorded, the time information items are separated from the digitized video data items and the digitized audio data items at every predetermined time interval without the application of the time synchronization processing and the format conversion processing to the digitized video data items and the digitized audio data items. Then, the digitized video data items, the time information items corresponding to the digitized video data items, the digitized audio data items, and the time information items corresponding to the digitized audio data items are separately recorded per se (the video time information file  14   a , the video data file  14   b , the audio time information file  24   a , and the audio data file  24   b ). In the restoration mode, the video time information file  14   a , the video data file  14   b , the audio time information file  24   a , and the audio data file  24   b  are subjected to the time synchronization processing and the format conversion processing so as to be combined with each other, thus generating the video/audio/time file  26 . In other words, if input data amounts exceed the processing performance of the information processing device related to time synchronization processing and the format conversion processing, the information processing device changes to the high load mode. In this mode, the information processing device leaves the application of the time synchronization processing and the format conversion processing to input data items on the back burner and temporarily stores the input data items. 
     Therefore, the input data items exceeding the processing performance of the information processing device related to the time synchronization processing and the format conversion processing are not required to be abandoned. That is, the input data items can be reliably recorded. 
     Here, the data amounts per unit time of the digitized video data items and the digitized audio data items, i.e., bit frames, frame rates, etc., are different. Therefore, when the digitized video data items and the digitized audio data items are successively read from the video buffer unit  13  and the audio buffer unit  23 , respectively, at a constant read speed, read digitized video data items and read digitized audio data items are not necessarily synchronized with each other. Accordingly, time synchronization processing is required to synchronize the digitized video data items and the digitized audio data items. 
       FIG. 5  is a flowchart for illustrating an operating procedure in the time synchronization processing by the time synchronization processing unit  15  according to the embodiment of the present invention. Note that the time synchronization processing at the time of reading storage data items from the video buffer unit  13  and the audio buffer unit  23  and the time synchronization processing at the time of reading storage data items from the temporary video buffer unit  14  and the temporary audio buffer unit  24  are described below one by one. 
     First, the time synchronization processing at the time of reading the storage data items from the video buffer unit  13  and the audio buffer unit  23  is described. 
     In step S 51 , the digitized video data item at time i (i=0 through N) is read from the video buffer unit  13  together with the time information item accompanied by time i. 
     Next, the digitized audio data item having the time information item indicating time i the same as the time of the video data item acquired from the video buffer unit  13  is searched for from the audio buffer unit  23 . 
     Generally, the data amounts per unit time of the video information items obtained through the camera  11  and the data amounts per unit time of the audio information items obtained through the microphone  21  are different from each other. Particularly, the data amounts per unit time of the audio information items are smaller than the data amounts per unit time of the video information items in many cases. Therefore, all the digitized video data items do not necessarily have the corresponding digitized audio data items. This is because, when the data amounts per unit time of the audio information items are relatively small, the generation frequency of the digitized audio data items becomes smaller than that of the digitized video data items at the time of digitizing input data items. Accordingly, the digitized audio data item corresponding to the digitized video data item at certain time may not exist. In this case, the flow proceeds to “No” in step S 52 . 
     If the flow proceeds to “No” in step S 52 , the digitized video data item and the time information item corresponding to the digitized video data items read in step S 51  are written in the video/audio/time file  26  in step S 53  (i.e., the time information item, the data size of the video data item, and the video data item at time i in the data structure on the right side of  FIG. 3 ). 
     If the corresponding digitized audio data item at time i is found (“Yes” in step S 52 ), it is read from the audio buffer unit  23  together with the time information item accompanied by the corresponding digitized audio data item in step S 54 . Then, in step S 55 , the corresponding digitized audio data item and the time information item are written in the video/audio/time file  26  (i.e., the time information item, the data size of the audio data item, and the audio data item at time i in the data structure on the right side of  FIG. 3 ). 
     In step S 56 , a determination is made as to whether all the digitized video data items of the video data file  14  have been processed (i.e., whether the digitized video data item at time N has been processed). 
     If all the digitized video data items have been processed (“Yes” in S 56 ), the time synchronization processing is completed. If all the digitized video data items have not been processed (“No” in S 56 ), the flow proceeds to step S 57  where 1 is added to the time variable i. Then, the flow returns to step S 51  to repeat the above processing. The video/audio/time file  26  is thus generated. 
     Next, the time synchronization processing at the time of reading the storage data items from the temporary video buffer unit  14  and the temporary audio buffer unit  24  is described. 
     In step S 51 , the digitized video data item at time i (i=0 through N) is read from the video data file  14   b  stored in the temporary video buffer unit  14 , while the time information item and the data size of the video data item each serving as the video time information item at the same time i are read from the video time information file  14   a.    
     Then, in step S 52 , the audio time information item or the digitized audio data item at the same time is searched for from the audio time information file  24   a  or the audio data file  24   b  stored in the temporary audio buffer unit  24 . 
     If the audio time information item or the digitized audio data item at the same time i is not found (“No” in step S 52 ), the digitized video data item and the time information item acquired in step S 51  are written in the video/audio/time file  26  (i.e., the time information item, the data size of the video data item, and the video data item at time i in the data structure on the right side of  FIG. 3 ). 
     On the other hand, if the audio time information item or the digitized audio data item at the same time is found (“Yes” in step S 52 ), the corresponding audio time information item and the digitized audio data item are read from the audio time information file  24   a  and the audio data file  24   b , respectively, stored in the temporary audio buffer unit  24   b  in step S 54 . In step S 55 , the read audio time information item and the digitized audio data item are written in the video/audio/time file  26  (i.e., the time information, the data size of the audio data item, and the audio data item at time i in the data structure on the right side of  FIG. 3 ). 
     In step S 56 , a determination is made as to whether all the time information items and all the digitized video data items in the video time information file  14  and the video data file  14   b , respectively, stored in the video temporary buffer  14  have been processed (i.e., whether the time information item and the digitized video data item at time N have been processed). 
     If all the time information items and all the digitized video data items have been processed (“Yes” in step S 56 ), the time synchronization processing is completed. If all the time information items and all the digitized video data items have not been processed (“No” in step S 56 ), the flow proceeds to step S 57  where 1 is added to the time variable i. Then, the flow returns to step S 51  to repeat the above processing. The video/audio/time file  26  is thus generated. 
     Next, with reference to  FIGS. 1 and 6  through  11 , the flow of moving image recording operations in the information processing device, particularly, a mode change operation between the low load mode, the high load mode, and the restoration mode is described. 
       FIG. 6  is an operations flowchart paying attention to the flow of the moving image recording operations in the information processing device, particularly, to the mode change operation. 
     When recording operations of moving images, i.e., photographing and recording operations by the user, are started, video information items and audio time information items are output from the camera  11  and the microphone  21 , respectively. The output video information items and the audio time information items are digitized by the video digitization unit  12  and the audio digitization unit  22 , respectively. Thus, the digitized video data items accompanied by the time information items and the digitized audio data items accompanied by the time information items are generated. 
     In step S 2 , a determination is made as to whether the storage data amount of the video buffer unit  13  or the audio buffer unit  23  exceeds 80% of its storage capacity. 
     If it is determined that the storage data amount has not exceeded 80% of the storage capacity (“No” in step S 2 ), a determination is made as to whether the storage destinations of the present digitized video data items and the digitized audio data items are the temporary video buffer unit  14  and the temporary audio buffer unit  24  in step S 3 . 
     If it is determined that the storage destinations are not the temporary video buffer unit  14  and the temporary audio buffer unit  24  (“No” in step S 3 ), i.e., if it is determined that the storage destinations are the video buffer unit  13  and the audio buffer unit  23 , the digitized video data items and the digitized audio data items are stored in the video buffer unit  13  and the audio buffer unit  23 , respectively, so as to be accompanied by the corresponding time information items (step S 4 ). 
     In step S 5 , a determination is made as to whether the storage data amount of the video buffer unit  13  or the audio buffer unit  23  exceeds 80% of its storage capacity. 
     If it is determined that the storage data amount does not exceed the storage capacity (“No” in step S 5 ), the time synchronization processing unit  15  and the data format conversion processing unit  16  of the combining unit  25  apply the time synchronization processing and the format conversion processing, respectively, to the digitized video data items accompanied by the time information items and the digitized audio data items accompanied by the time information items stored in the video buffer unit  13  and the audio buffer unit  23 , thereby generating the video/audio/time file  26  (step S 6 ). 
     The processing in steps S 1  through S 6  is repeatedly performed until it is determined that the storage data amount of the video buffer unit  13  or the audio buffer unit  23  exceeds 80% of the storage capacity in step S 2  or step S 5 . 
     The above state refers to the low load mode. 
     As illustrated in  FIGS. 7 and 8 , in the low load mode, the digitized video data items accompanied by the time information items and the digitized audio data items accompanied by the time information items are accumulated in the video buffer unit  13  and the audio buffer unit  23 , respectively. At the same time, the processing by the time synchronization processing unit  15  and the processing by the format conversion processing unit  16  of the combining unit  25  are applied to the digitized video data items and the digitized audio data items stored in the video buffer unit  13  and the audio buffer unit  23 , respectively, to generate the video/audio/time file  26 . The generated video/audio/time file  26  is stored in the video/audio/time file storage unit  26 A. 
       FIG. 7  illustrates a state in which the storage destinations of the digitized video data items and the digitized audio data items output from the video digitization unit  12  and the audio digitization unit  22 , respectively, are changed from the video buffer unit  13  and the audio buffer unit  23  to the temporary video buffer unit  14  and the temporary audio buffer unit  24  when the modes are changed from the low load mode to the restoration mode through the high load mode. Furthermore,  FIG. 7  illustrates a state in which the storage places of the data items to be processed when the digitized video data items and the digitized audio data items thus stored are processed by the combining unit  25  are changed from the video buffer unit  13  and the audio buffer unit  23  to the temporary video buffer unit  14  and the temporary audio buffer unit  24 . 
       FIG. 8  is a diagram for illustrating the operations of the information processing device along with  FIG. 1 . 
     As illustrated in times t 1  through t 4  in  FIG. 11 , when a processing load, i.e., the data amounts per unit time of the digitized video data items and the digitized audio data items output from the video digitization unit  12  and the audio digitization unit  22 , respectively, gradually increase, the processing by the time synchronization processing unit  15  and the processing by the format conversion processing unit  16  are not properly processed in time. As a result, the storage data amounts of the video buffer unit  13  and the audio buffer unit  23  gradually increase. 
       FIG. 9  is a diagram for illustrating the operations of the information processing device in  FIG. 1  in the high load mode.  FIG. 11  is a diagram for illustrating the mode change between the low load mode, the high load mode, and the restoration mode along with the change of the processing load in the information processing device in  FIG. 1 . 
     If the storage data amount of the video buffer unit  13  or the audio buffer unit  23  exceeds 80% of its storage capacity (time t 4  in  FIG. 11 ), the flow in  FIG. 6  proceeds to “Yes” in step S 2 , and the storage destinations of the digitized video data items and the digitized audio data items are changed from the video buffer unit  13  and the audio buffer unit  23  to the temporary video buffer unit  14  and the temporary audio buffer unit  24  in step S 8 . 
     Also, as the flow of another operation, the flow proceeds to “Yes” in step S 5  in  FIG. 6 , and the storage destinations of the digitized video data items and the digitized audio data items are changed from the video buffer unit  13  and the audio buffer unit  23  to the temporary video buffer unit  14  and the temporary audio buffer unit  24  in step S 7 . At the same time, in step S 6 , the time synchronization processing by the time synchronization processing unit  15  and the format conversion processing by the format conversion processing unit  16  are continuously applied to the digitized video data items accompanied by the time information items and the digitized audio data items accompanied by the time information items stored in the video buffer unit  13  and the audio buffer unit  23 , respectively. 
     In other words, if it is determined that the flow proceeds to “Yes” in step S 2  or S 5  in  FIG. 6 , the information processing device changes to the high load mode. 
     As illustrated in  FIGS. 7 and 9 , in the high load mode, the digitized video data items accompanied by the time information items and the digitized audio data items accompanied by the time information items output from the video digitization unit  12  and the audio digitization unit  22 , respectively, are temporarily stored in the temporary video buffer unit  14  and the temporary audio buffer unit  24 , respectively. 
     At that time, the time information items are separated from the digitized video data items accompanied by the time information items and the digitized audio data items accompanied by the time information items. That is, the video time information file  14   a , the video data file  14   b , the audio time information file  24   a , and the audio data file  24   b  are separately recorded (see  FIGS. 2B ,  4 A, and  4 B). 
     At the same time, the time synchronization processing by the time synchronization processing unit  15  and the format conversion processing by the format conversion processing unit  16  are continuously applied to the digitized video data items accompanied by the time information items and the digitized audio data items accompanied by the time information items stored in the video buffer unit  13  and the audio buffer unit  23 , respectively. 
     Thus, the digitized video data items, the digitized audio data items, and the time information items corresponding to the digitized video data items and the digitized audio data items are stored in the video/audio/time file storage unit  26 A as the video/audio/time file  26  as in the case of the low load mode. 
     As illustrated in  FIGS. 7 and 9 , in the high load mode, the storage destinations of the digitized video data items and the digitized audio data items are changed from the video buffer unit  13  and the audio buffer unit  23  to the temporary video buffer unit  14  and the temporary audio buffer unit  24 . Therefore, the storage data items in the video buffer unit  13  and the audio buffer unit  23  no longer increase, while the data items stored in the video buffer unit  13  and the audio buffer unit  23  are gradually reduced by the processing of the combining unit  25  (times t 4  through t 7  in  FIG. 11 ). 
     Referring back to  FIG. 6 , when the flow proceeds to “Yes” in step S 2  and then the storage destinations of the digitized video data items and the digitized audio data items are changed from the video buffer unit  13  and the audio buffer unit  23  to the temporary video buffer unit  14  and the temporary audio buffer unit  24  in step S 8 , a determination is made as to whether all the storage data items in the video buffer unit  13  and the audio buffer unit  23  have been processed in step S 9 . 
     As described above, in the high load mode, while the digitized video data items, the digitized audio data items, and the time information items corresponding to the digitized video data items and the digitized audio data items are stored in the temporary video buffer unit  14  and the temporary audio buffer unit  24 , respectively, the time synchronization processing by the time synchronization processing unit  15  and the format conversion processing unit by the format conversion processing unit  16  are applied to the digitized video data items accompanied by the time information items and the digitized audio data items accompanied by the time information items stored in the video buffer unit  13  and the audio buffer unit  23 , respectively. As the storage data items of the video buffer unit  13  and the audio buffer unit  23  are successively processed as described above, the storage data amounts (i.e., usage rates of the buffers) of the video buffer unit  13  and the audio buffer unit  23  gradually reduce as illustrated in times t 4  through t 7  in  FIG. 11 . As a result, if all the storage data items of the video buffer unit  13  and the audio buffer unit  23  have been processed, it is determined that the flow proceeds to “Yes” in step S 9 . 
     Note that the criterion of determining the flow in step S 9  is not limited to the “time at which all the storage data items of the video buffer unit  13  and the audio buffer unit  23  have been processed,” but it may be based on the “time at which the storage data amounts of the video buffer unit  13  and the audio buffer unit  23  have been reduced to a predetermined threshold (e.g., 80% of the storage capacity).” 
     If it is determined that the flow proceeds to “No” in step S 9 , i.e., if the storage data items still remain in the video buffer unit  13  and the audio buffer unit  23 , the flow returns to step S 1 . Then, the processing in steps S 1 , S 2 , S 8 , and S 9  is repeatedly performed until it is determined that the flow proceeds to “Yes” in step S 9 . 
     Furthermore, as the flow of another operation to change to the high load mode, the flow proceeds to “Yes” in step S 5  as described above. In addition, in step S 7 , the storage destinations of the digitized video data items and the digitized audio data items are changed from the video buffer unit  13  and the audio buffer unit  23  to the temporary video buffer unit  14  and the temporary audio buffer unit  24 . Moreover, in step S 6 , the time synchronization processing by the time synchronization processing unit  15  and the format conversion processing by the format conversion processing unit  16  are continuously applied to the digitized video data items accompanied by the time information items and the digitized audio data items accompanied by the time information items stored in the video buffer unit  13  and the audio buffer unit  23 , respectively. 
     In this case, the flow returns to step S 1 . Then, the flow proceeds to step S 2 , and it is determined that the flow proceeds to “Yes” in step S 2 . In addition, in step S 8 , the storage destinations of the digitized video data items and the digitized audio data items are changed from the video buffer unit  13  and the audio buffer unit  23  to the temporary video buffer unit  14  and the temporary audio buffer unit  24 . The following processing is the same as the processing described above. The processing in steps S 1 , S 2 , S 8 , and S 9  is repeatedly performed until it is determined that the flow proceeds to “Yes” in step S 9   
     On the other hand, if it is determined that the flow proceeds to “Yes” in step S 9 , i.e., if all the storage data items in the video buffer unit  13  and the audio buffer unit  23  have been processed by the time synchronization processing by the time synchronization processing unit  15  and the format conversion processing by the format conversion processing unit  16 , i.e., if the state at time t 8  is established in  FIG. 11 , the digitized video data items accompanied by the time information items and the digitized audio data items accompanied by the time information items output from the video digitization unit  12  and the audio digitization unit  22 , respectively, are stored in the video buffer unit  13  and the audio buffer unit  23 , respectively, so as to be accompanied by the corresponding time information items in step S 10 . 
     In step S 11 , the time synchronization processing by the time synchronization processing unit  15  and the format conversion processing by the format conversion processing unit  16  are applied to the video time information file  14   a , the video data file  14   b , the audio time information file  24   a , and the audio data file  24   b  separately stored in the temporary video buffer unit  14  and the temporary audio buffer unit  24  as described above. 
     This state refers to the restoration mode as illustrated in  FIG. 2C . 
       FIG. 10  is a diagram for illustrating the flow of processing data items in the restoration mode. 
     In the restoration mode, as in the case of the low load mode, the digitized video data items accompanied by the time information items and the digitized audio data items accompanied by the time information items output from the video digitization unit  12  and the audio digitization unit  22 , respectively, are stored in the video buffer unit  13  and the audio buffer unit  23  so as to be accompanied by the time information items, while the time synchronization processing by the time synchronization processing unit  15  and the format conversion processing by the format conversion processing unit  16  are applied to the video time information file  14   a , the video data file  14   b , the audio time information file  24   a , and the audio data file  24   b  separately stored in the temporary video buffer unit  14  and the temporary audio buffer unit  24  (see  FIG. 7 ). 
     In the restoration mode, the storage destinations of the digitized video data items and the digitized audio data items output from the video digitization unit  12  and the audio digitization unit  22 , respectively, are changed to the video buffer unit  13  and the audio buffer unit  23 . Therefore, as illustrated in times t 8  and t 9  in  FIG. 11 , the storage data amounts of the temporary video buffer unit  14  and the temporary audio buffer unit  24  are gradually reduced by the processing of the storage data items of the temporary video buffer unit  14  and the temporary audio buffer unit  24  through the time synchronization processing unit  15  and the format conversion processing unit  16  of the combining unit  25 . 
     At that time, the storage data amounts of the video buffer unit  13  and the audio buffer unit  23  are gradually increased when the digitized video data items accompanied by the time information items and the digitized audio data items accompanied by the time information items output from the video digitization unit  12  and the audio digitization unit  22 , respectively, are stored as described above. 
     Referring back to  FIG. 6 , a determination is made as to whether all the storage data items of the temporary video buffer unit  14  and the temporary audio buffer unit  24  have been processed by the time synchronization processing unit  15  and the format conversion processing unit  16  of the combining unit  25  in step S 12 . 
     As described above, the storage data amounts of the temporary video buffer unit  14  and the temporary audio buffer unit  24  are gradually reduced when the storage data items of the temporary video buffer unit  14  and the temporary audio buffer unit  24  are processed by the time synchronization processing unit  15  and the format conversion processing unit  16  of the combining unit  25 . As a result, if the storage data amounts of the temporary video buffer unit  14  and the temporary audio buffer unit  24  become zero, the flow proceeds to “Yes” in step S 12 . 
     Note that the criterion of determining the flow in step S 12  is not limited to the “time at which the storage data amounts of the temporary video buffer unit  14  and the temporary audio buffer unit  24  become zero” as described above, but it may be based on the “time at which the storage data amounts of the temporary video buffer unit  14  and the temporary audio buffer unit  24  are reduced to a predetermined threshold (e.g., 80% of the storage capacity).” 
     The processing in steps S 11  and S 12  is repeatedly performed until the storage data amounts of the temporary video buffer unit  14  and the temporary audio buffer unit  24  become zero and it is determined that the flow proceeds to “Yes” in step S 12 , whereby the restoration mode is maintained. 
     If the storage data amounts of the temporary video buffer unit  14  and the temporary audio buffer unit  24  become zero and it is determined that the flow proceeds to “Yes” in step S 12 , the flow proceeds to step S 6 . Then, as illustrated in  FIG. 7 , objects to be processed by the time synchronization processing unit  15  and the format conversion processing unit  16  of the combining unit  25  are changed to the digitized video data items accompanied by the time information items and the digitized audio data items accompanied by the time information items stored in the video buffer unit  13  and the audio buffer unit  23 , respectively, as described above. 
     As a result, the information processing device returns to the low load mode based on  FIG. 8 . 
     As illustrated in  FIG. 11 , if the processing load, i.e., the data amounts per unit time of the digitized video data items and the digitized audio data items output from the video digitization unit  12  and the audio digitization unit  22 , respectively, are reduced between the high load mode at the times t 1  through t 9  and the following restoration mode, the information processing device returns to the restoration mode in  FIG. 8 . Then, as described above, the low load mode is maintained until it is determined that the flow proceeds to “Yes” in step S 2  or S 5  in accordance with the increase in the processing load. 
     On the other hand, if the processing load, i.e., the data amounts per unit time of the digitized video data items and the digitized audio data items output from the video digitization unit  12  and the audio digitization unit  22 , respectively, are not reduced but remain at high level between the high load mode at the times t 1  through t 9  and the following restoration mode, the information processing device returns to the low load mode as illustrated in  FIG. 8 . Then, when the storage data items of the video buffer unit  13  and the audio buffer unit  23  are not properly processed in time by the time synchronization processing unit  15  and the format conversion processing unit  16  of the combining unit  25 , the storage data amounts of the video buffer unit  13  and the audio buffer unit  23  exceed 80% of the storage capacity and it is determined that the flow proceeds to “Yes” in step S 2  or S 5 . Then, the information processing device changes to the high load mode in  FIG. 9 . Subsequently, the processing of “high load mode→restoration mode→low load mode→high load mode→ . . . ” is repeatedly performed until the processing load is reduced as illustrated in  FIG. 7 . 
     According to the embodiment of the present invention, the following advantages are obtained. 
     In other words, when audio information items and time information items are recorded, it is required to perform the time synchronization processing and the format conversion processing that search for digitized video data items and digitized audio data items at the same time to ensure synchronization between them so as to simultaneously record the digitized video data items and the digitized audio data items in the same file. In this case, with an improvement in the quality of videos and audios to be recorded, the time synchronization processing and the format conversion processing require a certain processing time, which may result in a limitation in processing performance. 
     According to the embodiment of the present invention, in the high load mode, the digitized video data items and the digitized audio data items are separately stored in the different files, and the time information items corresponding to the digitized video data items and the digitized audio data items are stored in the different files. In other words, the digitized video data items and the digitized audio data items are successively recorded without being subjected to the time synchronization processing by which the digitized video data items and the digitized audio data items are synchronized with each other. 
     As a result, even if the improvement in the quality of videos and audios to be recorded is achieved, it is possible to reliably record input data items regardless of the processing performance of a device related to the time synchronization processing. 
     Furthermore, according to the embodiment of the present invention, the digitized video data items and the digitized audio data items separately recorded in the high load mode are read and subjected to the time synchronization processing and the format conversion processing. As a result, it is possible to store the video/audio/time file in a specified format in a recording medium. 
       FIG. 12  is a block diagram illustrating a configuration example of a computer for describing a case in which the information processing device according to the embodiment of the present invention is implemented by the computer. 
     As illustrated in  FIG. 12 , the computer  500  has a CPU  501  that issues instructions constituting specified programs to execute various operations; an operations unit  502  that is composed of a keyboard, a mouse, and the like and through which the user inputs operating contents or data items; a display unit  503  that is composed of a CRT, a liquid crystal display, or the like and displays processing processes, processing results, or the like by the CPU  501  to the user; a memory  504  that is composed of a ROM, a RAM, and the like and that stores programs, data items, and the like executed by the CPU  501  and serves as a work area; a hard disk unit  505  that stores programs, data items, and the like; a CD-ROM drive  506  that loads programs and data items from the outside through a CD-ROM  507  serving as a medium; and a modem  508  that downloads programs from outside servers via a communication network  509  such as the Internet and a LAN. 
     In addition, the computer  500  has the camera  11 , the microphone  21 , and the video digitization unit  12  and the audio digitization unit  22  that digitize the video information items and the audio information items, respectively, output from the camera  11  and the microphone  21  to generate the digitized video data items and the digitized audio data items. 
     The computer  500  loads or downloads the program having instructions for causing the CPU  501  to execute the processing by the information processing device according to the embodiment of the present invention through the CD-ROM  507  serving as a medium or through the communication network  509  serving as a medium. Then, the computer  500  installs the loaded or downloaded program in the hard disk unit  505  and appropriately loads the same in the memory  504  so as to be executed by the CPU  501 . As a result, the computer  500  implements the information processing device according to the embodiment of the present invention. 
     All examples and conditional language recited herein are intended for pedagogical purposes to aid the reader in understanding the present invention and the concepts contributed by the inventor to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions, and the organization of such examples in the specification does not relate to a showing of the superiority or inferiority of the present invention. Although the embodiment of the present invention has been described in detail, it should be understood that the various changes, substitutions, and alterations could be made hereto without departing from the spirit and scope of the present invention.