Patent ID: 12244845

DESCRIPTION OF THE EMBODIMENTS

The present invention will now be described in detail below with reference to the accompanying drawings showing an embodiment thereof. It should be noted that the embodiment described below does not limit the claimed invention. The embodiment described below has a plurality of features, but all of the features should not always be essential for the invention, and the features can be arbitrarily combined together. Moreover, in the accompanying drawings, the same or equivalent features are designated by the same reference symbols, and duplicate descriptions thereof are omitted.

In the following description, a digital camera that classifies subjects using an inference model is taken as an example of an image processing apparatus according to the present embodiment, but the image processing apparatus is not limited to the digital camera. For example, the image processing apparatus has only to be an apparatus such as a smartphone or a tablet PC that converts an image file such as an HEIF file stored in a storage unit into an image file in another format.

Moreover, in the present embodiment described below, an HEIF image file as a first-format image file is converted to a JPEG image file as a second-format image file, but the present invention is not limited to this. The first-format image file has only to have a configuration in which it includes image data and annotation information on the image data, and the storage capacity of a predetermined area where the annotation information is stored is greater than that of the image file in the second file format, and may be an image file in another format other than HEIF. The second-format image file has only to have a configuration in which the storage capacity of a predetermined area where the annotation information is stored is less than that of the first-file image file, and may be an image file in another format other than JPEG.

FIG.1is a block diagram showing an example of the configuration of the digital camera100that is the image processing apparatus according to the present embodiment.

A barrier10is a protective member that covers an image pickup unit including a taking lens11of the digital camera100, to protect the image pickup unit from dust and scratches. The taking lens11forms an optical image on an imaging surface of an image pickup device13. A shutter12has a diaphragm function. The image pickup device13converts an optical image formed on the imaging surface to an electrical signal. The image pickup device13is comprised of, for example, a CCD or a CMOS device. An A/D converter15(A/D15) converts analog signals output from the image pickup device13to digital signals. The digital signals obtained through the conversion by the A/D converter15are written as so-called RAW image data into a memory25. Development parameters for the respective pieces of the RAW image data are generated based on information obtained during shooting, and the development parameters, as well as the RAW image data, are written into the memory25. The development parameters include parameters for use in exposure control and parameters for use in image processing, such as white balance, color space, and contrast.

A timing generating unit14supplies clock signals and control signals to the image pickup device13, the A/D converter15, and a D/A converter21(D/A21). The timing generating unit14is controlled by a memory control unit22and a system control unit50A. An image processing unit20performs various types of image processing such as predetermined image interpolation, color conversion, correction, and resizing on data output from the A/D converter15or on data output from the memory control unit22. The image processing unit20also performs predetermined image processing and arithmetic processing using captured image data and provides the system control unit50A with obtained calculation results. The system control unit50A controls, based on the provided calculation results, an exposure control unit40and a distance measurement control unit41to implement AF (Auto Focus) processing, AE (Auto Exposure) processing, and EF (pre-flashing) processing.

It should be noted that the image processing unit20performs predetermined arithmetic processing using captured image data, and performs AWB (Auto White Balance) processing based on an obtained calculation result. The image processing unit20also compresses image data stored in the memory25using methods such as JPEG, MPEG-4 AVC, or HEVC (High Efficiency Video Coding). The image processing unit20also decompresses data that has been compressed using those methods. The image processing unit20writes the processed data into the memory25.

The image processing unit20also performs predetermined arithmetic processing using captured image data and edits various types of image data. Specifically, the image processing unit20carries out a trimming process that adjusts the size and the display area of an image by hiding an unnecessary part at the edge of image data and a resizing process that changes the size of image data, screen display element, or the like, by enlarging and/or reducing them. The image processing unit20also performs, on the compressed or decompressed data, RAW development that performs image processing, such as color conversion and then performs conversion to JPEG image data. The image processing unit20also carries out a process that cuts out a designated frame in such a video format as MPEG-4 and converts it to JPEG data.

The image processing unit20also carries out, for example, a process that, when displaying OSD (On-Screen Display), such as menus and/or arbitrary characters on the display unit23together with image data, superimposes them.

The image processing unit20also carries out a process that detects a subject area in image data using input image data and/or information on the distance to a subject obtained from the image pickup device13, or the like, during shooting. By this process, the position, size, etc., of the subject in the image data are obtained.

The memory control unit22controls the A/D converter15, the timing generating unit14, the image processing unit20, an image display memory24, the D/A converter21, and the memory25. RAW image data generated through the A/D converter15is written into the image display memory24and/or the memory25via the image processing unit20and the memory control unit22, or via only the memory control unit22without via the image processing unit20.

Image data to be displayed written into the image display memory24is displayed on the display unit23via the D/A converter21. An electronic finder function of displaying live images is implemented by captured image data being successively displayed on the display unit23. The memory25stores still image data and video data that have been shot. The memory25has a storage capacity enough to store a predetermined number of still image data and/or predetermined hours of video data. The memory25is also used as a work area for the system control unit50A.

The exposure control unit40controls the shutter12having the diaphragm function. The exposure control unit40also implements a flash light modulation function by operating in conjunction with a flash44. The distance measurement control unit41controls the focusing of the taking lens11. A zoom control unit42controls the zooming of the taking lens11. A barrier control unit43controls the operation of the barrier10, which is the protective member. The flash44has an AF assist light projecting function and a flash light modulation function.

The system control unit50A controls the entire digital camera100. A nonvolatile memory51is capable of electrically erasing and recording data. For example, an EEPROM is used as the nonvolatile memory51. It should be noted that map information, or the like, as well as programs, is recorded on the nonvolatile memory51.

When the shutter button60is pressed halfway down, a shutter switch61(SW1) is turned on to issue an instruction to start AF processing, AE processing, AWB processing, EF processing, or the like. When the shutter button60is pressed all the way down, a shutter switch62(SW2) is turned on to issue an instruction to start a sequence of shooting operations, including an exposure process, a development process, and a recording process. In the exposure process, a signal read out from the image pickup device13is written as RAW data into the memory25via the A/D converter15and the memory control unit22. In the development process, the RAW data written into the memory25is developed using the results of calculation by the image processing unit20and the memory control unit22, and written as image data into the memory25. In the recording process, the image data is read out from the memory25and compressed by the image processing unit20and stored in the memory25. And then, the compressed image data is written into an external recording medium91via a card controller90.

An operating unit63is equipped with various types of buttons, a touch panel, and so forth. For example, the operating unit63includes a power button, a menu button, a mode changing switch for changing a shooting mode among a shooting mode/a playback mode/other special shooting modes, and a cross key, a set button, a macro button, and a multi-screen reproducing page-break button. The operating unit63also includes, for example, a flash setting button, a single shooting/continuous shooting/self-timer selection button, a menu move plus (+) button, a menu move minus (−) button, a shooting image quality selection button, an exposure compensation button, a date/time setting button, and so forth.

A metadata generating/analyzing unit70, when recording image data in the external recording medium91, generates metadata conforming to the Exif (Exchangeable image file format) standard, or the like, for image data based on information obtained during shooting. The metadata includes, for example, information on settings in shooting, information on the image data, and characteristic information on a subject included in the image data. The metadata generating/analyzing unit70analyzes metadata added to image data loaded from the external recording medium91. The metadata generating/analyzing unit70is also capable of generating metadata for respective frames and adding to the metadata of all the frames to video data, when recording image data.

A power supply80is comprised of a primary battery, such as an alkali battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, an AC adapter, and so forth. The power supply control unit81supplies power supplied from the power supply80to each component of the digital camera100. The card controller90sends and receives data to and from the external recording medium91, such as a memory card. The external recording medium91is comprised of, for example, a memory card and records image data (still image data, video data) shot by the digital camera100.

An inference model recording unit72A stores an inference model obtained from an external apparatus101, or the like, by a communication unit71A. An inference engine73A inputs an image file, which includes image data and annotation information on the image data and is obtained from the system control unit50A, to the inference model stored in the inference model recording unit72A, and performs inference. It should be noted that, in the present embodiment, the inference engine73A can use an inference model obtained from the external apparatus101, or the like, by the communication unit71A and recorded in the inference model recording unit72A or use an inference model obtained by a training unit74A retraining.

The inference model is comprised of an input layer, an intermediate layer (neurons), and an output layer. An image file output from the system control unit50A is input to the input layer. The intermediate layer is comprised of a plurality of layers. The number of layers in the intermediate layer is determined appropriately for design, and the number of neurons in each layer is also determined appropriately for design. The output layer outputs annotation information for the image file input to the input layer.

In the present embodiment, it is assumed that the inference model used by the inference engine73A is an inference model that infers the classification of a subject area detected from image data. This inference model is generated in the external apparatus101, or the like, by deep learning using, as training data, image data on various subjects and its classification result (e.g., the classification of animals such as dogs and cats and the classification of subjects such as humans, animals, plants, and buildings). It should be noted that the digital camera100can obtain an inference model other than the above inference model from the external apparatus101, or the like, and use the obtained another inference model.

The training unit74A retrains the inference model stored in the inference model recording unit72A (namely, updates the inference model) in accordance with an instruction from the system control unit50A. The training unit74A has a training data recording unit74a. The training data recording unit74arecords information on training data for use in retraining the inference model. The training unit74A retrains the inference model using the training data recorded in the training data recording unit74aand stores the retrained inference model in the inference model recording unit72A. As a result, the inference engine73A can perform inference using the retrained inference model. It should be noted that, in the present embodiment, a management version for managing the update status of the inference model can be held in the inference model recording unit72A, making it possible to know the update status of the inference model based on the management version.

The communication unit71A has a communication circuit for sending and receiving data. Communication by the communication unit71A can be either wireless communication using Wi-Fi (registered trademark), Bluetooth (registered trademark), or the like, or wired communication using Ethernet, USB, or the like. For example, the communication unit71A is capable of communicating with a communication unit71B of the external apparatus101. The communication unit71A functions as a communication unit that sends and receives various types of information, such as inference models and training data as well as image files, to which annotation information created by the inference engine73A is added. It should be noted that, based on whether or not the external apparatus101is a camera-related apparatus, the digital camera100can place a limit on information to be sent.

The external apparatus101has a training unit74B, an inference engine73B, an inference model recording unit72B, a system recording unit50B, and the communication unit71B. The training unit74B creates an inference model in accordance with an instruction received from the system recording unit50B, or the like. The inference model recording unit72B records an inference model sent from the digital camera100and an inference model created by the training unit74B.

In the present embodiment, the digital camera100detects a target, such as a person, an animal, a plant, a building, or an object from image data obtained by shooting. The digital camera100then generates an image file, including the image data obtained by shooting and annotation information on the image data. The digital camera100records, for example, information indicating the area of the detected target by a point, a rectangle, a circle, or the like, as the annotation information, in the image file. The digital camera100also cuts out the image data of the detected target as image data (related image data) related to original image data and records the cut-out image data (related image data) in the image file. The digital camera100also records, in the image file, the properties of the cut-out image data such as color, shape, and type. In the present embodiment, for example, when receiving an instruction to shoot HEIF image data from a user, the digital camera100generates an HEIF image file200shown inFIG.2A.

FIGS.2A to2Dare views useful in explaining the configuration of the HEIF image file200that is used in the embodiment of the present invention. As shown inFIG.2A, the HEIF image file200is comprised of a plurality of areas each called as “box” and records data in a tree structure. Data can be directly recorded in the boxes, or in a certain box, other boxes can be recorded as child elements. The HEIF image file200has an ftype box201, in which the type of an image file is recorded, a meta box202, in which metadata including annotation information and others is recorded, and an mdat box218, in which image data is recorded. It should be noted thatFIG.2Aillustrates the minimum boxes necessary to the explanation of the present invention, but the HEIF image file200may have other boxes in addition to the above-mentioned boxes.

The meta box202has an item definition box203, an item association box207, an item attribute definition box211, and an item positional information box215, as child elements.

In the item definition box203, items desired to be defined as annotation information are recorded as child elements. In the item definition box203inFIG.2A, a main image item definition204, a related image item definition205, and an area item definition206are recorded. To define another annotation information, an item desired to be defined is added to the item definition box203. A definition of an item number of image data250(FIG.2B), which is main image data in the HEIF image file200, a definition of an item type as “Image”, and so forth are recorded in the main image item definition204. A definition of an item number of related image data251(FIG.2C) cut out from the image data250, a definition of an item type as “Image”, and so forth are recorded in the related image item definition205. A definition of an item number for an area corresponding to the related image data251in the image data250, for example, an area253(FIG.2D) and a definition of an item type as “Area” are recorded in the area item definition206.

Item reference information is recorded as child elements in the item association box207. InFIG.2A, the relationship between a main image and a related image208, the relationship between the main image and attribute information on the main image209, and the relationship between the related image and the attribute information on the related image210are recorded in the item association box207. It should be noted that, to define another annotation information, an item desired to be defined is added to the item association box207. In the relationship between the main image and the related image208, the item number of the main image data and the item number of the related image data are recorded in association with each other. In the relationship between the main image and the attribute information on the main image209, the item number of the main image data and attribute information on the main image data212(described below) are recorded in association with each other. In the relationship between the related image and attribute information on the related image210, the item number of the related image data and attribute information on the related image213(described below) are recorded in association with each other.

In the item attribute definition box211, properties such as item attribute information, are recorded as child elements. InFIG.2A, main image attribute information212, related image attribute information213, and area attribute information214are recorded in the item attribute definition box211. It should be noted that, to define another annotation information, an item desired to be defined is added to the item attribute definition box211. Size information, color space information, pixel information, rotational information, and so forth, on the main image data are recorded in the main image attribute information212. Size information, color space information, pixel information, rotational information, and so forth, on the related image data are recorded in the related image attribute information213. The attribute of a detected target is also recorded in the related image attribute information213. By recording the attribute of the detected target in the related image attribute information213, for example, as shown inFIG.2D, a character string254“Flower” indicating the attribute of the related image can be displayed in the area253corresponding to the related image data. The shape, coordinates, and the like, of the area of the detected target are recorded in the area attribute information214.

In the item positional information box215, positional information on the image data in the mdat box218is recorded as child elements. InFIG.2A, main image positional information216and related image positional information217are recorded in the item positional information box215. The positional information on the main image data in the mdat box218is recorded in the main image positional information216. The positional information on the related image data in the mdat box218is recorded in the related image positional information217. Main image data219and related image data220related to the main image data219are recorded in the mdat box218. It should be noted that, image data in formats other than HEIF, can be recorded in the mdat box218.

Shooting performed by the digital camera100is now described below.FIG.3is a flowchart showing the procedure of a shooting control process that is carried out by the digital camera100inFIG.1. The shooting control process inFIG.3Ais implemented by the system control unit50A executing a program stored in the nonvolatile memory51or the like. The shooting control process inFIG.3is carried out when the user has selected the shooting mode with the mode changing switch included in the operating unit63.

First, the system control unit50A determines whether or not the user has issued an instruction to obtain HEIF image data by depressing the shutter switch61(SW1) or the shutter switch62(SW2) (step S301). In a case when it is determined in the step S301that the user has issued an instruction to obtain HEIF image data, the process proceeds to step S302. In a case when it is determined in the step S301that the user has not issued an instruction to obtain HEIF image data, the process proceeds to step S308(described later).

In the step S302, the system control unit50A carries out a shooting process. In the shooting process, the system control unit50A performs AF processing and AE processing by controlling the distance measurement control unit41and the exposure control unit40, and stores picture data (still image data, or video data) output from the image pickup device13in the memory25. The system control unit50A also controls the image processing unit20to compress the picture data (still image data, or video data), which is stored in the memory25, to generate (obtain) HEIF image data.

Next, in step S303, the system control unit50A controls the image processing unit20to carry out a subject detection process on the picture data (still image data, or video data) stored in the memory25. As a result, a subject area in the picture data (still image data, or video data) is detected.

Then, in step S304, the system control unit50A stores the HEIF image file200, which includes the image data generated in the step S302and information on the subject area detected in the step S303, in the external recording medium91. The information on the subject area is recorded as information for use in an inference process (described later), in the meta box202of the HEIF image file200, by metadata generating/analyzing unit70.

After that, in step S305, the system control unit50A determines whether or not it is possible to carry out the inference process by the inference engine73A. For example, it is determined that it is not possible to carry out the inference process using the inference engine73A in a case when at least one of the following two conditions applies: the condition that the inference engine73A is currently processing another image file, and the condition that the user is currently issuing an instruction to perform continuous shooting by keeping depressing the shutter switch62. In a case when neither of the above two conditions applies, it is determined that it is possible to carry out the inference process using the inference engine73A. In a case when it is determined that it is not possible to carry out the inference process using the inference engine73A, the process proceeds to the step S308(described later). On the other hand, in a case when it is determined that it is possible to carry out the inference process using the inference engine73A, the process proceeds to step S306.

In the step S306, the system control unit50A subjects the HEIF image file200to the inference process using the inference engine73A. The inference engine73A carries out the inference process for the HEIF image file200as an input, using an inference model recorded in the inference model recording unit72A. In this inference process, subject areas in image data are identified from image data and annotation information included in the HEIF image file200, and the results of classification for subjects corresponding to the subject areas are output as inference results for each subject area. It should be noted that, information relating to the inference processn such as debug information and logs of operations during the inference processn can be output as alternatives to the inference results or in addition to the inference results.

Then, in step S307, the system control unit50A records the classification results for the subjects obtained in the step S306as annotation information in the HEIF image file200. It should be noted that in the HEIF image file200, the results of classification for the subjects, which are output as the inference results from the inference model, have only to be recorded as annotation information. Namely, the results of classification for the subjects can be recorded as they are in the HEIF image file200, and the results of classification for the subjects (annotation information) can be recorded in any recording format, such as text format and binary format. It is easier to manage data in a case when the image data and the results of classification for the subjects are recorded in the HEIF image file200in this way, than in a case when data is individually recorded. The system control unit50A also records the management version of the inference model held in the inference model recording unit72A, debug information for the inference model, and so forth as inference model management information, in the HEIF image file200.

Then, in the step S308, the system control unit50A determines whether or not an instruction to complete shooting has been received from the user. The user is able to issue the instruction to complete shooting by, for example, setting a mode other than the shooting mode with the mode changing switch in the operating unit63or by setting the power to OFF with the power button. In a case where it is determined in the step S308that the instruction to complete shooting has not been received from the user, the process returns to the step S301. In a case where it is determined in the step S308that the instruction to complete shooting has been received from the user, the present process is ended.

As described above, in the present embodiment, the HEIF image file200including image data generated by the digital camera100shooting subjects and annotation information such as the results of classification for the subjects is generated.

Moreover, the digital camera100according to the present embodiment is also capable of generating a JPEG image file, and, for example, is capable of converting the above-mentioned HEIF image file200into a JPEG image file.

The memory capacity of an annotation information recording space in a JPEG image file is less than that of an image file in another format, such as HEIF. For this reason, in a case when an image file in HEIF or the like (pre-conversion image file) is converted into a JPEG image file (post-conversion image file), all of annotation information recorded in the pre-conversion image file cannot be recorded in the JPEG image file. As a result, a part of the annotation information recorded in the pre-conversion image file cannot be used in the post-conversion image file.

On the other hand, in the present embodiment, in a case when the annotation information included in the HEIF image file is not predetermined annotation information, whose data amount is equal to or greater than the predetermined value, the annotation information is recorded in an application marker, which will be described later. In a case when the annotation information included in the HEIF image file is the predetermined annotation information, link information on the annotation information is recorded in the application marker, which will be described later.

FIG.4is a view useful in explaining the configuration of a JPEG image file400that is used in the embodiment of the present invention. As shown inFIG.4, in the JPEG image file400, a marker SOI401indicating the start of an image file is recorded at the head of the JPEG image file400, and an application marker (denoted by “APP1” inFIG.4)402is recorded on a row subsequent to the marker SOI401.

The application marker402is an area where annotation information or the like is recorded, and has memory capacity less than that of the meta box202in the HEIF image file200. An APP1 identifier code403is recorded in the application marker402. Typical examples of the APP1 identifier code403include Exif. Various types of metadata are recorded in the application marker402. Specifically, a main image item definition404, a related image item definition405, an area item definition406, the relationship between a main image and a related image407, and the relationship between the main image and attribute information on the main image408are recorded in the application marker402. The relationship between the related image and the attribute information on the related image409, main image attribute information410, related image attribute information411, and area attribute information412are also recorded in the application marker402.

Information similar to the information in the above-described main image item definition204is recorded in the main image item definition404. Information similar to the information in the above-described related image item definition205is recorded in the related image item definition405. Information similar to the information in the above-described area item definition206is recorded in the area item definition406. Information similar to the information in the above-described relationship between the main image and the related image208is recorded in the relationship between the main image and the related image407. Information similar to the information in the above-described relationship between the main image and the attribute information on the main image209is recorded in the relationship between the main image and the attribute information on the main image408. Information similar to the information in the above-described relationship between the related image and the attribute information on the related image210is recorded in the relationship between the related image and the attribute information on the related image409. Information similar to the information in the above-described main image attribute information212is recorded in the main image attribute information410. Information similar to the information in the above-described related image attribute information213is recorded in the related image attribute information411. Information similar to the information in the above-described area attribute information214is recorded in the area attribute information412.

An application marker (denoted by “APP2” inFIG.4)413is recorded on a row subsequent to the area attribute information412. As with the application marker402, the application marker413is an area where annotation information, or the like, is recorded, and has a memory capacity less than that of the meta box202in the HEIF image file200. An APP2 identifier code414is recorded in the application marker413. Link information on related image data is recorded as metadata in the application marker413. Specifically, related image positional information415is recorded in the application marker413. The size of related image data419(described later), and the location of a marker SOI418indicating the start of the related image data419are recorded in the related image positional information415.

Main image data416is recorded in a row subsequent to the related image positional information415. The main image data416is main image data in the JPEG image file400. A marker EOI417indicating the end of main image data is recorded in a row subsequent to the main image data416. A marker SOI418indicating the start of related image data, and the related image data419, are recorded in this order in rows subsequent to the marker EOI417. A marker EOI420indicating the end of related image data is recorded on a row subsequent to the related image data419.

FIG.5Ais a flowchart showing the procedure of a format conversion process that is carried out in the digital camera100inFIG.1. The format conversion process inFIG.5Ais implemented by the system control unit50A executing a program stored in the nonvolatile memory51, or the like. The format conversion process inFIG.5Ais carried out when the digital camera100has received an image conversion instruction from the user via the operating unit63. The user selects at least one image file to be processed from a plurality of image files stored in the external recording medium91and issues an image conversion instruction. Referring toFIG.5A, a case, as an example, in which the user selects one HEIF image file and issues an image conversion instruction to convert the selected HEIF image file to a JPEG image file, is described.

First, in step S500, the system control unit50A carries out a metadata loading process to load metadata from the HEIF image file selected by the user. Specifically, the system control unit50A reads information recorded in each of the ftype box201and the meta box202in the HEIF image file, and writes the read information into the memory25.

Next, in step S501, the system control unit50A carries out a metadata analyzing process to analyze the metadata for the HEIF image file. In the metadata analyzing process, the system control unit50A analyzes the information in the ftype box201and the information in the meta box202written into the memory25in the step S500, and extracts the metadata recorded in the HEIF image file. As a result of the metadata analyzing process, the main image positional information216and the related image positional information217recorded in the HEIF image file are obtained.

Then, in step S502, the system control unit50A carries out a main image data loading process to load main image data recorded in the HEIF image file. Specifically, the system control unit50A reads the main image data219from the HEIF image file using the main image positional information216obtained by the metadata analyzing process and writes the read main image data219into the memory25.

After that, in step S503, the system control unit50A carries out a format conversion process for the main image data219. Specifically, the system control unit50A decodes the main image data219written into the memory25in the step S502. The system control unit50A also encodes the decoded main image data219into the main image data416to be recorded in the JPEG image file, using a conversion engine (not shown).

Then, in step S504, the system control unit50A carries out a related image data loading process to load related image data recorded in the HEIF image file. Specifically, the system control unit50A reads the related image data220from the HEIF image file using the related image positional information217obtained by the metadata analyzing process and writes the read related image data220into the memory25.

After that, in step S505, the system control unit50A carries out a format conversion process for the related image data220. Specifically, the system control unit50A decodes the related image data220written into the memory25in the step S504. The system control unit50A also encodes the decoded related image data220into the related image data419to be recorded in the JPEG image file, using the conversion engine (not shown).

Then, in step S506, the system control unit50A carries out a header creating process inFIG.5B, to be described later, to create a header of the JPEG image file, which is a post-conversion image file. After that, in step S507, the system control unit50A carries out a JPEG image file storage process inFIG.5Cto be described later, to store the JPEG image file created by conversion from the HEIF image file, in the external recording medium91and ends the present process.

FIG.5Bis a flowchart showing the procedure of the header creating process in the step S506inFIG.5A.

Referring toFIG.5B, in step S520, the system control unit50A carries out a first storage area reservation process. As a result of the first storage area reservation process, a first storage area for use in creating the header of the JPEG image file, which is the post-conversion image file, is reserved in the memory25.

In step S521, the system control unit50A carries out a metadata obtaining process to obtain metadata for the HEIF image file. Specifically, the system control unit50A reads out the information in the ftype box201and the information in the meta box202written in the memory25in the step S500, from the memory25.

Then, in step S522, the system control unit50A carries out a first application marker creating process. As a result of the first application marker creating process, the application marker402in the JPEG image file is created. Here, APP1 is created as an example of the application marker402. It should be noted that the application marker402to be created in the step S522is not limited to APP1, but another application marker corresponding to an identifier code created in step S523to be described later, can be created.

After that, in the step S523, the system control unit50A carries out a first identifier code creating process. As a result of the first identifier code creating process, an APP1 identifier code403(an identifier code for APP1) in the JPEG image file is created. Here, Exif is created as an example of the APP1 identifier code403. It should be noted that the APP1 identifier code403to be created in the step S523is not limited to Exif, but another identifier code can be created.

Then, in step S524, the system control unit50A determines whether or not the metadata obtained in the step S521includes annotation information. In a case when it is determined, in the step S524, that the metadata obtained, in the step S521, includes annotation information, the process proceeds to step S525. In a case when it is determined in the step S524that the metadata obtained in the step S521includes no annotation information, the process proceeds to step S535to be described later.

In the step S525, the system control unit50A starts an annotation information generating process to generate annotation information to be recorded in the JPEG image file. In the annotation information generating process, processes in steps S526to S533are carried out. The annotation information generating process is carried out for all the annotation information recorded in the HEIF image file.

In the step S526, the system control unit50A obtains annotation information from the metadata obtained in the step S521. Then, in the step S527, the system control unit50A determines whether or not the obtained annotation information is the predetermined annotation information whose data amount is equal to or greater than the predetermined value. Specifically, the system control unit50A determines whether or not the obtained annotation information is the related image data. More specifically, based on, for example, the information in the item definition box203, in the metadata obtained in the step S521, the system control unit50A determines whether or not the obtained annotation information is the related image data.

In a case when it is determined in the step S527that the obtained annotation information is not the related image data, the process proceeds to the step S528. In the step S528, the system control unit50A carries out a first generating process. In the first generating process, the obtained annotation information, that is, the annotation information, which is not the related image data, is generated as it is (without being processed), as annotation information to be recorded in the JPEG image file400. For example, in a case when the obtained annotation information is the area attribute information214, the system control unit50A stores the area attribute information214as it is (without processing) in the first storage area. This annotation information is recorded as the area attribute information214, in the application marker402in the JPEG image file400, by the JPEG image file storage process inFIG.5C, which will be described later. Thus, in the present embodiment, annotation information other than the predetermined annotation information, whose data amount is equal to or greater than the predetermined value, out of the annotation information in the HEIF image file, is recorded as it is (without being processed) in the application marker in the JPEG image file. The process then proceeds to step S534described later.

In a case when it is determined in the step S527that the obtained annotation information is the related image data, the process proceeds to the step S529. In the step S529, the system control unit50A carries out a second generating process. In the second generating process, annotation information that is annotation information to be recorded in the application marker402in the JPEG image file and is related to the related image data is generated. For example, annotation information corresponding to the relationship between the related image and the attribute information on the related image409, and annotation information corresponding to the related image attribute information411, are generated. The generated annotation information is stored in the first storage area reserved in the step S520.

Then, in the step S530, the system control unit50A carries out a second storage area reservation process. As a result of the second storage area reservation process, a second storage area different from the first storage area is reserved in the memory25. The second storage area is a storage area for use in creating annotation information to be recorded in the application marker413in the JPEG image file.

Then, in the step S531, the system control unit50A carries out a second application marker creating process. As a result of the second application marker creating process, the application marker413in the JPEG image file is created. Here, APP2 is created as an example of the application marker413. It should be noted that the application marker413to be created in the step S531is not limited to APP2, but another application marker, corresponding to an identifier code created in the step S532described later, can be created.

After that, in the step S532, the system control unit50A carries out a second identifier code creating process. As a result of the second identifier code creating process, the APP2 identifier code414(an identifier code for APP2) in the JPEG image file is created.

Then, in the step S533, the system control unit50A carries out a third generating process. In the third generating process, link information on the related image data to be recorded in the application marker413in the JPEG image file is generated. For example, link information corresponding to the related image positional information415is generated. The generated link information on the related image data is stored in the second storage area reserved in the step S530. The various types of information generated in the step S529and S533are recorded in the application markers402and413in the JPEG image file by the JPEG image file storage process inFIG.5Cdescribed later. Thus, in the present embodiment, the predetermined annotation information, whose data amount is equal to or greater than the predetermined value, out of the annotation information in the HEIF image file, is not recorded in the application markers in the JPEG image file, but link information on the annotation information is recorded. It should be noted, that in a case when positional information on the related image data is determined in the JPEG image file storage process inFIG.5Cdescribed later, the link information corresponding to the related image positional information415can be updated at a timing when the positional information is determined. The process then proceeds to the step S534.

Upon completing the processing on all the annotation information recorded in the HEIF image file, the system control unit50A ends the annotation information generating process in the step S534.

Then, in the step S535, the system control unit50A generates metadata other than annotation information. For example, in a case when the APP1 identifier code403is Exif, the system control unit50A generates, in the step S535, metadata such as the date and time of shooting and shooting conditions. After that, the present process is ended.

FIG.5Cis a flowchart showing the procedure of the JPEG image file storage process in the step S507inFIG.5A.

In step S540, the system control unit50A writes the information on the header of the JPEG image file, which was generated in the above-described header creating process, for example, the information stored in the first storage area into the external recording medium91. Next, in step S541, the system control unit50A writes the information to be recorded in the application marker413(APP2), which was generated in the above-described header creating process, for example, the information stored in the second storage area, into the external recording medium91. Then, in step S542, the system control unit50A writes the main image data416in the JPEG image file400into the external recording medium91. After that, in step S543, the system control unit50A writes the related image data419in the JPEG image file400into the external recording medium91. Then, in step S544, the system control unit50A writes the other data into the external recording medium91. After that, the present process is ended.

According to the embodiment described above, in a case when the annotation information included in the HEIF image file is not predetermined annotation information whose data amount is equal to or greater than the predetermined value, the annotation information is recorded in the application marker in the JPEG image file. In a case when the annotation information in the HEIF image file is the predetermined annotation information, link information on the annotation information is recorded in the application marker in the JPEG file. It is thus possible to provide a post-conversion image file in which all of annotation information recorded in a pre-conversion image file can be used.

Moreover, in the embodiment described above, the predetermined annotation information is the related image data, and thus it is possible to provide a post-conversion image file in which related image data whose data amount is relatively large can be used.

Furthermore, in the embodiment described above, the related image data419is recorded in an area in the JPEG image file400, which is different from area for the application markers402,413. Thus, it is possible to read out the related image data419from a post-conversion image file.

Although the present invention has been described by way of the embodiment, the present invention should not be limited to the embodiment described above. For example, as shown inFIGS.6A and6B, related image data620can be recorded in another image file618different from a post-conversion JPEG image file600. In this case, for example, a file name of related image615is recorded as link information on the related image data620, in an application marker613in the post-conversion JPEG image file600. The file name of related image615is a file name of the image file618in which the related image data620is recorded. The configuration of the JPEG image file600differs from that of the JPEG image file400inFIG.4described above in that, in the JPEG image file600, the file name of related image615is recorded, instead of the related image positional information415, and the related image data419is not recorded. Otherwise, the configuration of the JPEG image file600is basically the same as that of the JPEG image file400inFIG.4described above.

FIG.7is a flowchart showing another procedure of the header creating process inFIG.5B. It should be noted that the header creating process inFIG.7is similar to the header creating process inFIG.5B, and description will be given below particularly of features different from those of the header creating process inFIG.5B.

Referring toFIG.7, processes in steps S700to S707that are similar to the processes in the steps S520to S527described above are carried out.

In a case when it is determined in the step S707that the obtained annotation information is not the related image data, the process proceeds to step S708, which is similar to the process in the step S528described above. Then, processes in steps S714and S715, which are similar to the processes in the steps S534and S535described above, are carried out, and the present process is ended.

In a case when it is determined in the step S707that the obtained annotation information is the related image data, the process proceeds to step S709, which is similar to the process in the step S529, described above. Then, processes in steps S710and S712, which are similar to the processes in the steps S530to S532described above, are carried out. Then, in step S713, the system control unit50A carries out a fourth generating process. In the fourth generating process, link information corresponding to the file name of related image615is generated as link information on the related image data. (In the third generating process, link information corresponding to the related image positional information415is generated as link information on the related image data as described above.) It should be noted that, in the present embodiment, the link information generated in the step S713has only to be information that can identify the related image data, and should not necessarily be the file name of an image file in which the related image data is recorded. For example, the link information generated in the step S713may be an ID, or the like, for identifying the related image data. The generated link information is stored in the second storage area reserved in the step S710. After that, processes in the steps S714and S715that are similar to the processes in the steps S534and S535described above are carried out, and the present process is ended.

FIG.8is a flowchart showing another procedure of the JPEG image file storage process inFIG.5C. It should be noted that the JPEG image file storage process inFIG.8is similar to the JPEG image file storage process inFIG.5C, and a description will be given below particularly of features different from those of the JPEG image file storage process inFIG.5C.

Referring toFIG.8, processes in steps S800to S802that are similar to the processes in the steps S540to S542described above are carried out. Next, a process in step S803that is similar to the process in the step S544described above is carried out. Then, in the step S804, the system control unit50A writes the related image data620as another image file618different from the JPEG image file600into the external recording medium91. After that, the present process is ended.

Thus, also in the configuration in which the related image data is recorded in another image file different from a post-conversion image file, the related image data recorded in a pre-conversion image file can be used in the post-conversion image file.

It should be noted that, although in the embodiment described above, the related image data is treated as the predetermined annotation information whose data amount is equal to or greater than a predetermined value, the predetermined annotation information is not limited to the related image data. For example, the predetermined annotation information can be video data and/or audio data that are not allowed to be recorded in image files according to the JPEG standard.

In a case when the predetermined annotation information is video data and/or audio data, the predetermined annotation information is not recorded in a post-conversion image file but is recorded in another image file different from the post-conversion image file. Link information on the predetermined annotation information is recorded in an application marker in the post-conversion image file. The link information on the predetermined annotation information is a file name of the image file in which the predetermined annotation information is recorded, an ID for identifying the predetermined annotation information, or the like. Thus, even in a case when the predetermined annotation information is video data and/or audio data, similar effects to those in the embodiment described above can be achieved.

OTHER EMBODIMENTS

Embodiment(s) of the present invention can also be realized by a computer of a system or an apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., an application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer may comprise one or more processors (e.g., a central processing unit (CPU), or a micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and to execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), a digital versatile disc (DVD), or a Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like.

While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.