IMAGE PROCESSING APPARATUS, IMAGE CAPTURING APPARATUS, IMAGE PROCESSING METHOD, AND STORAGE MEDIUM

There is provided an image processing apparatus. An obtainment unit obtains a first image, first subject information indicating a first subject detected from the first image, a second image, and second subject information indicating a second subject detected from the second image. A composition unit generates a composite image by compositing the first image and the second image. A recording unit records one of the first subject information and the second subject information in association with the composite image in a case where a similarity between the first subject information and the second subject information satisfies a predetermined criterion, and records both of the first subject information and the second subject information in association with the composite image in a case where the similarity between the first subject information and the second subject information does not satisfy the predetermined criterion.

BACKGROUND

Field of the Disclosure

The present disclosure relates to an image processing apparatus, an image capturing apparatus, an image processing method, and a storage medium.

Description of the Related Art

In recent years, artificial intelligence (AI) techniques, such as deep learning, have been utilized in a variety of technical fields. For example, conventionally, digital still cameras and the like are known to have a function to detect a human face from a shot image. Also, Japanese Patent Laid-Open No. 2015-099559 discloses a technique to accurately detect and recognize animals, such as dogs and cats, without limiting a detection target to humans.

Furthermore, there is a known technique whereby a composite image is generated by compositing a plurality of material images, such as HDR composition and multiple composition. In connection to this technique, Japanese Patent Laid-Open No. 2019-009577 discloses that only shooting information of an image including a main subject (a material image) is added to a post-composition image and recorded.

Assume a case where subject information pieces of material images are estimated and recorded using AI techniques and the like, and a composite image is generated by compositing a plurality of material images. In this case, if all subject information pieces of each material image are unconditionally associated with the composite image, there is a possibility that the subject information pieces associated with the composite image will become redundant and the usability of the subject information pieces will decrease, depending on the similarity among the subject information pieces of the material images. However, conventional techniques cannot address this problem.

SUMMARY

The present disclosure has been made in view of the foregoing situation, and provides a technique to associate subject information pieces of each material image with a composite image so as to reduce redundancy of the subject information pieces associated with the composite image.

According to a first aspect of the present disclosure, there is provided an image processing apparatus, comprising: an obtainment unit configured to obtain a first image, first subject information indicating a first subject detected from the first image, a second image, and second subject information indicating a second subject detected from the second image; a composition unit configured to generate a composite image by compositing the first image and the second image; and a recording unit configured to record one of the first subject information and the second subject information in association with the composite image in a case where a similarity between the first subject information and the second subject information satisfies a predetermined criterion, and record both of the first subject information and the second subject information in association with the composite image in a case where the similarity between the first subject information and the second subject information does not satisfy the predetermined criterion.

According to a second aspect of the present disclosure, there is provided an image capturing apparatus, comprising: the image processing apparatus according to the first aspect; and an image capturing unit configured to generate the first image and the second image, wherein the obtainment unit obtains the first image and the second image generated by the image capturing unit.

According to a third aspect of the present disclosure, there is provided an image processing method executed by an image processing apparatus, comprising: obtaining a first image, first subject information indicating a first subject detected from the first image, a second image, and second subject information indicating a second subject detected from the second image; generating a composite image by compositing the first image and the second image; and recording one of the first subject information and the second subject information in association with the composite image in a case where a similarity between the first subject information and the second subject information satisfies a predetermined criterion, and recording both of the first subject information and the second subject information in association with the composite image in a case where the similarity between the first subject information and the second subject information does not satisfy the predetermined criterion.

According to a fourth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium which stores a program for causing a computer to execute an image processing method comprising: obtaining a first image, first subject information indicating a first subject detected from the first image, a second image, and second subject information indicating a second subject detected from the second image; generating a composite image by compositing the first image and the second image; and recording one of the first subject information and the second subject information in association with the composite image in a case where a similarity between the first subject information and the second subject information satisfies a predetermined criterion, and recording both of the first subject information and the second subject information in association with the composite image in a case where the similarity between the first subject information and the second subject information does not satisfy the predetermined criterion.

DESCRIPTION OF THE EMBODIMENTS

Furthermore, the following description exemplarily presents a digital camera (an image capturing apparatus) as an image processing apparatus that performs subject classification with use of an inference model. However, in the following embodiments, the image processing apparatus is not limited to a digital camera. The image processing apparatus according to the following embodiments may be any apparatus as long as it is an apparatus that has digital camera functions to be described below, and may be, for example, a smartphone, a tablet PC, or the like.

First Embodiment

Configuration of Digital Camera100

FIG.1is a block diagram showing an exemplary configuration of a digital camera100. A barrier10is a protection member that covers an image capturing unit of the digital camera100, including a photographing lens11, thereby preventing the image capturing unit from being stained or damaged. The operations of the barrier10are controlled by a barrier control unit43. The photographing lens11causes an optical image to be formed on an image capturing surface of an image sensor13. A shutter12has a diaphragm function. The image sensor13is composed of, for example, a CCD or CMOS sensor or the like, and converts the optical image that has been formed on the image capturing surface by the photographing lens11via the shutter12into electrical signals.

An A/D converter15converts analog image signals output from the image sensor13into digital image signals. The digital image signals converted by the A/D converter15are written to a memory25as so-called RAW image data pieces. In addition to this, development parameters corresponding to respective RAW image data pieces are generated based on information at the time of shooting, and written to the memory25. Development parameters are composed of various types of parameters that are used in image processing for recording images using a JPEG method or the like, such as an exposure setting, white balance, color space, and contrast.

A timing generation unit14is controlled by a memory control unit22and a system control unit50, and supplies clock signals and control signals to the image sensor13, the A/D converter15, and a D/A converter21.

An image processing unit20executes various types of image processing, such as predetermined pixel interpolation processing, color conversion processing, correction processing, resize processing, and image composition processing, with respect to data from the A/D converter15or data from the memory control unit22. Also, the image processing unit20executes predetermined image processing and computation processing with use of image data obtained through image capture, and provides the obtained computation result to the system control unit50. The system control unit50realizes AF (autofocus) processing, AE (automatic exposure) processing, and EF (preliminary flash emission) processing by controlling an exposure control unit40and a focus control unit41based on the provided computation result.

In the present embodiment, the system control unit50can perform shooting under two exposure settings that differ from each other in exposure. The first exposure setting is “appropriate exposure setting”, which allows an appropriately-exposed image to be obtained as a result of the system control unit50providing the exposure control unit40with a feedback of the result of AE (automatic exposure) processing. The second exposure setting is “underexposure setting”, which allows an image with a dark exposure to be obtained as a result of the system control unit50providing the exposure control unit40with a feedback of the result of the AE (automatic exposure) processing to which an offset for reducing exposure has been given.

Furthermore, the image processing unit20executes predetermined computation processing with use of image data obtained through image capture, and also executes AWB (auto white balance) processing based on the obtained computation result. In addition, the image processing unit20reads in image data stored in the memory25, and executes compression processing or decompression processing with use of such methods as a JPEG method, an MPEG-4 AVC method, an HEVC (High Efficiency Video Coding) method, and a lossless compression method for uncompressed RAW data. Then, the image processing unit20writes the image data for which processing has been completed to the memory25.

Also, the image processing unit20executes predetermined computation processing with use of image data obtained through image capture, and executes editing processing with respect to various types of image data. For example, the image processing unit20can execute trimming processing in which the display range and size of an image is adjusted by causing unnecessary portions around image data not to be displayed, and resize processing in which the size is changed by enlarging or reducing image data, display elements of a screen, and the like. Furthermore, the image processing unit20can execute RAW development whereby image data is generated by applying image processing, such as color conversion, to data that has undergone compression processing or decompression processing with use of a lossless compression method for uncompressed RAW data, and converting the resultant data into a JPEG format. Moreover, the image processing unit20can execute moving image cutout processing in which a designated frame of a moving image format, such as MPEG-4, is cut out, converted into a JPEG format, and stored.

Also, the image processing unit20includes a composition processing circuit that composites a plurality of image data pieces. The image processing unit20can execute addition composition processing, weighted addition composition processing, and region designation composition processing. The region designation composition processing is processing to designate a region to be used for composition on a per-material-image basis and composite the designated region of each material image.

Furthermore, the image processing unit20also executes, for example, processing for causing OSD (On-Screen Display), such as a menu to be displayed on a display unit23and no particular characters, to be superimposed on image data to be displayed.

In addition, the image processing unit20executes subject detection processing for detecting a subject that exists within image data and detecting a subject region thereof with use of, for example, input image data and information of a distance to the subject at the time of shooting, which is obtained from, for example, the image sensor13. Examples of detectable information (subject detection information) include information of the position, size, inclination, and the like of a subject region within an image, and information indicating certainty.

The memory control unit22controls the A/D converter15, the timing generation unit14, the image processing unit20, an image display memory24, the D/A converter21, and the memory25. RAW image data generated by the A/D converter15is written to the image display memory24or the memory25via the image processing unit20and the memory control unit22, or directly via the memory control unit22.

Image data for display that has been written to the image display memory24is displayed on the display unit23, which is composed of a TFT LCD or the like, via the D/A converter21. An electronic viewfinder function for displaying live images can be realized by sequentially displaying image data pieces obtained through image capture with use of the display unit23.

The memory25has a storage capacity that is sufficient to store a predetermined number of still images and moving images of a predetermined length of time, and stores still images and moving images that have been shot. Furthermore, the memory25can also be used as a working area for the system control unit50.

The exposure control unit40controls the shutter12, which has a diaphragm function. Furthermore, the exposure control unit40also exerts a flash light adjustment function by operating in coordination with a flash44. The focus control unit41performs focus adjustment by driving a non-illustrated focus lens included in the photographing lens11based on an instruction from the system control unit50. A zoom control unit42controls zooming by driving a non-illustrated zoom lens included in the photographing lens11. The flash44has a function of emitting AF auxiliary light, and a flash light adjustment function.

The system control unit50controls the entirety of the digital camera100. A non-volatile memory51is an electrically erasable and recordable non-volatile memory; for example, an EEPROM or the like is used thereas. Note that not only programs, but also map information and the like are recorded in the non-volatile memory51.

A shutter switch61(SW1) is turned ON and issues an instruction for starting operations of AF processing, AE processing, AWB processing, EF processing, and the like in the midst of an operation on a shutter button60. A shutter switch62(SW2) is turned ON and issues an instruction for starting a series of shooting operations, including exposure processing, development processing, and recording processing, upon completion of the operation on the shutter button60. In the exposure processing, signals that have been read out from the image sensor13are written to the memory25as RAW image data via the A/D converter15and the memory control unit22. In the development processing, the image processing unit20and the memory control unit22perform computation to develop RAW image data that has been written to the memory25and write the same to the memory25as image data. In the recording processing, image data is read out from the memory25, the image data is compressed by the image processing unit20, the compressed image data is stored to the memory25, and then the stored image data is written to an external recording medium91via a card controller90.

An operation unit63includes such operation members as various types of buttons and a touchscreen. For example, the operation unit63includes a power button, a menu button, a mode changing switch for switching among a shooting mode, a reproduction mode, and other special shooting modes, directional keys, a set button, a macro button, and a multi-screen reproduction page break button. Also, for example, the operation unit63includes a flash setting button, a button for switching among single shooting, continuous shooting, and self-timer, a menu change+(plus) button, a menu change—(minus) button, a shooting image quality selection button, an exposure correction button, a date/time setting button, and so forth.

When image data is to be recorded in the external recording medium91, a metadata generation and analysis unit70generates various types of metadata, such as information of the Exif (Exchangeable image file format) standard to be attached to the image data, based on information at the time of shooting. Also, when image data recorded in the external recording medium91has been read in, the metadata generation and analysis unit70analyzes metadata added to the image data. Examples of metadata include shooting setting information at the time of shooting, image data information related to image data, feature information of a subject included in image data, and so forth. Furthermore, when moving image data is to be recorded, the metadata generation and analysis unit70can also generate and add metadata with respect to each frame.

A power source80includes, for example, a primary battery such as an alkaline battery and a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, and a Li battery, or an AC adapter. A power source control unit81supplies power supplied from the power source80to each component of the digital camera100.

The card controller90transmits/receives data to/from the external recording medium91, such as a memory card. The external recording medium91is composed of, for example, a memory card, and images (still images and moving images) shot by the digital camera100are recorded therein.

Using an inference model recorded in an inference model recording unit72, an inference engine73performs inference with respect to image data that has been input via the system control unit50. The system control unit50can record an inference model that has been input from an external apparatus (not shown) via a communication unit71in the inference model recording unit72. Also, the system control unit50can record, in the inference model recording unit72, an inference model that has been obtained by re-training the inference model with use of a training unit74. Note, there is a possibility that an inference model recorded in the inference model recording unit72is updated due to inputting of an inference model from an external apparatus, or re-training of an inference model with use of the training unit74. For this reason, the inference model recording unit72holds version information so that the version of an inference model can be identified.

Also, the inference engine73includes a neural network design73a. The neural network design73ais configured in such a manner that intermediate layers (neurons) are arranged between an input layer and an output layer. The system control unit50inputs image data to the input layer. Neurons in several layers are arranged as the intermediate layers. The number of layers of neurons is determined as appropriate in terms of design. Furthermore, the number of neurons in each layer is also determined as appropriate in terms of design. In the intermediate layers, weighting is performed based on an inference model recorded in the inference model recording unit72. An inference result corresponding to the image data input to the input layer is output to the output layer.

It is assumed that, in the present embodiment, an inference model recorded in the inference model recording unit72is an inference model that infers classification, that is to say, what kind of subject is included in an image. An inference model is used that has been generated through deep learning while using image data pieces of various subjects, as well as the result of classification thereof (e.g., classification of animals such as dogs and cats, classification of subject types such as humans, animals, plants, and buildings, and so forth), as supervisory data. Therefore, when an image has been input, together with information indicating a region of a subject that has been detected in this image, to the inference engine73that uses the inference model, an inference result indicating classification (type) of this subject is output.

Upon receiving a request from the system control unit50or the like, the training unit74re-trains an inference model. The training unit74includes a supervisory data recording unit74a. Information related to supervisory data for the inference engine73is recorded in the supervisory data recording unit74a. The training unit74can cause the inference engine73to be re-trained with use of the supervisory data recorded in the supervisory data recording unit74a, and update the inference engine73with use of the inference model recording unit72.

The communication unit71includes a communication circuit for performing transmission and reception. Communication performed by the communication circuit specifically may be wireless communication via Wi-Fi, Bluetooth®, or the like, or may be wired communication via Ethernet, a USB, or the like.

HDR Shooting Processing and HDR Composition Processing

Next, HDR shooting processing and HDR composition processing executed by the digital camera100will be described with reference toFIG.2toFIG.6.FIG.2is a flowchart of the HDR shooting processing executed by the digital camera100. Processing of each step in the present flowchart is realized by the system control unit50of the digital camera100controlling respective constituent elements of the digital camera100in accordance with a program, unless specifically stated otherwise. When the operation mode of the digital camera100has been set to an HDR shooting mode, the HDR shooting processing of the present flowchart is started. Note that a user can set the operation mode of the digital camera100to the HDR shooting mode by causing a menu screen to be displayed on the display unit23via an operation on the operation unit63and selecting the HDR shooting mode on the menu screen.

In step S202, the system control unit50determines whether the user has issued a shooting instruction. The user can issue the shooting instruction by depressing the shutter button60, thereby turning ON the shutter switches61(SW1) and 62 (SW2). The system control unit50repeats determination processing in step S202until the user issues the shooting instruction. Once the user has issued the shooting instruction, processing steps proceed to step S203.

In step S203, the system control unit50executes shooting processing under the appropriate exposure setting (appropriate shooting processing). In the appropriate shooting processing, the system control unit50executes AF (autofocus) processing and AE (automatic exposure) processing using the focus control unit41and the exposure control unit40, and then stores image signals that are output from the image sensor13via the A/D converter15into the memory25. At this time, the system control unit50performs control to obtain an appropriately-exposed image by providing the exposure control unit40with a feedback of the result of the AE (automatic exposure) processing under the appropriate exposure setting. Also, the image processing unit20generates image data of a format conforming to a user setting (e.g., a JPEG format) by executing compression processing conforming to the user setting with respect to the image signals stored in the memory25, and stores the generated image data into the memory25.

In step S204, the image processing unit20executes subject detection processing with respect to the image signals stored in the memory25, and obtains information of subjects included in the image (subject detection information).

In step S205, with use of the inference engine73, the system control unit50executes inference processing with respect to the subjects that were detected from the image signals (material image) stored in the memory25. The system control unit50specifies subject regions within the image based on the image signals stored in the memory25and on the subject detection information obtained in step S204. The system control unit50inputs the image signals (material image), as well as information indicating the subject regions in the material image, to the inference engine73. An inference result indicating classification (type) of the subjects included in the subject regions is output as the result of execution of the inference processing by the inference engine73for each subject region. Note that the inference engine73may output information related to the inference processing, such as debug information and logs associated with the operations of the inference processing, in addition to the inference result.

In step S206, the system control unit50records a file including the image data generated in step S203, the subject detection information obtained in step S204, and the inference result obtained in step S205as a material image file for HDR composition into the external recording medium91.

FIG.3is a diagram showing an exemplary configuration of a material image file. As shown inFIG.3, a material image file300is divided into a plurality of storage regions, and includes an Exif region301for storing metadata conforming to the Exif standard, as well as an image data region308in which compressed image data is recorded. Furthermore, the material image file300also includes an annotation information region310in which annotation information is recorded. In a case where the material image file300is a file of a JPEG format, each of the plurality of storage regions is defined by a marker. For example, in a case where the user has issued an instruction for recording images in the JPEG format, the material image file300is recorded in the JPEG format. In this case, the image data generated in step S203is recorded in the image data region308in the JPEG format, and information of the Exif region301is recorded in a region defined by, for example, an APP1 marker or the like. Also, information of the annotation information region310is recorded in a region defined by, for example, an APP11 marker or the like. In a case where the user has issued an instruction for recording images in an HEIF (High Efficiency Image File Format) format, the material image file300is recorded in an HEIF file format. In this case, information of the Exif region301and the annotation information region310is recorded in, for example, a Metadata Box. Also in a case where the user has issued an instruction for recording images in a RAW format, information of the Exif region301and the annotation information region310is similarly recorded in a predetermined region, such as a Metadata Box.

The metadata generation and analysis unit70records the subject detection information obtained in step S204into a subject detection information tag306within a MakerNote305(a region in which metadata unique to a maker can be described in a basically-undisclosed form) included in the Exif region301. Also, in a case where there are version information of the current inference model recorded in the inference model recording unit72, debug information output from the inference engine73in step S205, and so forth, these pieces of information are recorded inside the MakerNote305as inference model management information307.

The inference result obtained in step S205is recorded in the annotation information region310as annotation information. The location of the annotation information region310is indicated by an annotation information link303included in an annotation link information storage tag302. In the present embodiment, it is assumed that annotation information is described in a text format, such as XML and JSON.

Returning toFIG.2, in step S207, the system control unit50executes shooting processing under the underexposure setting (under shooting processing). In the under shooting processing, the system control unit50executes processing similar to that of step S203, but performs control to obtain an image with a dark exposure by providing the exposure control unit40with a feedback of the result of the AE (automatic exposure) processing under the underexposure setting. Also, similarly to step S203, the system control unit50generates image data of a format conforming to a user setting (e.g., a JPEG format), and stores the generated image data into the memory25.

In step S208, by way of processing similar to that of step S204, the image processing unit20obtains information of subjects included in the image obtained through the under shooting processing (subject detection information).

In step S209, by way of processing similar to that of step S205, the system control unit50obtains inference results related to the subjects included in the image signals (material images) obtained through the under shooting processing.

In step S210, similarly to step S206, the system control unit50records a file including the image data generated in step S207, the subject detection information obtained in step S208, and the inference results obtained in step S209, as a material image file for HDR composition, into the external recording medium91.

Thereafter, the processing step returns to step S202; when the next shooting instruction is issued, the system control unit50executes processing of step S203onward again.

With reference toFIG.5, examples of material images obtained through the HDR shooting processing ofFIG.2will be described. A material image501is an image generated through the appropriate shooting processing. Inference results corresponding to a mountain504, the sky507, a sloped surface510, and a cloud513have been obtained through the inference processing for the material image501. A material image502is an image generated through the under shooting processing. Inference results corresponding to a mountain505, the sky508, a sloped surface511, and a cloud514have been obtained through the inference processing for the material image502. The inference results for to the material image501are recorded in a material image file as annotation information601shown inFIG.6, and the inference results for the material image502are recorded in a material image file as annotation information602shown inFIG.6. As shown inFIG.6, annotation information includes, for each subject, an inference result including information indicating a region and a type of the subject (subject information indicating the subject).

Next, with reference toFIG.4, the HDR composition processing will be described. Processing of each step in the present flowchart is realized by the system control unit50of the digital camera100controlling respective constituent elements of the digital camera100in accordance with a program, unless specifically stated otherwise. When the operation mode of the digital camera100has been set to an HDR composition mode, the HDR composition processing of the present flowchart is started. Note that a user can set the operation mode of the digital camera100to the HDR composition mode by causing a menu screen to be displayed on the display unit23via an operation on the operation unit63and selecting the HDR composition mode on the menu screen.

In step S401, the system control unit50displays, on the display unit23, a user interface for a user to select material images for the HDR composition processing (an image selection UI). The image selection UI displays, for example, a thumbnail of a material image that has been generated through the appropriate shooting processing. The user can select the material image displayed as the thumbnail by operating the operation unit63. When the material image has been selected on the image selection UI, a material image that has been generated through the under shooting processing in correspondence with the selected material image is also selected as a material image for the HDR composition processing.

Note, it is assumed that two material image files that are generated through the appropriate shooting processing and the under shooting processing included in single HDR shooting processing are mutually associated in some way. As one example, the two material image files can be mutually associated by including, in the two material image files, a unique character string shared between file names as identification information.

In step S402, the system control unit50determines whether the user has completed the selection of material images. The system control unit50repeats the determination processing in step S402until the user completes the selection of material images. Once the user has completed the selection of material images, the processing step proceeds to step S403.

In step S403, the system control unit50reads out material image files corresponding to the material images selected in step S402from the external recording medium91, and stores the material image files into the memory25.

In step S404, the system control unit50parses (analyzes) the material image files that were stored into the memory25in step S403, and extracts image data (the material images), subject detection information pieces, and inference results.

In step S405, the image processing unit20composites two material images obtained in step S404, thereby generating an HDR composite image with an expanded luminance dynamic range. Here, any composition method existing in the technical fields for generating an HDR composite image through image composition can be used as a composition method. As one example, based on comparison of luminance between corresponding positions in the two material images, the image processing unit20determines which material image includes appropriately-exposed pixels on a per-pixel basis. Then, the image processing unit20generates an HDR composite image by compositing the appropriately-exposed pixels included in the two material images. The generated HDR composite image is stored into the memory25.

An HDR composite image503ofFIG.5is an example of the HDR composite image generated in step S405. In this example, with respect to the sloped surface510, pixels of the sloped surface510included in the material image501are composited, whereas with respect to the mountain505, the sky508, and the cloud514, pixels of the mountain505, the sky508, and the cloud514included in the material image502are composited.

In step S406, the system control unit50decides on an inference result to be excluded from the targets to be recorded. Specifically, the system control unit50identifies inference results that are similar between the two material images. For example, in the case of the material images501and502shown inFIG.5, each of pairs of the mountains504and505, the skies507and508, the sloped surfaces510and511, and the clouds513and514, is identified as the inference results that are similar between the material image501and the material image502. In a case where two inference results (e.g., the mountains504and505) are similar between the two material images, recording both of them in association with the HDR composite image causes the HDR composite image to include redundant inference results, which reduces the usability for the user. In view of this, in a case where two inference results (e.g., the mountains504and505) are similar between the two material images, the system control unit50excludes one of them from the targets to be recorded.

A description is now given of an example of a method of identifying inference results that are similar between the two material images. The system control unit50determines whether the similarity between the inference result for one material image (e.g., the mountain504) and the inference result for the other material image (the mountain505) satisfies a predetermined criterion. In a case where this predetermined criterion concerning the similarity is satisfied, the system control unit50can identify these two inference results as the inference results that are similar between the two material images. Although the predetermined criterion concerning the similarity is not limited in particular, a criterion based on the degree of overlap between subject regions can be used as one example. For example, in a case where the degree of overlap between a region of a subject specified by the inference result for one material image (e.g., a rectangular region of the mountain504specified by the coordinates (x11, y11, w11, h11) included in the annotation information601) and a region of a subject specified by the inference result for another material image (e.g., a rectangular region of the mountain505specified by the coordinates (x21, y21, w21, h21) included in the annotation information602) is equal to or higher than a predetermined degree, the system control unit50determines that the similarity between these two inference results satisfies the predetermined criterion.

Next, two examples will be described in relation to the method of deciding on an inference result to be excluded from the targets to be recorded. For the sake of explanation, focus is placed on the mountains504and505as the inference results that are similar between the two material images.

As a first example, the system control unit50determines whether the HDR composite image includes more components derived from the mountain504, or more components derived from the mountain505. In a case where the HDR composite image includes more components derived from the mountain504than components derived from the mountain505, it is considered that the exposure for the mountain504is more appropriate than that for the mountain505, and the inference result for the mountain504is more reliable or important than the inference result for the mountain505. Consequently, the system control unit50excludes the mountain505(that is to say, the mountain504is recorded). Conversely, in a case where the HDR composite image includes more components derived from the mountain505than components derived from the mountain504, the system control unit50excludes the mountain504(that is to say, the mountain505is recorded).

As a second example, the system control unit50decides on an inference result excluded from the targets to be recorded based on the degrees of detail of the subject types indicated by the inference results. As indicated by the annotation information601ofFIG.6, the inference result for each subject includes information indicating a subject type. Also, there are cases where the information indicating the subject type includes more detailed information related to that type. With regard to “subject 1” corresponding to the mountain504in the annotation information601, there is information indicating that the type is a mountain, and more specifically, Mount Fuji. On the other hand, with regard to “subject 1” corresponding to the mountain505in the annotation information602, although there is information indicating that the type is a mountain, more specific information does not exist. Accordingly, the system control unit50excludes an inference result in which information indicating a subject type is not relatively detailed, so that an inference result in which information indicating a subject type is more specific is recorded. Therefore, with regard to the mountain504and the mountain505, the inference result corresponding to the mountain505is excluded.

Note that it is permissible to adopt a configuration in which a similar inference result is not excluded in a case where some sort of condition is satisfied. For example, in a case where the two material images have been shot in a predetermined shooting mode, the system control unit50does not exclude an inference result regardless of whether the inference results are similar (therefore, for example, both of the mountains504and505are recorded). The predetermined shooting mode is, for example, a night view mode. As has been described in connection with the aforementioned first example related to the method of deciding on an inference result excluded from the targets to be recorded, it is generally considered that an inference result for a subject detected in an inappropriately-exposed region is of low reliability or importance. However, in the case of the night view mode, there is a possibility that an inference result for a subject detected in an underexposed region will be important to the user. For this reason, in a case where the two material images have been shot in the night view mode, the system control unit50does not exclude an inference result regardless of whether the inference results are similar.

In step S407, the system control unit50records the HDR composite image generated in step S405, as well as the inference results for each material image (except for the inference result that was excluded in step S406), into the external recording medium91as an HDR composite image file. Also, the system control unit50includes the subject detection information pieces corresponding to the recorded inference results (obtained in steps S204and S209ofFIG.2) in the HDR composite image file. A configuration similar to the configuration of the material image file that has been described with reference toFIG.3can be used as a configuration of the HDR composite image file.

Annotation information603ofFIG.6is an example of annotation information included in the HDR composite image file generated in step S407(i.e., the inference results recorded in association with the HDR composite image). As can be understood from comparison with the annotation information pieces601and602corresponding to the material images501and502, the annotation information603corresponds to a case where the aforementioned first example has been used as the method of deciding on an inference result to be excluded from the targets to be recorded.

Exemplary Modifications Related to Shooting Processing and Composition Processing

The foregoing description has been provided under the assumption that the two material images that are composited are images shot through shooting processing under different exposure settings (the appropriate exposure setting and the underexposure setting). However, the different exposure settings in the shooting processing according to the present embodiment are not limited to the appropriate exposure setting and the underexposure setting. For example, an overexposure setting may be used instead of the underexposure setting. The overexposure setting allows an image with a bright exposure to be obtained as a result of the system control unit50providing the exposure control unit40with a feedback of the result of the AE (automatic exposure) processing to which an offset for increasing exposure has been given.

Also, the number of material images that are composited is not limited to two. It is permissible to adopt, for example, a configuration in which three material images are shot under different exposure settings (e.g., the appropriate exposure setting, the underexposure setting, and the overexposure setting), and these three material images are composited.

Furthermore, although the foregoing description has been provided under the assumption that the composition processing for material images is the HDR composition processing (composition processing for expanding the luminance dynamic range), the composition processing according to the present embodiment is not limited to the HDR composition processing, and may be, for example, depth composition processing (composition processing for increasing the depth of field). In this case, the system control unit50shoots two (or more) material images under different focusing distance settings, instead of shooting two (or more) material images under different exposure settings.

As described above, according to the first embodiment, the digital camera100obtains a plurality of material images (e.g., the material image501and the material image502) and subject information pieces indicating the subjects detected in each material image (e.g., information pieces of the respective subjects in the annotation information pieces601and602). Furthermore, the digital camera100generates a composite image (e.g., the HDR composite image503) by compositing the plurality of material images. Then, the digital camera100records the subject information pieces in association with the composite image. At the time of this recording, in a case where the similarity between subject information of one material image (first subject information) and subject information of another material image (second subject information) satisfies the predetermined criterion, the digital camera100records one of the first subject information and the second subject information. In this way, redundancy of subject information (an inference result) associated with the composite image is reduced, and the usability of the subject information (inference result) is improved for a user.

OTHER EMBODIMENTS

This application claims priority to Japanese Patent Application No. 2022-028383, which was filed on Feb. 25, 2022 and which is hereby incorporated by reference herein in its entirety.