Image processing apparatus, control method, and storage medium

An apparatus generates an image file having a structure including a first region and a second region, the first region storing a plurality of instances of image data and the second region storing metadata pertaining to the plurality of instances of image data. The metadata includes synchronous group information that groups image data shot synchronously with respect to time, the synchronous group information indicating image data, among the plurality of instances of image data, belonging to a group created by the grouping, and indicator information that indicates an indicator of grouping for the group indicated by the synchronous group information. The apparatus configures the synchronous group information and the indicator information based on the shooting information, and generates the image file storing the plurality of instances of image data, and the configured synchronous group information and the configured indicator information.

BACKGROUND OF THE DISCLOSURE

Field of the Disclosure

The present disclosure relates to an image processing apparatus, a control method, and a storage medium, and particularly relates to a technique for generating an image file capable of storing the data of a plurality of still images, moving images, and the like.

Description of the Related Art

Using captured images obtained at the same time enables, for example, present extended information that is not limited to the information contained in a single captured image. Japanese Patent Laid-Open No. 2018-007031 discloses a shooting system that shoots images with two cameras synchronized by an internal clock and generates a 3D image using one image shot by one camera and another image shot by the other camera at a time closest to the time the one image was shot.

In recent years, file formats exist which encode a plurality of still images or moving images and store the result as a single image file, which is expected to facilitate the management of highly-related image groups and the like. For example, a file format called High Efficiency Image File Format (HEIF), which is internationally standardized in ISO/IEC 23008-12, can store still images encoded in H.265 (HEVC) as a single image file. For such a file format, a normative structure that includes metadata is defined, which specifies the method for associating the metadata with stored images and the configuration of metadata in a specific format. Additionally, by writing in a metadata region, a single image representation constituted by a plurality of still images, called a “derived image”, can be recorded as an image file.

As described in Japanese Patent Laid-Open No. 2018-007031, images in an image group obtained by synchronous shooting are highly interrelated, and if these images are grouped and stored as a synchronous group within a single image file, the convenience of displaying and processing the images is expected to improve. For HEIF, a standardization study is underway in ISO/IEC 23008-12:2017 FDAM2 to define an additional ‘tsyn’ group in the metadata, which indicates that a plurality of still images and moving images were obtained synchronously with respect to time.

However, no metadata configuration that takes into account the specific use of image files in which are stored image groups obtained by synchronous shooting has been proposed. The HEIF ‘tsyn’ was also considered to merely describe information that identifies an image group shot at the same time over the same period.

SUMMARY OF THE DISCLOSURE

Having been achieved in light of the above-described circumstances, the present disclosure provides an image processing apparatus, a control method, and a storage medium that generate an image file which stores an image group shot synchronously and which can be used appropriately.

The present disclosure in its first aspect provides an image processing apparatus that generates an image file having a structure including a first storage region and a second storage region, the first storage region storing a plurality of instances of image data and the second storage region storing metadata pertaining to the plurality of instances of image data, wherein the metadata includes: synchronous group information that groups image data shot synchronously with respect to time, the synchronous group information indicating image data, among the plurality of instances of image data, belonging to a group created by the grouping; and indicator information that indicates an indicator of grouping for the group indicated by the synchronous group information, and the image processing apparatus comprises at least one processor and/or circuit configured to function as the following units: an obtainment unit configured to obtain the plurality of instances of image data and shooting information pertaining to shooting of the image data; a configuration unit configured to configure the synchronous group information and the indicator information based on the shooting information obtained by the obtainment unit; and a generation unit configured to generate the image file storing the plurality of instances of image data obtained by the obtainment unit, and the synchronous group information and the indicator information configured by the configuration unit.

The present disclosure in its second aspect provides an image processing apparatus that processes an image file having a structure including a first storage region and a second storage region, the first storage region storing a plurality of instances of image data and the second storage region storing metadata pertaining to the plurality of instances of image data, wherein the metadata includes: synchronous group information that groups image data shot synchronously with respect to time, the synchronous group information indicating image data, among the plurality of instances of image data, belonging to a group created by the grouping; and indicator information that indicates an indicator of grouping for the group indicated by the synchronous group information, the indicator information including synchronization error information indicating simultaneity of shooting of the image data included in the group or permissible error information indicating a range of shooting times of image data that can be included in the group, and the image processing apparatus comprises at least one processor and/or circuit configured to function as the following units: an obtainment unit configured to obtain the image file to be processed; and a presenting unit configured to present synchronization accuracy of shooting, for the image data included in the group indicated by the synchronous group information, based on the indicator information stored in the image file to be processed.

The present disclosure in its third aspect provides an image processing apparatus that processes an image file having a structure including a first storage region and a second storage region, the first storage region storing a plurality of instances of image data and the second storage region storing metadata pertaining to the plurality of instances of image data, wherein the metadata includes: synchronous group information that groups image data shot synchronously with respect to time, the synchronous group information indicating image data, among the plurality of instances of image data, belonging to a group created by the grouping; and indicator information that indicates an indicator of grouping for the group indicated by the synchronous group information, the indicator information including permissible error information indicating a range of shooting times of image data that can be included in the group indicated by the synchronous group information, and the image processing apparatus comprises at least one processor and/or circuit configured to function as the following units: a first obtainment unit configured to obtain the image file to be processed; a second obtainment unit configured to obtain image data to be added to the first storage region of the image file to be processed and shooting information pertaining to shooting of the image data to be added; a determination unit configured to determine whether to add, to the group indicated by the synchronous group information, the image data to be added, based on the indicator information of the image file to be processed and the shooting information for the image data to be added; a configuration unit configured to configure the synchronous group information based on a determination result from the determination unit; and a generation unit configured to generate a new image file by storing the image data to be added in the first storage region of the image file to be processed and storing the synchronous group information configured by the configuration unit in the second storage region.

The present disclosure in its fourth aspect provides a control method for an image processing apparatus that generates an image file having a structure including a first storage region and a second storage region, the first storage region storing a plurality of instances of image data and the second storage region storing metadata pertaining to the plurality of instances of image data, wherein the metadata includes: synchronous group information that groups image data shot synchronously with respect to time, the synchronous group information indicating image data, among the plurality of instances of image data, belonging to a group created by the grouping; and indicator information that indicates an indicator of grouping for the group indicated by the synchronous group information, and the control method comprises: obtaining the plurality of instances of image data and shooting information pertaining to shooting of the image data; configuring the synchronous group information and the indicator information based on the shooting information obtained; and generating the image file storing the plurality of instances of image data obtained, and the synchronous group information and the indicator information configured.

The present disclosure in its fifth aspect provides a control method for an image processing apparatus that processes an image file having a structure including a first storage region and a second storage region, the first storage region storing a plurality of instances of image data and the second storage region storing metadata pertaining to the plurality of instances of image data, wherein the metadata includes: synchronous group information that groups image data shot synchronously with respect to time, the synchronous group information indicating image data, among the plurality of instances of image data, belonging to a group created by the grouping; and indicator information that indicates an indicator of grouping for the group indicated by the synchronous group information, the indicator information including synchronization error information indicating simultaneity of shooting of the image data included in the group or permissible error information indicating a range of shooting times of image data that can be included in the group, and the control method comprises: obtaining the image file to be processed; and presenting synchronization accuracy of shooting, for the image data included in the group indicated by the synchronous group information, based on the indicator information stored in the image file to be processed.

The present disclosure in its sixth aspect provides a control method for an image processing apparatus that processes an image file having a structure including a first storage region and a second storage region, the first storage region storing a plurality of instances of image data and the second storage region storing metadata pertaining to the plurality of instances of image data, wherein the metadata includes: synchronous group information that groups image data shot synchronously with respect to time, the synchronous group information indicating image data, among the plurality of instances of image data, belonging to a group created by the grouping; and indicator information that indicates an indicator of grouping for the group indicated by the synchronous group information, the indicator information including permissible error information indicating a range of shooting times of image data that can be included in the group indicated by the synchronous group information, and the control method comprises: obtaining the image file to be processed; obtaining image data to be added to the first storage region of the image file to be processed and shooting information pertaining to shooting of the image data to be added; determining whether to add, to the group indicated by the synchronous group information, the image data to be added, based on the indicator information of the image file to be processed and the shooting information for the image data to be added; configuring the synchronous group information based on a result of the determining; and generating a new image file by storing the image data to be added in the first storage region of the image file to be processed and storing the configured synchronous group information in the second storage region.

The present disclosure in its seventh aspect provides a computer-readable storage medium in which is stored a program for causing a computer to function as the respective units of the image processing apparatus of the first aspect.

DESCRIPTION OF THE EMBODIMENTS

First Embodiment

One embodiment, described below, will describe an example in which the present disclosure is applied in a synchronous shooting controller, serving as an example of an image processing apparatus, capable of obtaining image data shot by a plurality of camera devices communicatively connected over a network and generating image files. However, the present disclosure is applicable in any device capable of generating image files by obtaining image data and a shooting time according to a predetermined master clock at which the image data was shot.

In the present specification, “image data” may be any digital data expressing at least one images obtained by shooting, and may include still images, moving images, or both. “Image data” may be defined according to the file format of the image file generated by storing said image data, and is not limited to encoded data.

“(Shooting) synchronization accuracy” is a concept that quantifies the simultaneity of shooting for an image data group shot synchronously in terms of time. For example, in one aspect, the “synchronization accuracy” may be derived based on a difference in shooting timestamps between instances of image data. In another aspect, the “synchronization accuracy” may be derived based on a time difference between internal clocks in different devices when the devices shoot images having designated a shooting time. Or, in yet another aspect, the “synchronization accuracy” may be derived from a combination of such information.

Configuration of Synchronous Shooting System

FIG.1is a schematic diagram illustrating an example of the configuration of a synchronous shooting system according to the present embodiment. The synchronous shooting system is constructed to enable a synchronous shooting controller100to record, as a single image file, image data obtained by a plurality of camera devices200working in tandem and taking time-synchronized shots. In the example illustrated inFIG.1, the synchronous shooting controller100and four camera devices200ato200dare connected over a network300, and are configured to send and receive information. In the present embodiment, the configuration is such that the synchronous shooting controller100issues shooting instructions, shooting processing is executed in each camera device200, and the image data (encoded data) obtained from the shooting is sent back to the synchronous shooting controller100.

Although there are four camera devices200in the present embodiment, it is easy to understand that the number of camera devices200is not limited in carrying out the present disclosure. Additionally, although the present embodiment will describe a configuration in which shooting units that capture a subject from four respective points are provided in the different camera devices200, the present disclosure is not limited to such a configuration. For example, it goes without saying that a single camera device200may be provided with a plurality of shooting units. Furthermore, in the following, all four camera devices200are described as having the same functional configuration to facilitate understanding of the present disclosure, but it is to be understood that the present disclosure is not limited to such a configuration.

The network300may be a LAN, for example, or a public line such as the Internet. The connection between the network300and each device may be implemented by any method, whether wired or wireless.

Functional Configuration of Synchronous Shooting Controller

FIG.2is a block diagram illustrating an example of the functional configuration of the synchronous shooting controller100according to the present embodiment. Each functional configuration is configured to be capable of transmitting information over a system bus110.

Processing related to synchronous shooting by the plurality of camera devices200and the generation of image files storing a plurality of images obtained by the shooting, which is realized by the synchronous shooting system of the present embodiment, is realized by a CPU101executing a predetermined application program. This program, a system program involved in the operations of the synchronous shooting controller100, and the like are stored in ROM102, which is a nonvolatile storage device, and the CPU101reads out and loads the programs into RAM103to execute the programs. The RAM103is a temporary storage device and is used as a program loading area, a work area for the operations of each block, and the like. More specifically, the RAM103is also used as an input/output buffer for encoding and decoding performed by a codec105(described later) and as a data buffer for image file generation processing performed by a file generation unit104. The ROM102also stores user setting data and the like of the synchronous shooting controller100.

The file generation unit104performs processing involved in the generation, editing, and the like of image files. Image files that can be processed may include files in formats such as HEIF, JPEG, JPEG 2000, JPEG XR, WebP, MP4, MOV, AVI, WebM, and the like. The file generation unit104also performs processing for reading out encoded data (image data) from an image file stored in a recording medium109and transmitting the data to the codec105to be decoded.

The codec105is a video codec for moving images, still images, and the like, such as H.265 (HEVC), H.264 (AVC), AV1, JPEG, and so on. The codec105performs encoding and decoding processing for the data of still images, moving images, and the like handled by the synchronous shooting controller100.

A display unit106is a display device, such as a liquid crystal display (LCD), that is configured integrally with the synchronous shooting controller100or is provided so as to be detachable from the synchronous shooting controller100. In the present embodiment, the display unit106displays screens of an application related to synchronous shooting.

An operation input unit107includes a variety of user interfaces included in the synchronous shooting controller100, such as operation buttons, switches, a mouse, a keyboard, and the like. In the case where the display unit106is a touch panel, the operation input unit107may include a touch panel sensor. Upon detecting that an operation input has been made in the user interface, the operation input unit107outputs a corresponding control signal to the CPU101.

A communication control unit108is a network interface that connects to the network300, and transmits and receives transmission frames. The communication control unit108may be, for example, a PHY and MAC (transmission media control processing) capable of wired LAN connection using Ethernet (registered trademark). Alternatively, if the synchronous shooting controller100is capable of connecting to a wireless LAN, the communication control unit108may include a controller which executes wireless LAN control such as IEEE 802.11a/b/g/n/ac/ax or the like, RF circuitry, and an antenna.

The communication control unit108also incorporates a hardware clock for PTP. As will be described in detail later, the synchronous shooting system according to the present embodiment performs time-synchronous communication using the Precision Time Protocol (PTP) specified in IEEE 1588-2008 (also called “IEEE 1588 version 2”). Because synchronous shooting is realized by having the plurality of camera devices200work together under the control of the synchronous shooting controller100, the synchronous shooting controller100has the function of a PTP grand master clock (GMC). The PTP grand master clock function is implemented as software in the PTP stack executed by the CPU101, and the time of a hardware clock built in the communication control unit108is used as a reference thereof. The communication control unit108is configured to be capable of outputting timestamps of the transmission and reception of transmission frames based on the time of the hardware clock.

The recording medium109is a non-volatile recording device having a large storage capacity, such as a hard disk device, an SD card, a Compact Flash (registered trademark) card, or the like. In the present embodiment, the recording medium109is used as the storage location for image files generated based on synchronous shooting.

Functional Configuration of Camera Devices

An example of the functional configuration of each of the camera devices200according to the present embodiment will be described next with reference to the block diagram inFIG.3. As illustrated inFIG.3, each functional configuration in the camera device200is configured to be capable of transmitting information over a camera system bus210. Note that of the functional configurations of the camera device200, functional configurations having similar roles to the functional configurations of the synchronous shooting controller100are indicated by “camera” at the beginning to make a clear distinction.

Processing involved in time synchronization and shooting for the synchronous shooting is implemented by a camera CPU201executing a predetermined application program. This program, a system program involved in the operations of the camera device200, and the like are stored in camera ROM202, which is a nonvolatile storage device, and the camera CPU201reads out and loads the programs into camera RAM203to execute the programs. The camera RAM203is a temporary storage device and is used as a program loading area, a work area for the operations of each block, and the like. More specifically, the camera RAM203is also used as an input/output buffer involved in encoding and decoding performed by a camera codec205(described later). The camera ROM202also stores user setting data and the like for the camera device200.

A shooting unit204is a unit having an image capturing function, and is configured including a lens group, a CCD (a photoelectric conversion element), a CCD control unit, and an image processing unit. The lens group is constituted by a plurality of lenses that optically form a subject image on an image capturing surface of the CCD. The CCD control unit includes a timing generator for supplying transfer clock signals, shutter signals, and the like to the CCD, circuits for noise removal, gain processing, and the like for output signals from the CCD, an A/D conversion circuit for converting analog signals to digital signals, and the like. The image processing unit performs image processing such as gamma conversion, color space conversion, white balance, exposure correction, and the like on the digital signals output from the CCD control unit, and outputs digital image signal data (image data) that can be encoded by the camera codec205to the RAM203.

The camera codec205is a video codec for moving images, still images, and the like, such as H.265 (HEVC), H.264 (AVC), AV1, JPEG, and so on. The camera codec205performs encoding and decoding processing for the data of still images, moving images, and the like handled by the camera device200.

A camera display unit206is a display device, such as a liquid crystal display (LCD), that is configured integrally with the camera device200or is provided so as to be detachable from the camera device200. In the present embodiment, the camera display unit206displays screens of an application related to synchronous shooting.

A camera operation input unit207includes a variety of user interfaces included in the camera device200, such as operation buttons, switches, and the like. In the case where the camera display unit206is a touch panel, the camera operation input unit207may include a touch panel sensor. Upon detecting that an operation input has been made in the user interface, the camera operation input unit207outputs a corresponding control signal to the camera CPU201.

A camera communication control unit208is a network interface that connects to the network300, and transmits and receives transmission frames. The camera communication control unit208may be, for example, a PHY and MAC (transmission media control processing) capable of wired LAN connection using Ethernet (registered trademark). Alternatively, if the camera device200is capable of connecting to a wireless LAN, the camera communication control unit208may include a controller which executes wireless LAN control such as IEEE 802.11a/b/g/n/ac/ax or the like, RF circuitry, and an antenna.

The camera communication control unit208also incorporates a hardware clock for PTP. In the synchronous shooting system according to the present embodiment, the camera device200has the function of a PTP slave. The PTP slave function is implemented as software in the PTP stack executed by the camera CPU201, and synchronizes the hardware clock built into the camera device200based on the time information supplied from the synchronous shooting controller100. Time synchronization is then performed by synchronizing the time of a system timer of the operating system of the camera device200, executed by the camera CPU201, with the hardware clock of the camera device200. The camera communication control unit208is configured to be capable of outputting timestamps of the transmission and reception of transmission frames based on the time of the built-in hardware clock.

A camera recording medium209is a non-volatile recording device having a large storage capacity, such as a hard disk device, an SD card, a Compact Flash (registered trademark) card, or the like. In the present embodiment, the camera recording medium209may be used to store image data such as still images, moving images, and the like obtained by shooting.

Synchronous Shooting

The following will describe synchronous shooting performed by the synchronous shooting system according to the present embodiment and the generation of a single image file containing a plurality of instances of image data obtained through the synchronous shooting.

As described above, in the synchronous shooting system according to the present embodiment, the times of the system timers of each device in the synchronous shooting system are synchronized through time-synchronous communication using the synchronous shooting controller100as the grand master clock. Moreover, being supplied with the shooting time from the synchronous shooting controller100, each camera device200shoots an image when the corresponding internal clock reaches the corresponding shooting time. The image data obtained from the shooting is sent to the synchronous shooting controller100, and the synchronous shooting controller100receives the plurality of instances of image data shot at the specified shooting time, stores the data in a single image file, and records the data. At this time, the plurality of instances of image data can be regarded as image data taken synchronously in terms of time, and are grouped in the corresponding image file as having been shot synchronously. The grouping may be implemented by, for example, writing an identifier of the grouped image data in the metadata of the image file.

Here, the file format of the image file may be any format that stores a plurality of instances of image data and has metadata that describes the image data shot synchronously as being grouped, but the following will describe a case where the file format is HEIF.

HEIF File Structure

The file structure of an HEIF file will be described with reference toFIG.4. As illustrated inFIG.4, an HEIF file400is generally constituted by the following three boxes (storage regions). The first is a FileTypeBox (‘ftyp’)401, which stores a brand name for a reader of the HEIF file400to identify the specifications of the file. In the HEIF file400generated by the synchronous shooting controller100according to the present embodiment, ‘mifl’ is assumed to be set as the brand name and ‘heic’ is assumed to be set as a compatible brand name in the ‘ftyp’ box401. The second is a MetaBox (‘meta’)402, which stores metadata describing various information about the image data stored in the HEIF file400. As illustrated inFIG.4, the ‘meta’ box402stores a plurality of types of information pertaining to the image data, each separated into different boxes, which will be described in detail below. The third is a MediaDataBox (‘mdat’)412, which stores a plurality of instances of encoded data (image data) as image items. Note that in the following descriptions, the encoded data stored in the ‘mdat’ box412will be referred to as “image item” or “image data” with different wording as appropriate.

A HandlerReferenceBox (‘hdlr’)403stores a declaration of a handler type for analyzing the structure of the ‘meta’ box402. In the HEIF file400generated by the synchronous shooting controller100according to the present embodiment, it is assumed that all the encoded data to be stored is still images, and a handler type name ‘pict’ is set in the ‘hdlr’ box403.

A PrimaryItemBox (‘pitm’)404specifies an identifier (item ID) of the encoded data serving as a representative item among the image items stored by the HEIF file400.

An ItemLocationBox (‘iloc’)405stores information indicating the storage location of the data for each image item in the HEIF file400. The ‘Hoc’ box405typically describes the storage location of the image item as a byte offset from the beginning of the HEIF file400and a data length from that beginning. In other words, the information in the ‘iloc’ box405can identify the location of each instance of encoded data stored in the ‘mdat’ box412.

An ItemInfoBox (‘iinf’)406defines basic information about all the image items in the HEIF file400, including the item ID, an item type indicating the type of the item, an item name, and the like.

An ItemReferenceBox (‘iref’)407stores information describing associations among image items in the HEIF file400. The ‘iref’ box407is used to describe associations between an image item and shooting information (Exif data and the like) items thereof, in a situation where the image item is a shot image. In a situation where a plurality of image items are in a derived image relationship, the box is used to describe the associations among the image items.

An ItemPropertiesBox (‘iprp’)408stores various attribute information about the image items contained in the HEIF file400. More specifically, the ‘iprp’ box408contains an ItemPropertyContainerBox (‘ipco’)409describing the attribute information and an ItemPropertyAssociation (‘ipma’) box410array (Array) indicating associations among the attribute information and each image item. The ‘ipco’ box409may store an enumeration of data such as, for example, a creation date/time of the image item, information indicating the width and height of the image item in pixels, an HEVC parameter set required to decode the HEVC image item, and the like. On the other hand, the ‘ipma’ box410indicates an association for each instance of attribute information stored in the ‘ipco’ box409, e.g., by storing the item ID of the corresponding image item. Here, if a plurality of image items correspond to the same attribute information, this may be described in a single ‘ipma’ box410.

TimeSynchronizedEntityToGroupBox (‘tsyn’)411stores information indicating a group defined by grouping image items shot synchronously with respect to time (a synchronous group). The information stored in the ‘tsyn’ box411may be in the form of a list of item IDs, for example, for one synchronous group, and may include at least information that can identify the image items included in the group. Note that for one HEIF file400, a plurality of synchronous groups may be defined, and the ‘tsyn’ box411may store as many synchronous groups as there are ‘meta’ boxes402.

Note that the file structure of the HEIF file400illustrated inFIG.4is merely one example, and the storage order of each box, the manner of separation of the storage regions, and the types of stored information are not limited thereto.

Incidentally, as mentioned above, in conventional HEIF files, the ‘tsyn’ box411is simply a grouping of image data shot at the same time over the same period of time, and is not a configuration intended to be used in this manner.

For example, when adding new image data to an HEIF file, it was not possible to determine whether the new image data could be added to the synchronous group defined in the ‘tsyn’ box411, and the extensibility of the synchronous group could not be ensured. In other words, when image data was added at a different timing from than when the HEIF file was generated, it was difficult to determine whether the image data could be “image data shot at the same time over the same period of time” as the image data already stored in the synchronous group.

In addition, it is difficult to completely synchronize the system timers between devices even if time-synchronous communication is used, especially when image data shot by a plurality of different camera devices are stored in HEIF files, as in the present embodiment. Accordingly, for example, in cases such as generating 3D images as in Japanese Patent Laid-Open No. 2018-007031, where a plurality of instances of image data requiring simultaneous shooting are used for processing, whether the image data included in a single synchronous group meets a condition for the required synchronization accuracy of the shooting could not be determined. In other words, when using a plurality of instances of image data contained in a synchronous group in an HEIF file, it has been difficult to determine whether the plurality of instances of image data ensure the synchronization accuracy of shooting for the intended use. In other words, it has been difficult to determine the extent to which a plurality of instances of image data in a synchronous group were taken synchronously with respect to time.

In view of this type of file use, a new way of configuring the ‘meta’ box402, which makes it possible to easily evaluate the synchronization accuracy of image data included in a synchronous group, will be described for the synchronous shooting controller100according to the present embodiment. To be more specific, in the framework of the file structure of the HEIF file400illustrated inFIG.4, the present embodiment defines the data structure of the ‘tsyn’ box411and the data structure of the ‘iinf’ box406so as to include information for evaluating the synchronization accuracy for the synchronous group.

FIG.5indicates a definition501of the data structure of the ‘tsyn’ box411for the HEIF file according to the present embodiment. As indicated in definition501, the ‘tsyn’ box411according to the present embodiment, for one synchronous group, contains synchronous group information502that identifies the grouped image items and indicator information503indicating the synchronization accuracy of the image data shooting for that group.

The synchronous group information502describes, for example, a list of group IDs (group_id), which are identifiers of synchronous groups, the number of entries in the group (num_entities_in_group), and entry IDs (entity_id), as illustrated inFIG.5. Here, the entry ID is assumed to specify the item ID of the image item stored in the HEIF file. In other words, the synchronous group information502is information that identifies the image data, among the image data stored in the ‘mdat’ box412, that is grouped together as having been taken synchronously with respect to time, as conventionally assumed.

On the other hand, it is desirable that the indicator information503, which indicates the synchronization accuracy of the shooting, be configured to enable a variety of determinations not limited to when the HEIF file is generated, assuming future forms of use not limited within the HEIF file synchronous shooting system. Accordingly, the indicator information503according to the present embodiment is configured including synchronization method information504, reference time information505, synchronization error information506, and permissible error information507.

The synchronization method information504describes information that identifies the time synchronization method used for the devices that shot the image data included in the synchronous group. Specifically, the synchronization method information504may be information that specifies a compatible time synchronization standard, protocol, or the like. In addition, the synchronization method information504may include information specific to the time synchronization standard, protocol, or the like. For example, when a synchronous group is constituted by image data shot by the synchronous shooting system according to the present embodiment, the information may be information indicating IEEE 1588-2008, which is the time synchronization standard, and that the grand master clock is the synchronous shooting controller100.

The reference time information505describes information that identifies a time serving as a reference for the synchronous shooting (a reference time) for the image data included in the synchronous group. When the synchronous group is constituted by image data shot by the synchronous shooting system according to the present embodiment, the reference time information505may be, for example, information indicating a specified shooting time (a shooting timestamp or time) supplied from the synchronous shooting controller100to each camera device200.

The synchronization error information506describes information expressing the state of synchronization error among the devices with respect to the synchronous shooting for the synchronous group. In other words, the synchronization error information506describes information indicating error in the simultaneity of the image data included in the synchronous group. For example, when PTP or NTP is used for time synchronization, the synchronization error information506may be a maximum value of the time differences between the system clocks of the camera devices200and the master clock of the synchronous shooting controller100at the time each instance of image data was captured. In other words, the synchronization error information may be the maximum value of the deviation of the shooting time of the image data from the specified shooting time on the time axis with respect to the grand master clock. Alternatively, the synchronization error information506may be the maximum value of the time differences among the system clocks among the camera devices200that shot the image data in the synchronous group. In other words, the synchronization error information may be the time difference between the earliest and the latest of the shooting times of the plurality of instances of image data, on the time axis with respect to the grand master clock.

The permissible error information507describes information indicating an error range of synchronization times permitted to be included in the synchronous group. In other words, the permissible error information507describes information, defined for a synchronous group, that indicates the minimum simultaneity that is to be maintained for the image data in that synchronous group. The permissible error information507may indicate conditions under which image data can be included in the synchronous group in an absolute manner, in the form of a time difference between the system clock of the camera device200that shot the image data and the master clock of the synchronous shooting controller100. Alternatively, the permissible error information507may indicate the stated conditions in a relative manner, in the form of a time difference among the system clocks of the camera devices200that shot the image data. As will be described in detail later, the information described in the permissible error information507is determined based on the intended use of the file, and the options for the synchronization accuracy to be ensured, and the like, which are specified, for example, when performing the processing involved in HEIF file generation in the synchronous shooting controller100. In other words, the information in the permissible error information507is not determined based on the image data obtained through the file generation or the state of the camera device200that shot the image data, but is instead determined in advance in consideration of the extensibility of the HEIF file.

In other words, while the synchronization error information506is information indicating the synchronization accuracy of shooting that is finalized based on the image data included in the synchronous group, the permissible error information507is information indicating the synchronization accuracy of shooting that is absolutely fixed for the synchronous group. Accordingly, for the plurality of instances of image data included in a synchronous group, the minimum synchronization accuracy of shooting specified by the permissible error information507is guaranteed, and a more specific synchronization accuracy is indicated by the synchronization error information506.

By defining the data structure of the ‘tsyn’ box411in this manner, it is possible to know the synchronization accuracy of the shooting of the corresponding synchronous group by referring to the indicator information503when using the HEIF file. To be more specific, the synchronization method information504and reference time information505enable to understand that image data shot at the specified shooting time is grouped together in devices that are time-synchronized through specific time-synchronous communication. Additionally, by further referring to the synchronization error information506, the maximum value of the time difference from the specified shooting time of the image data included in the synchronous group, or the maximum value of the time difference of the shooting times among the image data in the group, can be obtained, and the synchronization accuracy of the shooting can be ascertained. Alternatively, by further referring to the permissible error information507, a permissible value of the time difference from the specified shooting time of the image data that can be included in the synchronous group, or a permissible value of the time difference of the shooting times among the image data in the group, can be obtained, and the synchronization accuracy for the synchronous group can be ascertained. In other words, the indicator information503has not only the aspect of indicating the synchronization accuracy of the shooting of the image data in the synchronous group, but also the aspect of expressing an indicator of the grouping of the image data for that synchronous group.

In a synchronous shooting system that generates HEIF files using image data obtained by supplying the specified shooting time from the synchronous shooting controller100, as in the present embodiment, the shooting timestamps of the obtained image data are identical. In other words, because the system clock of each camera device200after the time-synchronous communication indicates that images are to be shot at the specified shooting time, the shooting timestamp attached to the image data shot by each camera device200indicates the specified shooting time. Accordingly, in the synchronous shooting system according to the present embodiment, when image data is provided by each of the camera devices200, information on the synchronization error of the system clocks of the camera devices200with respect to the grand master clock (device error information) is assumed to be sent in association with the image data. Then, in the synchronous shooting controller100, the ‘tsyn’ box411(the synchronous group information502and the indicator information503) of the HEIF file is configured based on the shooting timestamps and the device error information of the image data. Although the present embodiment describes the device error information has being derived in each of the camera devices200, the implementation of the present disclosure is not limited thereto, and may be derived in at least one of the synchronous shooting controller100and the camera devices200.

On the other hand, considering the extensibility of the synchronous group, it is preferable that the HEIF file be configured such that image data shot without supplying a specified shooting time, by a camera device time-synchronized through the same method as the devices in the synchronous shooting system, can be added. In other words, it is preferable that the HEIF file be configured such that synchronous groups can be defined without being limited to image data shot synchronously having specified the shooting time under the control of the synchronous shooting controller100. Because the simultaneity of shooting pertaining to a synchronous group can be measured if it is possible to compare the shooting time on a common time axis for the plurality of camera devices200that shoot the image data, grouping is possible even for image data obtained without controlling the shooting time. In other words, for example, based on shooting instructions transmitted simultaneously from the synchronous shooting controller100, the image data obtained by the camera devices200shooting upon receiving the shooting instructions may be grouped in the HEIF file. Alternatively, for example, in a situation where a plurality of camera devices (shooting units) are operated independently without forming a synchronous shooting system, the image data obtained by shooting at the same time may be grouped in an HEIF file.

Here, the time synchronization among the devices need not be implemented in the manner of the synchronous shooting system according to the present embodiment. That is, in the synchronous shooting system according to the present embodiment, the synchronous shooting controller100is described as having a grand master clock function and time-synchronizing the camera devices200belonging to the system, but the reference time for synchronization may be supplied by an external device. In other words, the device that obtains the image data and generates the HEIF file need not be the device that supplies the reference time for synchronization, and at least the time synchronization of the plurality of camera devices that shoot images may be achieved through time-synchronous communication with any device. To rephrase, the generation of HEIF files according to the present disclosure can be realized using image data shot by any device that is time-synchronized using the same method, in addition to image data shot by the camera devices200belonging to the synchronous shooting system.

Accordingly, the ‘meta’ box402may be configured to be capable of storing information indicating the respective shooting times (shooting timestamps) for the image data stored in the ‘mdat’ box412. The shooting timestamp may be obtained, for example, from the shooting information (including the device error information) that is provided in association with the image data and stores various information pertaining to the shooting of the image data. The information pertaining to the image data is stored by defining (601) the data structure of the ‘iinf’ box406as illustrated inFIG.6.

As illustrated inFIG.6, the definition601includes an entry number602(16 or 32 bits) indicating the number of image items to be stored in the ‘mdat’ box412, and an ItemInfoEntry (‘infe’) array603that stores various information pertaining to each image item. The ‘infe’ array603is an array having as many data as the entry number602, and stores various information about each of the image items.

The data structure of ‘infe’ for a single image item may be defined (701) as illustrated inFIG.7. As illustrated inFIG.7, the definition701defines basic information702such as the item ID of the image item, the item type indicating the type of the image item, the item name of the image item, and the like, as well as timestamp information703describing the shooting timestamp. In the example inFIG.7, the timestamp information703stores the shooting timestamp as a 64-bit integer value for the number of elapsed seconds in Coordinated Universal Time (UTC) in item_ts_sec and a 32-bit integer value for nanoseconds less than integer seconds in item_ts_nsec. The description format of the timestamp information703is not limited thereto, and for example, the information may describe the shooting timestamp in NTP timestamp format with a 32-bit second value and a 32-bit nanosecond value. Alternatively, International Atomic Time (TAI) may be used to describe the shooting timestamp in the form of a PTP timestamp with a 48-bit second value and a 32-bit nanosecond value.

By making it possible to store the shooting timestamp of each instance of image data in the ‘meta’ box402in this manner, grouping can also be performed using the shooting timestamp of the image data already stored in the ‘mdat’ box412. In this aspect, for example, rather than using a configuration that stores the specified shooting time in the reference time information505in the ‘tsyn’ box411, the shooting timestamp of the image data may be specified as the reference time for the synchronous group by describing the item ID of the image data. In this case, the synchronization error information506may store the maximum value of the time differences in the shooting times between the image data specified in the reference time information505and other image data in the synchronous group. The permissible error information507may describe conditions under which image data can be included in the synchronous group in the form of the time difference from the shooting time of the image data specified in the reference time information505.

File Generation Sequence

The following will describe, with reference toFIG.8, specific processing for a file generation sequence for generating an HEIF file storing a plurality of instances of image data shot synchronously in the synchronous shooting system according to the present embodiment. As illustrated inFIG.8, the file generation sequence is implemented by the synchronous shooting controller100and the camera devices200performing the corresponding processing, i.e., the CPU101and the camera CPU201executing the corresponding processing. The sequence for the camera devices200illustrated inFIG.8is executed by each of the camera devices200ato200dcommunicating with the synchronous shooting controller100.

Note that in generating the HEIF file, the intended use of the plurality of instances of image data shot synchronously is assumed to be selected. Information on the selected intended use is used to determine the permissible upper limit value to be stored in the permissible error information507in the ‘tsyn’ box411. The intended use may be selected in a manner that specifies the application that uses the HEIF file generated in the synchronous shooting, the shooting synchronization accuracy required when compositing image data included in the synchronous group, and the like.

In step801, time-synchronous communication is performed between the synchronous shooting controller100and the camera devices200, and the system timers of the devices are synchronized. As described above, in the synchronous shooting system according to the present embodiment, the synchronous shooting controller100has a PTP grand master clock function, and thus the times of the system timers of the camera devices200are synchronized to the grand master clock through the time-synchronous communication. In the descriptions of the file generation sequence inFIG.8, the time-synchronous communication is assumed to be executed only in step801to facilitate understanding of the present disclosure, but the implementation of the present disclosure is not limited thereto, and time-synchronized communication may be performed at a predetermined frequency. In the camera device200, when time-synchronous communication is performed, a PTP offset value used to adjust the system timer or the hardware clock is stored. Here, the system timer of the synchronous shooting controller100and the system timers of the camera devices200are temporarily synchronized through the time-synchronous communication, but strictly speaking, these can still be slightly out of sync even after the time synchronization. Additionally, the PTP offset value can fluctuate each time the time-synchronous communication is executed. Therefore, it is assumed that the camera device200holds the PTP offset values for a plurality of instances of time-synchronous communication.

When the time-synchronous communication is complete, in step802, the synchronous shooting controller100transmits shooting instructions for synchronous shooting to the camera devices200, along with information on the specified shooting time. Note that in the present embodiment, the camera device200is described as shooting images in response to the supply of information on the specified shooting time, but as described above, the implementation of the present disclosure is not limited thereto.

In step803, the camera device200executes the shooting processing at the specified shooting time. The shot image obtained by the shooting processing is encoded by the camera codec205and stored as image data.

In step804, the image data generated in step803is transmitted from the camera device200to the synchronous shooting controller100along with the shooting information. At this time, the device error information to be included in the shooting information may be the PTP offset value or the moving median value of the last predetermined number of PTP offset values.

In step805, an HEIF file is generated by the synchronous shooting controller100based on the plurality of instances of image data received in the synchronous shooting. To simplify the descriptions of the present embodiment, the plurality of instances of image data obtained by shooting in each of the camera devices200in response to shooting instructions for synchronous shooting are assumed to be stored in the ‘mdat’ box412of a single HEIF file. In addition, based on the information on the synchronization accuracy of the shooting to be ensured, which is set in advance for the synchronous shooting, the image data among the plurality of instances of image data, i.e., the image data that meets the conditions for permitting inclusion in the synchronous group, are included in the synchronous group. In addition, only one synchronous group is assumed to be defined in the single generated HEIF file.

At this time, under the control of the CPU101, the file generation unit104configures various information to be stored in the ‘meta’ box402pertaining to the HEIF file to be generated based on the shooting information received for the plurality of instances of image data. For example, for synchronous shooting performed having supplied a specified shooting time as in the present embodiment, in a situation where the permissible upper limit value for the PTP offset value of the camera devices200is set as a condition for inclusion in the synchronous group, the ‘meta’ box402is configured as follows.

For the ‘tsyn’ box411, the file generation unit104configures the synchronization method information504and the reference time information505based on information on the synchronous time communication and the specified shooting time, stores that information in the permissible error information507along with the permissible upper limit value based on the information on the selected intended use. The file generation unit104also determines whether to include each instance of image data in the synchronous group based on the information in the permissible error information507, and updates the synchronization error information506accordingly while configuring the synchronous group information502. Specifically, when image data meets the conditions for inclusion in the synchronous group, the file generation unit104adds the information for that image data as an entry in the list of the synchronous group and updates the synchronous group information502. Additionally, if the image data meets the conditions for inclusion in the synchronous group, the file generation unit104also updates (the absolute value of) the PTP offset value of the camera device200that shot the image data if that PTP offset value is greater than the current value of the synchronization error information506. Here, the synchronization error information506has an initial value of 0, and when the absolute value of the PTP offset value for the image data to be included in the group is high, is updated to that value. In this manner, the file generation unit104can configure the synchronous group information502and the indicator information503for the synchronous group defined in the HEIF file to be generated.

For the ‘iinf’ box406, the file generation unit104also configures the entry number602and the ‘infe’ array603based on the received image data. When configuring the ‘infe’ for each instance of image data, a shooting timestamp based on the shooting information received in association with the image data is stored.

By doing so, for each synchronous group that is defined, an HEIF file can be generated that contains information indicating the synchronization accuracy of the shooting for the plurality of instances of image data included in that synchronous group.

Although the present embodiment describes an aspect in which the specified shooting time is supplied and an HEIF file storing the plurality of instances of image data shot at that time is generated, the implementation of the present disclosure is not limited thereto. For example, in an aspect where the camera devices200shoot images in response to receiving shooting instructions from the synchronous shooting controller100, the ‘tsyn’ box411may be configured in a different manner and the HEIF file may be generated, as described above. That is, when generating an HEIF file containing a plurality of instances of image data shot without the specified shooting time being supplied from the synchronous shooting controller100, the ‘meta’ box402may be configured in a different manner, as described hereinafter.

In this case, the shooting timestamps of each instance of image data can be different, and thus the shooting timestamps stored in ‘infe’ for each instance of image data in the ‘iinf’ box406will not be uniform. Therefore, for the reference time information505in the ‘tsyn’ box411, the file generation unit104stores, for example, the item ID of the image data indicating the most recent shooting timestamp among the image data to be grouped, and identifies the shooting timestamp as the reference time. The file generation unit104then identifies the image data having shooting timestamps that fall within the time range determined based on that reference time and the permissible upper limit value, and configures the synchronous group information502. Furthermore, the file generation unit104configures the synchronization error information506based on the shooting timestamp, among the shooting timestamps of the image data included in the synchronous group, that is farthest from the reference time. In this manner as well, the synchronous group can be defined, and an HEIF file can be generated that contains information indicating the synchronization accuracy of the shooting for the plurality of instances of image data included in that synchronous group.

Use of HEIF File

The manner in which the HEIF file generated in this way is used will be described next. Here, it can easily be understood that the HEIF file can be used in any device, including, but not limited to, the synchronous shooting controller100that generated the file. In use, a processor of the device such as a CPU reads out the ‘meta’ box402of the HEIF file to be processed to reproduce or change the image data stored in the ‘mdat’ box412, edit the ‘meta’ box402as necessary, and the like.

One aspect of the use of the HEIF file includes presenting, based on the information about the synchronous group defined in the ‘tsyn’ box411of the HEIF file, the synchronization accuracy of the shooting for the plurality of instances of image data included in the synchronous group. The presentation of the synchronization accuracy of the shooting may, for example, be in the aspect of displaying at least one instance of information in the indicator information503of the HEIF file, or displaying the HEIF file if the required synchronization accuracy is met.

In the former case, for example, a display may be made indicating that the image data of the group was shot within the time difference indicated by the synchronization error information506, with respect to the shooting time indicated by the reference time information505. Alternatively, for example, a display may be made indicating that at least the synchronization accuracy of the shooting indicated by the permissible error information507is ensured in the image data in the group.

In the latter case, for example, when the synchronization accuracy of shooting is set as an extraction condition in a file viewing application, an HEIF file that meets the extraction condition may be displayed based on the information in the synchronization error information506or the permissible error information507. Alternatively, for example, in an application that performs compositing processing using image data shot synchronously, an HEIF file that satisfies the synchronization accuracy required for the compositing processing may be displayed based on the information in the synchronization error information506or the permissible error information507.

When presenting the synchronization accuracy for these synchronous groups, some or all of the plurality of instances of image data, classified into synchronous groups defined by the synchronous group information502, may be further presented.

Another aspect of the use of HEIF files includes, when new image data is added to the HEIF file (stored in the ‘mdat’ box412), determining to add the new image data to the synchronous group. For example, when an instruction is given to add new image data to an HEIF file, a determination is made as to whether a predefined synchronous group is present by referring to the ‘tsyn’ box411. If such a group is present, a determination is made as to whether the new image data can be added to the synchronous group (grouping) by comparing the shooting information associated with the new image data and the indicator information503.

For example, if the synchronization method information504of the device that shot the new image data is different, it may be determined that the image data cannot be added to the synchronous group. If the synchronization method information504is the same, for example, it may be determined that the new image data can be added to the synchronous group if the shooting timestamp of the new image data falls within the time range defined by the reference time information505and the permissible error information507. Alternatively, in addition to the shooting timestamp, the PTP offset value of the device that shot the new image data may be taken into account in making the determination. If the result of the determination indicates that the data can be added to the synchronous group, the information pertaining to the new image data is added to the synchronous group information502, and grouping is performed. If the synchronization accuracy of the shooting for the image data in the group changes due to the addition of new image data to the synchronous group, the information in the synchronization error information506is updated.

Note that the addition of new image data to the synchronous group need not be made on the condition that the synchronization method information504is the same. For example, even if the device that shot the new image data was not time-synchronized at the time of the shooting using the same method as that indicated by the synchronization method information504pertaining to the synchronous group, the time at the time of shooting may be estimated, and the determination to add be made, based on, information indicating time synchronization performed at any timing after the shooting.

As described thus far, for a synchronous group defined in an HEIF file according to the present embodiment, the convenience of the image data included in a group can be increased by including the indicator information503. More specifically, including the indicator information503for a synchronous group makes it possible to quantitatively indicate the degree to which the image data included in the group is image data that was shot synchronously with respect to time, which in turn makes it easier to determine whether the group is suitable for the intended use. In addition, grouping conditions are indicated by the indicator information503and it is therefore easy to determine whether a predefined synchronous group is updated when image data is added to the HEIF file. In addition, for example, an operation for directly adding new image data to a synchronous group already defined can be accepted, and if the synchronization accuracy of the group changes as a result of the addition, the information in the indicator information503can be updated. Alternatively, for example, for a synchronous group already defined, an operation for deleting given image data in the group from that group can be accepted, and if the synchronization accuracy of the group changes as a result of the deletion, the information in the indicator information503can be updated.

Second Embodiment

The foregoing embodiment described a situation in which synchronous group information and indicator information is included in the ‘tsyn’ box411of the HEIF file and the shooting timestamp of each instance of image data is included in the ‘iinf’ box406, but the implementation of the present disclosure is not limited thereto. In the synchronous shooting system according to the present embodiment described hereinafter, the shooting timestamp of each instance of image data and the indicator information pertaining to the synchronous group are stored in the ‘iprp’ box408instead of the ‘tsyn’ box411, and the HEIF file is generated.

HEIF File Structure

The structure of the HEIF file generated by the synchronous shooting controller100in the synchronous shooting according to the present embodiment will be described hereinafter. More specifically, the structure of the HEIF file to be generated is configured in the same manner as that described in the first embodiment with reference toFIG.4, but the definition of the data structure is different for some boxes, and includes indicator information related to the synchronous group.

As mentioned above, the ‘iprp’ box408contains various attribute information about the image items contained in ‘mdat’ box412(the ‘ipco’ box409) and information identifying the image item corresponding to each attribute information (the ‘ipma’ box410). In other words, the ‘ipco’ box409and the ‘ipma’ box410are attribute information which the image item included in the HEIF file can have, and information indicating which image item that attribute information is associated with. From this perspective, the ‘iprp’ box408expresses the attribute information for a given entry within the HEIF file as being stored in the ‘ipco’ box409and the association between that attribute information and the entry as being defined in the ‘ipma’ box410. As such, in the present embodiment, the indicator information for each of the synchronous groups defined in the ‘tsyn’ box411is stored in the ‘ipco’ box409as attribute information, and information indicating the association between the indicator information and the synchronous group is stored in the ‘ipma’ box410.

To realize such information storage, the data structure of the ‘tsyn’ box411is defined as illustrated inFIG.9, unlike in the first embodiment. As illustrated inFIG.9, a definition901of the data structure of the ‘tsyn’ box411for one synchronous group in the HEIF file according to the present embodiment contains only the synchronous group information502that identifies the grouped image items and does not contain the indicator information.

On the other hand, for the indicator information, attribute information for each of the synchronization method information, the reference time information, the synchronization error information, and the permissible error information is newly defined to enable storage in the ‘ipco’ box409, for use when generating and using the HEIF file. As described in the first embodiment, the information of the shooting timestamp for each image item is also assumed to be defined to be capable of being stored in the ‘ipco’ box409, to cover a situation where the synchronization error information and the permissible error information are stored as a difference value of the shooting timestamps. That is, in the HEIF file according to the present embodiment, the information on the shooting timestamp of the image item is stored in the ‘iprp’ box408, not in the ‘iinf’ box406, unlike in the first embodiment.

First, the storage of the shooting timestamp is realized, for example, by newly defining a CaptureTimeProperty (‘catt’) data structure, illustrated inFIG.10, as attribute information. As illustrated inFIG.10, a definition1001of ‘catt’ has a structure that can store the shooting timestamp in the format of a 64-bit timestamp (1002) or in the format of a timestamp having a set of a 64-bit second value and a 32-bit nanosecond value (1003). The former format (1002) can store the shooting timestamp in a 64-bit NTP timestamp format as a most significant 32-bit second value and a least significant 32-bit sub-second value. The latter format (1003) can also store the shooting timestamp as a PTP timestamp. The definition1001enables the ‘ipco’ box409to store the attribute information of the shooting timestamp of the image item as ‘catt’. Because the HEIF specification already defines the CreationTimeProperty (‘crtt’) attribute information that stores the creation date/time of the image item, the storage of the shooting timestamp can be realized by extending the data structure of that attribute information.

Next, the storage of the synchronization method information in the indicator information is realized by newly defining a SynchronizedTimeMethodProperty (‘sytm’) data structure, illustrated inFIG.11, as attribute information, for example. As illustrated inFIG.11, a definition1101of ‘sytm’ has a structure that can store character string type information1102indicating the time synchronization standard, protocol, or the like, and an identifier1103of the 64-bit PTP grand master clock, for the time synchronization executed in the synchronous shooting. The information1102may further include proprietary information according to the standard, protocol, or the like.

Additionally, the storage of the reference time information in the indicator information is realized by newly defining a SynchronizedCaptureTimeProperty (‘syct’) data structure, illustrated inFIG.12, as attribute information, for example. As illustrated inFIG.12, a definition1201of ‘syct’ has a structure can store information identifying the reference time of synchronous shooting in the form of a shooting timestamp (1202), or an item ID of an image item to which the shooting timestamp of the reference time corresponds.

Additionally, the storage of the synchronization error information in the indicator information is realized by newly defining a SynchronizedTimeDifferenceProperty (‘sytd’) data structure, illustrated inFIG.13, as attribute information, for example. As illustrated inFIG.13, a definition1301of ‘sytd’ has a structure that can store information indicating error in the simultaneity for image items included in the synchronous group as a maximum value (1302) of a time difference among the shooting timestamps. Alternatively, the definition has a structure that can store the same information as a maximum value (offset value) (1303) of the time difference between the system clock at the time of shooting by the camera device200that performed the shooting and the master clock of the synchronous shooting controller100.

Additionally, the storage of the permissible error information in the indicator information is realized by newly defining a SyncronizedTimeCriteria (‘sytc’) data structure, illustrated inFIG.14, as attribute information, for example. As illustrated inFIG.14, a definition1401of ‘sytc’ has a structure that can store a format (1402) of the time difference between the system clock at the time of shooting by the camera device200that performed the shooting and the master clock of the synchronous shooting controller100. Alternatively, the definition has a structure that can store information indicating an error range of the simultaneity permitting for the image items to be included in the synchronous group in a format (1403) of a time difference among 64-bit shooting timestamps.

By defining five new types of attribute information in this manner, the indicator information pertaining to the synchronous group can be stored in the ‘iprp’ box408of the HEIF file. In other words, the indicator information can be included by storing the attribute information of the defined ‘sytm’, ‘syct’, ‘sytd’ and ‘sytc in the ‘ipco’ box409, and storing information associating those items with a group ID of the synchronous group in the ‘ipma’ box410.

Therefore, in the file generation sequence in the synchronous shooting system according to the present embodiment, the file generation unit104stores the ‘catt’ attribute information in the ‘ipco’ box409based on the received shooting information for the plurality of instances of image data. Additionally, in the grouping process, the file generation unit104stores the ‘sytm’ and ‘syct’ attribute information in the ‘ipco’ box409based on the information of the synchronous time communication and the specified shooting time. The file generation unit104also stores the ‘sytc’ attribute information in the ‘ipco’ box409based on the information of the selected intended use. The file generation unit104also determines whether to include each instance of image data in the synchronous group based on the information in ‘sytc’, and stores/updates ‘sytd’ accordingly while configuring the synchronous group information502. Specifically, when image data meets the conditions for inclusion in the synchronous group, the file generation unit104adds the information for that image data as an entry of the synchronous group and updates the synchronous group information502. Additionally, if the image data meets the conditions for inclusion in the synchronous group, the file generation unit104also updates (the absolute value of) the PTP offset value of the camera device200that shot the image data if that PTP offset value is greater than the current value of ‘sytd’, for example. Here, the ‘sytd’ has an initial value of 0, and when the absolute value of the PTP offset value for the image data to be included in the group is high, is updated to that value, for example. Then, after the completion of the determination on whether to include the received image data in the synchronous group, the file generation unit104stores information associating the group ID of the synchronous group with the four types of attribute information pertaining to the configured indicator information in the ‘ipma’ box410. In this manner, the file generation unit104can configure the synchronous group information502and the indicator information for the synchronous group defined in the HEIF file to be generated.

Note that the use of the HEIF file generated in this manner may be similar to that described in the first embodiment, aside from the method for reading out the indicator information being different, and will therefore not be described.

First Variation

The foregoing first and second embodiments described an example of an HEIF file containing synchronous group information, which indicates the grouping of a plurality of instances of image data shot in a time-synchronized manner, and indicator information, which indicates the synchronization accuracy or a grouping index for the shooting of the plurality of instances of image data. However, as mentioned above, the implementation of the present disclosure need not be limited to HEIF format files, and can be applied to any file constituted by including the synchronous group information and the indicator information.

For example, the present disclosure can also be applied in an aspect in which a plurality of camera devices200in a synchronous shooting system each shoots a moving image and stores a plurality of instances of moving image data obtained in a single file. In other words, according to the present disclosure, a single video file can be generated which stores a plurality of instances of moving image data and contains, as metadata, synchronous group information that groups and defines the moving image data shot synchronously with respect to time and indicator information serving as a reference for the grouping. At this time, the generated moving image file may be in a file format based on ISO Base Media File Format (ISOBMFF), similar to HEIF. For example, in an aspect where the camera devices200generate moving image data encoded in H.264, the synchronous shooting controller100may use an MP4 file that stores the moving image data shot by each camera device200as different video tracks and audio tracks.

In MP4 files, metadata is described in a MovieBox (‘moov’), and thus the ‘tsyn’ box in ‘moov’ can be included to describe the grouping of tracks. Note that the list of entry IDs in the ‘tsyn’ box may specify track IDs. In the case of moving image data, the shooting timestamp is the start time of moving image shooting by each camera device200, and can be stored in a TrackHeaderBox (‘tkhd’) box in ‘moov’. The indicator information may be stored in the ‘tsyn’ box, for example, as in the first embodiment described above.

Second Variation

Although the foregoing embodiments described the permissible upper limit value stored in the permissible error information as being determined based on the selection of the intended use of the plurality of instances of image data shot synchronously when generating the HEIF file, the implementation of the present disclosure is not limited thereto. For example, the permissible upper limit value may be set in a fixed manner for the synchronous shooting system, or the permissible upper limit value may be set dynamically according to the method of the synchronous shooting (e.g., methods that do or do not supply a specified shooting time), the state of time synchronization among the devices, and so on.

Third Variation

Although the foregoing embodiments and variations described the indicator information as including the synchronization method information, the reference time information, the synchronization error information, and the permissible error information, the implementation of the present disclosure is not limited thereto, and only some of this information may be included. In other words, from the perspective of using the plurality of instances of image data included in the synchronous group, the indicator information may be any information indicating the simultaneity of the shooting of the image data, and need not be configured including all of these pieces of information.

Additionally, from the perspective of indicating the synchronization accuracy of the shooting of the plurality of instances of image data included in the synchronous group, it is sufficient for one of the synchronization error information and the permissible error information to be included. From the perspective of indicating the time range in which image data can be included in the synchronous group, it is sufficient for the reference time information and the permissible error information, or the reference time information and the synchronization error information, to be included. At this time, for image data to be stored in/added to the HEIF file, the indicator information need not include the synchronization method information in situations where it can be ensured that the time synchronization method is the same among the devices that performed the shooting.

Other Embodiments

This application claims the benefit of Japanese Patent Application No. 2021-133671, filed Aug. 18, 2021, which is hereby incorporated by reference herein in its entirety.