Patent Description:
One of the recent most popular streaming services is the over-the-top video (OTT-V) via the Internet. The moving picture experts group phase-dynamic adaptive streaming over HTTP (MPEG-DASH) is widely used as its underlying technology (see, for example, Non-Patent Literature <NUM>).

In MPEG-DASH, a delivery server prepares a group of video data having different screen sizes and coding rates for one video content item, and a playback terminal requests a group of video data having an optimal screen size and coding rate depending on transmission line conditions, thus adaptive streaming delivery is achieved.

Non-Patent Literature <NUM>: MPEG-DASH (Dynamic Adaptive Streaming over HTTP) (URL:http://mpeg. chiariglione. org/standards/mpeg-dash/media-presentation-description-and-segment-formats/text-isoiec-<NUM>-<NUM>-dam-<NUM>).

Further previously proposed arrangements are disclosed in: <CIT>; <CIT>; <CIT>; <CIT>; <CIT>; <NPL>; and <NPL>. Summary of Invention.

However, no consideration is given to the recognition by a playback terminal of an acquisition position of voice data on video content.

The present disclosure is made in view of such circumstances, and is intended to be capable of recognizing an acquisition position of voice data on an image.

The invention is defined by the appended independent claims and will be more readily understood after reading the following detailed description in conjunction with the accompanying drawings.

According to the first aspect of the present disclosure, it is possible to transmit information used to recognize an acquisition position of voice data on an image.

According to the second aspect of the present disclosure, it is possible to recognize an acquisition position of voice data on an image.

<FIG> is a diagram illustrated to describe an overview of a first embodiment of an information processing system to which the present disclosure is applied.

An information processing system <NUM> shown in <FIG> is configured to include a Web server <NUM> and a video playback terminal <NUM>, which are connected to each other through the Internet <NUM>. The Web server <NUM> is connected to a file generation device <NUM>.

In the information processing system <NUM>, the Web server <NUM> delivers image data of video content to the video playback terminal <NUM> in unit of tiles (tiled streaming) using a manner compliant with MPEG-DASH.

Specifically, the file generation device <NUM> acquires the image data of video content and encodes the image data in units of tiles to generate a video stream. The file generation device <NUM> processes the video stream of each tile into a file format at time intervals ranging from several seconds to approximately ten seconds, which is called a segment. The file generation device <NUM> uploads the resulting image file of each tile to the Web server <NUM>.

The file generation device <NUM> acquires voice data of video content for each object (described late in detail) and encodes the voice data in unit of objects to generate an audio stream. The file generation device <NUM> processes an audio stream of each object into a file format in unit of segments, and uploads the resulting voice file of each object to the Web server <NUM>.

The object is a sound source. The voice data of each object is acquired through a microphone or other audio equipment attached to the object. The object may be a material body such as a fixed microphone stand, or may be a moving body such as a person.

The file generation device <NUM> encodes audio metadata including object position information (voice position information) indicating the position of each object (the position at which voice data is acquired) and an object ID that is a unique ID to the object. The file generation device <NUM> processes the encoded data obtained by encoding the audio metadata into a file format in unit of segments, and the file generation device <NUM> uploads the resulting voice metafile to the Web server <NUM>.

The file generation device <NUM> generates a media presentation description (MPD) file (control information) used to manage an image file or a voice file. The media presentation description file may contain image frame size information that indicates the frame size of images of video content and contains tile position information that indicates the position of each tile on an image. The file generation device <NUM> uploads the MPD file to the Web server <NUM>.

The Web server <NUM> stores the image file, voice file, voice metafile, and MPD file which are uploaded from the file generation device <NUM>.

In the example shown in <FIG>, the Web server <NUM> stores a segment group of a plurality of segments composed of image files of a tile with a tile ID "<NUM>" and a segment group of a plurality of segments composed of image files of a tile with a tile ID "<NUM>". The Web server <NUM> also stores a segment group of a plurality of segments composed of voice files of an object with an object ID "<NUM>" and a segment group of a plurality of segments composed of voice files of an object with an object ID "<NUM>". Although not shown, a segment group composed of voice metafiles is similarly stored.

A tile with a tile ID of i is referred to as "tile #i", and an object with an object ID of i is referred to as "object #i", hereinafter.

The Web server <NUM> functions as a transmitter configured to transmit the stored image file, voice file, voice metafile, and MPD file to the video playback terminal <NUM> in response to a request from the video playback terminal <NUM>.

The video playback terminal <NUM> executes, for example, software for control of streaming data (hereinafter, referred to as control software) <NUM>, video playback software <NUM>, and client software for hypertext transfer protocol (HTTP) access (hereinafter, referred to as access software) <NUM>.

The control software <NUM> is software to control data delivered via streaming from the Web server <NUM>. Specifically, the control software <NUM> allows the video playback terminal <NUM> to acquire the MPD file from the Web server <NUM>.

The control software <NUM> specifies a tile in a display area based on the display area that is an area in an image used to display video content indicated by the video playback software <NUM> and the tile position information contained in the MPD file. The control software <NUM> instructs the access software <NUM> to issue a request to transmit an image file of the specified tile.

The control software <NUM> instructs the access software <NUM> to issue a request to transmit the voice metafile. The control software <NUM> specifies an object corresponding to an image in the display area, based on the display area, the image frame size information contained in the MPD file, and object position information contained in the voice metafile. The control software <NUM> instructs the access software <NUM> to issue a request to transmit a voice file of the specified object.

The video playback software <NUM> is software to play back the image file and voice file acquired from the Web server <NUM>. Specifically, when the user specifies a display area, the video playback software <NUM> indicates the specified display area to the control software <NUM>. The video playback software <NUM> decodes the image file and voice file acquired from the Web server <NUM> in response to the indication, and the video playback software <NUM> synthesizes the decoded file for output.

The access software <NUM> is software to control communication with the Web server <NUM> via the Internet <NUM> using HTTP. Specifically, the access software <NUM> allows the video playback terminal <NUM> to transmit the request to transmit the image file, voice file, and voice metafile in response to the instruction from the control software <NUM>. The access software <NUM> allows the video playback terminal <NUM> to receive the image file, voice file, and voice metafile transmitted from the Web server <NUM> in response to the transmission request.

<FIG> is a diagram illustrating an example of a tile.

As shown in <FIG>, an image of video content is partitioned into a plurality of tiles. A tile ID that is a sequential number starting from <NUM> is assigned to each tile. In the example shown in <FIG>, an image of video content is partitioned into four tiles #<NUM> to #<NUM>.

<FIG> is a diagram illustrated to describe an object.

The example of <FIG> illustrates eight voice objects in an image acquired as a voice of video content. An object ID that is a sequential number starting from <NUM> is assigned to each object. Objects #<NUM> to #<NUM> are moving bodies, and objects #<NUM> to #<NUM> are stationary material bodies. In the example shown in <FIG>, the image of video content is partitioned into <NUM> (width) × <NUM> (height) tiles.

In this case, as shown in <FIG>, when the user specifies a display area <NUM> composed of <NUM> (width) × <NUM> (height) tiles, the display area <NUM> contains only objects #<NUM>, #<NUM>, and #<NUM>. Thus, the video playback terminal <NUM> may acquire and play back voice files of the objects #<NUM>, #<NUM>, and #<NUM> from the Web server <NUM>.

The objects in the display area <NUM> can be specified based on the image frame size information and the object position information as described above.

<FIG> is a diagram illustrated to describe the object position information.

As shown in <FIG>, the object position information contains a horizontal angle θA (-<NUM>° ≤ θA ≤ <NUM>°), a vertical angle γA (-<NUM>° ≤ γA ≤ <NUM>°), and a distance rA (<NUM> < rA). For example, a shooting position in the center of an image may be set to an origin (base point) O, the horizontal direction of the image may be set to X-axis direction, the vertical direction of the image may be set to Y-axis direction, and the depth direction perpendicular to the X-Y plane may be set to Z-axis direction. In this case, the horizontal angle θA is the angle in the horizontal direction formed by the Y-Z plane and the straight line connecting the object <NUM> and the origin O. The vertical angle γA is the angle in the vertical direction formed by the X-Z plane and the straight line connecting the object <NUM> and the origin O. The distance rA is the distance between the object <NUM> and the origin O.

Herein, the angle of the left and up rotation is set to a positive angle, and the angle of the right and down rotation is set to a negative angle.

<FIG> is a diagram illustrated to describe image frame size information.

As shown in <FIG>, the image frame size information contains a horizontal angle θV1 of the left edge, a horizontal angle θV2 of the right edge, a vertical angle γV1 of the upper edge, a vertical angle γV2 of the lower edge, and a distance rV in the image frame.

A shooting position of the center of an image may be set to the origin O, the horizontal direction of the image may be set to X-axis direction, the vertical direction of the image may be set to Y-axis direction, and the depth direction perpendicular to the X-Y plane may be set to Z-axis direction. In this case, the horizontal angle θV1 is the angle in the horizontal direction formed by the Y-Z plane and the straight line connecting the left end of the image frame and the origin O. The horizontal angle θV2 is the angle in the horizontal direction formed by the Y-Z plane and the straight line connecting the right end of the image frame and the origin O. Thus, an angle obtained by combining the horizontal angle θV1 and the horizontal angle θV2 becomes a horizontal angle of view.

The vertical angle γV1 is the angle formed by the X-Z plane and the straight line connecting between the upper end of the image frame and the origin O, and the vertical angle γV2 is the angle formed by the X-Z plane and the straight line connecting between the lower end of the image frame and the origin O. An angle obtained by combining the vertical angles γV1 and γV2 becomes a vertical angle of view. The distance rA is the distance between the origin O and the image plane.

As described above, the object position information represents the positional relationship between the object <NUM> and the origin O, and the image frame size information represents the positional relationship between the image frame and the origin O. Thus, it is possible to detect (recognize) the position of each object on the image based on the object position information and the image frame size information. As a result, it is possible to specify an object in the display area <NUM>.

<FIG> is a diagram illustrating the structure of an MPD file.

In the analysis (parsing) of an MPD file, the video playback terminal <NUM> selects an optimum one among attributes of "Representation" contained in "Period" of the MPD file (Media Presentation).

The video playback terminal <NUM> acquires a file by referring to a uniform resource locator (URL) or other reference of "Initialization Segment" at the head of the selected "Representation", and processes the acquired file. The video playback terminal <NUM> acquires a file by referring to a uniform resource locator (URL) or other reference of the subsequent "Media Segment", and plays back the acquired file.

In the MPD file, the relationship among "Period", "Representation", and "Segment" becomes as shown in <FIG>. In other words, a single video content item can be managed in a longer time unit than the segment by "Period", and can be managed in unit of segments by "Segment" in each "Period". In each "Period", it is possible to manage the video content in unit of stream attributes by "Representation".

Thus, the MPD file has a hierarchical structure shown in <FIG>, starting from the "Period". When the structure of the MPD file arranged on the time axis becomes the configuration as shown in <FIG>. As is clear from <FIG>, there are a plurality of "Representation" elements in the same segment. The video playback terminal <NUM> selects any one from among these elements adaptively, and thus it is possible to acquire an image file and voice file in the display area selected by the user and to play back the acquired file.

<FIG> is a diagram illustrated to describe the description of MPD file.

As described above, in the information processing system <NUM>, the image frame size information is contained in the MPD file to allow an object in the display area to be specified by the video playback terminal <NUM>. As shown in <FIG>, Scheme (urn:mpeg:DASH:viewingAngle:<NUM>) used to define new image frame size information (viewing angle) is extended by utilizing a DescriptorType element of Viewpoint, and thus the image frame size information is arranged in an "Adaptation Set" for voice and an "Adaptation Set" for image. The image frame size information may be arranged only in the "Adaptation Set" for image.

The "Representation" for voice metafile is described in the "Adaptation Set" for voice of the MPD file. A URL or other reference as information for specifying the voice metafile (audiometadata. mp4) is described in "Segment" of the "Representation". In this case, it is described that the file to be specified in "Segment" is the voice metafile (objectaudiometadata) by utilizing Role element.

The "Representation" for voice file of each object is also described in "Adaptation Set" for voice of the MPD file. A URL or other reference as information for specifying the voice file (audioObje1. mp4, audioObje5. mp4) of each object is described in "Segment" of the "Representation". In this case, object IDs (<NUM> and <NUM>) of the objects corresponding to the voice file are also described by extending Viewpoint.

Although not shown, the tile position information is arranged in the "Adaptation Set" for image.

<FIG> is a block diagram illustrating an exemplary configuration of the file generation device <NUM> shown in <FIG>.

The file generation device <NUM> shown in <FIG> is configured to include a screen split processor <NUM>, an image encoding processor <NUM>, an image file generator <NUM>, an image information generator <NUM>, a voice encoding processor <NUM>, a voice file generator <NUM>, an MPD generator <NUM>, and a server upload processor <NUM>.

The screen split processor <NUM> of the file generation device <NUM> splits image data of video content inputted from the outside into tile units. The screen split processor <NUM> supplies the image information generator <NUM> with the tile position information. The screen split processor <NUM> supplies the image encoding processor <NUM> with the image data configured in units of tiles.

The image encoding processor <NUM> encodes the image data, which is configured in tile units and is supplied from the screen split processor <NUM>, for each tile to generate a video stream. The image encoding processor <NUM> supplies the image file generator <NUM> with the video stream of each tile.

The image file generator <NUM> processes the video stream of each tile supplied from the image encoding processor <NUM> into a file format in units of segments and supplies the MPD generator <NUM> with the resulting image file of each tile.

The image information generator <NUM> supplies the MPD generator <NUM> with the tile position information supplied from the image split processor <NUM> and with the image frame size information inputted from the outside.

The voice encoding processor <NUM> encodes voice data, which is configured in units of objects of video content inputted from the outside, for each object, and generates an audio stream. The voice encoding processor <NUM> encodes the object position information of each object inputted from the outside and the audio metadata that contains the object ID to generate encoded data. The voice encoding processor <NUM> supplies the voice file generator <NUM> with the audio stream of each object and the encoded data of the audio metadata.

The voice file generator <NUM> functions as a voice file generator. The voice file generator <NUM> processes the audio stream of each object supplied from the voice encoding processor <NUM> into a file format in units of segments and supplies the MPD generator <NUM> with the resulting voice file of each object.

The voice file generator <NUM> functions as a metafile generator. The voice file generator <NUM> processes the encoded data, which is obtained by encoding the audio metadata, supplied from the voice encoding processor <NUM> into a file format in units of segments, and supplies the MPD generator <NUM> with the resulting voice metafile.

The MPD generator <NUM> determines a URL or other reference of the Web server <NUM> for storing the image file of each tile supplied from the image file generator <NUM>. The MPD generator <NUM> determines a URL or other reference of the Web server <NUM> for storing the voice file of each tile and the voice metafile, which are supplied from the voice file generator <NUM>.

The MPD generator <NUM> arranges the image information supplied from the image information generator <NUM> in "AdaptationSet" for image of the MPD file. The MPD generator <NUM> arranges the image frame size information among the image information in "AdaptationSet" for voice of the MPD file. The MPD generator <NUM> arranges a URL or other reference of the image file of each tile in "Segment" of "Representation" for image file of the tile.

The MPD generator <NUM> arranges the URL or other reference of the image file of each object in "Segment" of "Representation" for voice file of the object. The MPD generator <NUM> functions as an information generator. The MPD generator <NUM> arranges a URL or other reference as information for specifying a voice metafile in "Segment" of "Representation" for voice metafile. The MPD generator <NUM> supplies the server upload processor <NUM> with the MPD file in which various types of information are arranged as described above, the image file, the voice file, and the voice metafile.

The server upload processor <NUM> uploads the image file of each tile, the voice file of each object, the voice metafile, and the MPD file, which are supplied from the MPD generator <NUM>, to the Web server <NUM>.

<FIG> is a flowchart illustrated to describe a file generation process to be performed by the file generation device <NUM> shown in <FIG>.

In step S11 of <FIG>, the screen split processor <NUM> of the file generation device <NUM> splits image data of video content inputted from the outside into tile units. The screen split processor <NUM> supplies the image information generator <NUM> with the tile position information. The screen split processor <NUM> supplies the image encoding processor <NUM> with the image data configured in units of tiles.

In step S12, the image encoding processor <NUM> encodes the image data, which is configured in tile units and is supplied from the screen split processor <NUM>, for each tile to generate a video stream of each tile. The image encoding processor <NUM> supplies the image file generator <NUM> with the video stream of each tile.

In step S13, the image file generator <NUM> processes the video stream of each tile supplied from the image encoding processor <NUM> into a file format in units of segments to generate an image file of each tile. The image file generator <NUM> supplies the MPD generator <NUM> with the image file of each tile.

In step S14, the image information generator <NUM> acquires the image frame size information from the outside. In step S15, the image information generator <NUM> generates image information that contains the tile position information supplied from the image split processor <NUM> and the image frame size information, and supplies the MPD generator <NUM> with the generated image information.

In step S16, the voice encoding processor <NUM> encodes voice data, which is configured in units of objects of video content inputted from the outside, for each object, and generates an audio stream of each object. The voice encoding processor <NUM> encodes the object position information of each object inputted from the outside and the audio metadata that contains the object ID to generate encoded data. The voice encoding processor <NUM> supplies the voice file generator <NUM> with the audio stream of each object and the encoded data of the audio metadata.

In step S17, the voice file generator <NUM> processes the audio stream of each object supplied from the voice encoding processor <NUM> into a file format in units of segments to generate a voice file of each object. The voice file generator <NUM> processes the encoded data, which is obtained by encoding the audio metadata, supplied from the voice encoding processor <NUM> into a file format in units of segments to generate a voice metafile. The voice file generator <NUM> supplies the MPD generator <NUM> with the voice file of each object and the voice metafile.

In step S18, the MPD generator <NUM> generates an MPD file that contains the image information supplied from the image information generator <NUM>, the URL of each file, or other information. The MPD generator <NUM> supplies the server upload processor <NUM> with the MPD file, the image file of each tile, the voice file of each object, and the voice metafile.

In step S19, the server upload processor <NUM> uploads the image file of each tile, the voice file of each object, the voice metafile, and the MPD file, which are supplied from the MPD generator <NUM>, to the Web server <NUM>. Then, the process is terminated.

<FIG> is a block diagram illustrating an exemplary configuration of a streaming playback section implemented by the execution of the control software <NUM>, the video playback software <NUM>, and the access software <NUM> in the video playback terminal <NUM> shown in <FIG>.

The streaming playback section <NUM> shown in <FIG> is configured to include an MPD acquisition unit <NUM>, an MPD processor <NUM>, a metafile acquisition unit <NUM>, a voice selector <NUM>, a voice file acquisition unit <NUM>, a voice decoding processor <NUM>, a voice synthesis processor <NUM>, an image selector <NUM>, an image file acquisition unit <NUM>, an image decoding processor <NUM>, and an image synthesis processor <NUM>.

The MPD acquisition unit <NUM> of the streaming playback section <NUM> functions as a receiver. The MPD acquisition unit <NUM> acquires an MPD file from the Web server <NUM> and supplies the MPD processor <NUM> with the acquired MPD file.

The MPD processor <NUM> extracts information such as a URL, which is described in "Segment" for voice metafile, from the MPD file supplied from the MPD acquisition unit <NUM>, and supplies the metafile acquisition unit <NUM> with the extracted information. The MPD processor <NUM> extracts image frame size information, which is described in "AdaptationSet" for image, from the MPD file, and supplies the voice selector <NUM> with the extracted information. The MPD processor <NUM> extracts information such as a URL, which is described in "Segment" for voice file of the object requested from the voice selector <NUM>, from the MPD file. Then, the MPD processor <NUM> supplies the voice selector <NUM> with the extracted information.

The MPD processor <NUM> extracts the tile position information described in "AdaptationSet" for image from the MPD file and supplies the image selector <NUM> with the extracted information. The MPD processor <NUM> extracts information such as a URL, which is described in "Segment" for image file of the tile requested from the image selector <NUM>, from the MPD file. Then, the MPD processor <NUM> supplies the image selector <NUM> with the extracted information.

The metafile acquisition unit <NUM> requests a voice metafile from the Web server <NUM> based on the information supplied from the MPD processor <NUM>. The voice metafile to be requested is specified in a URL among the information supplied from the MPD processor <NUM>. Then, the metafile acquisition unit <NUM> acquires the voice metafile. The metafile acquisition unit <NUM> supplies the voice selector <NUM> with object position information contained in the voice metafile.

The voice selector <NUM> functions as a position determination unit. The voice selector <NUM> calculates a position of each object on the image, based on the image frame size information supplied from the MPD processor <NUM> and the object position information supplied from the metafile acquisition unit <NUM>. The voice selector <NUM> selects an object in the display area specified by the user based on the position of each object on the image. The voice selector <NUM> requests information such as a URL of the voice file of the selected object from the MPD processor <NUM>. The voice selector <NUM> supplies the voice file acquisition unit <NUM> with the information such as a URL supplied from the MPD processor <NUM> in response to the request.

The voice file acquisition unit <NUM> functions as a receiver. The voice file acquisition unit <NUM> requests a voice file in units of objects from the Web server <NUM> based on the information supplied from the voice selector <NUM>. The voice file in units of objects to be requested is specified in a URL among the information supplied from the voice selector <NUM>. Then, the voice file acquisition unit <NUM> acquires the voice file in units of objects, and supplies the voice decoding processor <NUM> with the acquired voice file in units of objects.

The voice decoding processor <NUM> decodes an audio stream contained in the voice file in units of object supplied from the voice file acquisition unit <NUM> to generate voice data in units of objects. The voice decoding processor <NUM> supplies the voice synthesis processor <NUM> with the voice data in units of objects.

The voice synthesis processor <NUM> synthesizes the voice data in units of objects supplied from the voice decoding processor <NUM> and outputs the synthesized data.

The image selector <NUM> selects a tile in the display area specified by the user based on the tile position information supplied from the MPD processor <NUM>. The image selector <NUM> requests information such as a URL for the image file of the selected tile from the MPD processor <NUM>. The image selector <NUM> supplies the image file acquisition unit <NUM> with the information such as a URL supplied from the MPD processor <NUM> in response to the request.

The image file acquisition unit <NUM> requests an image file in units of tiles from the Web server <NUM> based on the information supplied from the image selector <NUM>. The image file in units of tiles to be requested is specified in a URL among the information supplied from the image selector <NUM>. Then, the image file acquisition unit <NUM> acquires the image file in units of tiles, and supplies the image decoding processor <NUM> with the acquired image file in units of tiles.

The image decoding processor <NUM> decodes a video stream contained in the image file in units of tiles supplied from the image file acquisition unit <NUM> to generate image data in units of tiles. The image decoding processor <NUM> supplies the image synthesis processor <NUM> with the image data in units of tiles.

The image synthesis processor <NUM> synthesizes the image data in units of tiles supplied from the image decoding processor <NUM> and outputs the synthesized data.

<FIG> is a flowchart illustrated to describe a streaming playback process by the streaming playback section <NUM> (see <FIG>) of the video playback terminal <NUM>.

In step S31 of <FIG>, the MPD acquisition unit <NUM> of the streaming playback section <NUM> acquires the MPD file from the Web server <NUM> and supplies the MPD processor <NUM> with the acquired MPD file.

In step S32, the MPD processor <NUM> acquires the image frame size information and the tile position information, which are described in "AdaptationSet" for image, from the MPD file supplied from the MPD acquisition unit <NUM>. The MPD processor <NUM> supplies the voice selector <NUM> with the image frame size information and supplies the image selector <NUM> with the tile position information. The MPD processor <NUM> extracts information such as a URL described in "Segment" for voice metafile and supplies the metafile acquisition unit <NUM> with the extracted information.

In step S33, he metafile acquisition unit <NUM> requests a voice metafile from the Web server <NUM> based on the information supplied from the MPD processor <NUM>. The voice metafile to be requested is specified in a URL among the information supplied from the MPD processor <NUM>. Then, the metafile acquisition unit <NUM> acquires the voice metafile. The metafile acquisition unit <NUM> supplies the voice selector <NUM> with object position information contained in the voice metafile.

In step S34, the voice selector <NUM> selects an object in the display area specified by the user, based on the image frame size information supplied from the MPD processor <NUM> and the object position information supplied from the metafile acquisition unit <NUM>. The voice selector <NUM> requests the information such as a URL for the voice file of the selected object from the MPD processor <NUM>.

The MPD processor <NUM> extracts information such as a URL, which is described in "Segment" for voice file of the object requested from the voice selector <NUM>, from the MPD file. Then, the MPD processor <NUM> supplies the voice selector <NUM> with the extracted information. The voice selector <NUM> supplies the voice file acquisition unit <NUM> with the information such as a URL supplied from the MPD processor <NUM>.

In step S35, the voice file acquisition unit <NUM> requests and acquires, based on the information such as a URL supplied from the voice selector <NUM>, a voice file of the selected object specified in the URL from the Web server <NUM>. Then, the voice file acquisition unit <NUM> supplies the voice decoding processor <NUM> with the acquired voice file in units of objects.

In step S36, the image selector <NUM> selects a tile in the display area specified by the user based on the tile position information supplied from the MPD processor <NUM>. The image selector <NUM> requests information such as a URL for the image file of the selected tile from the MPD processor <NUM>.

The MPD processor <NUM> extracts information such as a URL, which is described in "Segment" for image file of the object requested from the image selector <NUM>, from the MPD file, and the MPD processor <NUM> supplies the image selector <NUM> with the extracted information. The image selector <NUM> supplies the image file acquisition unit <NUM> with the information such as a URL supplied from the MPD processor <NUM>.

In step S37, the image file acquisition unit <NUM> requests an image file in units of tiles from the Web server <NUM> based on the information supplied from the image selector <NUM>. The image file in selected tiles to be requested is specified in a URL among the information supplied from the image selector <NUM>. Then, the image file acquisition unit <NUM> acquires the image file in units of tiles, and supplies the image decoding processor <NUM> with the acquired image file in units of tiles.

In step S38, the voice decoding processor <NUM> decodes an audio stream contained in the voice file in units of object supplied from the voice file acquisition unit <NUM> to generate voice data in units of objects. The voice decoding processor <NUM> supplies the voice synthesis processor <NUM> with the voice data in units of objects.

In step S39, the image decoding processor <NUM> decodes a video stream contained in the image file in units of tiles supplied from the image file acquisition unit <NUM> to generate image data in units of tiles. The image decoding processor <NUM> supplies the image synthesis processor <NUM> with the image data in units of tiles.

In step S40, the voice synthesis processor <NUM> synthesizes the voice data in units of objects supplied from the voice decoding processor <NUM> and outputs the synthesized data. In step S41, the image synthesis processor <NUM> synthesizes the image data in units of tiles supplied from the image decoding processor <NUM> and outputs the synthesized data. Then, the process is terminated.

As described above, the Web server <NUM> transmits the image frame size information and the object position information. Thus, the video playback terminal <NUM> can specify an object in the display area to acquire selectively a voice file of the specified object so that the voice file corresponds to the image in the display area. This allows the video playback terminal <NUM> to acquire only a necessary voice file, which leads to the improvement of transmission efficiency.

As shown in <FIG>, an object ID (object specifying information) may be described in "AdaptationSet" for an image of the MPD file. This object ID may be used as information for specifying an object corresponding to the voice that is intended to be played back simultaneously with the image of the MPD file. The object ID may be described by extending Scheme (urn:mpeg:DASH:audioObj:<NUM>) for defining new object ID information (audioObj) by utilizing a DescriptorType element of Viewpoint. In this case, the video playback terminal <NUM> selects a voice file of the object corresponding to the object ID described in "AdaptationSet" for image, and acquires the voice file for playback.

The encoded data of all objects may be multiplexed into a single audio stream, rather than the generation of voice file in units of objects, to generate a single voice file.

In this case, as shown in <FIG>, one "Representation" for voice file is provided in "AdaptationSet" for voice of the MPD file, and a URL or other reference for the voice file (audioObje. mp4) that contains the encoded data of all objects is described in "Segment". At this time, object IDs (<NUM>, <NUM>, <NUM>, <NUM>, and <NUM>) of all objects corresponding to the voice file are described by extending Viewpoint.

In addition, in this case, as shown in <FIG>, the encoded data (Audio object) of each object is arranged, as a subsample, in an "mdat" box of the voice file (hereinafter, also referred to as voice media file, as appropriate) acquired by referring to "Media Segment" of the MPD file.

Specifically, data is arranged in the voice media file in units of subsegments that are any time shorter than a segment. The position of data in units of subsegments is specified by a "sidx" box. The data in units of subsegments is composed of moof and mdat boxes. The mdat box is composed of a plurality of samples, and the encoded data of each object as each subsample of the sample is arranged.

A gsiz box in which information on sample is described is placed next to the sidx box of the voice media file. In this way, the gsiz box in which the information on sample is provided separately from the moof box, and thus the video playback terminal <NUM> can acquire the information on sample in a short time.

A grouping_type for indicating the types of Sample group entry composed of one or more samples or subsamples managed by the gsix box is described in the gsix box, as shown in <FIG>. For example, when the Sample group entry is a subsample of the encoded data in units of objects, the type of the Sample group entry is "obja". The gsix box of a plurality of grouping_type is arranged in the voice media file, as shown in <FIG>.

As shown in <FIG>, an index (entry_index) of each Sample group entry and a byte range as data position information indicating the position in the voice media file are described in the gsix box. When the index (entry_index) is zero, the corresponding byte range indicates a byte range of the moof box (a1 in the example of <FIG>).

Information indicating which object is used for allowing each Sample group entry to corresponds to a subsample of the encoded data obtained by encoding the object is described in the voice file acquired by referring to "Initialization Segment" of the MPD file (hereinafter, also referred to as voice initialization file, as appropriate).

Specifically, as shown in <FIG>, this information is indicated by using a type assignment box (typa), which is associated with AudioObjectSampleGroupEntry of a sample group description box (sgpd) in a sbtl box of the voice initialization file.

In other words, as shown in the portion A of <FIG>, an object ID (audio_object_id) corresponding to the encoded data contained in the sample is described in each AudioObjectSampleGroupEntry box. As shown in the portion B of <FIG>, an object ID such as, for example, <NUM>, <NUM>, <NUM>, and <NUM> is described in each of four AudioObjectSampleGroupEntry boxes.

On the other hand, as shown in <FIG>, in the type assignment box, an index as a parameter (grouping_type_parameter) of the Sample group entry corresponding to the AudioObjectSampleGroupEntry is described in each AudioObjectSampleGroupEntry.

The voice media file and the voice initialization file are configured as described above. Thus, when the video playback terminal <NUM> acquires the encoded data of the object selected as an object in the display area, the AudioObjectSampleGroupEntry in which the object ID of the selected object is described is retrieved from the stbl box of the voice initialization file. Then, an index of the Sample group entry corresponding to the retrieved AudioObjectSampleGroupEntry is read from the mvex box. The position of data in units of subsegments is read from the sidx box of the voice file, and the byte range of the Sample group entry of the read index is read from the gsix box. The encoded data arranged in the mdat box is acquired based on the position of data in units of subsegments and the byte range. This allows encoded data of the selected object to be acquired.

Although, in the description mentioned above, the index of Sample group entry and the object ID of AudioObjectSampleGroupEntry are associated with each other through the mvex box, they can be associated with each other directly. When they are associated with each other directly, the index of Sample group entry is described in the AudioObjectSampleGroupEntry.

When the voice file is composed of a plurality of tracks, the sgpd box can be stored in the mvex box, which allows the sgpd box to be shared among tracks.

<FIG> is a diagram illustrated to describe an overview of a second embodiment of the information processing system to which the present disclosure is applied.

In <FIG>, the same elements as those shown in <FIG> are denoted with the same reference signs.

In the example shown in <FIG>, as is the case with <FIG>, the image of video content is partitioned into <NUM> (width) × <NUM> (height) tiles, and voices of objects #<NUM> to #<NUM> are acquired as a voice of video content.

In this case, when the user specifies the display area <NUM> composed of <NUM> (width) × <NUM> (height) tiles, the display area <NUM> is converted (extended) to an area having the same size as the size of the image of video content, resulting in a display image <NUM> in the second embodiment as shown in <FIG>. The voices of the objects #<NUM> to #<NUM> are synthesized based on the positions of the objects #<NUM> to #<NUM> in the display image <NUM> and are outputted along with the display image <NUM>. In other words, voices of the objects #<NUM> to #<NUM>, #<NUM>, and #<NUM>, which are outside the display area <NUM>, are outputted, in addition to the voices of the objects #<NUM>, #<NUM>, and #<NUM>, which are inside the display area <NUM>.

The configuration of the second embodiment of the information processing system to which the present disclosure is applied is the same as the configuration of the information processing system <NUM> shown in <FIG> except for the configuration of the streaming playback section, and thus the following description will be given only of the streaming playback section.

<FIG> is a block diagram illustrating an exemplary configuration of the streaming playback section of the information processing system to which the present disclosure is applied.

In <FIG>, the same configurations as those shown in <FIG> are denoted with the same reference signs, and repeated description is omitted as appropriate.

A streaming playback section <NUM> shown in <FIG> is provided with an MPD processor <NUM>, a voice synthesis processor <NUM>, and an image synthesis processor <NUM> instead of the MPD processor <NUM>, the voice synthesis processor <NUM>, and the image synthesis processor <NUM>, respectively, and the streaming playback section <NUM> is configured to further include a position determination unit <NUM>, which is different from the streaming playback section <NUM> shown in <FIG>.

The MPD processor <NUM> of the streaming playback section <NUM> extracts information such as a URL, which is described in "Segment" for voice metafile, from the MPD file supplied from the MPD acquisition unit <NUM>, and supplies the metafile acquisition unit <NUM> with the extracted information. The MPD processor <NUM> extracts image frame size information of an image of the video content (hereinafter, referred to as content image frame size information) that is described in "AdaptationSet" for image from the MPD file and supplies the position determination unit <NUM> with the extracted information. The MPD processor <NUM> extracts information such as a URL, which is described in "Segment" for voice file of all objects, from the MPD file, and supplies the voice file acquisition unit <NUM> with the extracted information.

The position determination unit <NUM> acquires object position information that is contained in the voice metafile obtained by the metafile acquisition unit <NUM> and the content image frame size information that is supplied from the MPD processor <NUM>. The position determination unit <NUM> acquires display area image frame size information that is the image frame size information of the display area specified by the user. The position determination unit <NUM> determines (recognizes) the position of objects in the display area, based on the object position information, the content image frame size information, and the display area image frame size information. The position determination unit <NUM> supplies the voice synthesis processor <NUM> with the determined position of the object.

The voice synthesis processor <NUM> synthesizes voice data in units of objects supplied from the voice decoding processor <NUM> based on the object position supplied from the position determination unit <NUM>. Specifically, the voice synthesis processor <NUM> determines voice data to be assigned to each speaker for each object based on the object position and the position of each speaker that outputs sound. The voice synthesis processor <NUM> synthesizes voice data of each object for each speaker and outputs the synthesized voice data as voice data for each speaker. A detailed description of the method of synthesizing voice data of each object based on the object position is disclosed, for example, in <NPL>.

The image synthesis processor <NUM> synthesizes image data in units of tiles supplied from the image decoding processor <NUM>. The image synthesis processor <NUM> functions as a converter. The image synthesis processor <NUM> converts the size of the image corresponding to the synthesized image data to the size of the video content to generate a display image. The image synthesis processor <NUM> outputs the display image.

<FIG> are diagrams illustrated to describe a method of determining a position of an object by the position determination unit <NUM> shown in <FIG>.

The display area <NUM> is extracted from the video content and the size of the display area <NUM> is converted to the size equivalent to that of the video content, and then a display image <NUM> is generated. Thus, the display image <NUM> has the size equivalent to the size obtained by shifting the center C of the display area <NUM> to the center C' of the display image <NUM> as shown in <FIG> and by converting the size of the display area <NUM> to the size of the video content as shown in <FIG>.

Thus, the position determination unit <NUM> calculates a shift amount θshift in horizontal direction when the center O of the display area <NUM> is shifted to the center O' of the display image <NUM>. This calculation uses the following Equation (<NUM>). <NUM>] <MAT>.

In Equation (<NUM>), θv1' is the horizontal angle at the left edge in the display area <NUM> contained in the display area image frame size information, and θv2' is the horizontal angle at the right edge in the display area <NUM> of the display area image frame size information. In Equation (<NUM>), θv1 is a horizontal angle at the left edge of the content image frame size information, and θv2 is a horizontal angle at the right edge contained in the content image frame size information.

The position determination unit <NUM> calculates a horizontal angle at the left edge θv1_shift' and a horizontal angle at the right edge θv2_shift' in the display area <NUM> after the center O of the display area <NUM> is shifted to the center O' of the display image <NUM>. This calculation uses the shift amount θshift and is obtained by the following Equation (<NUM>). <NUM>] <MAT>.

According to Equation (<NUM>), the horizontal angle θv1_shift' and the horizontal angle θV2 shift' are calculated so that they do not exceed the range of -<NUM>° to <NUM>°.

As described above, the display image <NUM> has the size equivalent to the size obtained by shifting the center O of the display area <NUM> to the center O' of the display image <NUM> and by converting the size of the display area <NUM> to the size of the video content. Thus, the following Equation (<NUM>) is satisfied for the horizontal angles θv1 and θv2. <NUM>] <MAT>.

The position determination unit <NUM> calculates the shift amount θshift, the horizontal angle θv1_shift', and the horizontal angles θv2_shift', in the way described above and then calculates a horizontal angle of objects in the display image <NUM>. Specifically, the position determination unit <NUM> calculates a horizontal angle θAi_shift of an object #i after the center C of the display area <NUM> is shifted to the center C' of the display image <NUM>. This calculation uses the shift amount θshift and is obtained by the following Equation (<NUM>). <NUM>] <MAT>.

In Equation (<NUM>), θAi is the horizontal angle of an object #i contained in the object position information. According to Equation (<NUM>), the horizontal angle θAi_shift is calculated so that it does not exceed the range of -<NUM>° to <NUM>°.

When an object #i is present in the display area <NUM>, that is, if the condition of θv2_shif' < θAi_shift < θv1_shift' is satisfied, the position determination unit <NUM> calculates a horizontal angle θAi' of an object #i in the display image <NUM> using the following Equation (<NUM>). <NUM>] <MAT>.

According to Equation (<NUM>), the horizontal angle θAi' is calculated by extending the distance between the position of an object #i in the display image <NUM> and the center C' of the display image <NUM> depending on the ratio between the sizes of the display area <NUM> and the display image <NUM>.

On the other hand, when no object #i is present in the display area <NUM>, that is, if the condition of -<NUM>° ≤ θAi_shift ≤ θv2_shift' or θv1_shift' ≤ θAi_shift ≤ <NUM>° is satisfied, the position determination unit <NUM> calculates the horizontal angle θAi' of an object #i in the display image <NUM> using the following Equation (<NUM>). <NUM>] <MAT>.

According to Equation (<NUM>), when an object #i is present in a position <NUM> on the right side of the display area <NUM> (-<NUM>° ≤ θAi_shift ≤ θv2_shift') as shown in <FIG>, the horizontal angle θAi' is calculated by extending the horizontal angle θAi_shift depending on the ratio between angles R1 and R2. The angle R1 is the angle measured from the right edge of the display image <NUM> to the position <NUM> just behind a viewer <NUM>. The angle R2 is the angle measured from the right edge of the display area <NUM> whose center is shifted to the position <NUM>.

According to Equation (<NUM>), when an object #i is present in a position <NUM> on the left side of the display area <NUM> (θv1_shift' ≤ θAi_shift ≤ <NUM>°), the horizontal angle θAi' is calculated by extending the horizontal angle θAi_shift depending on the ratio between angles R3 and R4. The angle R3 is the angle measured from the left edge of the display image <NUM> to the position <NUM>. The angle R4 is the angle measured from the left edge of the display area <NUM> whose center is shifted to the position <NUM>.

The position determination unit <NUM> calculates a vertical angle γAi' of an object #i in the display image <NUM> based on the horizontal angles θAi and θAi'. Specifically, when the object #i is located in front of the viewer, the position determination unit <NUM> calculates the vertical angle γAi' in a similar way to the horizontal angle θAi'.

On the other hand, when the object #i is located behind the viewer, the extension of the display area <NUM> allows the position of the object #i indicated by the horizontal angle θAi to be shifted to the position just behind the viewer. Thus, when the vertical angle γAi' is calculated in a similar way to the horizontal angle θAi', the position indicated by the vertical angle γAi contained in the object position information of the object #i is shifted to the position just above or just below the viewer. However, when the object #i is located behind the viewer, unless the object #i is shifted behind the viewer by change in the horizontal angle θAi, it is desirable to prevent the vertical angle γAi' from being changed even when the display area <NUM> is extended or reduced, or in this case, it is desirable to change the vertical angle γAi' in the same direction as the direction in which the horizontal angle θAi is changed.

Thus, when the object #i indicated by the horizontal angle θAi and the object #i indicated by the horizontal angle θAi' are all located in front of the viewer, i.e., if the condition of abs (θAi) < <NUM>° and abs (θAi') < <NUM>° is satisfied, the position determination unit <NUM> calculates the vertical angle γAi' using the following Equation (<NUM>). <NUM>] <MAT>.

In Equation (<NUM>), γv1' is the vertical angle at the upper edge in the display area <NUM> contained in the display area image frame size information, and γv2' is the vertical angle at the lower edge. In Equation (<NUM>), γv1 is the vertical angle at the upper edge contained in the content image frame size information, and γv2 is the vertical angle at the lower edge. when the object #i indicated by the horizontal angle θAi and the object #i indicated by the horizontal angle θAi' are all located behind the viewer, i.e., if the condition of abs (θAi) ≥ <NUM>° and abs (θAi') ≥ <NUM>° is satisfied, the position determination unit <NUM> calculates the vertical angle γAi' using the following Equation (<NUM>). In other words, the vertical angle γAi becomes equal to the vertical angle γAi. <NUM>] <MAT>.

Furthermore, when one of the object #i indicated by the horizontal angle θAi and the object #i indicated by the horizontal angle θAi' is located in front of the viewer and the other is located behind the viewer, i.e., if the condition of abs (θAi) < <NUM>° and abs (θAi') ≥ <NUM>° or abs (θAi) ≥ <NUM>° and abs (θAi') < <NUM>° is satisfied, the position determination unit <NUM> calculates the vertical angle γAi' as described below.

In other words, in this case, the object #i is shifted from a position in front of the viewer to a position behind the viewer, or is shifted from a position behind the viewer to a position in front of the viewer. Thus, it will be difficult to simply conclude that the object #i is located at a position in front of or behind the viewer. In this case, it is necessary that the calculation procedure be split into two stages. One stage is to allow the horizontal angle of an object #i to have a range between an angle indicating the position of the object #i in front of the viewer and the angle of <NUM>° that is the angle indicating the left side of the viewer or the angle of -<NUM>° indicating the right side of the viewer. The other stage is to allow the horizontal angle of an object #i to have a range between the angle of <NUM>° or -<NUM>° and an angle indicating the position of the object #i behind the viewer.

Specifically, the position determination unit <NUM> sets the shift amount of the vertical angle γAi in the stage in which the horizontal angle of the object #i has a range between <NUM>° or -<NUM>° and an angle indicating the position of the object #i behind the viewer to be equal to zero. The position determination unit <NUM> calculates the shift amount of the vertical angle γAi in the stage in which the horizontal angle of an object #i has a range between an angle indicating the position of the object #i in front of the viewer and the angle of <NUM>° that is an angle indicating the left side of the viewer or the angle of -<NUM>° indicating the right side of the viewer in a similar way to Equation (<NUM>). As given in the following Equation (<NUM>), the vertical angles γv2', γv2, γv1', and γv1 defined in Equation (<NUM>) are substituted by the vertical angle γv2f', γv2f, γv1f', and γv1f defined in the following Equation (<NUM>), respectively. <NUM>] <MAT>
[Math. <NUM>] <MAT>.

In Equation (<NUM>), Fside indicates the ratio between the vertical angle of the display area <NUM> and the vertical angle of the display image <NUM> when the horizontal angle of the object #i is <NUM>° or -<NUM>°, and is calculated using the following Equation (<NUM>). <NUM>] <MAT>.

In Equation (<NUM>), Fside is calculated by multiplying the ratio between the vertical angle of the display area <NUM> and the vertical angle of the display image <NUM> when the horizontal angle of the object #i is <NUM>° or -<NUM>° by TransForm_Factor.

In Equation (<NUM>), TransForm_Factor is the factor indicating the ratio between the vertical angles γv1 and γv2 and the horizontal angles θv1 and θv2, and is expressed by the following Equation (<NUM>). This factor TransForm_Factor enables the variation in the vertical angle corresponding to the variation in the horizontal angle to be estimated. <NUM>] <MAT>.

As described above, when the object #i is shifted from a position in front of the viewer to a position behind the viewer, i.e., if the condition of abs (θAi) < <NUM>° and abs (θAi') ≥ <NUM>° is satisfied, the variation in the vertical angle of the object #i becomes equal to a variation when the horizontal angle of the object #i is shifted from the horizontal angle θAi to the angle of <NUM>° or -<NUM>°. Thus, the vertical angle γv1' in Equation (<NUM>) becomes equal to the image frame size for abs (θAi') = <NUM>°, i.e., the product of Fside and the vertical angle γv1, and the vertical angle γv2' in Equation (<NUM>) becomes equal to the image frame size for abs (θAi') = <NUM>°, i.e., the product of Fside and the vertical angle γv2. when the object #i is shifted from a position behind the viewer to a position in front of the viewer, i.e., if the condition of abs abs (θAi) ≥ <NUM>° and abs (θAi') < <NUM>° is satisfied, the variation in the vertical angle of the object #i becomes equal to a variation when the horizontal angle of the object #i is shifted from the angle of <NUM>° or -<NUM>° to the horizontal angle θAi'. Thus, the vertical angle γv1 in Equation (<NUM>) becomes equal to the product of Fside and the vertical angle γv1, and the vertical angle γv2 in Equation (<NUM>) becomes equal to the product of Fside and the vertical angle γv2.

In the above case, although the vertical angle is intended not to change when the object #i is located behind the viewer, the vertical angle may be set to change in the same direction as the horizontal angle. In other words, when the position indicated by the horizontal angle is shifted to a position just behind the viewer, the position indicated by the vertical angle also may be shifted to a position just behind the viewer. When the position indicated by the horizontal angle is shifted to a position right in front of the viewer, the position indicated by the vertical angle also may be shifted to a position right in front of the viewer.

The position determination unit <NUM> may set the distance rAi' of the object #i in the display image <NUM> to the distance rAi contained in the object position information of the object #i. The position determination unit <NUM> supplies the horizontal angle θAi', vertical angle γAi', and distance rAi of the object #i which are obtained as described above to the voice synthesis processor <NUM>, which accepts these values as the position of the object #i.

<FIG> is a graph illustrating the relationship between the horizontal angle θAi and the horizontal angle θAi'.

In the graph shown in <FIG>, the horizontal axis represents the horizontal angle θAi, and the vertical axis represents the horizontal angle θAi'.

As shown in <FIG>, if the condition of θAi' < θAi < θAi' is satisfied, the horizontal angle θAi is shifted by the shift amount θshift and is extended, then the horizontal angle θAi becomes equal to the horizontal angle θAi'. If the condition of - <NUM>° ≤ θAi ≤ θv2' or θv1' ≤ θAi ≤ <NUM>° is satisfied, the horizontal angle θAi is shifted by the shift amount θshift and is reduced, then the horizontal angle θAi becomes equal to the horizontal angle θAi'.

<FIG> is a graph illustrating the relationship between the vertical angle γy1' and the vertical angle γAi'.

In the graph shown in <FIG>, the horizontal axis represents the vertical angle γv1' and the vertical axis represents the vertical angle γAi'.

In the example shown in <FIG>, the horizontal angle θAi and the vertical angle γAi are both <NUM>°. The vertical angles γv1 and γv2 are changed in the same ratio as the horizontal angles θv1 and θv2. In other words, TransForm_Factor is equal to <NUM>. The angles θv1, θv2, γv1, and γv2 are equal to <NUM>°, -<NUM>°, <NUM>°, and -<NUM>°, respectively.

In this case, as shown in <FIG>, if the condition of <NUM>° < γv1' ≤ (Fside * γv1) is satisfied, the vertical angle γAi' increases as the image frame size in the vertical direction of the display image <NUM> increases. If the condition of (Fside * γv1) ≤ γv1' ≤ <NUM>° is satisfied, the vertical angle γAi' becomes equal to the vertical angle γAi' when γy1' = (Fside * γv1), i.e., when the horizontal angle θAi' of the object #i is <NUM>°, regardless of the magnitude the image frame size in the vertical direction of the display image <NUM>.

<FIG> is a flowchart illustrated to describe the streaming playback process to be performed by the streaming playback section <NUM> of <FIG>.

In step S131 of <FIG>, the MPD acquisition unit <NUM> of the streaming playback section <NUM> acquires the MPD file from the Web server <NUM> and supplies the MPD processor <NUM> with the acquired MPD file.

In step S132, the MPD processor <NUM> acquires the content image frame size information and the tile position information, which are described in "AdaptationSet" for image, from the MPD file supplied from the MPD acquisition unit <NUM>. The MPD processor <NUM> supplies the position determination unit <NUM> with the image frame size information and supplies the image selector <NUM> with the tile position information. The MPD processor <NUM> extracts information such as a URL described in "Segment" for voice metafile and supplies the metafile acquisition unit <NUM> with the extracted information.

In step S133, he metafile acquisition unit <NUM> requests a voice metafile from the Web server <NUM> based on the information supplied from the MPD processor <NUM>. The voice metafile to be requested is specified in a URL among the information supplied from the MPD processor <NUM>. Then, the metafile acquisition unit <NUM> acquires the voice metafile. The metafile acquisition unit <NUM> supplies the voice selector <NUM> with object position information contained in the voice metafile.

In step S134, the position determination unit <NUM> performs a position determination process of determining the position of an object in a display image, based on object position information, content image frame size information, and display area image frame size information. The position determination process will be described in detail with reference to <FIG> described later herein.

In step S135, the MPD processor <NUM> extracts information such as a URL, which is described in "Segment" for voice file of all objects, from the MPD file. Then, the MPD processor <NUM> supplies the voice file acquisition unit <NUM> with the extracted information.

In step S136, the voice file acquisition unit <NUM> requests and acquires, based on the information such as a URL supplied from the MPD processor <NUM>, voice files of all the objects specified in the URL from the Web server <NUM>. Then, the voice file acquisition unit <NUM> supplies the voice decoding processor <NUM> with the acquired voice file in units of objects.

The process in steps S137 to S140 is substantially similar to the process in steps S36 to S39 of <FIG>, and thus a description thereof is omitted.

In step S141, the voice synthesis processor <NUM> synthesizes voice data in units of objects supplied from the voice decoding processor <NUM> based on the object position supplied from the position determination unit <NUM> and outputs the synthesized voice data.

In step S142, the image synthesis processor <NUM> synthesizes image data in units of tiles supplied from the image decoding processor <NUM>.

In step S143, the image synthesis processor <NUM> converts the size of the image corresponding to the synthesized image data to the size of the video content to generate a display image. The image synthesis processor <NUM> outputs the display image, and then the process is terminated.

<FIG> is a flowchart illustrated to describe in more detail the position determination process in step S134 of <FIG>. The position determination process is performed, for example, for each object.

In step S151 of <FIG>, the position determination unit <NUM> performs a horizontal angle θAi' estimation process of estimating the horizontal angle θAi' in a display image. The detailed description of the horizontal angle θAi' estimation process will be given later with reference to <FIG>.

In step S152, the position determination unit <NUM> performs a vertical angle γAi' estimation process of estimating the vertical angle γAi' in a display image. The detailed description of the vertical angle γAi' estimation process is similar to the horizontal angle θAi' estimation process in step S151 except that the vertical direction is used in place of the horizontal direction, and thus a detailed description thereof will be omitted.

In step S153, the position determination unit <NUM> sets the distance rAi' in a display image to be equal to the distance rAi contained in the object position information supplied from the metafile acquisition unit <NUM>.

In step S154, the position determination unit <NUM> outputs the horizontal angle θAi', vertical angle γAi', and distance rAi to the voice synthesis processor <NUM>, which accepts these values as the position of the object #i. Then, the process returns to step S134 of <FIG> and proceeds to step S135.

<FIG> is a flowchart illustrated to describe in detail the horizontal angle θAi' estimation process in step S151 of <FIG>.

In step S171 of <FIG>, the position determination unit <NUM> acquires the horizontal angle θAi contained in the object position information supplied from the metafile acquisition unit <NUM>.

In step S172, the position determination unit <NUM> acquires the content image frame size information supplied from the MPD processor <NUM> and the display area image frame size information specified by the user.

In step S173, the position determination unit <NUM> calculates the shift amount θshift by the above-mentioned Equation (<NUM>) based on the content image frame size information and the display area image frame size information.

In step S174, the position determination unit <NUM> calculates the horizontal angles θv1_shift' and θv2_shift' by the above-mentioned Equation (<NUM>) using the shift amount θshift and the display area image frame size.

In step S175, the position determination unit <NUM> calculates the horizontal angle θAi_shift by the above-mentioned Equation (<NUM>) using the horizontal angle θAi and the shift amount θshift.

In step S176, the position determination unit <NUM> determines whether the object #i is located in the display area <NUM> (the horizontal angle of the object #i ranges between horizontal angles at both edges of display area <NUM>) or whether the condition of θv2_shift' < θAi_shift < θv1_shift' is satisfied.

If it is determined in step S176 that the object #i is located in the display area <NUM> or that the condition of θv2_shift' < θAi_shift < θv1_shift' is satisfied, the process proceeds to step S177. In step S177, the position determination unit <NUM> calculates the horizontal angle θAi' by the above-mentioned Equation (<NUM>) based on the content image frame size information, horizontal angles θv1_shift' and θv2_shift', and vertical angle γAi_shift'.

On the other hand, if it is not determined in step S176 that the object #i is located in the display area <NUM> or that the condition of -<NUM>° ≤ θAi_shift ≤ θv2_shift' or θv1_shift' ≤ θAi_shift ≤ <NUM>° is satisfied, the process proceeds to step S178. In step S178, the position determination unit <NUM> calculates the horizontal angle θAi' by the above-mentioned Equation (<NUM>) based on the content image frame size information, horizontal angle θv1_shift' or θv2_shift', and horizontal angle θAi_shift'.

When the process in steps S177 and <NUM> is performed, the process returns to step S151 of <FIG> and proceeds to step S152.

Although the size of the display image is set to be equal to the size of the video content in the second embodiment, the size of both may be different from each other.

In the second embodiment, voice data of some objects (e.g. objects in a display area or objects within a predetermined range from a display area) may be synthesized and outputted, rather than the voice data of all objects is synthesized and outputted. A method of selecting an object of voice data to be outputted may be determined in advance or may be decided by the user.

A third embodiment of the information processing system to which the present disclosure is applied is substantially similar to the second embodiment except for a method of determining a position of an object in a display area by the position determination unit <NUM> of <FIG>. Thus, a description will be given only of a method of determining a position of an object in a display area by the position determination unit <NUM> of <FIG>.

When a viewer views a video content item, the change in a distance from the viewer to the image frame causes the angle of the image frame viewed from the viewer to be changed accordingly. For example, as the viewer approaches the image frame, the image frame is visible larger. As the viewer is away from the image frame, the image frame is visible smaller. Thus, the change in size of the display area or the image frame can be represented by the movement of the viewer. In the third embodiment, the change in size of the display area is represented by the movement of the viewer.

<FIG> are diagrams illustrated to describe a method of determining a position of an object in a display area according to the third embodiment of the information processing system to which the present disclosure is applied.

<FIG> and <FIG> are diagrams illustrated to describe a state in which the size of the display area <NUM> in the video content is changed. <FIG> is a top view seen from the top of the head of the viewer, and <FIG> is a side view seen from the right side of the viewer.

<FIG> and <FIG> are diagrams illustrated to describe a state in which the viewer is moved and the size of the display area <NUM> seen from the viewer is equal to the size of the video content with the movement of the viewer. <FIG> is a top view seen from the top of the head of the viewer, and <FIG> is a side view seen from the right side of the viewer.

As shown in <FIG>, the relative positional relationships between the viewer and an object #<NUM> within the display area <NUM> and between the viewer and objects #<NUM> and #<NUM> outside the display area <NUM> are changed with the movement of the viewer. Thus, the position determination unit <NUM> estimates object position information when the size of the display area <NUM> is changed to be equal to the size of the video content, i.e., object position information in the display image, by obtaining object position information of the object changed by the movement of the viewer.

Specifically, the position determination unit <NUM> calculates a horizontal width ScreenWidth in the display area <NUM> before the movement, a width ScreenLeftwidth extending between the left edge and the center, and a height ScreenTopHeight extending between the top edge and the center, using the following Equation (<NUM>). <NUM>] <MAT>.

In Equation (<NUM>), rv is the distance from the viewer to the image frame, θv1' is the horizontal angle at the left edge in the display area <NUM> contained in the display area image frame size information, θv2' is the horizontal angle at the right edge, and γv1' is the vertical angle at the upper edge.

Then, the position determination unit <NUM> estimates a distance rv' from the viewer to the image frame when the size of the display area <NUM> seen from the viewer is equal to the size of the video content. This estimation is performed using the following Equation (<NUM>). <NUM>] <MAT>.

In Equation (<NUM>), θv1 is the horizontal angle at the left edge contained in the display area image frame size information of the display area <NUM> after the movement, i.e., the content image frame size information, and θv2 is the horizontal angle at the right edge.

Then, the position determination unit <NUM> estimates a shift amount Lengthshift in the back and forth direction of the viewer, a shift amount Widthshif in the side-to-side direction of the viewer, and a shift amount Heightshift in the up-and-down direction of the viewer when the size of the display area <NUM> seen from the viewer is equal to the size of the video content with the movement of the viewer. This estimation is performed using the following Equation (<NUM>). <NUM>] <MAT>.

In Equation (<NUM>), vertical angle γv1 is the vertical angle at the upper edge contained in the display area image frame size information of the display area <NUM> after the movement, i.e., the content image frame size information. In Equation (<NUM>), FAspect_ratio is the factor indicating the variation of the ratio between the width in the vertical direction and the width in the horizontal direction of the display area <NUM> seen from the viewer before and after the movement, and is expressed by the following Equation (<NUM>). This factor FAspect_ratio allows the variation in width in the vertical direction corresponding to the variation in width in the horizontal direction to be estimated. <NUM>] <MAT>.

In Equation (<NUM>), γv2 is the vertical angle at the lower edge contained in the display area image frame size information of the display area <NUM> after the movement, i.e., the content image frame size information, and γv2' is the vertical angle at the lower edge contained in the display area image frame size information of the display area <NUM>.

Consequently, the position determination unit <NUM> estimates the object position information of the object in the display image using the obtained information as described above.

Specifically, when an object is located in the display area <NUM>, i.e., if the condition of θv2' ≤ θAi ≤ θv1' and γv2' ≤ γAi ≤ γv1' is satisfied (in the examples of <FIG>, the object is object #<NUM>), the position determination unit <NUM> estimates the object position information in the display image using the following Equation (<NUM>). <NUM>] <MAT>.

In Equation (<NUM>), θAi' is the horizontal angle of the object #<NUM> after the movement, θAi is the horizontal angle of the object #<NUM> before the movement, γA1' is the vertical angle of the object #<NUM> after the movement, and a vertical angle γAi is the horizontal angle of the object #<NUM> before the movement. In Equation (<NUM>), rAi' is the distance of the object #<NUM> after the movement and rAi is the distance of the object #<NUM> before the movement.

On the other hand, when no object is located in the display area <NUM>, i.e., if the condition of θv2' ≤ θAi ≤ θv1' and γv2' ≤ γAi ≤ γv1' is not satisfied (in the examples of <FIG>, objects #<NUM> and #<NUM> are used), the position determination unit <NUM> estimates the object position information in the display image using the following Equation (<NUM>). <NUM>] <MAT>.

Note that Equation (<NUM>) may be modified like the following equation (<NUM>). <NUM>] <MAT>.

In Equation (<NUM>), xAi', yv1', and zAi' are positions in the back and forth direction (depth direction), in the side-to-side direction, and in the up-and-down direction, respectively, in the xyz coordinate system of the object #i after the movement. Equation (<NUM>) is obtained as a modification of Equation (<NUM>), and Equations (<NUM>) and (<NUM>) are equivalent to each other.

As described above, the object position information after the movement is estimated using the distance rv from the viewer to the image frame and the distance rAi of the object #i before the movement. However, when any of rv and rAi is not given, it may be assumed that rv = rAi. Furthermore, when none of them are given, it may be assumed that rv = rAi = <NUM>.

When at least one of rv and rAi is not given, a restriction that rv' calculated by Equation (<NUM>) does not exceed <NUM> * rv may be provided to prevent the shift amount in the back and forth direction of the viewer from exceeding rv. In other words, rv' calculated by Equation (<NUM>) may be used to obtain a final value of rv' using an equation of rv' = min (<NUM> * rv, rv').

A streaming playback process to be performed by the streaming playback section according to the third embodiment of the information processing system to which the present disclosure is applied is the same as the streaming playback process shown in <FIG> except for the position determination process in step S134, and thus the following description will be given only of the position determination process.

<FIG> is a flowchart illustrated to describe in detail the position determination process according to the third embodiment of the information processing system to which the present disclosure is applied.

In step S181 of <FIG>, the position determination unit <NUM> acquires the display area image frame size information and the content image frame size information of the display area <NUM> before the movement. Specifically, the position determination unit <NUM> acquires the horizontal angle θv1' at the left edge, horizontal angle θv2' at the right edge, vertical angle γv1' at the upper edge, and vertical angle γv2' at the lower edge, of the image frame in the display area <NUM> before the movement. The position determination unit <NUM> also acquires the horizontal angle θv1 at the left edge, horizontal angle θv2 at the right edge, vertical angle γv1 at the upper edge, vertical angle γv2' at the lower edge, and distance rv, of the image frame of the video content.

In step S182, the position determination unit <NUM> calculates a horizontal width ScreenWidth, a width ScreenLeftwidth, and a height ScreenTopHeight by the above-mentioned Equation (<NUM>) using the horizontal angles θv1' and θv2', vertical angle γv1', and distance rv.

In step S183, the position determination unit <NUM> estimates a distance rv' after the movement by the above-mentioned Equation (<NUM>) using the horizontal width ScreenWidth and horizontal angles θv1 and θv2.

In step S184, the position determination unit <NUM> estimates the shift amounts Lengthshift, Widthshift, and Heightshift of the viewer by the above-mentioned Equations (<NUM>) and (<NUM>) using the width ScreenLeftWidth, height ScreenTopHeight, distances rv and rv', horizontal angles θv1, θv2, θv1', and θv2', and vertical angles γvi, γv2, γv1', and γv2'.

In step S185, the position determination unit <NUM> selects an object #i as a first target to be played.

In step S186, the position determination unit <NUM> acquires a horizontal angle θAi, a vertical angle γAi, and a distance rAi that are object position information before the movement of the selected object #i.

In step S187, the position determination unit <NUM> determines whether the object #i is located in the display area <NUM>, i.e., whether the condition of θv2' ≤ θAi ≤ θv1' and γv2' ≤ γAi ≤ γv1' is satisfied.

If it is determined in step S187 that the object #i is located in the display area <NUM>, i.e., the condition of θv2' ≤ θAi ≤ θv1' and γv2' ≤ γAi ≤ γv1' is satisfied, the process proceeds to step S188.

In step S188, the position determination unit <NUM> estimates object position information of the object #i in the display image by the above-mentioned Equation (<NUM>) using the shift amounts Widthshif and Heightshift, distances rv and rv', horizontal angles θAi and θAi', and vertical angles γAi and γAi'. Then, the process proceeds to step S190.

On the other hand, if it is not determined in step S187 that the object #i is located in the display area <NUM>, i.e., the condition of θv2' ≤ θAi ≤ θv1' and γv2' ≤ γAi ≤ γv1' is not satisfied, the process proceeds to step S189.

In step S189, the position determination unit <NUM> estimates object position information of the object #i in the display image by the above-mentioned Equation (<NUM>) using the shift amounts Lengthshift, Widthshift and Heightshift, the distance rAi, horizontal angles θAi and θAi', and vertical angles γAi and γAi'. Then, the process proceeds to step S190.

In step S190, the position determination unit <NUM> determines whether the object position information for all the objects to be played is estimated. If it is not determined in step S190 that the object position information for all the objects to be played is estimated, the process proceeds to step S191.

In step S191, the position determination unit <NUM> selects an object following the currently selected object among objects to be played, and then the process returns to step S186.

On the other hand, if it is determined in step S190 that the object position information for all the objects is estimated, the process proceeds to step S192.

In step S192, the position determination unit <NUM> outputs the object position information for all the objects to be played in the display image to the voice synthesis processor <NUM>. After the process in step S192, the process returns to step S134 of <FIG>, and then proceeds to step S135.

In the third embodiment, voice data of some objects (e.g. objects in a display area or objects within a predetermined range from a display area) may be synthesized and outputted, rather than the voice data of all objects to be played is synthesized and outputted. A method of selecting an object of voice data to be outputted may be determined in advance or may be decided by the user.

The series of processes by the Web server <NUM> described above can be executed by hardware or software. When the series of processes are executed by software, a program constituting the software is installed in a computer. Here, the computer includes a computer incorporated into dedicated hardware, a general-purpose personal computer, for example, that can execute various functions by installing various programs, and the like.

<FIG> is a block diagram illustrating an example of a hardware configuration of a computer which executes the above-described series of processes by the Web server <NUM> using a program.

In the computer, a central processing unit (CPU) <NUM>, a read only memory (ROM) <NUM>, and a random access memory (RAM) <NUM> are connected to one another by a bus <NUM>.

The bus <NUM> is further connected with an input and output interface <NUM>. The input and output interface <NUM> is connected with an input unit <NUM>, an output unit <NUM>, a storage unit <NUM>, a communication unit <NUM>, and a drive <NUM>.

The input unit <NUM> includes a keyboard, a mouse, a microphone, or the like. The output unit <NUM> includes a display, a speaker, or the like. The storage unit <NUM> includes a hard disk, a non-volatile memory, or the like. The communication unit <NUM> includes a network interface or the like. The drive <NUM> drives a removable medium <NUM> such as a magnetic disk, an optical disc, a magneto-optical disc, or a semiconductor memory.

The series of processes described above are performed in the computer configured as described above when the CPU <NUM> loads, for example, the program stored in the storage unit <NUM> in the RAM <NUM> through the input and output interface <NUM> and the bus <NUM> for execution.

The program executed by the computer (CPU <NUM>) can be provided by being recorded in the removable medium <NUM> serving as, for example, a package medium or the like. In addition, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

The program can be installed in the storage unit <NUM> of the computer using the input and output interface <NUM> by loading the removable medium <NUM> in the drive <NUM>. In addition, the program can be received by the communication unit <NUM> and installed in the storage unit <NUM> via the wired or wireless transmission medium. Further, the program can be installed in advance in the ROM <NUM> or the storage unit <NUM>.

Note that the program executed by the computer may be a program which performs the processes in a time series manner in the order described in the present specification, or may be a program which performs the processes in parallel or at necessary timings when they are invoked, or the like.

The video playback terminal <NUM> may have hardware configuration that is substantially similar to that of the computer shown in <FIG>. In this case, the CPU <NUM> may execute the control software <NUM>, the video playback software <NUM>, and the access software <NUM>. Processing by the video playback terminal <NUM> may be executed using hardware.

Further, in the present disclosure, a system has the meaning of a set of a plurality of configured elements (such as an apparatus or a module (part)), and does not take into account whether or not all the configured elements are in the same casing. Therefore, the system may be either a plurality of apparatuses, stored in separate casings and connected through a network, or a plurality of modules within a single casing.

An embodiment of the disclosure is not limited to the embodiments described above, and various changes and modifications may be made without departing from the scope of the disclosure.

For example, the file generation device <NUM> may be configured to generate a video stream by multiplexing encoded data of all tiles rather than the generation of an image file in units of tiles to generate a single image file.

The voice corresponding to the display area <NUM> may include a voice from a fixed object outside the display area <NUM> in addition to the voice from an object within the display area <NUM>.

Claim 1:
An information processing device (<NUM>) for use with MPEG-DASH content, the device comprising:
a screen split processor (<NUM>) configured to split image data of MPEG-DASH video content into tile units and to generate tile position information;
an image encoding processor (<NUM>) configured to encode the image data for each tile to generate a video stream;
an image file generator (<NUM>) configured to process the video stream of each tile into a file format in units of segments, and generate a resulting image file of each tile;
an image information generator (<NUM>) configured to supply an information generator (<NUM>) with the tile position information and with image frame size information, the image frame size information indicating an image frame size of image data;
an audio file generator (<NUM>) configured to generate an audio file including audio data of a plurality of the objects and audio position information indicating a position of each of the objects in the audio file of the audio data;
a metafile generator (<NUM>) configured to generate a metadata file of the audio data including the audio position information;
the information generator (<NUM>) configured to generate, as an MPD file, information used to specify the image frame size information, tile size information, and the metadata file; and
a transmitter (<NUM>) configured to transmit the MPD file generated by the information generator and the metadata file generated by the metafile generator,
wherein the image frame size information is configured as information indicating an angle of view in horizontal and vertical directions of the image data and a distance between a base point of the angle of view and an image plane,
wherein the audio position information is configured as information indicating an angle in horizontal and vertical directions of a line connecting a position in which the audio data is acquired and a base point and a distance between the position in which the audio data is acquired and the base point,
wherein the transmitter is configured to transmit the audio data corresponding to a display area that is an area to be displayed in an image corresponding to the image data, the audio data being requested based on the image frame size information and the audio position information,
wherein the audio data is audio data in units of objects, and
wherein the transmitter is configured to transmit audio data of the object corresponding to the display area.