Patent Description:
Compared with classic content consumption, for VR the content creators cannot any-longer control what the user visualises at various moments in time - the current viewport. The user has the freedom to choose different viewports at each instance of time, out of the allowed or available viewports.

A common issue of VR content consumption is the risk that the user will miss the important events in the Video scene due to wrong viewport selection. For addressing this issue, the notion of Region Of Interest (ROI) was introduced and several concepts for signaling the ROI are considered. Although, the ROI is commonly used to indicate to the user the region containing the recommended viewport, it can also be used with other purposes, such as: indicating the presence of a new character/object in the scene, indicating accessibility features associated with objects in the scene, basically any feature that can be associated with an element composing the video scene. For example, visual messages (e.g., "Turn your head to left") can be used and overlaid over the current viewport. Alternatively, audible sounds can be used, either natural or synthetic sounds, by playing them back at the position of the ROI. These Audio messages are known as "Earcons".

In the context of this application the notion of Earcon will be used to characterise Audio messages conveyed for signaling the ROls, but the signaling and the processing proposed can be used also for generic Audio messages with other purpose than signaling ROIs. One example of such Audio messages is given by Audio messages for conveying information/indication of various options the user has in an interactive AR/VR/MR environment (e.g., "jump over the box to your left for entering room X"). Additionally, the VR example will be used, but the mechanisms described in this document apply to any media consumption environment.

The following terminology is used in the technical field:.

In the context of this application the notions of the Adaptation Sets are used more generic, sometimes referring actually to the Representations. Also, the media streams (audio/video streams) are generally encapsulated first into Media segments that are the actual media files played by the client (e.g., DASH client). Various formats can be used for the Media segments, such as ISO Base Media File Format (ISOBMFF), which is similar to the MPEG-<NUM> container format, and MPEG-TS. The encapsulation into Media Segments and in different Representations/Adaptation Sets is independent of the methods described in here, the methods apply to all various options.

Additionally, the description of the methods in this document may be centered around a DASH Server-Client communication, but the methods are generic enough to work with other delivery environments, such as MMT, MPEG-<NUM> Transport Stream, DASH-ROUTE, File Format for fileplayback etc..

A delivery mechanisms for <NUM>-degree content is given by the ISO/IEC <NUM>-<NUM>, Omnidirectional Media Format [<NUM>]. This standard specifies the media format for coding, storage, delivery, and rendering of omnidirectional images, Video and the associated Audio. It provides information about the media codecs to be used for Audio and Video compression and additional metadata information for correct consumption of the <NUM>-degree A/V content.

It also specifies constrains and requirements on the delivery channels, such as Streaming over DASH/MMT or file-based playback.

The Earcon concept was first introduced in M41184, "Use of Earcons for ROI Identification in <NUM>-degree Video" [<NUM>], which provides a mechanism for signaling of the Earcon Audio data to the user.

However, some users have reported disappointing comments of these systems. Often, a great quantity of Earcons has resulted annoying. When the designers have reduced the number of Earcons, some users have lost important information. Notably, each user has his/her own knowledge and level of experience, and would prefer a system suitable for himself/herself. Just to give an example, each user would prefer to have Earcons reproduced at a preferred volume (independent, for example, from the volume used for the other Audio signals). It has been proven difficult, for the system designer, to obtain a system which provides a good level of satisfaction for all the possible users. A solution has therefore been searched for permitting an increase of satisfaction for almost all the users.

Further, it has been proven difficult to reconfigure the systems even for the designers. For example, they have experienced difficulty in preparing new releases of the Audio Streams and to update the Earcons.

Further, a restricted system imposes certain limitations on the functionality, such as the Earcons cannot be accurately identified into one Audio Stream. Moreover, the Earcons have to be always active and can become annoying to the user if played back when they are not needed.

Further, the Earcon spatial information cannot be signaled nor modified by, for example, a DASH Client. Easy access to this information on the Systems level can enable additional feature for better user experience.

Moreover, there is no flexibility in addressing various types of Earcons (e.g., natural sound, synthetic sound, sound generated in the DASH Client etc).

All these issues lead to a poor user Quality of Experience. A more flexible architecture would therefore be preferable.

<CIT> discloses a method for generating and transmitting metadata for virtual reality which does not refer to audio messages. Further, the reproduction of an audio message is not carried out according to modified associated audio information message metadata.

Experiments for Omnidirectional MediA Formats are provided by "<NPL>; <NPL>.

A semantics for earcon signaling is disclosed by <NPL>; (<NPL>.

<CIT> discloses a medical image system for alerting a user to a region of interest within an image of interest.

<CIT> discloses a method for performing virtual reality content control.

<NPL>; [<NPL>, discloses a method for speech synthesis.

<FIG> shows an example of a system <NUM> for a virtual reality, VR, augmented reality, AR, mixed reality, MR, or <NUM>-degree Video environment. The system <NUM> may be associated, for example, to a content consumption device (e.g., Head-Mounted Display or the like), which reproduces visual data in a spherical or hemispherical display intimately associated to the head of the user.

The system <NUM> may comprise at least one media Video decoder <NUM> and at least one media Audio decoder <NUM>. The system <NUM> may receive at least one Video Stream <NUM> in which a Video signal is encoded for the representation of a VR, AR, MR or <NUM>-degree Video environment scene 118a to a user. The system <NUM> may receive at least one first Audio Stream <NUM>, in which an Audio signal is encoded for the representation of an Audio scene 118b to a user.

The system <NUM> may also comprise a region of interest, ROI, processor <NUM>. The ROI processor <NUM> may process data associated to a ROI. In general terms, the presence of the ROI may be signalled in viewport metadata <NUM>. The viewport metadata <NUM> may be encoded in the Video Stream <NUM> (in other examples, the viewport metadata <NUM> may be encoded in other Streams). The viewport metadata <NUM> may comprise, for example, positional information (e.g., coordinate information) associated to the ROI. For example, the ROI may, in examples, be understood as a rectangle (identified by coordinates such as the position of one of the four vertexes of the rectangles in the spherical Video and the length of the sides of the rectangle). The ROI is normally projected in the spherical Video. The ROI is normally associated to a visible element which is believed (according to a particular configuration) to be of interest of the user. For example, the ROI may be associated to a rectangular area displayed by the content consumption device (or somehow visible to the user).

The ROI processor <NUM> may, inter alia, control operations of the media Audio decoder <NUM>.

The ROI processor <NUM> may obtain data <NUM> associated to the user's current viewport and/or position and/or head orientation and/or movement (also virtual data associated to the virtual position may understood, in some examples, as being part of data <NUM>). These data <NUM> may be provided at least partially, for example, by the content consumption device, or by positioning/detecting units.

The ROI processor <NUM> may check correspondences between the ROI and the user's current viewport and/or position (actual or virtual) and/or head orientation and/or movement data <NUM> (in examples, other criteria may be used). For example, the ROI processor may check if the ROI is represented in the current viewport. In case a ROI is only partially represented in the viewport (e.g., on the basis of the user's head movements), it may determined, for example, if a minimum percentage of the ROI is displayed in the screen. In any case, the ROI processor <NUM> is capable of recognizing if the ROI is not represented or visible to the user.

In case the ROI is considered to be outside the user's current viewport and/or position and/or head orientation and/or movement data <NUM>, the ROI processor <NUM> may audibly signal the presence of the ROI to the user. For example, the ROI processor <NUM> may request the reproduction of an Audio information message (Earcon) in addition to the Audio signal decoded from the at least one first Audio Stream <NUM>.

In case the ROI is considered to be within the user's current viewport and/or position and/or head orientation and/or movement data <NUM>, the ROI processor may decide to avoid the reproduction of the Audio information message.

The Audio information message may be encoded in an Audio Stream <NUM> (Audio information message Stream), which may be the same of the Audio Stream <NUM> or a different Stream. The Audio Stream <NUM> may be generated by the system <NUM> or may be obtained from an external entity (e.g., server). Audio Metadata, such as Audio information message metadata <NUM>, may be defined for describing properties of the Audio information Stream <NUM>.

The Audio information message is superposed (muxed) to the signal encoded in the Audio Stream <NUM> or may not be selected, e.g., simply on the basis of a decision of the ROI processor <NUM>. The ROI processor <NUM> may base its decision on the viewport and/or position and/or head orientation and/or movement data <NUM>, metadata (such as the viewport metadata <NUM> or other metadata) and /or other criteria (e.g., selections, state of the system, number of Audio information message reproductions that have been already performed, particular functions and/or operations, user preferred settings that can disable the usage of Earcons and so on).

A metadata processor <NUM> may be implemented. The metadata processor <NUM> may be interposed, for example, between the ROI processor <NUM> (by which it may be controlled) and the media Audio decoder <NUM> (which may be controlled from the metadata processor). In examples, the metadata processor is a section of the ROI processor <NUM>. The metadata processor <NUM> may receive, generate, process and/or manipulate the Audio information message metadata <NUM>. The metadata processor <NUM> may also process and/or manipulate metadata of the Audio Stream <NUM>, for example for muxing the Audio Stream <NUM> with the Audio information message Stream <NUM>. In addition or alternative, the metadata processor <NUM> may receive metadata of the Audio Stream <NUM>, for example from a server (e.g., a remote entity).

The metadata processor <NUM> may therefore change the Audio scene reproduction and adapt the Audio information message to particular situations and/or selections and/or states.

Some of the advantages of some implementations are here discussed.

The Audio information messages can be accurately identified, e.g., using the Audio information message metadata <NUM>.

The Audio information messages may be easily activated/deactivated, e.g., by modifying the metadata (e.g., by the metadata processor <NUM>). The Audio information messages may be, for example, enabled/disabled based on the current viewport and the ROI information (and also special functions or effects that are to be achieved).

Audio information message (containing for example status, type, spatial information and so on) can be easily signalled and modified by common equipment, such as a Dynamic Adaptive Streaming over HTTP (DASH) Client, for example.

Easy access to the Audio information message (containing for example status, type, spatial information and so on) on the systems level can therefore enable additional feature for better user experience. Hence, the system <NUM> may be easily customized and permit further implementations (e.g., specific applications) which may be performed by personnel which is independent from the designers of the system <NUM>.

Moreover, flexibility is achieved in addressing various types of Audio information messages (e.g., natural sound, synthetic sound, sound generated in the DASH Client etc.).

Other advantages (which will be also apparent with the following examples):.

Therefore, in a VR/AR environment the user can usually visualize full <NUM>-degree content using for example a Head Mounted Display (HMD) and listen to it over headphones. The user can usually move in the VR/AR space or at least change the viewing direction - the so-called "viewport" for video. Compared with classic content consumption, for VR the content creators cannot any-longer control what the user visualizes at various moments in time - the current viewport. The user has the freedom to choose different viewports at each instance of time, out of the allowed or available viewports. In order to indicate to the user the Region Of Interest (ROI), audible sounds can be used, either natural or synthetic sounds, by playing them back at the position of the ROI. These audio messages are known as "Earcons". This invention proposes a solution for efficient delivery of such messages and proposes an optimized receiver behaviour for making use of the Earcons without affecting the user experience and the content consumption. This leads to an increased Quality of Experience. This can be achieved by using dedicated metadata and metadata manipulation mechanisms on systems level for enabling or disabling the Earcons in the final scene.

The metadata processor <NUM> may be configured to receive and/or process and/or manipulate metadata <NUM> so as to cause, at the decision that the information message is to be reproduced, the reproduction of the Audio information message according to the metadata <NUM>. Audio signals (e.g., those for representing the scene) may be understood as being part of the audio scene (e.g., an audio scene downloaded from a remote server). Audio signals may be in general semantically meaningful for the audio scene and all audio signals present together construct the audio scene. Audio signals may be encoded together in one audio bitstream. Audio signals may be created by the content creator and/or may be associated to the particular scene and/or may be independent from the ROI.

The audio information message (e.g., earcon) may be understood as not semantically meaningful to the audio scene. It may be understood as an independent sound that can be generated artificially, such as recorded sound, a recorder voice of a person, etc. It can be also device-dependent (a system-sound generated at the press of a button on the remote control, for example). The audio information message (e.g., earcon) may be understood as being meant to guide the user in the scene, without being part of the scene.

The audio information message may be independent of the audio signals as above.

According to different examples, it may be either included in the same bitstream, or transmitted in a separate bitstream, or generated by the system <NUM>.

An example of an audio scene composed of multiple audio signals may be:.

The audio information message may be, for example, a recorded sound like "look to the piano player" (the piano being the ROI). If the user is already looking at the piano player, the audio message will not be playedback.

Another example: a door (e.g., a virtual door) is opened behind the user and a new person enters the room; the user is not looking there. The Earcon can be triggered, based on this (information regarding the VR environment, such as virtual position), to announce the user that something happens behind him.

In examples, each scene (e.g., with the related audio and video streams) is transmitted from the server to the client when the user changes the environment.

The audio information message may be flexible. In particular:.

The Audio information message is in general independent of any Audio Signal part of the Audio Scene and not is not used for the representation of the Audio Scene.

Examples of systems embodying or including parts which embody system <NUM> are provided below.

<FIG> shows a system <NUM> (which may contain at least a part embodying system <NUM>) which is here represented as being subdivided into a server side <NUM>, a media delivery side <NUM>, a client side <NUM>, and/or a media consumption device side <NUM>. Each of the sides <NUM>, <NUM>, <NUM>, and <NUM> is a system itself and may be combined with any other system to obtain another system. The Audio information messages are Earcons.

The client side <NUM> may receive the at least one Video Stream <NUM> and/or the at least one Audio Stream <NUM> from the server side <NUM> though a media delivery side <NUM>.

The delivery side <NUM> may be, for example, based on a communication system such as a cloud system, a network system, a geographical communication network or well-known media transport formats (MPEG-<NUM> TS Transport Stream, DASH, MMT, DASH ROUTE etc) or even a file based storage. The delivery side <NUM> may be capable of performing communications in form of electric signals (e.g., on cable, wireless etc) and/or by distributing data packets (e.g., according to a particular communication protocol) with bitStreams in which Audio and Video signals are encoded. The delivery side <NUM> may however be embodied by a point-to-point link, a serial or parallel connection, and so on. The delivery side <NUM> may perform a wireless connection e.g., according to protocols such as WiFi, Bluetooth, and so on.

The client side <NUM> may be associated to a media consumption device, e.g., a HND, for example, into which the user's head may be inserted (other devices may be used, however). Therefore, the user may experience a Video and Audio scene (e.g., a VR scene) prepared by the client side <NUM> on the basis of Video and Audio data provided by the server side <NUM>. Other implementations are, however, possible.

The server side <NUM> is here represented as having a media encoder <NUM> (that can cover Video encoders, Audio encoders, subtitle encoders, etc). This encoder <NUM> may be associated, for example, to an Audio and Video scene to be represented. The Audio scene may be, for example, for recreating an environment and is associated to the at least one Audio and Video data Streams <NUM>, <NUM>, which may be encoded on the basis of the position (or virtual position) reached by the user in the VR, AR, MR environment. In general terms, the Video Stream <NUM> encodes spherical images, only a part of which (viewports) will be seen by the user in accordance to its position and movements. The Audio Stream <NUM> contains Audio data which participates to the Audio scene representation and is meant at being heard by a user. According to examples, the Audio Stream <NUM> may comprise Audio metadata <NUM> (which refer to the at least one Audio signal that is intended to participate to the Audio scene representation) and/or Earcon metadata <NUM> (which may describe Earcons to be reproduced only in some cases).

The system <NUM> is here represented as being at the client side <NUM>. For simplicity, the media Video decoder <NUM> is not represented in <FIG>.

In order to prepare the reproduction of the Earcon (or other Audio information messages), Earcon metadata <NUM> may be used. The Earcon metadata <NUM> may be understood as metadata (which may be encoded in an Audio Stream) which describe and provide attributes associated to the Earcon. Hence, the Earcon (if to be reproduced) may be based on the attributes of the Earcon metadata <NUM>.

Advantageously, the metadata processor <NUM> may specifically be implemented for processing the Earcon metadata <NUM>. For example, the metadata processor <NUM> may control the reception, processing, manipulation, and/or the generation of the Earcon metadata <NUM>. When processed, the Earcon metadata may be represented as modified Earcon metadata <NUM>. According to the invention, the Earcon metadata are manipulated for obtaining a particular effect, and/or for performing Audio processing operations, by multiplexing or muxing, for adding the Earcon to the Audio signal to be represented in the Audio scene.

The metadata processor <NUM> may control the reception, processing, manipulation of the Audio metadata <NUM> associated to the at least one Stream <NUM>. When processed, the Audio metadata <NUM> may be represented as modified Audio metadata <NUM>.

The modified metadata <NUM> and <NUM> may be provided to the media Audio decoder <NUM> (or a plurality of decoders in some examples) for the reproduction of the Audio scene 118b to the user.

In examples, there may be provided, as an optional component, a synthetic Audio generator and/or storing device <NUM>. The generator may synthesize an Audio Stream (e.g., for generating an Earcon which is not encoded in a Stream). The storing device permits to store (e.g., in a cache memory) Earcon Streams (e.g., for future use) which have been generated by the generator and/or obtained in a received Audio Stream.

Hence, the ROI processor <NUM> may decide for the representation of an Earcon on the basis of the user's current viewport and/or position and/or head orientation and/or movement data <NUM>. However, the ROI processor <NUM> may also base its decision on criteria which involve other aspects.

For example, the ROI processor may enable/disable the Earcon reproduction on the basis of other conditions, such as, for example, user's selections or higher layer selections, e.g., on the basis of the particular application that is intended to be consumed. For a Video game application, for example, Earcons or other Audio information messages may be avoided for high-Videogame-levels. This may be simply obtained, by the metadata processor, by disabling the Earcons in the Earcon metadata.

Further, it is possible to disable the Earcons on the basis of the state of the system: if, for example, the Earcon has already been reproduced, its repetition may be inhibited. A timer may be used, for example, for avoiding too quick repetitions.

The ROI processor <NUM> may also request the controlled reproduction of a sequence of Earcons (e.g., the Earcons associated to all the ROIs in the scene), e.g., for instructing the user on the elements which he/she may see. The metadata processor <NUM> may control this operation.

The ROI processor <NUM> may also modify the Earcon position (i.e., the spatial location in the scene) or the Earcon type. For example, some users may prefer to have as Earcon one specific sound play back at the exact location/position of the ROI, while other users can prefer to have the Earcon always played-back at one fixed location (e.g., center, or top position "voice of God" etc) as a vocal sound indication the position where the ROI is located.

It is possible to modify the gain (e.g., to obtain a different volume) of the Earcon's reproduction. This decision may follow a user's selection, for example. Notably, on the basis of the ROI processor's decision, the metadata processor <NUM> will perform the gain modification by modifying, among the Earcon metadata associated to the Earcon, the particular attribute associated to the gain.

The original designer of the VR, AR, MR environment may also be unaware of how the Earcons will be actually reproduced. For example, user's selections may modify the final rendering of the Earcons. Such an operation may be controlled, for example, by the metadata processor <NUM> which may modify the Earcon metadata <NUM> on the basis of the ROI processor's decisions.

Thus, the operations performed on the Audio data associated to the Earcon are therefore in principle independent of the at least one Audio Stream <NUM> used for representing the Audio scene and may be differently managed. The Earcons may even be generated independently of the Audio and Video Streams <NUM> and <NUM> which constitute the Audio and Video scene and may be produced by different and independent entrepreneurial groups.

Hence, the examples permit to increase the satisfaction for users. For example, a user may perform his/her own selections, e.g., by modifying the volume of the Audio information messages, by disabling the Audio information messages, and so on. Therefore, each user may have the experience more suited to his/her preference. Further, the obtained architecture is more flexible. The Audio information messages may be easily updated, for example, by modifying the metadata, independently of the Audio Streams, and/or by modifying the Audio information message Streams independently of the metadata and of the main Audio Streams.

The obtained architecture is also compatible with legacy systems: legacy Audio information message Streams may be associated to new Audio information message metadata, for example. In case of absence of a suitable Audio information message Stream, in examples the latter may be easily synthesized (and, for example, stored for subsequent use).

The ROI processor may keep track of metrics associated to historical and/or statistical data associated to the reproduction of the Audio information message, so as to disable the Audio information message's reproduction if the metrics is over a predetermined threshold (this may be used as criteria).

The ROI processor's decision may be based, as a criteria, on a prediction of user's current viewport and/or position and/or head orientation and/or movement data <NUM> in relationship to the position of the ROI.

The ROI processor may be further configured to receive the at least one first Audio Stream <NUM> and, at the decision that the information message is to be reproduced, to request an Audio message information Stream from a remote entity.

The ROI processor and/or the metadata generator may be further configured to establish whether to reproduce two Audio information messages at the same time or whether to select a higher-priority Audio information message to be reproduced with priority with respect to a lower-priority Audio information message. In order to perform this decision, Audio information metadata may be used. A priority may be, for example, obtained by the metadata processor <NUM> on the basis of the values in the audio information message metadata.

In some examples, the media encoder <NUM> may be configured to search, in a database, intranet, internet, and/or geographical network, an additional Audio Stream and/or Audio information message metadata and, in case of retrieval, delivery the additional Audio Stream and/or the Audio information message metadata. For example, the search may be performed on the request of the client side.

As explained above, a solution is here proposed for efficient delivery of Earcon messages together with the Audio content. An optimised receiver behaviour is obtained, for making use of the Audio information messages (e.g., Earcons) without affecting the user experience and the content consumption. This will lead to an increased Quality of Experience.

This can be achieved by using dedicated metadata and metadata manipulation mechanisms on systems level for enabling or disabling of the Audio information messages in the final Audio scenes. The metadata can be used together with any Audio codecs and complements in a nice fashion the Next Generation Audio codecs metadata (e.g., MPEG-H Audio metadata).

The delivery mechanisms can be various (e.g., Streaming over DASH/HLS, broadcast over DASH-ROUTE/MMT/MPEG-<NUM> TS, file playback etc). In this application DASH delivery is considered, but all concepts are valid for the other delivery options.

In most of the cases the Audio information messages will not overlap in time domain, i.e., at a specific point in time only one ROI is defined. But, considering more advanced use cases, for example in an interactive environment where the user can change the content based on his selections/movements, there could be also use cases which require multiple ROIs. For this purpose, more than one Audio information message can be required at one moment in time. Therefore, a generic solution is described for supporting all different use cases.

The delivery and processing of the Audio information messages should complement the existing delivery methods for Next Generation Audio.

One way of conveying multiple Audio information messages for several ROIs, which are independent in time domain, is to mix together all Audio information messages into one Audio element (e.g., Audio object) with associated metadata describing the spatial position of each Audio information message at different instances of time. Because the Audio information messages don't overlap in time, they can be independently addressed in the one, shared Audio element. This Audio element could contain silence (or no Audio data) in-between the Audio information messages, i.e., whenever there is no Audio information message. The following mechanisms may apply in this case:.

In alternative examples, multiple Earcons for several ROIs, independent in time domain or overlapping in time domain, can be delivered in multiple Audio elements (e.g., Audio objects) and embedded either in one elementary Stream together with the main Audio scene or in multiple auxiliary Streams, e.g., each Earcon in one ES or a group of Earcons in one ES based on a shared property (e.g., all Earcons located on the left side share one Stream).

Alternatively, one common (generic) Earcon can be used for signaling all the ROIs in one Audio scene. This can be achieved by using the same Audio content with different spatial information associated with the Audio content at different instances of time. In this case, the ROI processor <NUM> may request the metadata processor <NUM> to gather the Earcons associated to the ROIs in the scene, and to control the reproduction of the Earcons in sequence (e.g., at a user's selection or at a higher-layer application request).

Alternatively, one Earcon can be transmitted only once and cached in the Client. The Client can re-use it for all the ROIs in one Audio scene with different spatial information associated with the Audio content at different instances of time.

Alternatively, the Earcon Audio content can be generated synthetically in the Client. Together with that, a Metadata Generator can be used for creating the necessary metadata for signaling the spatial information of the Earcon. For example, the Earcon Audio content can be compressed and fed into one Media decoder together with the main Audio content and the new metadata or it can be mixed into the final Audio scene after the Media Decoder, or several Media Decoders can be used.

Alternatively, the Earcon Audio content can, in examples, be generated synthetically in the Client (e.g., under the control of the metadata processor <NUM>), while the Metadata describing the Earcon is embedded already in the Stream. Using specific signaling of the Earcon type in the encoder, the metadata can contain the spatial information of the Earcon, the specific singling for a "Decoder generated Earcon" but no Audio data for the Earcon.

Alternatively, the Earcon Audio content can be generated synthetically in the Client, and a Metadata Generator can be used for creating the necessary metadata for signaling the spatial information of the Earcon. For example, the Earcon Audio content can be.

An example of Audio information message (Earcons) metadata <NUM>, as described above, is provided here.

One structure for describing the Earcon properties and offer possibility to easily adjust these values:.

Each identifier in the table may be intended as being associated to an attribute of the Earcon metadata <NUM>.

numEarcons - This field specifies the number of Earcons Audio Elements available in the Stream. Earcon_isIndependent - This flag defines if the Earcon Audio Element is independent from any Audio Scene. If Earcon_islndependent == <NUM> the Earcon Audio Element is independent from the Audio Scene. If Earcon_islndependent == <NUM> the Earcon Audio Element is part of the Audio Scene and the Earcon_id shall have the same value as the mae_grouplD associated with the Audio Element. EarconType - This field defines the type of the Earcon. The following table specifies the allowed values.

One structure for identifying the Earcons on system level and associate them with existing viewports. The following two tables offer two ways of implementing such structure that can be used in different implementations:
aligned(<NUM>) class EarconSample() extends SphereRegionSample {
for (i = <NUM>; i < num_regions; i++) {
unsigned int(<NUM>) reserved;
unsigned int(<NUM>) hasEarcon;
if (hasEarcon == <NUM>) {
unsigned int(<NUM>) numRegionEarcons;
for (n=<NUM>; n<numRegionEarcons; n++) {
unsigned int(<NUM>) Earcon_id;
unsigned int(<NUM>) Earcon_track_id;
}
}
}
}
or alternatively:
aligned(<NUM>) class EarconSample() extends SphereRegionSample {
for (i = <NUM>; i < num_regions; i++) {
unsigned int(<NUM>) Earcon_track_id;
unsigned int(<NUM>) Earcon_id;
}
}.

For an easy identification of the Earcon(s) track on MPD level, the following Attribute/Element can be used EarconComponent@tag:
Summary of relevant MPD elements and attributes for MPEG-H Audio.

For MPEG-H Audio this can be implemented, in examples, by making use of the MHAS packets:.

With respect to the metadata, the metadata processor <NUM> may have at least some of the following capabilities:.

<FIG> shows a system <NUM> comprising, at the client side <NUM>, a system <NUM> (client system) which may embody, for example, the system <NUM> or <NUM>.

The system <NUM> may comprise the ROI processor <NUM>, the metadata processor <NUM>, a decoder group <NUM> formed by a plurality of decoders <NUM>.

In this example, different Audio Streams are decoded (each at by a respective media Audio decoder <NUM>) and are subsequently mixed together and/or rendered together to provide the final Audio scene.

The at least one Audio Stream is here represented as comprising two Streams <NUM> and <NUM> (other examples may provide one single Stream, as in <FIG>, or more than two Streams). These are the Audio Streams that are meant at reproducing the Audio scene that the user is expected to experience. Here, reference is made to Earcons, even it is possible to generalize the concept to any Audio information messages.

Additionally, an Earcon Stream <NUM> may be provided by the media encoder <NUM>. On the basis of the user's movements and of the ROIs as indicated in the viewport metadata <NUM> and/or other criteria, the ROI processor will cause the reproduction of an Earcon from the Earcon Stream <NUM> (also indicated as additional Audio Stream as being in addition to the Audio Streams <NUM> and <NUM>).

Notably, the actual representation of the Earcon will be based on the Earcon metadata <NUM> and on the modifications performed by the metadata processor <NUM>.

In examples, the Stream may be requested by the system <NUM> (client) to the media encoder <NUM> (server) in case of necessity. For example, the ROI processor may decide that, on the basis of the user's movements, a particular Earcon will be soon needed and, therefore, may request an appropriate Earcon Stream <NUM> to the media encoder <NUM>.

It is possible to note the following aspects of this example:.

<FIG> shows a system <NUM> comprising, at the client side <NUM>, a system <NUM> (client system) which may embody, for example, the system <NUM> or <NUM>. Here, reference is made to Earcons, even it is possible to generalize the concept to any Audio information messages.

The system <NUM> may comprise the ROI processor <NUM>, the metadata processor <NUM>, a Stream multiplexer or muxer <NUM>. By virtue of the multiplexer or muxer <NUM>, the number of operations to be performed by the hardware are advantageously reduced with respect to the number of operations to be performed when multiple decoders and one mixer or renderer are used.

In this example, different Audio Streams are processed, on the basis of their metadata and multiplexed or muxer at the element <NUM>.

The at least one Audio Stream is here represented as comprising two Streams <NUM> and <NUM> (other examples may provide one single Stream, as in <FIG>, or more than two Streams). These are the Audio Streams that are meant at reproducing the Audio scene that the user is expected to experience.

Additionally, an Earcon Stream <NUM> may be provided by the media encoder <NUM>. On the basis of the user's movements and of the ROIs as indicated in the viewport metadata <NUM> and/or other criteria, the ROI processor <NUM> will cause the reproduction of an Earcon from the Earcon Stream <NUM> (also indicated as additional Audio Stream as being in addition to the Audio Streams <NUM> and <NUM>).

Each Audio Stream <NUM>, <NUM>, <NUM> may include metadata <NUM>, <NUM>, <NUM>, respectively. At least some of these metadata are manipulated and/or processed to be provided to the Stream muxer or multiplexer <NUM> in which the packets of the Audio Streams are merged together. Accordingly, the Earcon is represented as a part of the Audio scene.

The Stream muxer or multiplexer <NUM> therefore provides an Audio Stream <NUM> wich comprises modified Audio metadata <NUM> and modified Earcon metadata <NUM>, which may be provided to an Audio decoder <NUM> and decoded and reproduced to the user.

<FIG> shows a system <NUM> according to the invention and comprising, at the client side <NUM>, a system <NUM> (client system) which may embody, for example, the system <NUM> or <NUM>. Here, reference is made to Earcons.

The system <NUM> may comprise the ROI processor <NUM>, the metadata processor <NUM>, a Stream multiplexer or muxer <NUM>.

In this example, an Earcon Stream is not provided by a remote entity (at the client side), but is generated by the synthetic Audio generator <NUM> (which may also have the capability of storing a Stream, for re-using subsequently, or to use a stored compress/uncompressed version of a natural sound). The Earcon metadata <NUM> are notwithstanding provided by the remote entity, e.g., in an Audio Stream <NUM> (which is not an Earcon Stream). Therefore, the synthetic Audio generator <NUM> may be activated to create an Audio Stream <NUM> on the basis of the attributes of the Earcon metadata <NUM>. For example, the attributes may refer to a type of the synthesized voice (natural sound, synthetic sound, spoken text, and so on) and/or text labels (the Earcon can be generated by creating synthetic sound based on the text in the metadata). In examples, after that the Earcon Stream has been created, the same may be stored for being re-used in the future. Alternatively, the synthetic sound can be a generic sound stored permanently in the device.

A Stream muxer or multiplexer <NUM> is used to merge packets of the Audio Stream <NUM> (and in case also of other Streams, such as the auxiliary Audio Stream <NUM>) with the packets of the Earcon Stream generated by the generator <NUM>. After that, an Audio Stream <NUM> is obtained which is associated to modified Audio metadata <NUM> and modified Earcon metadata <NUM>. The Audio Stream <NUM> is decoded by the decoder <NUM> and reproduced to the user at the media consumption device side <NUM>.

<FIG> shows a system <NUM> comprising, at the client side <NUM>, a system <NUM> (client system) which may embody, for example, the system <NUM> or <NUM>. Here, reference is made to Earcons.

In this example, an Earcon Stream is not provided by a remote entity (at the client side), but is generated by the synthetic Audio generator <NUM> (which may also have the capability of storing a Stream, for re-using subsequently).

In this example, the Earcon metadata <NUM> are not provided by the remote entity. The Earcon metadata are generated by a metadata generator <NUM> which may generate Earcon metadata to be used (e.g., processed, manipulated, modified) by the metadata processor <NUM>. The Earcon metadata <NUM> generated by the Earcon metadata generator <NUM> may have the same structure and/or format and/or attribute than the Earcon metadata discussed for the previous examples.

The metadata processor <NUM> may operate as in the example of <FIG>. A synthetic Audio generator <NUM> may be activated to create an Audio Stream <NUM> on the basis of the attributes of the Earcon metadata <NUM>. For example, the attributes may refer to a type of the synthesized voice (natural sound, synthetic sound, spoken text, and so on), and/or to the gain, and/or to the activation/non-activation state, and so on. In examples, after that the Earcon Stream <NUM> has been created, the same may be stored (e.g., cached) for being re-used in the future. It is also possible to store (e.g., cache) the Earcon metadata generated by the Earcon metadata generator <NUM>.

It is possible to implement a function which permits to reproduce an Earcon only when a user does not see the ROI.

The ROI processor <NUM> may periodically check, for example, the user's current viewport and/or position and/or head orientation and/or movement data <NUM>. If the ROI is visible to the user, no reproduction of the Earcon is caused.

If, from the user's current viewport and/or position and/or head orientation and/or movement data, the ROI processor determines that the ROI is not visible to the user, the ROI processor <NUM> may request a reproduction of the Earcon. In this case, the ROI processor <NUM> may cause the metadata processor <NUM> to prepare the reproduction of the Earcon. The metadata processor <NUM> may use one of the techniques described for the examples above. For example the metadata may be retrieved in a Stream delivered by the server side <NUM>, may be generated by the Earcon metadata generator <NUM>, and so on. The attributes of the Earcon metadata may be easily modified on the basis of the ROI processor's requests and/or various conditions. For example, if a user's selection has previously disabled the Earcon, the Earcon will not be reproduced, even if the user does not see the ROI. For example, if a (previously set) timer has not expired yet, the Earcon will not be reproduced, even if the user does not see the ROI.

Additionally, if from the user's current viewport and/or position and/or head orientation and/or movement data, the ROI processor determines that the ROI is visible to the user, the ROI processor <NUM> may request that no reproduction of the Earcon is done, especially if the Earcon metadata already contains signalling for an active Earcon.

In this case, the ROI processor <NUM> may cause the metadata processor <NUM> to disable the reproduction of the Earcon. The metadata processor <NUM> may use one of the techniques described for the examples above. For example, the metadata may be retrieved in a Stream delivered by the server side <NUM>, may be generated by the Earcon metadata generator <NUM>, and so on. The attributes of the Earcon metadata may be easily modified on the basis of the ROI processor's requests and/or various conditions. If the metadata already contains the indication that an Earcon should be reproduced, the metadata is modified, in this case, to indicate that the Earcon is inactive and it should not be reproduced.

Therefore, the final Audio scene experienced by the user will be obtained on the basis of the metadata modifications performed by the metadata processor.

<FIG> shows a system <NUM> comprising, at the client side <NUM>, a system <NUM> (client system) which may embody, for example, the system <NUM> or <NUM> or <NUM> or <NUM> or <NUM>. Here, reference is made to Earcons, even it is possible to generalize the concept to any Audio information messages.

The system <NUM> may comprise the ROI processor <NUM>, the metadata processor <NUM>, a Stream multiplexer or muxer <NUM>. (In examples, different Audio Streams are decoded (each at by a respective media Audio decoder <NUM>) and are subsequently mixed together and/or rendered together to provide the final Audio scene).

Additionally, an Earcon Stream <NUM> may be provided by the media encoder <NUM>.

The Audio Streams may be encoded at different bitrates, that allow efficient bitrate adaptation depending on the network connection (i.e., for users using high speed connection the high bitrate coded version is delivered while for users with lower speed network connection a lower bitrate version is delivered).

The Audio Streams may be stored on a Media Server <NUM>, where for each Audio Stream the different encodings at different bitrates are grouped in one Adaptation Set <NUM> with the appropriate data signalling the availability of all the created Adaptation Sets. Audio adaptation sets <NUM> and Video adaptation sets <NUM> may be provided.

On the basis of the user's movements and of the ROIs as indicated in the viewport metadata <NUM> and/or other criteria, the ROI processor <NUM> will cause the reproduction of an Earcon from the Earcon Stream <NUM> (also indicated as additional Audio Stream as being in addition to the Audio Streams <NUM> and <NUM>).

An Adaptation Set may be formed by a set of Representations containing interchangeable versions of the respective content, e.g., different audio bitrates (e.g., different streams at different bitrates). Although one single Representation could be theoretically enough to provide a playable stream, multiple Representations may give the client the possibility to adapt the media stream to its current network conditions and bandwidth requirements and therefore guarantee smoother playback.

All the examples above may be implemented by method steps. Here, a method <NUM> (which may be performed by any of the examples above) is described for completeness. The method may comprise:.

Notably, the sequence may also vary. For example, the receiving steps <NUM>, <NUM>, <NUM> may have different order, according to the actual order in which the information is delivered.

Line <NUM> refers to the fact that the method may be reiterated. Step <NUM> may be skipped in case of ROI processor's decision of non-reproducing the Audio information message.

<FIG> shows a system <NUM> which may implement one of the system (or a component thereof) or perform the method <NUM>. The system <NUM> may comprise a processor <NUM> and a non-transitory memory unit <NUM> storing instructions which, when executed by the processor <NUM>, may cause the processor to perform at least the Stream processing operations discussed above and/or the metadata processing operations discussed above. The system <NUM> may comprise an input/output unit <NUM> for the connection with external devices.

The system <NUM> may implement at least some of (or all) the functions of the ROI processor <NUM>, the metadata processor <NUM>, the generator <NUM> the muxer or multiplexer <NUM>, the decoder <NUM> the Earcon metadata generator <NUM>, and so on.

Depending on certain implementation requirements, examples may be implemented in hardware. The implementation may be performed using a digital storage medium, for example a floppy disk, a Digital Versatile Disc (DVD), a Blu-Ray Disc, a Compact Disc (CD), a Read-only Memory (ROM), a Programmable Read-only Memory (PROM), an Erasable and Programmable Read-only Memory (EPROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM) or a flash memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed.

Generally, examples may be implemented as a computer program product with program instructions, the program instructions being operative for performing one of the methods when the computer program product runs on a computer. The program instructions may for example be stored on a machine readable medium.

Other examples comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier. In other words, an example of method is, therefore, a computer program having a program instructions for performing one of the methods described herein, when the computer program runs on a computer.

A further example of the methods is, therefore, a data carrier medium (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein. The data carrier medium, the digital storage medium or the recorded medium are tangible and/or non-transitionary, rather than signals which are intangible and transitory.

A further example comprises a processing unit, for example a computer, or a programmable logic device performing one of the methods described herein.

A further example comprises a computer having installed thereon the computer program for performing one of the methods described herein.

A further example comprises an apparatus or a system transferring (for example, electronically or optically) a computer program for performing one of the methods described herein to a receiver.

In some examples, a programmable logic device (for example, a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some examples, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods may be performed by any appropriate hardware apparatus.

Claim 1:
A system configured to:
receive at least one first Audio Stream (<NUM>, <NUM>),
wherein the system comprises:
a media Audio decoder (<NUM>) for the representation of an Audio scene (118b) to a user;
a processor (<NUM>), configured to:
decide, based on the user's current head orientation and/or movement data (<NUM>), provided by positioning/detecting unit(s), whether an Audio information message is to be reproduced, wherein the Audio information message is an earcon; and
cause, at the decision that the Audio information message is to be reproduced, the reproduction of the Audio information message,
the processor further comprising a metadata processor (<NUM>) configured to process Audio information message metadata (<NUM>) so as to cause, at the decision that the Audio information message is to be reproduced, the reproduction of the Audio information message according to the Audio information message metadata (<NUM>),
wherein the metadata processor (<NUM>) is configured to control a muxer or multiplexer (<NUM>) to merge, on the basis of the Audio information message metadata, packets of an additional Audio Stream (<NUM>) in which the Audio information message (<NUM>) is encoded with packets of the at least one first Audio Stream (<NUM>) in one merged Stream (<NUM>),
wherein the media Audio decoder (<NUM>) is configured to decode at least one Audio signal from the merged Stream (<NUM>).