Patent Description:
The term 'multimedia service' refers to call services such as Video Call, streaming services such as Video on Demand (VOD) service, or services such as Multicast and Broadcast Service. Real-time multimedia services may be divided into call services, interactive services and streaming services depending on the service type, and may also be divided into unicast, multicast and broadcast depending on the number of participating users.

The conventional broadcast network usually uses Moving Picture Experts Group-<NUM> Transport Stream (MPEG-<NUM> TS), for transmission of multimedia content. MPEG-<NUM> TS is used as a typical transmission technology for transmitting a bit stream, with which multiple broadcast programs (or multiple coded video bit streams) are multiplexed, in an error-ridden transmission environment. MPEG-<NUM> TS is suitable for use in digital TV broadcasting or the like in the multimedia age.

MPEG-<NUM> TS may have a few limitations in supporting multimedia services. In other words, MPEG-<NUM> TS may have limitations such as the one-way communication, the inefficiency of transmission due to the fixed frame size, and the unnecessary overhead which occurs when data is transmitted using the transport protocol and Internet Protocol (IP) that are specific to audio/video.

Therefore, MPEG has newly proposed an MPEG Media Transport (MMT) standard as one of the multimedia transmission technologies for supporting multimedia services based on the MPEG technology. For example, the MMT standard may be applied to efficiently transmit hybrid content over the heterogeneous network. The term `hybrid content' as used herein may refer to a collection of content having multimedia elements by video/audio/application. The heterogeneous network may be, for example, a network in which a broadcasting network and a mobile communication network coexist.

The MMT standard aims at defining an IP-friendly transmission technology which is the basic technology in a transmission network for multimedia services. To this end, there is a need for a structure of an MMT transport packet for more efficiently delivering logical data objects in a variety of formats, including coded media data, in a changing multimedia service environment.

Document <CIT> relates to an interface for providing a media service, including an encapsulation layer for encapsulating coded media data; a delivery layer for transmitting the encapsulated media data to another entity; and a control layer for controlling transmission of the media data.

Document "<NPL>; held in Geneva, discloses a method for providing "composition information" in layer E. <NUM>, by combining MMT Assets into MMT Packages.

Document "<NPL>; held in Geneva, discloses an encapsulation layer design (E. <NUM>) to receive MMT Packages and produce MMT payload units that can be transported to the other end using an application transport protocol and/or underlying network transport protocols.

Aspects of the present disclosure provide apparatuses for efficiently transmitting and receiving coded media data in a multimedia transmission system.

A first aspect of the present invention provides an apparatus, according to independent claim <NUM>, for transmitting media content in a multimedia system supporting a moving picture experts group, MPEG media transport, MMT, protocol.

A second aspect of the present invention provides an apparatus, according to independent claim <NUM>, for receiving media data in a multimedia system supporting a moving picture experts group, MPEG media transport, MMT, protocol.

Further embodiments are set out in the dependent claims.

Now, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the following description of the present disclosure, a detailed description of known functions and configurations incorporated herein will be omitted when it may obscure the subject matter of the present disclosure. The terms used herein are defined in consideration of their functions in the present disclosure, and may vary depending on the users, the operator's intention, or the usual practices. Therefore, the definition should be made based on the contents throughout the entire specification.

A hierarchical structure defined in the MMT standard will be described herein below. Embodiments of the present disclosure provide the structure of a multimedia data packet. To this end, a detailed description will be made of layers for generating a multimedia data packet in the hierarchical structure defined in the MMT standard. The term 'multimedia data packet' as used herein may refer to a delivery frame for an MMT service.

An MMT Protocol (MMTP) defines an application layer protocol for delivering a delivery frame configured in an MMT Payload Format (PF) over an IP network. An MMT payload is designed to be efficiently delivered by being configured in an MMT payload format. To efficiently deliver MPEG media data over heterogeneous IP networks, MMT defines encapsulation formats, delivery protocols, and signaling message formats.

<FIG> illustrates a hierarchical structure of an MMT system according to an example not including all the claimed feature.

Referring to <FIG>, shown are a media coding layer <NUM>, an encapsulation function layer (Layer E) <NUM>, a delivery function layer (Layer D) <NUM>, a transport protocol layer <NUM>, an Internet Protocol (IP) layer <NUM>, and a control function layer (Layer C) <NUM>, which are for configuring a multimedia data packet and transmitting the configured multimedia data packet.

In accordance with an embodiment of the present disclosure, the media coding layer <NUM> and the encapsulation function layer <NUM> may operate as a multimedia data generation unit for generating multimedia data that is based on multimedia content and/or multimedia services. The delivery function layer <NUM> may operate as a multimedia data configuration unit for configuring a multimedia data packet based on the multimedia data received from the multimedia data generation unit. The delivery function layer <NUM> corresponding to the multimedia data configuration unit may configure header information by identifying at least one multimedia data provided from the multimedia data generation unit, and configure a multimedia data packet by combining the header information with the least one multimedia data.

The multimedia data compressed in the media coding layer <NUM> may be packetized in the form similar to a file format by the encapsulation function layer <NUM>. The encapsulation function layer <NUM> may generate data segments, which are small units for an MMT service, by receiving the coded media data provided from the media coding layer <NUM> or the stored media data, and generate access units for an MMT service using the data segments. The encapsulation function layer <NUM> may generate a packet format for generation/storage and transmission of hybrid content by combining and/or dividing the access units.

The delivery function layer <NUM> may convert a data unit(s) output from the encapsulation function layer <NUM> into an MMT payload format and then, add an MMT transport packet header thereto to configure an MMT transport packet, or may configure a Real-Time Protocol (RTP) packet using an RTP which is the existing transport protocol.

The packets configured in the delivery function layer <NUM> may be finally IP-packetized in the IP layer <NUM> after undergoing the transport protocol layer <NUM> such as User Datagram Protocol (UDP) or Transport Control Protocol (TCP). The transport protocol layer <NUM> and the IP layer <NUM> may operate as a data transmission unit. The control function layer <NUM>, which is optional, may generate control information or signaling information necessary for transmission of data, add the generated information to data, and transmit the data, or may transmit the data through a separate signaling means.

An MMT payload format generated in the delivery function layer <NUM> may define a logical structure of a media unit(s) to be delivered, by means of the MMT protocol or the RTP. An MMT payload may be specified by a payload format for delivering the encapsulated data unit or other information by MMT layer protocols or other existing application transport protocols. The MMT payload may provide information about streaming and information about file transfer. In streaming, a data unit may be an MMT Media Fragment Unit (MFU) or an MMT Processing Unit (MPU). In file transfer, a data unit may be an MMT asset and an MMT package.

<FIG> illustrates a structure of an MMT payload according to an example not including all the claimed feature.

Referring to <FIG>, an MMT payload <NUM> is configured to include at least one of at least one MMT MFU <NUM>, at least one MMT MPU <NUM>, at least one MMT asset <NUM>, and at least one MMT package <NUM>.

The MFU <NUM> may be a common container format including coded media data, which is independent of any media codec, and can be independently processed by a media decoder. The MFU <NUM>, which represents part of segmented data of the MPU <NUM>, may be the minimum unit that can be decoded independently. As an example, if coding is performed by using one frame as an access unit, the MFU <NUM> may be one video frame. In another case, the MFU <NUM> may be one slice included in one frame.

The MPU <NUM>, which is a container format including one or more MFUs and additional delivery and processing-related information, may include a variety of numbers of MFUs generated from a plurality of different access units. The MPU <NUM>, which means a coded media data unit that can be completely and independently processed by an MMT compliant entity, may have a specific size (e.g., one Group of Picture (GOP) in the case of video) depending on the application environment. As an example, the MPU <NUM> may have (or consist of) a plurality of picture frames constituting one GOP (e.g., <NUM>-second video), and the PFU <NUM> may include each picture frame.

The MMT asset <NUM>, which is a data entity having one or more MPUs, may be the largest data unit, to which the same composition information and transport characteristics are applied. The MMT asset <NUM> may include only one type of data, including packaged or multiplexed data. For example, at least a part of an Elementary Stream (ES) of audio, at least a part of an ES of video, at least a part of an MPEG-U (User Interface) widget package, at least a part of an MPEG-<NUM> Transport Stream (TS), at least a part of an MPEG-<NUM> (MP4) file, and all or at least a part of an MMT package may be each MMT asset <NUM>.

The ES, which is defined by a specific media codec, may be logically one or more MMT assets. The MMT asset <NUM> supporting a layered codec and a multi-view codec may overlap with other MMT assets.

MMT Composition Information (MMT-CI) means the information defining a spatial and temporal relationship of MMT assets, and MMT Transport Characteristics (MMT-TC) defines Quality of Service (QoS) required for delivery of MMT assets. MMT-TC may be expressed as Asset Delivery Characteristics (ADC), for a specific delivery environment.

The MMT package <NUM> may be defined as a collection of coded media data and its related information, which are processed by an MMT compliant entity.

<FIG> illustrates a logical structure of an MMT package according to an example not including all the claimed feature. As illustrated, an MMT package <NUM> may have (or consist of) one or more MMT assets <NUM>, MMT composition information <NUM>, and one or more ADCs <NUM> representing MMT transport characteristics. The MMT package <NUM> may include description information such as an identifier, and locations of the MMT assets <NUM>, and the MMT assets <NUM> in the MMT package <NUM> may be multiplexed or concatenated.

The MMT package <NUM> may be processed in units of MPUs, and the MMT asset <NUM> may be a collection of one or more MPUs having the same MMT asset ID. The transport characteristics related to each MMT asset <NUM> may be expressed by the ADC <NUM>. The ADC <NUM> may be used by an entity of packetizing an MMT package, to configure parameters of an MMT payload and below-described header information of an MMT packet.

Among the logical entities that can be included in the MMT package, the MPU may include header fields that are based on the given definition and some of these fields may be essential for processing the MPU. In a case where the outer parts of the boundaries of MFUs in a payload of MPUs are cut off, if the packet is lost, an error may be propagated longer. In media data requiring a low delay, an MFU needs to be immediately transmitted at the time of its generation. In this case, to prevent the transmission delay, the transmission order may be determined to transmit header field information of the MPU after transmitting the generated PFU data.

Given the above characteristics, an MMT payload needs to be configured to maintain the compatibility between MPU packets and MFU packets.

<FIG> illustrates a flowchart for configuring and transmitting a multimedia data packet including an MMT payload in an MMT system according to an embodiment of the present disclosure. The operation shown in <FIG> is performed by the delivery function layer <NUM> among the layers in <FIG>.

Referring to <FIG>, in operation <NUM>, the delivery function layer <NUM> receives multimedia data provided from the encapsulation function layer <NUM>. The encapsulation function layer <NUM> may provide multimedia data necessary to configure a multimedia data packet, and information related to each multimedia data.

In operation <NUM>, the delivery function layer <NUM> configures payload header information related to an MMT payload of the multimedia data packet based on the header information of the multimedia data. As an example, the payload header information may provide a length of an MMT payload, a type of payload data, and information related to fragmentation and aggregation, and the payload header format may be defined by the system operator or the protocol standard.

In operation <NUM>, the delivery function layer <NUM> generates a payload of a multimedia data packet for delivery of multimedia data. In other words, the delivery function layer <NUM> configures an MMT payload by combining at least a part of the multimedia data received in operation <NUM>, with the payload header information configured in operation <NUM>.

In operation <NUM>, an MMT packet, which is a multimedia data packet including the MMT payload, may be may be transmitted to a counterpart entity (e.g., a receiver) through a given transport protocol.

A description will now be made of a detailed procedure for configuring a multimedia data packet by a delivery function layer in charge of transmission of multimedia data for MMT services, which is proposed in an embodiment of the present disclosure, and also made of a payload format. Specifically, a transport data structure that is based on the type of the data generated in the encapsulation function layer <NUM> may be provided. For configuration of a multimedia data packet, payload header information to be recorded in a header region and multimedia data to be recorded in a payload region may be generated according to a given format.

<FIG> illustrates an operation of generating a multimedia data packet according to an embodiment of the present disclosure.

Referring to <FIG>, an MPU file <NUM> means data that the encapsulation function layer desires to transmit, and an MPU/MFU building block <NUM> has a data structure for configuring transmission data. The MPU file <NUM> may include header information <NUM> such as a File Type (FTYP) box indicating a type of a file, an MMPU box indicating a structure of an MPU, an MOOV box indicating codec setting information, and Movie Fragment (MOOF) box; and an MDAT box <NUM> having Video Samples (VSs) <NUM> and MMT Hint Samples (HSs) <NUM>. The MMPU box indicates that a type of the data stored in the MDAT box is MPU, and the MOOV box contains information about a media frame stored in the MDAT box. The MPU/MFU building block <NUM> may include MPU metadata <NUM> such as FTYP, MMPU, MOOV and MOOF, and MFUs <NUM> having related HSs <NUM> and VSs <NUM>.

An MMT payload may be configured based on the MPU/MFU building block <NUM>, and carried or transmitted in an MMT packet.

A detailed procedure for configuring an MMT payload will be as follows.

A data configuration unit may analyze data <NUM> in the form of an MPU file, which is provided from a data generation unit and desired to be transmitted, to determine type information, configuration information of an MPU, setting information of a codec, starting point and size (or length) information of data, from the header information <NUM> of the MPU data <NUM>. The data configuration unit may analyze the structure of MFUs (corresponding to VSs in <FIG>) <NUM> and MFU HSs <NUM> constituting the MPU data <NUM>, to determine an MFU header containing the size, mutual importance and interconnection (length or the like) information of each MFU, and MFU data which is actual media data, and may also analyze the size of each MFU through the MFU HS <NUM> indicating the starting offset and length of each MFU.

Based on the analyzed MPU format, the MPU/MFU building block <NUM> for configuring a transport packet may be configured. The MFU HS <NUM> may be an MFU header of the MPU/MFU building block <NUM>.

The data configuration unit may configure each MFU as a transport unit based on the size of each MFU, which is analyzed from the MFU HS <NUM>. In this case, depending on the size of the MFU, one or multiple payload may be configured. The start, middle or end of each payload may be indicated by start_end_indicator which is optional bits. The MPU metadata <NUM> may also be treated as one configuration unit of the MPU/MFU building block <NUM>.

The data type of each payload element may be presented as follows.

Shown below is syntax of an MMT payload according to an embodiment of the present disclosure. The below MMT payload format may be used for UDP transport, exchange of point-to-point chunks, and transport of files. <IMG>
<IMG>.

As described above, an MMT payload may the MMT payload may include a type field indicating a data type of the payload, and asset_id indicating an identifier of an asset to which the payload (MPU or MFU) belongs. The asset_id may exist to multiplex an MMT bit stream. The asset_id may be omitted, if the MMT payload is delivered through the RTP and a single asset is used.

Depending on the type field, the following various cases of elements may be included in the MMT payload.

case <NUM> may be used if the MMT payload contains configuration data of the data to be transmitted. If asset_id=<NUM>, the MMT payload may include package configuration data. Otherwise, asset configuration data may be included.

An MFU_Partial() element corresponding to case <NUM> may be used when the MMT payload carries configuration data (header information) of the MFU data. An MFU_chunk() element corresponding to case <NUM> may be used when the MMT payload contains part of the MFU data together with configuration data (header information) of the MFU data, or carries all or part of the MFU data. An MFU_set() element corresponding to case <NUM> may be used when the MMT payload contains all (header and data) of the MFU data. An MPU_Partial() element corresponding to case <NUM> may be used when the MMT payload contains the MPT data. An MPU_Set() element corresponding to case <NUM> may be used when the MMT payload contains all (header and data) of the MPU data.

Additionally, the MMT payload may further include other units such as a file unit, signaling information of the control layer, and a protection message.

MFU_Partial() may be defined to be a set of all of headers that are added to a start of payload bytes and defined in the MFU, when the size of the MFU exceeds the size of the Maximum Transmission Unit (MTU).

MFU_chunk() may be defined to be a middle or a last part of a payload of the MFU for the headers transmitted in the MFU_Partial() packets.

MFU_set() may be defined to be a packet that includes a predetermined number of complete MFUs.

MPU_Partial() may be defined to be a set of all of headers that are added to the additional MFU_set element that is added to the optional MFU_Partial() element, and that are defined in the MPU. The presence/absence of the additional elements may be determined depending on the remaining bytes and the MFU length of the packet.

A description will now be made of examples of the MPU and MFU elements, together with the mapping between the MMT payload format and the MMT transport packet. A description of the parameter fields that are known in the art or that have nothing to do with embodiments of the present disclosure will be omitted herein, but it should be noted that such parameter fields do not limit the scope of the present disclosure.

The following represents an example of an MFU element.

In the above example, start_end_indicator may be optional bits indicating that the payload is a start, middle or end of an MFU. Additionally, start_end_indicator may indicate that the MFU includes Random Access Point (RAP) data, and may include optimal bits indicating a start and/or end position where the data is to be decoded. rap fragment indicator may be optional bits indicting an address of a start point of the data that can be directly decoded, if the payload contains a start point of the randomly accessible data.

scrambling_key_index may contain information about a series of values capable of decrypting encryption, if the payload is encrypted.

layer_dependency_info may be information indicating the dependency between picture layers. Additionally, it may further include layer identifiers (LayerIDs), a dependency count, signal rolling keys similar to a MPEG-<NUM> TS packet, and the like.

The following represents an example of an MPU element. <IMG>
<IMG>
<IMG>.

In the above example, start_endindicator may be optional bits indicating that the payload is a start, middle or end of an MPU. multiple mfu flag may be set to '<NUM>', if the MPU has multiple MFUs. no_mfu may be set to '<NUM>', if no MFU is included in the MPU data like in the case of file transfer.

timing flag may be used to correct the time stamp by indicating whether the current MFU belongs to the same access unit as the previous one, if the previous packet is lost. codec_update_flag may be set to '<NUM>', if the payload has codec configuration information, header first may be set to '<NUM>', if the payload starts at the header information of the media data. event list flag may be set, if the payload has an event list to be represented for the user. The event list may list events belonging to the content.

multiple_aus_flag may be set to '<NUM>', if the payload contains multiple frames. If a codec such as Advanced Audio Coding (AAC) is used, units for inband codec configuration may be included in the MPU element.

codec_update_flag may include codec_config_size indicating the size of codec configuration information and codec_config_data indicating the codec configuration information. In an alternative embodiment, if a codec is changed based on the codec configuration information or if the codec configuration information is modified, at the beginning or in the middle of the multimedia streaming service, the presence/absence of the information may be set through the codec_update_flag. downcounter may be used to indicate that a few more payloads containing the codec configuration information exist after the current payload, if the codec configuration information is divided into multiple payloads. As an example, if the codec configuration information is divided into multiple segments and downcounter is set to '<NUM>', a receiver may determine that payloads having downcounters of <NUM>, <NUM> and <NUM> are further later required to complete the codec configuration information. Upon receiving a payload of downcounter <NUM>, the receiver may determine that the transmission of all payloads related to the codec configuration information is completed.

If event list flag is set, an event list to be delivered through a transmission unit may exist, and the event list may include num_events indicating the number of events and at least one event data. In an alternative embodiment, if the event data that has already been transmitted is used, the event may be expressed, by including only the event identifier `event ID' instead of the event data. In another alternative embodiment, event data may be transmitted in advance using a predetermined transport means before transmission of the payload. Listed events may be executed after being synchronized with the display of the frame. In a possible embodiment, audio may be optionally changed through the event list during playback of video data.

If multiple mfu flag is set, num_MFUs indicating how many MFUs the payload has may exist in the MPU element. If multiple mfu flag is not set, num_MFUs is set to '<NUM>', and header_first and multiple_au_flag are set to '<NUM>'. In a possible embodiment, if a multimedia data unit desired to be transmitted has multiple frames, or multiple slices in a single frame, the information for setting the number of multiple entities may be set.

If timing flag is set, timing information for units in the MPU may be included in the MPU element.

Additionally, as many MFUs as num_MFUs may be listed in the MPU element. Since the order of MFUs may be different from the order of decoder inputs, decode_order_number indicating the order that the MFU is input to the decoder may be optionally included.

If multiple mfu flag is set to '<NUM>', mfu_size indicating the size of each MFU may be further included. In this specification, it is assumed that the MFUs that are input to the data configuration unit have the same size.

If headers_first is not set to '<NUM>', one mmt_mfu_element() may be included.

If headers_first is set to '<NUM>', as many mmt_mfu_element() as num_MFUs may be included in the MPU element.

The receiver configured in accordance with an embodiment of the present disclosure may include a reception unit configured to receive a multimedia data packet from a counterpart entity, and an interpretation unit configured to extract and interpret at least one MFU or MPU included in the multimedia data packet or at least a part thereof depending on the above-described format of the MMT payload.

The above-described format of the MMT payload may enable the efficient delivery and aggregation/fragmentation of MPUs and MFUs.

Claim 1:
An apparatus for transmitting media content in a multimedia system supporting a moving picture experts group, MPEG media transport, MMT, protocol, the apparatus comprising:
a processor configured to generate at a delivery function layer (<NUM>), one or more MMT packets based on a media processing unit, MPU, (<NUM>) fragmented into one or more media fragment units, MFUs, (<NUM>), each MMT packet including a MMT packet header and a MMT payload (<NUM>); and
a transmitter configured to transmit the one or more MMT packets,
wherein payload data included in the MMT payload comprises one of MPU metadata of the MPU (<NUM>) or media data element derived from the one or more MFUs (<NUM>),
wherein each of the one or more MMT packets comprises a data type field indicating that the respective payload data comprises the MPU metadata of the MPU (<NUM>) in case that the MMT payload comprises the MPU metadata or indicating that the respective payload data comprises the media data element derived from the one or more MFUs (<NUM>) in case that the MMT payload comprises the media data element,
wherein the MPU metadata of the MPU (<NUM>) comprises file type information of a file to be transmitted,
wherein the data type field is set to a first value indicating that the respective payload data includes the MPU metadata of the MPU if the respective payload data includes the MPU metadata of the MPU, and
wherein the data type field is set to a second value indicating that the respective payload data includes the media data element derived from the one or more MFUs if the payload data includes the media data element derived from the one or more MFUs.