Patent Description:
A new generation video compression standard H. <NUM>/HEVC (High Efficiency Video Coding) jointly developed by ITU-T/VCEG (Video Coding Experts Group) and ISO/IEC MPEG continues the style of last generation video standard H. <NUM>/AVC and is compatible with scalable video coding (Scalable Video Coding). But different from H. <NUM>/AVC, H. <NUM>/HEVC puts forward a request for multiple coding standards jointed scalable coding (Different Coding Standards with Scalability). The currently released H. <NUM>/HEVC standard (Version <NUM>-<NUM>) is known as the "H. <NUM>/HEVC Version <NUM>" standard. In Version <NUM>, information used for indicating a base layer (Base Layer) is provided for hybrid coding (Hybrid Scalability), while how to put a picture of the base layer (Base Layer) into a reference frame management of an enhancement layer (Enhancement Layer) is implemented by using an external means (External Means), and the external means is not explained in Version <NUM>.

It is expected that devices using H. <NUM>/AVC will be widely used in next few years, and H. <NUM>/AVC will be the main video coding protocol for digital videos throughout the world. Therefore, developing a new generation scalable video coding standard compatible with H. <NUM>/AVC will be very useful in a standard migration process. The multiple coding standards jointed scalable coding refers to the combination of two bitstreams of different video standards according to a certain rule, and is also known as multi-protocol hybrid coding (Hybrid Coding). For example, in spatial scalability (Spatial Scalability), the base layer (Base Layer) contains a bitstream in which Constrained Baseline Profile, Main Profile and High Profile of H. <NUM>/AVC standard are compatible, and the enhancement layer (Enhancement Layer) includes a basic bitstream of HEVC. Pictures in the two layers may have the same size or different sizes, i.e., low resolution is adopted in H. <NUM>/AVC, and high resolution is adopted in H. <NUM>/HEVC. The base layer of the spatial scalability coding may also use a bitstream of MPEG-<NUM>, which is combined with a bitstream of the enhancement layer of H. <NUM>/HEVC to form the hybrid coding. For another example, in a view scalability (View Scalability), the base layer is a view using an H. <NUM>/AVC coding manner, and the enhancement layer is another view using a H. <NUM>/HEVC format and taking the base layer as a reference. The base layer of multiple coding standards jointed hybrid coding is not limited to H. <NUM>/AVC or MPEG-<NUM>.

A scalable bitstream composed of the bitstreams with the same protocol may be synchronized in various manners. For example, a video picture obtained at the same time in H. <NUM>/AVC and its corresponding coded bits form an access unit (Access Unit, AU). Therefore, the pictures in different layers may be synchronized through an AU scope. <NUM>/HEVC, syntactic information such as POC (Picture Order Count) in a video bitstream may be used. For bitstreams of two layers or multiple layers of different video coding protocols, codecs thereof are respectively independent codecs, and syntactic information contained in different video bitstreams is slightly different. Therefore, to pack the bitstreams in what form or to provide what syntactic information to ensure synchronization is a problem that needs to be solved for hybrid coding (Hybrid Coding).

To this issue, one of the existing methods is to pack the bitstreams of H. <NUM>/AVC and H. <NUM>/HEVC as a whole and transmit the bitstreams by an internal means. In other words, the existing form of the bitstream of H. <NUM>/AVC may be indicated by adding some syntactic elements in H. <NUM>/HEVC. There are many related proposals in JCT-VC standard conference. For example, JCTVC-N0050 proposes a solution regarding hybrid bitstreams of AVC-HEVC and adds a new NAL header of H. <NUM>/HEVC (nal_unit_type = ENC_NUT) to a H. <NUM>/AVC NAL unit, which is packed as a HEVC NAL unit; avc_base_layer_flag is added in a VPS extension to indicate whether avc_base_profile_level_idc exists, while the avc_base_profile_level_idc indicates the capability set (profile) of H. The disadvantage of this means is that a standard H. <NUM>/AVC decoder cannot decode such a bitstream, and it needs to unpack the header information of the NAL unit corresponding to H. <NUM>/HEVC and then it can decode correctly.

JCTVC-O0190 proposes another different packing strategy with respect to JCTVC-N0050. It also proposes to pack the bitstreams of H. <NUM>/AVC and H. <NUM>/HEVC as a whole, but the H. <NUM>/AVC NAL unit of the base layer is unchanged while the H. <NUM>/HEVC NAL unit of the enhancement layer is added with one prefix byte (0x16) for the H. <NUM>/AVC decoder to identify that this portion of bitstreams cannot be decoded and discard the bitstreams, and the internal means is still used in the bitstream portion thereof. Meanwhile, it is suggested to add a variable HevcPrefixBytePresent, which indicates to the decoder that this prefix byte needs to be parsed. This means needs to add information that can identify different bitstreams in the NAL header or in front of the NAL header, while the decoder has to be capable of identifying the bitstreams belonging to it and the bitstreams not belonging to it. Therefore, a corresponding processing needs to be added in either the H. <NUM>/AVC decoder or the H. <NUM>/HEVC decoder if it wants to support the function of hybrid coding (Hybrid Coding). This function cannot be achieved without changing or slightly changing the decoder in the existing device.

It is apparent that the existing means transmits the bitstreams of different video standards on the basis of the internal means, and its alignment or synchronization processing needs to modify the existing coding and decoding devices, which has the effects of complicated implementation process, poor universality and high implementation cost.

A US patent application, whose publication number is <CIT>), discloses techniques for encoding and decoding layered video where the non-temporal enhancement layers and their respective temporal enhancement layers, comply with a scalable video coding standard or technology, and the base layer and its respective temporal enhancement layers does not comply with the same scalable video coding standard or technology. A Video Parameter Set that comprises information about the relationship of layers includes a syntax element indicative of the derivation mechanism for a temporal layer associated with a NAL unit coded in a first coding technology, for example HEVC. For one value of the syntax element, the derivation mechanism is to set the temporal layer of the base layer NAL unit to the value coded in the header of the encapsulating NAL unit, which can be an HEVC NAL unit. For another value, the derivation mechanism is to imply the value of temporal base layer for the first NAL unit.

In document of<NPL>;<NPL>, proposed is a design for signalling and deriving picture order count (POC) in SHEVC and MV-HEVC for POC alignment. In enhancement layer (EL) slice headers, when nuh_layer_id is greater than <NUM> and POC alignment flag is set to <NUM>, the value of the most significant bit (MSB) for POC calculation is explicitly signalled, and the value of least significant bit (LSB) is conveyed by slice_pic_order_cnt_lsb. Therefore, the proposed design only introduces additional bits to slice headers of enhancement layer (EL) pictures without any changes to base layer, and makes the POC values for both BL and EL pictures unique and static. The encoder sets the MSB and LSB values in EL slice headers so that the decoded POC value of the EL picture is equal to the POC value of the existing or hypothetically existing base layer (BL) picture in the same access unit (AU), which also facilitates the POC alignment for hybrid scalability cases. Furthermore, as full POC is signalled, this design can also be applied to the pictures, which do not need POC alignment, to improve error resilience performance for cases of possible picture loss.

A US patent application, whose publication number is <CIT>), discloses a device for coding video data. The device includes a video coder configured to code data representative of whether a tile of an enhancement layer picture can be predicted using inter-layer prediction, and predict data of the tile using inter-layer prediction only when the data indicates that the tile can be predicted using inter-layer prediction.

In order to reduce changes to the existing coding and decoding devices as much as possible, the transmitting end, when determining that the bitstreams need to be aligned, sets the alignment auxiliary information (the alignment auxiliary information is used with regard to the transmitting end, and is used for synchronization with regard to the receiving end, and thus can also be called synchronization auxiliary information for the receiving end) into various bitstreams of different standards, and the receiving end, after acquiring the bitstreams, may complete synchronization processing according to the alignment auxiliary information in various bitstreams. The present disclosure directly sets the alignment auxiliary information in various bitstreams, which is easy to achieve, does not need additional modifications to a codec basically, and has a good universality and a low implementation cost. The present disclosure will be further described hereinafter in detail with reference to the drawings and specific embodiments.

In this embodiment, a transmitting end, when determining that current multiple bitstreams need to be aligned, sets alignment auxiliary information in the multiple bitstreams respectively, and then stores the multiple bitstreams, or transmits the multiple bitstreams to a receiving terminal. The specific process is as shown in <FIG>, including the following steps.

In step <NUM>, a transmitting terminal determines whether current multiple bitstreams need to be aligned; if yes, the process goes to step <NUM>; otherwise, the process goes to step <NUM>.

In step <NUM>, the transmitting terminal sets alignment auxiliary information in the multiple bitstreams.

In step <NUM>, the transmitting terminal stores the multiple bitstreams, or transmits the multiple bitstreams to the receiving terminal. Here, the storage may means that the multiple bitstreams are stored locally or stored on a corresponding server, or stored locally and on the server at the same time, and the bitstreams may be stored in a form of files.

The above-mentioned step <NUM> is conducted when the transmitting terminal needs to transmit the current multiple bitstreams. To be specific, the transmitting terminal may judge whether it is necessary to conduct an alignment processing on the current multiple bitstreams according to an application demand. The application demand may be determined according to a specific demand of a user, or a specific application environment, or other factors, for example, whether a coder uses an AVC picture of a base layer as a reference for an HEVC picture of an enhancement layer, or whether the AVC picture and the HEVC picture need to be outputted synchronously.

It is to be understood that the bitstream in this embodiment may either be a video bitstream, or other bitstreams in the system layer; and the embodiment does not make any restrictions on the specific bitstream types. At the same time, it should be understood that the multiple bitstreams in this embodiment may be multiple bitstreams of the same standard, and may also be multiple bitstreams of different standards, which depends on specific application scenarios. The alignment or synchronization operation of the multiple bitstreams with the above conditions may be achieved by the solution shown in <FIG>. In the following, the embodiment is exemplarily illustrated by taking the video bitstream as an example.

In the above-mentioned step <NUM>, when the transmitting terminal sets the alignment auxiliary information in the multiple bitstreams, it is necessary to determine a specific location in each video bitstream where the alignment auxiliary information is needed to be set. In this embodiment, the transmitting terminal may set the alignment auxiliary information in at least one of a video layer, a system layer and an application layer of the multiple bitstreams. When the transmitting terminal sets the alignment auxiliary information in various bitstreams, the existing information in the bitstreams may be directly used as the alignment auxiliary information, or new content may be added in the bitstreams and used as the alignment auxiliary information.

To be specific, when the transmitting terminal sets the alignment auxiliary information in the video layer of the multiple bitstreams, the method includes the step of setting the alignment auxiliary information in at least one of the following fields of the video layer:.

For the contents in the above fields which are not necessary for the codec, the coder may discard them according to standard rules.

In this embodiment, the transmitting terminal may set the alignment auxiliary information in any one of the above fields of the various video bitstreams, or may set the alignment auxiliary information in multiple fields of the above fields of the various video bitstreams at the same time. For one video bitstream, the alignment auxiliary information may be theoretically set in any field of any layer mentioned above. However, in order to further improve the reliability and avoid missing or lack of the alignment auxiliary information caused by factors like transmission errors, the alignment auxiliary information may also be set in multiple fields at the same time. For example, the alignment auxiliary information may be set in at least one field of at least two layers of the video layer, the system layer, and the application layer of one bitstream.

Moreover, the positions at which the transmitting terminal sets the alignment auxiliary information in various video bitstreams may be the same or different; for example, when the multiple bitstreams are two video bitstreams of different standards, the alignment auxiliary information may be set at the same position for the two video bitstreams, for example, the alignment auxiliary information may be set in the field corresponding to enhancement information of the video layer for both of the two video bitstreams; the alignment auxiliary information may also be set at different positions; for example, the alignment auxiliary information of one bitstream is set in the field corresponding to enhancement information of the video layer, and the alignment auxiliary information of the other bitstream is set in the field corresponding to video shared information of the video layer; for another example, the alignment auxiliary information of one bitstream is set in the field of the system layer for describing a video media attribute, and the alignment auxiliary information of the other bitstream is set in the field of the application layer for describing a video media attribute.

In this embodiment, contents of the alignment auxiliary information set by the transmitting terminal in the multiple bitstreams may be the same or different, as long as the contents can characterize that an alignment operation needs to be performed and the contents can be correctly identified.

In this embodiment, numbers of the alignment auxiliary information set by the transmitting terminal in the multiple bitstreams may be the same or different, for example, when the multiple bitstreams are two video bitstreams of different standards, one alignment auxiliary information may be set in one of the two video bitstreams, and multiple alignment auxiliary information may be set in the other bitstream. The contents of the multiple alignment auxiliary information set in the same layer for one bitstream are the same.

In this embodiment, the alignment auxiliary information set by the transmitting terminal at least includes an alignment scalar content, or further includes alignment instruction information when including an alignment scalar content. The alignment scalar content in this embodiment includes at least one of picture absolute time information (i.e., a system time when performing the hybrid coding), picture relative time information, picture decoding sequence information, picture playing sequence information, a picture separator and a picture specific delimiter. As mentioned above, when the alignment scalar content is the picture decoding sequence information, and when picture decoding sequence information exists in the bitstream itself, then the original picture decoding order information in the bitstream may be directly used as the alignment auxiliary information for the bitstream; and for other bitstreams, the picture decoding sequence information may be added as the alignment auxiliary information.

When the alignment auxiliary information includes the alignment instruction information, the alignment instruction information may specifically be alignment flag information and/or type information. An exemplary explanation is given hereinafter with a specific example.

The alignment flag information and the type information of the alignment instruction information used in the following examples with respect to the alignment or synchronization operation adopt structures as shown in Table <NUM> and/or Table <NUM> respectively. The alignment scalar content adopts a structure as shown in Table <NUM>. The bitstream as shown in Table <NUM> includes flag information that identifies whether the alignment or synchronization operation is adopted in the entire or a part of the bitstream. The bitstream as shown in Table <NUM> includes the type of the flag information that identifies the alignment or synchronization operation of the entire or a part of the bitstream. The bitstream as shown in Table <NUM> includes content information needed by the alignment or synchronization.

The semantic of the field in Table <NUM> is as follows: a value of pic_alignment_flag equal to <NUM> indicates that video pictures of different coding formats in the bitstream need an alignment or synchronization operation. The value of pic_alignment_flag equal to <NUM> represents that the video pictures of different coding formats in the bitstream do not need an alignment or synchronization operation. For those skilled in the art, other manners may also be adopted for the value of the pic_alignment_flag and the meaning represented thereof, which are not limited in the embodiment. Coding and decoding methods corresponding to different coding protocols are used in the pic_alignment_flag.

Optionally, if a bit field corresponding to the pic_alignment_flag does not exist in the bitstream, then the value of the corresponding variable pic_alignment_flag is <NUM> by default.

The semantic of the field in Table <NUM> (corresponding control operation) is as follows: pic_alignment_type () represents to acquire type information. The type here refers to the field for storing the aligned or synchronized content information to be further extracted. For those skilled in the art, other manners may also be adopted for inner forms of the pic_alignment_type() and the meaning represented thereof, which are not limited in the embodiment.

Optionally, if the pic_alignment_type() does not exist in the bitstream, then the corresponding type information thereof does not exist.

The semantic of the field in Table <NUM> (corresponding control operation) is as follows: alignment_code() includes data information needed for alignment or synchronization. The field in Table <NUM> may be used in combination with the fields in Table <NUM> and/or Table <NUM>. One or more of the following information may be used for the alignment_code() in Table <NUM>, including, but not limited to the following information: picture absolute time information, picture relative time information, picture decoding sequence information, picture playing sequence information, a picture separator, a picture special delimiter and so on.

The syntactic elements in Table <NUM>, Table <NUM>, and Table <NUM> may be set at a position where at least one field of the above (<NUM>) to (<NUM>) is located or a plurality of fields shown in the above (<NUM>) to (<NUM>) are mixed; and the bit fields corresponding to the syntactic elements in Table <NUM>, Table <NUM> and Table <NUM> are used in the bitstream corresponding to the data structure.

In the above-mentioned step <NUM>, when transmitting the multiple bitstreams to the receiving terminal, the transmitting terminal can flexibly select to respectively and independently transmit the multiple bitstreams to the receiving terminal according to the specific application scenarios, or, select to pack at least two of the multiple bitstreams into one bitstream, and then transmit the packed bitstream to the receiving terminal. For instance, when there are three bitstreams (respectively A, B, and C), the three bitstreams may be packed into one bitstream and then transmitted; it is also possible to pack two of the three bitstreams (such as A and B) into one bitstream, and then the one bitstream obtained after packing and the bitstream C are transmitted respectively and independently; and a similar method is adopted for processing the situations of more than three bitstreams. The multiple bitstreams may also be split and recombined into multiple bitstreams, and then transmitted to the receiving terminal.

In the above-mentioned step <NUM>, when the transmitting terminal stores the multiple bitstreams, no matter the transmitting terminal stores the multiple bitstreams locally or on a server, the multiple bitstreams may be written into one file for storing, or the multiple bitstreams are respectively written into independent files for storing, or the multiple bitstreams are split and recombined, and then written into different files for storing.

Alternatively, the multiple bitstreams in this embodiment are specifically set to be two video bitstreams of different video standards, and one video bitstream (for example, an enhancement layer) is a video bitstream of H. <NUM>/HEVC and the other video bitstream (for example, a base layer) is a non video bitstream of H. <NUM>/HEVC, for instance, a video bitstream of H. <NUM>/AVC, a video bitstream of MPGE-<NUM>, a video bitstream of MPGE-<NUM>, a video bitstream of VC-<NUM>, a video bitstream of VP8, a video bitstream of AVS, etc. The multiple bitstreams in this embodiment may also be specifically configured as two video bitstream of the same video standard, and one video bitstream (for example, the base layer) is the video bitstream of H. <NUM>/HEVC, and the other video bitstream (for example, the enhancement layer) is also the video bitstream of H. <NUM>/HEVC.

In this embodiment, the operation of a receiving terminal for bitstream synchronization is as shown in <FIG>, including the following steps.

In step <NUM>, the receiving terminal acquires multiple bitstreams; the acquiring manner is receiving the multiple bitstreams from a transmitting terminal or a server, or directly reading the multiple bitstreams from a corresponding storage medium. It is to be understood that, in this step, when the transmitting terminal respectively and independently transmits the multiple bitstreams to the receiving terminal, the receiving terminal respectively and independently receives various bitstreams; and when the transmitting terminal packs the multiple bitstreams into one bitstream and transmits the packed bitstream to the receiving terminal, the receiving terminal receives the one packed bitstream.

In step <NUM>, when the multiple bitstreams carry alignment auxiliary information set by a transmitting terminal, the receiving terminal synchronizes the multiple bitstreams according to the alignment auxiliary information.

It is to be understood that the bitstreams in the embodiment may either be video bitstreams, or other bitstreams in the system layer; the embodiment does not make any restrictions on the specific bitstream types; at the same time, it should be understood that the multiple bitstreams in the embodiment may be multiple bitstreams of the same standard, and may also be multiple bitstreams of different standards, which depends on the specific application scenarios. In the following, the embodiment is exemplarily illustrated by taking the video bitstreams as an example. In this embodiment, the receiving terminal, after receiving the bitstreams, searches the alignment auxiliary information in at least one of a video layer, a system layer and an application layer of the multiple bitstreams. For example, when there is an agreement between the transmitting and receiving terminals about which layer or several layers that the alignment auxiliary information needs to be added in when alignment or synchronization is needed, it only needs to search the alignment auxiliary information in this layer or these layers. Otherwise, the alignment auxiliary information needs to be searched in the application layer, the system layer and the video layer of the bitstreams in sequence.

In this embodiment, when inconsistent alignment auxiliary information exists in multiple layers of one bitstream, the alignment auxiliary information of one layer is selected from the alignment auxiliary information according to a preset rule for synchronization processing. The preset rule of this embodiment includes, but is not limited to:.

It should be understood that in addition to the two preset rules of the above-mentioned examples in the embodiment, the user may further customize other rules according to the application scenarios or other factors, as long as the effective alignment auxiliary information can be selected to complete the alignment or synchronization operation.

Moreover, in this embodiment, when completed alignment auxiliary information exists in a layer of one bitstream with a relatively lower channel transmission reliability, while alignment auxiliary information existing in the layer with a relatively higher channel transmission reliability incomplete or missing, the alignment auxiliary information of the layer with the relatively lower channel transmission reliability is selected for synchronization processing; for example, when the alignment auxiliary information of the application layer is incomplete or missing, but the alignment auxiliary information of the video layer is completed, although the channel transmission reliability of the application layer is higher than the channel transmission reliability of the video layer, the alignment auxiliary information of the video layer is selected.

Alternatively, when completed alignment auxiliary information exists in other layer of one bitstream excluding the video layer, while the alignment auxiliary information lacks in the video layer, the alignment auxiliary information of the other layer excluding the video layer is selected for synchronization processing. For example, when the alignment auxiliary information of the video layer is incomplete or missing, while the alignment auxiliary information of the application layer is completed, then the alignment auxiliary information of the application layer is selected.

In this embodiment, the alignment auxiliary information set by the transmitting terminal at least includes an alignment scalar content, or when the alignment auxiliary information includes an alignment scalar content, it further includes alignment instruction information. The alignment scalar content in the embodiment includes at least one of picture absolute time information (i.e., a system time when hybrid coding is performed), picture relative time information, picture decoding sequence information, picture playing sequence information, a picture delimiter, and a picture specific delimiter. When the alignment auxiliary information includes the alignment instruction information, the alignment instruction information may specifically be alignment flag information and/or type information. The receiving terminal then performs synchronization processing on the multiple bitstreams according to the alignment scalar content.

Alternatively, the multiple bitstreams in the embodiment are specifically set to be two video bitstreams of different video standards, and one video bitstream is a video bitstream of H. <NUM>/HEVC and the other video bitstream is a non video bitstream of H. <NUM>/HEVC, for instance, a video bitstream of H. <NUM>/AVC, a video bitstream of MPGE-<NUM>, a video bitstream of MPGE-<NUM>, a video bitstream of VC-<NUM>, a video bitstream of VP8, a video bitstream of AVS, etc. The multiple bitstreams in this embodiment may also be specifically configured as two video bitstream of the same video standard, and one video bitstream (for example, the base layer) is the video bitstream of H. <NUM>/HEVC, and the other video bitstream (for example, the enhancement layer) is also the video bitstream of H. <NUM>/HEVC.

As shown in <FIG>, the present embodiment provides a communication system, which includes a transmitting terminal <NUM> and a receiving terminal <NUM>, wherein structures of the transmitting terminal <NUM> and the receiving terminal <NUM> are as shown in <FIG> respectively.

As shown in <FIG>, the transmitting terminal <NUM>, i.e., a source device, includes a processing module <NUM> and a transmitting module <NUM>.

The processing module <NUM> is configured to, when current multiple bitstreams need to be aligned, set alignment auxiliary information in the multiple bitstreams; when setting the alignment auxiliary information in various bitstreams, existing information in the bitstreams may be directly used as the alignment auxiliary information, or new contents may be added in the bitstreams and used as the alignment auxiliary information.

The transmitting module <NUM> is configured to store the multiple bitstreams, or transmit the multiple bitstreams to a receiving terminal. When storing, the multiple bitstreams may either be stored locally in the transmitting terminal, or stored on a server. Moreover, the multiple bitstreams may be written into one file for storage, or the multiple bitstreams may be respectively written into independent files for storage, or the multiple bitstreams may be split and recombined, and then written into different files for storage. When transmitting, the multiple bitstreams may be flexibly, respectively and independently sent to the receiving terminal according to the specific application scenarios; or the multiple bitstreams may be packed into one bitstream and then transmitted to the receiving terminal; or, the multiple bitstreams may be split and recombined into multiple bitstreams, and then transmitted to the receiving terminal.

The processing module <NUM> specifically includes a determining submodule <NUM> and a writing submodule <NUM>; the determining submodule <NUM> is configured to determine whether the current multiple bitstreams need to be aligned according to an application demand. The application demand may be determined according to the specific demand of a user or specific application environment or other factors, for example, whether a coder uses an AVC picture of a base layer as a reference for an HEVC picture of an enhancement layer, or whether the AVC picture and the HEVC picture need to be outputted synchronously. The writing submodule <NUM> is configured to write the alignment auxiliary information into a corresponding position of the bitstream.

It is to be understood that the bitstreams in the embodiment may either be video bitstreams, or other bitstreams in the system layer; the embodiment does not make any restrictions on the specific bitstream types; at the same time, it should be understood that the multiple bitstreams in the embodiment may be multiple bitstreams of the same standard, and may also be multiple bitstreams of different standards, which depends on the specific application scenarios. In the following, the embodiment is exemplarily illustrated by taking the video bitstreams as an example. The writing submodule <NUM> is configured to set the alignment auxiliary information in at least one of a video layer, a system layer and an application layer of the multiple bitstreams respectively. To be specific, when the transmitting terminal sets the alignment auxiliary information in the video layer of the multiple bitstreams, the method includes the step of setting the alignment auxiliary information in at least one of the following fields of the video layer:.

The writing submodule <NUM> may set the alignment auxiliary information in any one of the above fields of various video bitstreams, or may set the alignment auxiliary information in multiple fields of the above fields of various video bitstreams at the same time. For one video bitstream, the alignment auxiliary information may be theoretically set in any field of any layer mentioned above. However, in order to further improve the reliability and avoid missing or lacking of the alignment auxiliary information caused by factors like transmission errors, the alignment auxiliary information may also be set in multiple fields at the same time. For example, the alignment auxiliary information may be set in at least one field of at least two of the video layer, the system layer, and the application layer of one bitstream.

Moreover, the positions at which the writing submodule <NUM> in the embodiment sets the alignment auxiliary information in various video bitstreams may be the same or different; for example, when the multiple bitstreams are two video bitstreams of different standards, the alignment auxiliary information may be set at the same position for the two video bitstream, for example, the alignment auxiliary information may be set in the field corresponding to enhancement information of the video layer for both of the two video bitstreams; the alignment auxiliary information may also be set in different positions; for example, the alignment auxiliary information of one bitstream is set in the field corresponding to enhancement information of the video layer, and the alignment auxiliary information of the other bitstream is set in the field corresponding to video shared information of the video layer; for another example, the alignment auxiliary information of one bitstream is set in the field of the system layer for describing the video media attribute, and the alignment auxiliary information of the other bitstream is set in the field of the application layer for describing the video media attribute.

In this embodiment, the contents of the alignment auxiliary information set by the writing submodule <NUM> in the multiple bitstreams may be the same or different, as long as the contents can characterize that an alignment operation needs to be performed and the contents can be correctly identified.

In this embodiment, the numbers of the alignment auxiliary information set by the writing submodule <NUM> in the multiple bitstreams may be the same or different, for example, when the multiple bitstreams are two video bitstreams of different standards, one alignment auxiliary information may be set in one of the two video bitstreams, and multiple alignment auxiliary information may be set in the other bitstream. The contents of the multiple alignment auxiliary information set in the same layer for one bitstream are the same.

In this embodiment, the alignment auxiliary information set by the writing submodule <NUM> at least includes an alignment scalar content, or further includes alignment instruction information except including the alignment scalar content. The alignment scalar content in the embodiment includes at least one of picture absolute time information (i.e., a system time when hybrid coding is performed), picture relative time information, picture decoding sequence information, picture playing sequence information, a picture delimiter, and a picture specific delimiter. When the alignment auxiliary information includes the alignment instruction information, the alignment instruction information may specifically be alignment flag information and/or type information.

As shown in <FIG>, the receiving terminal <NUM> includes an acquisition module <NUM> and an execution module <NUM>.

The acquisition module <NUM> is configured to acquire multiple bitstreams; and the acquiring manner is that the multiple bitstreams may be received from a transmitting terminal or a server, or directly read from a corresponding storage medium.

The execution module <NUM> is configured to, when the multiple bitstreams carry alignment auxiliary information set by a transmitting terminal, synchronize the multiple bitstreams according to the alignment auxiliary information.

The execution module <NUM> includes a searching submodule <NUM> configured to search the alignment auxiliary information from at least one of an application layer, a system layer and a video layer of the multiple bitstreams. When the searching submodule <NUM> finds that inconsistent alignment auxiliary information exists in multiple layers of one bitstream, the alignment auxiliary information of one layer is selected from the alignment auxiliary information according to a preset rule for synchronization processing. The preset rule of the embodiment includes, but is not limited to:.

It should be understood that the user may further customize other rules according to the application scenarios or other factors excluding the two preset rules of the above-mentioned examples in the embodiment, as long as the effective alignment auxiliary information can be selected to complete the alignment or synchronization operation.

Moreover, in the embodiment, when the searching submodule <NUM> finds that completed alignment auxiliary information exists in a layer of one bitstream with relatively lower channel transmission reliability, while the layer with high channel transmission reliability lacks the alignment auxiliary information, the alignment auxiliary information of the layer with relatively lower channel transmission reliability is selected for synchronization processing; for example, when the alignment auxiliary information of the application layer is incomplete or missing, but the alignment auxiliary information of the video layer is completed, although the channel transmission reliability of the application layer is higher than the channel transmission reliability of the video layer, the alignment auxiliary information of the video layer is selected.

Or, when completed alignment auxiliary information exists in other layer of one bitstream excluding the video layer, while the alignment auxiliary information in the video layer is incomplete or missing, the alignment auxiliary information of the other layer excluding the video layer is selected for synchronization processing. For example, when the alignment auxiliary information of the video layer is incomplete or missing, while the alignment auxiliary information of the application layer is completed, then the alignment auxiliary information of the application layer is selected.

The execution module <NUM> further includes a synchronization submodule <NUM> configured to conduct a synchronization processing on the multiple bitstreams according to the alignment scalar content.

It should be understood that the functions of each module and sub-module in the embodiment may be implemented by using dedicated hardware, software, or a combination of hardware and software. The hardware or dedicated hardware may include an application specific integrated circuit (ASIC), various other circuits, various processors, and the like. When being implemented by a processor, the functions may be provided by a single dedicated processor, a single shared processor, or multiple independent processors (some of which may be shared). In addition, the processor should be exclusively interpreted as the hardware capable of executing software; and it may implicitly include, but is not limited to, digital signal processor (DSP) hardware, read-only memory (ROM) for storing software, random access memory (RAM), and non-volatile storage device.

The terminal in the embodiment may be a related bitstream receiving and playing device in a video communication application, such as a mobile phone, a computer, a server, a set-top box, a portable mobile terminal, a digital camera, a television broadcast system device, or the like. The device may be located in a host device, and the processing device may process at least one of the following multilayer video signals: scalable videos, multi-view videos, multi-view depths, multi-view videos + multi-view depths. A stereoscopic video is a special form of the multi-view videos whose view number is equal to <NUM>.

In order to better understand the present disclosure, several specific application scenarios are taken as examples to further illustrate the present disclosure.

In this example, a load type (Payload Type) of an SEI is pic_alignment_type, and the control range thereof is a picture using the SEI. When the load type is an SEI unit of time information, the time information is further given by using the bit field in Table <NUM>.

In this example, transmission of video bitstreams with different coding formats mainly includes the following steps.

In step <NUM>, a source device (i.e., a transmitting terminal) determines whether it is necessary to conduct an alignment operation on the videos with different coding formats according to an application demand (for example, whether a coder uses an AVC picture of a base layer as a reference for an HEVC picture of an enhancement layer, or whether the AVC picture and the HEVC picture need to be outputted synchronously).

If it is needed to use the alignment information, an HEVC coder in the source device codes one time_code type SEI unit (Time code SEI message), i.e., the payloadType of the SEI is equal to <NUM>. An AVC coder in the source device codes one pic_timing type SEI unit (Picture timing SEI message), i.e., the payloadType of the SEI is equal to <NUM>.

If the alignment information is not needed to be used, then the SEI unit is not set. In step <NUM>, the source device uses a coding method corresponding to u(n) to write values of the payloadType in different coding protocols into a SEI packet header of an AVC video bitstream and an SEI packet header of an HEVC video bitstream respectively.

In step <NUM>, the source device acquires a system time (i.e., picture absolute time information) when the current coder performs a hybrid coding, and writes the time information into a payload field of the SEI of the AVC video bitstream and a payload field of the SEI of the HEVC video bitstream respectively as alignment information. In step <NUM>, a host device (i.e., a receiving terminal), after receiving the bitstreams, searches SEI data in the AVC video bitstream and the HEVC video bitstream respectively.

In step <NUM>, the host device uses a decoding method corresponding to u(n) to acquire the payloadType values from the SEI packet header of the AVC video bitstream and the SEI packet header of the HEVC video bitstream respectively, and determines whether there is the time information (the payloadType is equal to <NUM> for the pic_timing type in AVC, and the payloadType is equal to <NUM> for the time_code type in HEVC).

In step <NUM>, if there is the time information, the host device obtains the fields shown in the corresponding Table <NUM> according to the payloadType, and uses a corresponding decoding method to acquire the time information of the AVC picture and the HEVC picture from the bitstreams respectively.

In step <NUM>, the host device determines whether the pictures in different coding formats need to be synchronized according to the acquired time information.

In step <NUM>, if the synchronization is needed, i.e., the time information acquired by AVC and HEVC is consistent, the host device puts reconstructed picture decoded by the current AVC decoder into a reference frame management list of the HEVC for being used by the HEVC decoder. For a multi-view application, the host device uses this information to conduct synchronous output and display operations on pictures of different viewpoints.

In this example, vui_timing_info_present_flag in the load type of the VUI is pic_alignment_flag, and the control range thereof is a picture using the VUI. When the vui_timing_info_present_flag is <NUM>, the bit field of Table <NUM> is used to further give the time information.

In step <NUM>, a source device determines whether it is necessary to conduct an alignment operation on videos with different coding formats according to an application demand (for example, whether a coder uses an AVC picture of the base layer as a reference for the HEVC picture of the enhancement layer, or whether the AVC picture and the HEVC picture need to be outputted synchronously).

If it is needed to use alignment information, the HEVC coder in the source device codes one VUI unit, wherein the vui_timing_info_present_flag in the VUI is equal to <NUM>. The AVC coder in the source device codes one VUI unit, wherein the vui_timing_info_present_flag in the VUI is equal to <NUM>.

If it is not needed to use the alignment information, then the
vui_timing_info_present_flag is equal to <NUM>.

In step <NUM>, the source device uses a coding method corresponding to u(<NUM>) to write the value of the vui_timing_info_present_flag into the VUI of an AVC video bitstream and the VUI of a HEVC video bitstream respectively.

In step <NUM>, the source device acquires the system time when the current coder performs hybrid coding, and writes the time information into a time information field of the VUI of the AVC video bitstream and a time information field of the VUI of the HEVC video bitstream respectively as the alignment information.

In step <NUM>, a host device, after receiving the bitstreams, searches VUI data in the AVC video bitstream and HEVC video bitstream respectively.

In step <NUM>, the host device uses a decoding method corresponding to u(<NUM>) to acquire values of the vui_timing_info_present_flag from the VUI of the AVC video bitstream and the VUI of the HEVC video bitstream respectively, and determines whether there is time information (whether the vui_timing_info_present_flag is equal to <NUM>).

In step <NUM>, if there is time information (the vui_timing_info_present_flag is equal to <NUM>), the host device uses a corresponding decoding method to acquire the time information of the AVC picture and the HEVC picture from the bitstreams respectively according to the fields shown in the corresponding Table <NUM>.

In step <NUM>, if synchronization is needed, i.e., the time information acquired by AVC and HEVC is consistent, the host device puts the reconstructed picture decoded by the current AVC decoder into a reference frame management list of the HEVC for the HEVC decoder to use. For a multiple-viewpoint application, the host device uses this information to conduct synchronous output and display operations on pictures of different viewpoint.

In this example, extension_flag in the field corresponding to video shared information (i.e., a field corresponding to a parameter set) is namely pic_alignment_flag, and the control range thereof is a picture using the parameter set. When the extension _flag corresponding to the parameter set is <NUM>, extension_data is namely alignment code, and the time information is further given by using the bit field in Table <NUM>. The operation methods of the source device and the host device on the pic_alignment_flag and alignment code are similar to that in the second example.

Different from the second example, in the present example, a user may define the time information independently, and the rules may be unified by the source device and the host device through negotiation.

The positions of the time information appearing in the video parameter set, the sequence parameter set and the picture parameter set may be in any combination. The set time information may be the same, or may be generated by combination.

In this example, slice_segement_header_extension_present_flag in a field corresponding to slice header information is pic_alignment_flag namely. When the slice_segement_header_extension_present_flag is <NUM>, slice_segment_header_extension_data is namely alignment_code, and the time information is further given by using the bit field in Table <NUM>. The operation methods of the source device and the host device on the pic_alignment_flag and alignment_code are similar to that in the second example.

The present example is different from the second example in that the position of the time information is not in the control range of the entire frame of picture, but is in a slice (Slice). It is required that the time information of slices in the same frame of picture has to be consistent. Moreover, the user may define the time information independently, and the rules may be unified by the source device and the host device through negotiation.

In this example, the alignment auxiliary information is located in other data structure having an effective range at least including one picture (frame picture and/or field picture); for the other data structure having the effective range at least including one picture (frame picture and/or field picture), if the data structure includes data information that has to be used in other decoding process, then the data structure is the data structure required in the decoding process. If the data structure uses type information similar to pic_alignment_type, then the operation methods of the source device and the host device on the pic_alignment_type and alignment_code are similar to that in the first example. If the data structure uses a flag similar to pic_alignment_flag, then the operation methods of the source device and the host device on the pic_alignment_flag are similar to that in the second example.

The present example differs from the first example and the second example in that the source device uses a corresponding coding method to write the pic_alignment_flag and/or pic_alignment_type, and the alignment_code into a corresponding bitstream of the data structure at least including one picture (frame picture and/or field picture), and the host device uses a corresponding decoding method to parse fields corresponding to the pic_alignment_flag and/or pic_alignment_type, and the alignment_code from the corresponding bitstream of the data structure at least including one picture (frame picture and/or field picture) to acquire the values of the fields.

In this example, alignment or synchronization information is located in a system layer, such as a MPEG-<NUM> TS (Transport Stream) layer, PTS_DTS_flags is namely pic_alignment_flag, and is located in a PES_packet of TS. The control range thereof is the Transport Stream to which the current PES belongs. The PTS_DTS_flags is equal to <NUM>, representing that there is time information PTS (Presentation Time Stamps) subsequently. The PTS_DTS_flags is equal to <NUM>, representing that there are time information PTS and DTS (Decoding Time Stamps) subsequently. The PTS_DTS_flags is equal to <NUM>, representing that there is no time information PTS or DTS subsequently.

The bit field in Table <NUM> is used after the PTS _DTS_flags field to further give the time information.

In this example, transmission of video bitstream with different coding formats mainly includes the following steps.

In step <NUM>, the source device determines whether it is necessary to conduct an alignment operation on the videos with different coding formats according to an application demand (for example, whether the coder uses the AVC picture of the base layer as a reference for the HEVC picture of the enhancement layer, or whether the AVC picture and the HEVC picture need to be outputted synchronously).

If alignment information is needed to be used, the source device sets the PTS_DTS_flags as <NUM> or <NUM>; otherwise, the source device sets the PTS_DTS_flags as <NUM>. In step <NUM>, the source device uses a corresponding coding method to write the value of the PTS_DTS_flags into a PES packet header of MPEG-2TS for packing the video bitstream of AVC and a PES packet header of MPEG-2TS for packing the video bitstream of HEVC. At this time, the video bitstreams of non-HEVC and the video bitstreams of HEVC are in different TS packets.

In step <NUM>, the source device acquires the system time when the current coder performs hybrid coding, and writes the time information into the PES packet header of MPEG-2TS for packing the video bitstream of AVC and the PES packet header of MPEG-2TS for packing the video bitstream of HEVC respectively as alignment information.

In step <NUM>, after receiving the bitstreams, the source device searches PTS_DTS_flags in the PES packet header of MPEG-<NUM> TS for packing the video bitstream of AVC and the PES packet header of MPEG-<NUM> TS for packing the video bitstream of HEVC respectively.

In step <NUM>, the host device uses a corresponding decoding method to acquire the values of the PTS_DTS_flags from the PES packet header of MPEG-<NUM> TS for packing the video bitstream of AVC and the PES packet header of MPEG-<NUM> TS for packing the video bitstream of HEVC respectively, and determines whether there is time information (whether the PTS_DTS_flags is equal to <NUM> or <NUM>).

In step <NUM>, if there is the time information (the PTS_DTS_flags is equal to <NUM> or <NUM>), the host device uses a corresponding decoding method to acquire the time information in the PES packet header of MPEG-<NUM> TS for packing the video bitstream of AVC and the PES packet header of MPEG-<NUM> TS for packing the video bitstream of HEVC respectively according to the fields shown in the corresponding Table <NUM>.

In step <NUM>, if synchronization is needed, i.e., the time information acquired by AVC and HEVC is consistent, the host device puts the reconstructed picture decoded by the current AVC decoder into the reference frame management list of the HEVC for the HEVC decoder to use. For a multiple viewpoint application, the host device uses this information to conduct synchronous output and display operations on pictures of different viewpoint.

In this example, the above-mentioned alignment auxiliary information is located in a field corresponding to a video media attribute of a file; when an effective range of the field corresponding to video media attribute including pic_alignment_flag and/or pic_alignment_type is the entire video bitstream with different coding formats in the bitstreams of a media file, the operation methods of the source device and the host device on the pic_alignment_flag and pic_alignment_type are similar to that in the first and second examples.

This example is different from the first and second examples in that the source device uses the corresponding coding method to write the values of the pic_alignment_flag and/or pic_alignment_type, and the alignment_code into a system bitstream corresponding to the field corresponding to the video media attribute. The host device uses the corresponding decoding method to parse the fields corresponding to the pic_alignment_flag and/or pic_alignment_type, and the alignment_code from the corresponding system bitstream of the field for describing the video media attribute, and acquire the values thereof.

In this example, the above-mentioned alignment auxiliary information is located in a media data area of the application layer. When the effective range of the data area containing the pic_alignment_flag and/or pic_alignment_type is the entire video bitstreams of different coding formats in the bitstreams of the application layer, the operation methods of the source device and the host device on the pic_alignment_flag and/or pic_alignment_type are similar to those in the first and second examples.

This example is different from the first and second examples in that the source device uses the corresponding coding method to write the values of the pic_alignment_flag and/or pic_alignment_type, and the alignment_code into the system bitstream corresponding to the media data area. The host device uses the corresponding decoding method to parse the fields corresponding to the pic_alignment_flag and/or pic_alignment_type, and the alignment_code from the corresponding system bitstream of the media data area, and acquire the values thereof.

For example, the above alignment or synchronization information is written into a private data byte area (Private Data Bytes) in the bitstream of a DVB application layer containing video data of different coding formats.

The following steps are mainly included.

If alignment information is needed to be used, the source device sets a private data flag (private _data _flag) as <NUM>, an sets a private data packet header type (data_field_tag) as one of the user-defined values (0xA0 to 0xFF, not limited to this value range, and a specific defined value may be agreed by the source device and the host device); otherwise, the private_data_flag is set as <NUM>.

In step <NUM>, the source device uses a corresponding coding method to respectively write the value of the data_field_tag into a DVB private data area for packing the video bitstream of AVC, and a DVB private data area for packing the video bitstream of HEVC.

In step <NUM>, the source device uses a corresponding coding method to respectively write the value of the data_field_tag into a DVB private data area for packing the video bitstream of non-AVC, and a DVB private data area for packing the video bitstream of HEVC.

In step <NUM>, the source device acquires the system time when the current coder performs hybrid coding, and writes the time information into the DVB private data area for packing the video bitstream of AVC and the DVB private data area for packing the video bitstream of HEVC respectively as alignment information.

In step <NUM>, after receiving the bitstreams, the host device searches and determines whether the private_data_flag exists in the DVB private data area for packing the video bitstream of AVC and the DVB private data area for packing the video bitstream of HEVC respectively.

In step <NUM>, the host device uses a decoding method corresponding to u(<NUM>) to acquire the value of the private_data_flag from the DVB bitstream for packing the video bitstream of non-HEVC video data and the DVB bitstream for packing the video bitstream of HEVC video data, and determines whether there is private data information.

In step <NUM>, if there is the time information (the private_data_flag is equal to <NUM>), the host device obtains private data, and uses a corresponding decoding method to decode data_field_tag from the PES packet header of MPEG-<NUM> TS for packing the video bitstream of AVC and the PES packet header of MPEG-<NUM> TS for packing the video bitstream of HEVC from the bitstreams respectively; if it is the user-defined value (a certain value from 0xA0 to 0xFF, not limited to this value range) agreed with the source device, then there is time information; otherwise, there is no time information.

In step <NUM>, if there is the time information, the host device uses a corresponding decoding method to acquire the time information from the DVB private data area containing video data of non-HEVC and the DVB private data area containing video data of HEVC according to the fields shown in Table <NUM>.

In step <NUM>, if synchronization is needed, i.e., the time information acquired by AVC and HEVC is consistent, the host device puts a reconstructed picture, which is obtained by decoding by the current AVC decoder, into a reference frame management list of the HEVC for being used by the HEVC decoder. For a multiple viewpoint application, the host device uses this information to conduct synchronous output and display operations on pictures of different viewpoints.

The above first to eighth examples are all directed to the processing when the multiple bitstreams are bitstreams of different standards. The present example illustrates the case where the multiple bitstreams are of the same standard. The alignment or synchronization processing of the multiple bitstreams in this example is similar to that of the multiple bitstreams of different standards. Two same bitstreams are illustrated hereinafter, for example: both the base layer and the enhancement layer contain video bitstreams of H. <NUM>/HEVC. At this moment, the source device determines a form of adding alignment or synchronization information into the two video bitstreams of H. <NUM>/HEVC according to an application demand. The host device acquires the required alignment or synchronization information.

The source device may use the method according to the first to eighth examples to set the values of the pic_alignment_flag and/or pic_alignment_type, and the alignment_code in a high layer data structure and a low layer data structure in the video bitstream layer, the system layer and the application layer, and uses a corresponding coding method to write the values into the bitstreams.

The host device processes the received bitstreams, and searches to acquire the values of the pic_alignment_flag and/or pic_alignment_type, and the alignment_code from the application layer, the system layer and the video bitstream layer using the method in the first to eighth examples, and converts the values into a uniform time format according to the unified agreement with the source device. The host device uses the above-mentioned methods to set an error control and play control module during receiving and decoding.

In this example, the positions of the bitstreams of different videos or same video standards for carrying the above-mentioned alignment auxiliary information may be different. The source device determines the positions of adding alignment or synchronization information into the video bitstreams according to an application demand, and writes the same information format. The host device searches according to a sequence of the application layer, the system layer, and the video encoding layer until the required alignment or synchronization information is acquired.

When using the hybrid method, the source device firstly determines the alignment manner of various bitstreams with different video standards or the same video standard according to the application demand, including determining the layer where the alignment information locates, and a data form of the alignment information. The source device uses the method in the first to eighth examples to set the values of the pic_alignment_flag and/or pic_alignment_type, and the alignment_code in the high layer data structure and the low layer data structure in the video bitstream layer, the system layer and the application layer according to the above determined information, and uses a corresponding coding method to write the values into the bitstream.

The host device processes the received bitstreams, and searches to acquire the values of the pic_alignment_flag and/or pic_alignment_type, and the alignment_code from the application layer, the system layer and the video bitstream layer using the method in the first to eighth examples. The host device uses the above-mentioned methods to set an error control and play control module during receiving and decoding.

In this example, forms of the various bitstreams with different video standards or same video standard for carrying the above-mentioned alignment auxiliary information may be different. The source device determines the forms of adding alignment or synchronization information into various bitstreams of different video standards or same video standard according to an application demand. The rule is unified by the source device and the host device through negotiation, for example, the time information is uniformly converted into the system time, or the number is unified. The host device searches according to a sequence of the application layer, the system layer, and the video encoding layer until the required alignment or synchronization information is acquired.

In the used hybrid method, the source device firstly determines the alignment methods of various bitstreams with different video standards or same video standard according to the application demand, including determining the layer where the alignment or synchronization information locates, and a data form. The source device uses the method in the first to eighth examples to set the values of the pic_alignment_flag and/or pic_alignment_type, and the alignment_code in the high layer data structure and the low layer data structure in the video bitstream layer, the system layer and the application layer according to the above determined information, and uses a corresponding coding method to write the values into the bitstream.

In the example, the hybrid method is used to carry the above-mentioned alignment auxiliary information. For the various bitstreams with different video standards or same video standard, the application layer is a data structure higher than the system layer, and the system layer is a data structure higher than the video coding layer.

In the example, the pic_alignment_flag and/or pic_alignment_type, and the alignment_code may be coded in the data structures with different levels. When the values of the pic_alignment_flag and/or pic_alignment_type, and the alignment_code in the high layer data structure and the low layer data structure are different, the values in the low layer data structure cover the values in the high layer data structure. When the values of the pic_alignment_flag and/or pic_alignment_type, and the alignment_code in the low layer data structure are incomplete or missing, the values in the high layer data structure may be borrowed for alignment or synchronization. In the hybrid methods used, the source device firstly determines the alignment methods of various bitstreams with different video standards or same video standard according to the application demand, including determining the layer where the alignment or synchronization information locates, and the data form. The source device uses the method in the first to eighth examples to set the values of the pic_alignment_flag and/or pic_alignment_type, and the alignment_code in the high layer data structure and the low layer data structure in the video bitstream layer, the system layer and the application layer according to the information determined, and uses a corresponding coding method to write the values into the bitstream.

The host device processes the received bitstreams, and acquires the values of the pic_alignment_flag and/or pic_alignment_type, and the alignment_code from the application layer, the system layer and the video bitstream layer using the method in the first to eighth examples. The host device uses the above-mentioned methods to set an error control and play control module during receiving and decoding.

The host device may use the alignment or synchronization information acquired from different layers as a condition for bitstream error detection and play control condition thereof. If there is a difference among the alignment or synchronization information acquired from different layers, the host device executes an error control mechanism to report an error to the source device through feedback information. The host device may directly conduct output and play control between the picture of this layer and the pictures of other layers according to the alignment or synchronization information.

In conclusion, descriptions of video alignments in different coding formats may be added in descriptions of a high layer, the system layer and the video layer of bitstreams through the method provided by the embodiments of the present disclosure. Meanwhile, a layered description mechanism is used in the high layer structure of bitstreams, which is beneficial for the flexible control during the bitstream generation process.

It will be apparent to those skilled in the art that the above-described modules or steps of the present disclosure may be implemented by a general purpose computing device which may be focused on a single computing device or distributed over a network consisting of a plurality of computing devices, which may optionally be implemented by a program code executable by the computing device, so that they may be stored in a storage device for executing by the computing device and, in some cases, the steps shown or described herein may be performed in a different sequence; or they are achieved by separately making them into individual integrated circuit modules, or by making multiple modules or steps in them into a single integrated circuit module. Thus, the present disclosure is not limited to any particular combination of hardware and software.

Claim 1:
A method for multi-protocol hybrid coding, wherein multiple bitstreams of different video standards with different coding formats are aligned, the method comprising:
when the multiple bitstreams need to be aligned, setting (S102), by a transmitting terminal, alignment auxiliary information in each bitstream of the multiple bitstreams; the method characterized in
transmitting the multiple bitstreams respectively and independently to a receiving terminal, by the transmitting terminal (S103).