Grouping Of Video Streaming Messages

A video streaming server transmitting coded video for a current picture in a first set of network packets. The server generates a set of configuration data for the current picture. The server transmits the set of configuration data in a second set of network packets. The server transmits a particular network packet comprising a group identifier identifying a group that is applicable to the current picture, the identified group comprising the second set of network packets. A video streaming client receives the first set of network packets and reconstructs the current picture. The client receives the second set of network packets and the particular network packet. The client uses the group identifier in the particular network packet to identify the second set of network packets as being in the group that is applicable to the current picture. The client outputs the reconstructed current picture by applying the set configuration data.

TECHNICAL FIELD

The present disclosure relates generally to video streaming. In particular, the present disclosure relates to methods of grouping Neural Network Post Filter (NNPF) and Supplemental Enhancement Information (SEI) messages.

BACKGROUND

Video streaming is a continuous transmission of video files from a server to a client. Video streaming enables users to view videos online without having to download them. In video streams, content is sent in a compressed form over the internet and is displayed by the viewer in real time. The media is sent in a continuous stream of data and is played as it arrives. A video player is a program or device that uncompresses and provides video data to the display and audio data to speakers. Video streams begin with a prerecorded media file hosted on a remote server. Once the server receives a client request, the data in the video file is compressed and sent to the requesting device in pieces. Audio and video files are broken into data packets, where each packet contains a small piece of data. A transmission protocol such as Transmission Control Protocol (TCP) or User Datagram Protocol (UDP) is used to exchange data over a network. Once the requesting client receives the data packets, a video player on the user end will decompress the data and interpret video and audio. The video files are automatically deleted once played.

Versatile video coding (VVC) is the latest international video coding standard developed by the Joint Video Expert Team (JVET) of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11. The input video signal is predicted from the reconstructed signal, which is derived from the coded picture regions. The prediction residual signal is processed by a block transform. The transform coefficients are quantized and entropy coded together with other side information in the bitstream. The reconstructed signal is generated from the prediction signal and the reconstructed residual signal after inverse transform on the de-quantized transform coefficients. The reconstructed signal is further processed by in-loop filtering for removing coding artifacts. The decoded pictures are stored in the frame buffer for predicting the future pictures in the input video signal.

The coded video data is organized into Network Abstraction Layer (NAL) units, each of which is effectively a packet that contains an integer number of bytes. NAL units are classified into Video Coding Layer (VCL) and non-VCL NAL units. The VCL NAL units contain the data that represents the values of the samples in the video pictures, and the non-VCL NAL units contain any associated additional information such as parameter sets (e.g., important header data that can apply to a large number of VCL NAL units) and supplemental enhancement information (SEI) (e.g., timing information and other supplemental data that may enhance usability of the decoded video signal but are not necessary for decoding the values of the samples in the video pictures).

Supplemental enhancement information (SEI) is additional data that can be inserted into a bitstream of video content during encoding and transmission. SEI messages are metadata that are inserted into the bitstream as NAL units during encoding and transmission. SEI messages can be used for a variety of purposes, including decoding, display, and other applications. They can convey technical information related to the bitstream, such as camera or encoder parameters, time code, closed captions, lyrics, or copyright information.

SUMMARY

Some embodiments of the disclosure provide a video streaming method. A video streaming server transmitting coded video for a current picture in a first set of network packets (e.g., VCL NAL units). The server generates a set of configuration data for the current picture. The server transmits the set of configuration data in a second set of network packets (e.g., non-VCL NAL units or SEI messages). The server transmits a particular network packet comprising a group identifier identifying a group that is applicable to the current picture, the identified group comprising the second set of network packets. A video streaming client receives the first set of network packets and reconstructs the current picture. The client receives the second set of network packets and the particular network packet. The client uses the group identifier in the particular network packet to identify the second set of network packets as being in the group that is applicable to the current picture. The client outputs the reconstructed current picture by applying the set configuration data.

The particular network data packet may assign processing order to the network data packets in the second set of network data packets. The particular network data packet may be a SEI processing order group characteristic (SPOGC) SEI message that associates persistence, purpose, grouping type, or other processing characteristics with the group identifier. The particular network data packet may be a SPOG activation (SPOGA) SEI message that activates or de-activates a processing function that uses the configuration data in the second set of network data packets. The activated/deactivated function may be a neural network post filter to be applied to the current picture and the generated configuration data is for configuring the neural network post filter. The group may be one of a plurality of groups that are associated with the current picture, the plurality of groups comprising multiple sets of network data packets for supporting respective multiple processing functions, which may be multiple neural network post filters to be applied to the current picture.

In some embodiments, the group identifier is a first group identifier for a first group comprising one or more network data packets and at least a second (nested) group associated with a second group identifier, with the second group including one or more network data packets.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are set forth by way of examples in order to provide a thorough understanding of the relevant teachings. Any variations, derivatives and/or extensions based on teachings described herein are within the protective scope of the present disclosure. In some instances, well-known methods, procedures, components, and/or circuitry pertaining to one or more example implementations disclosed herein may be described at a relatively high level without detail, in order to avoid unnecessarily obscuring aspects of teachings of the present disclosure.

Neural Network Post Filter (NNPF) SEI messages enable the use of neural networks for post-processing video streaming such as super-resolution, frame rate upsampling, chroma format conversion, and colorization. The NNPF characteristics (NNPFC) SEI message signals neural-network parameters/weights and additional information needed for a receiver to determine if it can implement the indicated neural network. Several neural networks can be signaled using multiple NNPFC SEI messages to support different receiver capabilities and different post-processing operations. Specific neural networks may be invoked from available neural networks using an NNPF activation (NNPFA) SEI message.

FIG.1conceptually illustrates a video streaming system100that supports neural network post filters (NNPFs) by NNPF SEI messages. In the video streaming system100, a video encoder110generates encoded video content. The encoded video content is delivered as video content NAL units to a video decoder120to be reconstructed. The reconstructed video is post-processed by a neural network post filter125before being displayed at a display device130.

The operations of the neural network post filter125are configured by post filter control data in SEI messages (specifically NNPF SEI messages), which are received along with the video content NAL units. (In the figure, the video content NAL units are labeled as “VCL” and SEI message NAL units are labeled as “SEI”, which are non-VCL NAL units.) The neural network control data in the SEI messages are generated by a post filter controller115based on information provided by the video encoder110regarding the video content.

An encoder side streaming multiplexer device140splices video content NAL units from the encoder110and the SEI messages from the post filter controller115as a bitstream150. A decoder side streaming demultiplexer device145receives the bitstream150to provide the video content NAL to the video decoder120and SEI messages (at least those related to the neural network post filtering) to the neural network post filter125.

In some embodiments, some or all of the devices in the encoder side (the encoder110, the post filter controller115, the multiplexer140, etc.) are implemented by one computing device or even one integrated circuit. In some embodiments, some or all of the devices in the decoder side (the demultiplexer145, the decoder120, the neural network post filter125, the display130, etc.) are implemented by one computing device or even one integrated circuit.

I. Specifying Multiple Neural Network Post Filters

In some embodiments, the neural network post filter125at the decoder side inFIG.1may include multiple NNPFs. These NNPFs may be organized into NNPF groups (also termed NNPF list). For an NNPF group, SEI messages may be used to specify some properties or operations, including group relationships and corresponding processing steps. In some embodiments, the operations and/or properties of a NNPF group are specified by a NNPF group characteristic SEI message, which provides group id, type, and purpose of the NNPF group. In some embodiments, the operations and/or properties of a NNPF group are specified by a NNPF group activation SEI message, which provides target NNPF group id.

In some embodiments, the SEI messages in a bitstream support multiple activated NNPFs (with the same or different purposes) for a picture encoded in the bitstream, in a cascading or alternative manner. In some embodiments, the SEI messages provide instructions for multiple activated NNPFs for a picture while the receiver may process the bitstream multiple times to generate multiple different results.

In some embodiments, the properties or operations of an NNPF group are addressed before the NNPF group can be used correctly for an intended purpose. In some embodiments, the properties or operations of an NNPF group is specified for a specific processing order with other SEI messages by the SEI processing order (SPO) SEI messages in the bitstream.

In some embodiments, NNPF group SEI messages are incorporated into to a SEI payload according to the following Tables 1-3:

A neural-network post-filter group (NNPFG) SEI message specifies a neural network post-filter (NNPF) group comprising two or more combined NNPFs and/or NNPFGs applied in terms of the group operation. The use of NNPFG for specific pictures is indicated with neural-network post-filter group activation (NNPFGA) SEI messages. An NNPFG can be included in other NNPFGs, i.e., in a nested manner.

nnpfg_purpose indicates the purpose of the NNPF group and has the semantics of nnpfc_purpose.

nnpfg_id contains an identifying number that may be used to identify an NNPFG. The value of nnpfg_id shall be in the same range as that of the value of nnpfc_id. The value of nnpfg_id shall be unique and different from all values of nnpf_id and other values of nnpfg_id in the bitstream.

nnpfg_operation indicates the group relation and processing steps for an NNPFG. The value of nnpfg_operation shall be in the range of 0 to 7, inclusive, in bitstreams conforming to this edition of this document. Values of 4 to 7, inclusive, for nnpfg_operation are reserved for future use by ITU-T|ISO/IEC and shall not be present in bitstreams conforming to this edition of this document. Decoders conforming to this edition of this document shall ignore NNPFG SEI messages with nnpfg_operation in the range of 4 to 7, inclusive. The table below provides descriptions of nnpfg_operation:

TABLE 3ValueDescription0Optional - Applying the NNPF and/or NNPFG is optional1Cascade - The NNPFs and/or NNPFGs in an NNPF group areapplied in a way that the input pictures to the (i + 1)-th NNPFor NNPFG in an NNPF group are derived from the output ofthe i-th NNPF or NNPFG in the NNPF group.2Alternate - The NNPF or NNPFG in an NNPF group is appliedalternatively (only one is chosen) to the associated picture3Coherence - The NNPFs and/or NNPFGs in an NNPF group areapplied in parallel (in contrast to the cascade) to theassociated picture4-7Reserved

FIGS.2A-Cillustrate the operations of a group of neural network post filters (NNPFs) in three different cases: Cascade, Alternate, and Coherence. The value of nnpfg_operation is used to indicate one of the three cases according to Table 3.FIG.2Aillustrates a group of NNPFs or NNPFGs in cascade configuration (nnpfg_operation=1).FIG.2Billustrates a group of NNPFs or NNPFGs in alternate configuration (nnpfg_operation=2).FIG.2Cillustrates a group of NNPFs or NNPFGs in coherence configuration (nnpfg_operation=3). Some constraints may further impose on input to an NNPF or NNPFG, for example, for a cascade case, a coherence NNPFG (with two outputs) may be followed by an NNPF (with one input).

num_nnpf_nnpfg_minus2 plus 2 indicates the number of NNPFs and/or NNPFGs in the NNPF group that this SEI message defines.

nnpf_nnpfg_id[i] indicates that the i-th NNPF or NNPFG in the NNPF group has nnpfc_id or nnpfg_id equal to nnpf_nnpfg_id[i]. Each NNPF or NNPFG with nnpfc_id or nnpfg_id equal to nnpf_nnpfg_id[i] is defined by one or more NNPFC messages or one NNPFG SEI message.

nnpfg_complexity_info_present_flag equal to 1 specifies that one or more syntax elements that indicate the complexity of the NNPFG associated with the nnpfg_id are present. nnpfg_complexity_info_present_flag equal to 0 specifies that no syntax elements that indicates the complexity of the NNPFG associated with the nnpfg_id are present.

nnpfg_region_info_present_flag equal to 1 specifies that one or more syntax elements that indicate the regions of the NNPFG associated with the nnpfg_id are present. nnpfg_region_info_present_flag equal to 0 specifies that no syntax elements that indicates the regions of the NNPFG associated with the nnpfg_id are present.

b. NNPF Group Activation SEI Message

A neural-network post-filter group activation (NNPFGA) SEI message activates or de-activates the possible use of the target neural-network post-processing filter group (NNPFG), identified by nnpfga_target_id, for post-processing filtering of a set of pictures. For a particular picture for which the NNPFG is activated, the target NNPFG is the NNPFG specified by the NNPFG SEI message with nnpfg_id equal to nnpfga_target_id, that precedes the first VCL NAL unit of the current picture in decoding order and the NNPFs or NNPFGs of the target NNPFG are defined by the NNPFC SEI messages or the NNPFG SEI messages that have nnpfc_id or nnpfg_id equal to any nnpf_nnpfg_id[i] value of the target NNPFG and are present in the current picture unit or precede the current picture in decoding order. The syntax of NNPFGA SEI message is provided in Table 4 below:

nnpfga_target_id indicates the target NNPFG, which is specified by an NNPFG SEI message that pertain to the current picture and have nnpfg_id equal to nnpfga_target_id. The value of nnpfga_target_id shall be in the same range as that of the value of nnpfg_id.

nnpfga_cancel_flag equal to 1 indicates that the persistence of the target NNPFG established by any previous NNPFGA SEI message with the same nnpfga_target_id as the current SEI message is cancelled, i.e., the target NNPFG is no longer used unless it is activated by another NNPFGA SEI message with the same nnpfga_target_id as the current SEI message and nnpfga_cancel_flag equal to 0. nnpfga_cancel_flag equal to 0 indicates that the nnpfga_persistence_flag follows.

nnpfga_persistence_flag specifies the persistence of the target NNPFG for the current layer.

nnpfga_region_info_present_flag equal to 1 specifies that one or more syntax elements that indicate the regions of the NNPFG associated with the nnpfg_id are present. nnpfg_region_info_present_flag equal to 0 specifies that no syntax elements that indicates the complexity of the NNPFG associated with the nnpfg_id are present.

nnpfga_region_info_present_flag and nnpfga_region_info_present_flag may be used to specify the presence of static region information or dynamic region information. When all NNPFs of an NNPFG are the same, nnpfga_region_info_present_flag and nnpfga_region_info_present_flag may be used to specify the presence of static region information or dynamic region information for the NNPF rather than the NNPFG.

II. Generic SEI Processing Order SEI Messages

SEI processing order (SPO) SEI messages are used for establishing processing order among SEI messages. SPO SEI message is useful not only for indicating stages of processing operations, but also for indicating the properties and intended uses of the video content after particular stages of processing. For some embodiments, the syntax for a SPO SEI message is shown in Table 5 below:

In some embodiments, the SEI processing order (SPO) SEI message is further extended to support various processing order types in addition to the cascade processing and to include the neural network post filter (NNPF) SEI messages in the SPO SEI message. The following sub-sections describes these improvements for some embodiments.

The NNPF SEI messages comprise the NNPF characteristics SEI message and the NNPF activation SEI message. The NNPF characteristic SEI messages precede the NNPF activation SEI message when present in the bitstream. The NNPF activation SEI message, when present, signals the NNPF processing in terms of the NNPF characteristic SEI message applied to the associated picture. And, like other non-NNPF SEI messages, the NNPF activation SEI message specifies the scope of persistence. Table 6 below is an example SPO SEI message that supports the NNPF SEI messages in the SPO SEI message for some embodiments.

po_num_sei_messages_minus2 plus 2 indicates the number of SEI messages that have a processing order indicated in the SEI processing order SEI message. The indicated SEI messages are processed in the sequential order as specified with the index i from 0 to po_num_sei_messages_minus2+1, inclusive, which means the indicated SEI messages are ordered with increasing values of the index i.

sei_activation_present_flag[i] equal to 1 indicates the activation SEI message following the i-th SEI message is present. sei_activation_present_flag[i] equal to 0 indicates the activation SEI message following the i-th SEI message is not present. The activation SEI message, when present, is applicable for the i-th SEI message as the activation for use.

Thus, in addition to indicating an SEI message, the corresponding activation SEI message is also indicated (sei_activation_present_flag) if applicable. This is for indicating whether NNPF SEI messages are present in pair in the SPO SEI message, i.e., an NNPF characteristic SEI message is indicated like other non-NNPF SEI messages followed by indicating the corresponding NNPF activation SEI message. In some embodiments, SPO SEI message may be modified according to the following Table 7.

For each picture in the video, there can be multiple persisting or activated SEI messages belonging to one or more groups of SEI messages. Groups of SEI messages can be alternatives to each other, i.e., such that at most one group is chosen to be applied, or they can be complementary, i.e., such that more than one group is chosen and applied separately, with each group generating one output.

In some embodiments, two types of SEI messages are used for specifying processing order for groups of SEI messages: SEI processing order group characteristic (SPOGC) SEI messages and SEI processing order group activation (SPOGA) SEI messages. A SPOGC SEI message may identify one or more SEI messages as a SEI processing order group (SPOG) using a unique SPOG identifier and an initial processing order type. A SPOGA SEI message, when present, specifies the final processing order type for one or more SPOGs that were specified by the SPOGC SEI messages. The SPOGA SEI message is applied to one or more associated pictures and specifies the scope of persistence of the indicated one or more SPOGs. The SPOGC SEI messages (one or more) when present in the bitstream precede the SPOGA SEI message.

The SPOGC SEI message carries information indicating the preferred processing order type, as determined by the encoder (i.e., the content producer), for different types of SEI messages that may be present in a coded video sequence (CVS). Table 8 below shows the syntax of a SPOG characteristic (SPOGC) SEI message for some embodiments:

spogc_id contains an identifying number that may be used to identify an SPOG. The value of spogc_id shall be in the range of 0 to 255. The value of spogc_id shall be unique and different from other values of spogc_id in the bitstream.

spogc_purpose indicates the purpose of the SPOG and has the semantics of spogc_purpose. Definition of spogc_purpose and spoga_purpose is specified in Table 9 below:

spogc_type indicates the processing order type for the SPOG. The semantics of spogc_type as specified in Table 10 below. The value of spogc_type shall be in the range of 0 to 7, inclusive, in bitstreams conforming to this edition of this document. Values of 4 to 7, inclusive, for spogc_type are reserved for future use may not be present in bitstreams conforming to this document. Decoders conforming to this document shall ignore SPOGC SEI messages with spogc_type in the range of 4 to 7, inclusive.

spogc_num_sei_messages_minus1 plus 1 indicates the number of SEI messages that have a processing order type spogc_type (specified in Table 10) indicated in SPOGC SEI message below.

TABLE 10Values ofspogc_typeandspoga_typeDescription0Unspecified (e.g., it may be used for a single SEImessage in an SPOGC SEI message with spogc_type ora single SPOG in an SPOGA SEI message withspoga_type.)1Cascade - The SEI messages in an SPOGC SEI messagewith spogc_type or the SPOGs in an SPOGA SEI messagewith spoga_type is applied in the sequential orderwith the index i, which means in a way that theindicated SEI messages or SPOGs are ordered withincreasing values of the index i.2Alternate - The SEI messages in an SPOGC SEI messagewith spogc_type or the SPOGs in an SPOGA SEI messagewith spoga_type is applied alternatively (only oneis chosen).3Parallel - The SEI messages in an SPOGC SEI messagewiths pogc_type or the SPOGs in an SPOGA SEI messagewith spoga_type is applied in parallel.4-7Reserved

sei_activation_present_flag[i] equal to 1 indicates the activation SEI message following the i-th SEI message is present. sei_activation_present_flag[i] equal to 0 indicates the activation SEI message following the i-th SEI message is not present. The activation SEI message, when present, is applicable for the i-th SEI message as the activation for use.

FIG.3conceptually illustrates SPOGC SEI messages for setting the characteristics of SPOGs. The figure illustrates two example SPOGC SEI messages301and302complying with syntax of Table 8. In the figure, the SPOGC SEI message301having spogc_id=1 defines a SPOG (SPOG1) that includes several SEI messages A, B, C, and D. The SPOGC SEI sets the processing order of the SEI messages in the SPOG, as well as the purpose and type parameters of the SPOG. The SPOGC SEI message302having spogc_id=2 defines a SPOG (SPOG2) that includes SEI messages E, F, G, H, and I. The SPOGC SEI sets the processing order of the SEI messages in the SPOG, as well as the purpose and type parameters of the SPOG.

The use of SPOGs for specific pictures is indicated with SPOG activation (SPOGA) SEI messages indicating the preferred final processing order types. SPOGA SEI messages identifies one or more SPOGs for specific pictures (individual SPOGs are identifiable by spogc_id). A SPOGA SEI message may indicate multiple SPOGs for a specific picture with a collective SPOG activation id. The SPOGA SEI message may also indicate the preferred processing order type for SPOGs identified by target values of SPOG id. Table 11 below shows syntax of a SPOGA SEI message for some embodiments:

The SPOGA SEI message activates or de-activates the use of the target SEI processing order groups (SPOGs), identified by spoga_target_id[i] for i=0 to spoga_num_pogc_minus1, inclusive, with the preferred processing order type, as determined by the encoder (i.e., the content producer), collectively identified by spoga_activation_id, for processing of a set of pictures.

FIG.4illustrates SPOGA SEI messages that activates or deactivate one or more target SPOGs. The figure illustrates two example SPOGC SEI messages401and402complying with syntax of Table 11. In the figure, the SPOGA SEI message401having activation_id=101 activates two SPOGs (SPOG1and SPOG2) with respective group identifiers (target_id=1 and target_id=2) by setting the persistence, purpose, and type parameters. The SPOGA SEI message402having activation_id=102 deactivates two SPOGs (SPOG3and SPOG4) by setting the cancel parameter.

For a particular picture for which the SPOGs are activated, the target SPOGs are the SPOGs (each SPOG comprising the applied SEI messages) specified by the SPOG characteristic (SPOGC) SEI messages, identified by spoga_target_id[i], for i=0 to spoga_num_pogc_minu1+1, inclusive, that precedes the first VCL NAL unit of the current picture in decoding order and the SPOGs of the target SPOGs are defined by the SPOGC SEI messages that have spogc_id equal to any spoga_target_id[i] value of the target SPOGs and are present in the current picture unit or precede the current picture in decoding order.

spoga_activation_id contains an identifying number that may be used to identify an SPOGA. The value of spogc_activation_id shall be in the range of 0 to 255. The value of spoga_activation_id shall be unique and different from other values of spoga_activation_id in the bitstream.

spoga_cancel_flag equal to 1 indicates that the SEI message cancels the persistence of the target SPOG specified by any previous SPOGA SEI message with the same spoga_target_id as the current SEI message in output order that applies to the current layer. spoga_cancel_flag equal to 0 indicates that the target SPOG is activated for use.

spoga_persistence_flag specifies the persistence of the target SPOGs for the current layer. spoga_persistence_flag equal to 0 specifies that the target SPOGs applies to the current decoded picture only. spoga_persistence_flag equal to 1 specifies that the target SPOGs applies to the current decoded picture and persists for all subsequent pictures of the current layer in output order until one or more of the following conditions are true:A new coded layer video sequence (CLVS) of the current layer begins.The bitstream ends.A picture in the current layer in an AU associated with a SPOGA SEI message with the same spoga_activation_id as the current SEI message that follows the current picture in output order.

spoga_purpose indicates the purpose of the SPOG and has the semantics of spoga_purpose as specified in Table 9.

spoga_type indicates the SEI processing order type for the SPOG and has the semantics of spoga_type as specified in Table 10. The value of spoga_type shall be in the range of 0 to 7, inclusive, in bitstreams conforming to this edition of this document. Values of 4 to 7, inclusive, for spoga_type are reserved for future use and shall not be present in bitstreams conforming to this edition of this document. Decoders conforming to this document shall ignore SPOGA SEI messages with spoga_type in the range of 4 to 7, inclusive.

spoga_num_spog_minus1 plus 1 indicates the number of SPOGs that have a processing order type spoga_type as specified in Table 10 indicated in the SPOGA SEI message.

spoga_target_id[i] indicates the i-th target SPOG, which is specified by an SPOGC SEI message that pertain to the current picture and have spogc_id equal to spoga_target_id[i]. The value of spoga_target_id[i] shall be in the range of 0 to 255.

In some embodiments, a SPOG can be defined recursively to include other SPOGs as well as individual SEI messages. In some embodiments, a SPOGC SEI message may operate on a combination of two or more individual SEI messages and/or SPOGs by treating the combination as a SPOG and assigning the combination a unique SPOG identifier and a processing order type. Table 12 below shows a SPOGC SEI message that defines a SPOG recursively (may include SPOGs and SEI messages).

The SPOGC SEI message carries information indicating the preferred processing order type, as determined by the encoder (i.e., the content producer), for a combination of different types of SEI messages and/or SPOGs. The recursive combination is identified by spogc_id in the SPOGC SEI message, which may be present in a CVS. The use of SPOG, identified by spogc_id, for specific pictures is indicated with SPOG activation (SPOGA) SEI messages with a target SPOG id.

spogc_num_sei_messages_minus2 plus 2 indicates the combined number of SEI messages and/or SPOGs that have a processing order type spogc_type (as specified in Table 10) indicated in the SPOGC SEI message.

In some embodiments, as shown in Table 12, identifiers of individual members of a SPOG that is itself a nested SPOG may be identified in the SPOGC SEI message using spogc_member_id.

spogc_member_id_present_flag[i] equal to 1 specifies that spogc_member_id[i] is present. spogc_member_id_present_flag[i] equal to 0 specifies spogc_member_id[i] is not present.

spogc_member_id[i] when present, indicates that the i-th SPOG indicated in this SPOGC SEI message has spogc_id equal to spogc_member_id[i]. Each SPOG with spogc_id equal to spogc_member_id[i] is defined by an SPOGC SEI message.

FIG.5conceptually illustrates a SPOGC SEI message for setting the characteristics a SPOG that includes both SPOGs and SEI messages. The figure illustrates a SPOGC SEI message500that sets the characteristics of a SPOG (SPOG101), including purpose and type. The SPOG101recursively includes several SPOGs (SPOG1and SPOG2) and several SEI messages, to which the SPOG101assigns processing order.

One or more such SPOGC SEI messages may precede a corresponding SPOGA SEI message when present in the bitstream. The SPOGA SEI message, when present, signals a target SPOG (which may recursively include SEI messages and SPOGs) with a target SPOG identifier that is applied to the associated picture. The SPOGA SEI also specifies a scope of persistence for the targeted SPOG.

In some embodiments, instead of using a collective activation id to target several SPOGs, the SPOGA SEI message activates or de-activates one specific target SPOG, identified by spoga_target_id, for processing of a set of pictures. Table 13 below shows a SPOGA SEI message with SPOGA target identifier.

For a particular picture for which the SPOG is activated, the target SPOG is the SPOG specified by an earlier SPOGC SEI message, identified by spoga_target_id that precedes the first VCL NAL unit of the current picture in decoding order and the SPOG of the target SPOG is defined by the SPOGC SEI message that have spogc_id equal to spoga_target_id of the target SPOG and is present in the current picture unit or precede the current picture in decoding order.

spoga_target_id indicates the target SPOG, which is specified by an SPOGC SEI message that pertain to the current picture and have spogc_id equal to spoga_target_id. The value of spoga_target_id shall be in the range of 0 to 255.

spoga_cancel_flag equal to 1 indicates that the SEI message cancels the persistence of the target SPOG specified by any previous SPOGA SEI message with the same spoga_target_id as the current SEI message in output order that applies to the current layer. spoga_cancel_flag equal to 0 indicates that the target SPOG is activated for use.

spoga_persistence_flag specifies the persistence of the target SPOGs for the current layer. spoga_persistence_flag equal to 0 specifies that the target SPOGs applies to the current decoded picture only. spoga_persistence_flag equal to 1 specifies that the target SPOGs applies to the current decoded picture and persists for all subsequent pictures of the current layer in output order until one or more of the following conditions are true:A new CLVS of the current layer begins.The bitstream ends.A picture in the current layer in an AU associated with a SPOGA SEI message with the same spoga_activation_id as the current SEI message that follows the current picture in output order.

In the example SPOGC SEI message of Table 12, identifiers of individual members of a SPOG (whether SEI messages or nested SPOGs) are indicated in the SPOGC SEI message by spogc_member_id. In some other embodiments, a SPOGC SEI message does not specify the identifiers of individual members. Table 14 below shows such a SPOGC SEI message:

spogc_num_sei_messages_minus2 plus 2 indicates the number of SEI messages that have a processing order type spogc_type (as specified in Table 10) indicated in the SPOGC SEI message.

The SEI processing order group (SPOG) characteristic (SPOGC) SEI message carries information indicating the preferred processing order type, as determined by the encoder (i.e., the content producer), for different types of SEI messages that may be present in a CVS.

The use of combination of SPOGs, identified by spogc_id, and/or other SEI messages for specific pictures is indicated with SPOGA SEI messages. The SPOGA SEI message may target a specific SPOG using a target id as in the example of Table 13 above. The SPOGA SEI message may also target a set of SPOGs with a collective SPOG activation id, indicating the preferred processing order types, as determined by the encoder (i.e., the content producer). Each of the set of SPOGs identified by the collective SPOG activation id may be a combination of nested SPOGs and SEI messages. For a particular picture for which the SEI messages are activated, for which a SPOG may include nested SPOGs and SEI messages, a SPOGA SEI message that activates a collection of SPOGs is shown in Table 15.

The SPOGA SEI message of Table 15 activates or de-activates the use of the combination of target SEI messages identified with the index i, and/or target SPOGs identified by spoga_target_id [i] or spoga_member_id [i], for i in the range of 0 to spoga_num_sei_messages_minus1, inclusive. The combination is collectively identified by spoga_activation_id for processing of a set of pictures with the preferred processing order type (as determined by the encoder or the content producer).

FIG.6conceptually illustrates a SPOGA SEI message that activates or deactivates a combination that includes both SPOGs and SEI messages. As illustrated, a SPOGA SEI message600collectively activates several SPOGs and SEI messages as a combination identified by an activation id (activation_id=104). The SPOGs of the combination are identified by their respective spoga_member_id, while the SEI messages (P, Q, and R) of the combination are assigned processing orders. The SPOGA SEI message600may de-activate the entire activation group by setting the cancel parameter, or setting the persistence, purpose, and type parameters of the group.

For a particular picture for which the SPOGs are activated, the target SPOGs are the SPOGs (comprising the applied SEI messages) specified by an earlier corresponding SPOGC SEI messages. The target SPOGs are identified by spoga_member_id[i], for i in the range of 0 to spoga_num_sei_messages_minus1, inclusive, that precedes the first VCL NAL unit of the current picture in decoding order. The target SPOGs are defined by the SPOGC SEI messages to have spogc_id equal to any spoga_member_id[i] value in the SPOGA SEI messages and are present in the current picture unit or precede the current picture in decoding order.

For a particular picture for which the SEI messages are activated, the target SEI messages are the SEI messages indicated in this SEI message with the index i, for i in the range of 0 to spoga_num_sei_messages_minus1, inclusive.

spoga_activation_id contains an identifying number that may be used to identify an SPOGA. The value of spogc_activation_id shall be in the range of 0 to 255. The value of spoga_activation_id shall be unique and different from other values of spoga_activation_id in the bitstream.

spoga_cancel_flag equal to 1 indicates that the SEI message cancels the persistence of the combination of target SEI messages and/or target SPOGs specified by any previous SPOGA SEI message with the same spoga_activation_id as the current SEI message in output order that applies to the current layer. spoga_cancel_flag equal to 0 indicates that the combination of target SEI messages and/or target SPOGs are activated for use and the information follows.

spoga_persistence_flag specifies the persistence of the combination of target SEI messages and/or target SPOGs for the current layer. spoga_persistence_flag equal to 0 specifies that the combination target SEI messages and/or target SPOGs applies to the current decoded picture only. spoga_persistence_flag equal to 1 specifies that the combination target SEI messages and/or target SPOGs applies to the current decoded picture and persists for all subsequent pictures of the current layer in output order until one or more of the following conditions are true:A new CLVS of the current layer begins.The bitstream ends.A picture in the current layer in an AU associated with a SPOGA SEI message with the same spoga_activation_id as the current SEI message that follows the current picture in output order.

spogc_num_sei_messages_minus2 plus 2 indicates the combined number of SEI messages and/or SPOGs that have a processing order type spogc_type as specified in Table X2 indicated in the SPOGC SEI message.

spoga_member_id_present_flag[i] equal to 1 specifies that spoga_member_id[i] is present. spoga_member_id_present_flag[i] equal to 0 specifies spoga_member_id[i] is not present.

spoga_member_id[i] when present, indicates that the i-th SPOG indicated in this SPOGA SEI message has spogc_id equal to spoga_member_id[i]. Each SPOG with spogc_id equal to spoga_member_id[i] is defined by an SPOGC SEI message.

III. Example Video Encoder

FIG.7illustrates an example video encoder700that may be part of a video streaming server. As illustrated, the video encoder700receives input video signal from a video source705and encodes the signal into bitstream795. The video encoder700has several components or modules for encoding the signal from the video source705, at least including some components selected from a transform module710, a quantization module711, an inverse quantization module714, an inverse transform module715, an intra-picture estimation module720, an intra-prediction module725, a motion compensation module730, a motion estimation module735, an in-loop filter745, a reconstructed picture buffer750, a MV buffer765, and a MV prediction module775, and an entropy encoder790. The motion compensation module730and the motion estimation module735are part of an inter-prediction module740.

In some embodiments, the modules710-790are modules of software instructions being executed by one or more processing units (e.g., a processor) of a computing device or electronic apparatus. In some embodiments, the modules710-790are modules of hardware circuits implemented by one or more integrated circuits (ICs) of an electronic apparatus. Though the modules710-790are illustrated as being separate modules, some of the modules can be combined into a single module.

The video source705provides a raw video signal that presents pixel data of each video frame without compression. A subtractor708computes the difference between the raw video pixel data of the video source705and the predicted pixel data713from the motion compensation module730or intra-prediction module725as prediction residual709. The transform module710converts the difference (or the residual pixel data or residual signal708) into transform coefficients (e.g., by performing Discrete Cosine Transform, or DCT). The quantization module711quantizes the transform coefficients into quantized data (or quantized coefficients)712, which is encoded into the bitstream795by the entropy encoder790.

The inverse quantization module714de-quantizes the quantized data (or quantized coefficients)712to obtain transform coefficients, and the inverse transform module715performs inverse transform on the transform coefficients to produce reconstructed residual719. The reconstructed residual719is added with the predicted pixel data713to produce reconstructed pixel data717. In some embodiments, the reconstructed pixel data717is temporarily stored in a line buffer (not illustrated) for intra-picture prediction and spatial MV prediction. The reconstructed pixels are filtered by the in-loop filter745and stored in the reconstructed picture buffer750. In some embodiments, the reconstructed picture buffer750is a storage external to the video encoder700. In some embodiments, the reconstructed picture buffer750is a storage internal to the video encoder700.

The intra-picture estimation module720performs intra-prediction based on the reconstructed pixel data717to produce intra prediction data. The intra-prediction data is provided to the entropy encoder790to be encoded into bitstream795. The intra-prediction data is also used by the intra-prediction module725to produce the predicted pixel data713.

The motion estimation module735performs inter-prediction by producing MVs to reference pixel data of previously decoded frames stored in the reconstructed picture buffer750. These MVs are provided to the motion compensation module730to produce predicted pixel data.

Instead of encoding the complete actual MVs in the bitstream, the video encoder700uses MV prediction to generate predicted MVs, and the difference between the MVs used for motion compensation and the predicted MVs is encoded as residual motion data and stored in the bitstream795.

The MV prediction module775generates the predicted MVs based on reference MVs that were generated for encoding previously video frames, i.e., the motion compensation MVs that were used to perform motion compensation. The MV prediction module775retrieves reference MVs from previous video frames from the MV buffer765. The video encoder700stores the MVs generated for the current video frame in the MV buffer765as reference MVs for generating predicted MVs.

The MV prediction module775uses the reference MVs to create the predicted MVs. The predicted MVs can be computed by spatial MV prediction or temporal MV prediction. The difference between the predicted MVs and the motion compensation MVs (MC MVs) of the current frame (residual motion data) are encoded into the bitstream795by the entropy encoder790.

The entropy encoder790encodes various parameters and data into the bitstream795by using entropy-coding techniques such as context-adaptive binary arithmetic coding (CABAC) or Huffman encoding. The entropy encoder790encodes various header elements, flags, along with the quantized transform coefficients712, and the residual motion data as syntax elements into the bitstream795. The bitstream795is in turn stored in a storage device or transmitted to a decoder over a communications medium such as a network.

The in-loop filter745performs filtering or smoothing operations on the reconstructed pixel data717to reduce the artifacts of coding, particularly at boundaries of pixel blocks. In some embodiments, the filtering or smoothing operations performed by the in-loop filter745include deblock filter (DBF), sample adaptive offset (SAO), and/or adaptive loop filter (ALF). In some embodiments, luma mapping chroma scaling (LMCS) is performed before the loop filters.

FIG.8illustrates a video streaming server device800that implements grouping of SEI messages. The video streaming server800includes the video encoder700, a NN post filter controller810, a SEI generator820, and a NAL multiplexer830. The NAL multiplexer830multiplexes between VCL NAL units generated by the video encoder700(at the entropy encoder790) and non-VCL NAL units generated by the SEI generator820to produce the bitstream795, which is provided to the network895to reach video streaming clients.

The NN post filter controller810receives information related to encoded video from the video encoder700, which may include the reconstructed pixel data717or the content of the reconstructed picture buffer750. The NN post filter controller810may also receive other information regarding the encoded video or specifically the current picture, such as the size of video, type of video frame, the prediction mode or coding tools used encode the current picture, etc. The NN post filter controller810uses the information provided by the video encoder700to calculate NN configuration data for configuring one or more NN post filters at the decoder side or the video streaming clients. The generated NN configuration data is provided to the SEI generator820to be delivered to the video streaming clients as SEI messages.

The SEI generator820generates SEI messages as non-VCL NAL units to be injected into the bitstream795. The SEI generator generates various SEI messages, including SPO SEI messages for setting the processing orders of SEI messages in SEI processing order groups (SPOGs). Each SPOG is identified by a group identifier. The generated SEI messages may use the group identifier of a SPOG to set parameters for the SPOG. In Sections II.b, II.c, II.d, the group identifier is set by spogc_id in SPOGC SEI messages or by spoga_target_id or member_id in SPOGA SEI messages. The SEI generator820may reference a storage for SPO group definitions840and SEI templates845when generating SEI messages related to SPOGs.

Each SPOG may be related to a corresponding video picture so the parameters assigned to a SPOG using the group identifier may be applicable to only that specific picture or a particular set of pictures. For example, SEI messages in a SPOG related to a particular picture may be active for only that particular picture, and the SEI messages of the SPOG may carry NN configuration data that is valid for only that particular picture.

FIG.9conceptually illustrates a process900for using grouping of SEI messages by a video streaming server. In some embodiments, one or more processing units (e.g., a processor) of a computing device implementing the video streaming server800performs the process900by executing instructions stored in a computer readable medium. In some embodiments, an electronic apparatus implementing the video streaming server800performs the process900.

The server transmits (at block910) coded video for a current picture in a first set of network data packets (which may be VCL NAL units). The server generates (at block920) a set of configuration data for the current picture. The server transmits (at block930) the set of configuration data in a second set of network data packets (which may be non-VCL NAL units or SEI messages).

The server transmits (at block940) a particular network data packet comprising a group identifier identifying a group that is applicable to the current picture. The identified group includes the second set of network data packets. The particular network data packet (e.g., SPO SEI message, SPOGA SEI message, SPOGC SEI message) may assign processing order to the network data packets in the second set of network data packets. The particular network data packet may be a SPOGC SEI message that associates persistence, purpose, grouping type, or other processing characteristics with the group identifier. The particular network data packet may be a SPOGA SEI message that activates or de-activates a processing function that uses the configuration data in the second set of network data packets. The activated/deactivated function may be a neural network post filter to be applied to the current picture and the generated configuration data is for configuring the neural network post filter. The group may be one of a plurality of groups that are associated with the current picture, the plurality of groups comprising multiple sets of network data packets for supporting respective multiple processing functions, which may be multiple neural network post filters to be applied to the current picture.

In some embodiments, the group identifier is a first group identifier for a first group comprising one or more network data packets and at least a second (nested) group associated with a second group identifier, with the second group including one or more network data packets.

IV. Example Video Decoder

In some embodiments, an encoder may signal (or generate) one or more syntax element in a bitstream, such that a decoder may parse said one or more syntax element from the bitstream.

FIG.10illustrates an example video decoder1000that may be part of a video streaming client. As illustrated, the video decoder1000is an image-decoding or video-decoding circuit that receives a bitstream1095and decodes the content of the bitstream into pixel data of video frames for display. The video decoder1000has several components or modules for decoding the bitstream1095, including some components selected from an inverse quantization module1011, an inverse transform module1010, an intra-prediction module1025, a motion compensation module1030, an in-loop filter1045, a decoded picture buffer1050, a MV buffer1065, a MV prediction module1075, and a parser1090. The motion compensation module1030is part of an inter-prediction module1040.

In some embodiments, the modules1010-1090are modules of software instructions being executed by one or more processing units (e.g., a processor) of a computing device. In some embodiments, the modules1010-1090are modules of hardware circuits implemented by one or more ICs of an electronic apparatus. Though the modules1010-1090are illustrated as being separate modules, some of the modules can be combined into a single module.

The parser1090(or entropy decoder) receives the bitstream1095and performs initial parsing according to the syntax defined by a video-coding or image-coding standard. The parsed syntax element includes various header elements, flags, as well as quantized data (or quantized coefficients)1012. The parser1090parses out the various syntax elements by using entropy-coding techniques such as context-adaptive binary arithmetic coding (CABAC) or Huffman encoding.

The inverse quantization module1011de-quantizes the quantized data (or quantized coefficients)1012to obtain transform coefficients, and the inverse transform module1010performs inverse transform on the transform coefficients1016to produce reconstructed residual signal1019. The reconstructed residual signal1019is added with predicted pixel data1013from the intra-prediction module1025or the motion compensation module1030to produce decoded pixel data1017. The decoded pixels data are filtered by the in-loop filter1045and stored in the decoded picture buffer1050. In some embodiments, the decoded picture buffer1050is a storage external to the video decoder1000. In some embodiments, the decoded picture buffer1050is a storage internal to the video decoder1000.

The intra-prediction module1025receives intra-prediction data from bitstream1095and according to which, produces the predicted pixel data1013from the decoded pixel data1017stored in the decoded picture buffer1050. In some embodiments, the decoded pixel data1017is also stored in a line buffer (not illustrated) for intra-picture prediction and spatial MV prediction.

In some embodiments, the content of the decoded picture buffer1050is used for display. A display device1005either retrieves the content of the decoded picture buffer1050for display directly or retrieves the content of the decoded picture buffer to a display buffer. In some embodiments, the display device receives pixel values from the decoded picture buffer1050through a pixel transport.

The motion compensation module1030produces predicted pixel data1013from the decoded pixel data1017stored in the decoded picture buffer1050according to motion compensation MVs (MC MVs). These motion compensation MVs are decoded by adding the residual motion data received from the bitstream1095with predicted MVs received from the MV prediction module1075.

The MV prediction module1075generates the predicted MVs based on reference MVs that were generated for decoding previous video frames, e.g., the motion compensation MVs that were used to perform motion compensation. The MV prediction module1075retrieves the reference MVs of previous video frames from the MV buffer1065. The video decoder1000stores the motion compensation MVs generated for decoding the current video frame in the MV buffer1065as reference MVs for producing predicted MVs.

The in-loop filter1045performs filtering or smoothing operations on the decoded pixel data1017to reduce the artifacts of coding, particularly at boundaries of pixel blocks. In some embodiments, the filtering or smoothing operations performed by the in-loop filter1045include deblock filter (DBF), sample adaptive offset (SAO), and/or adaptive loop filter (ALF). In some embodiments, luma mapping chroma scaling (LMCS) is performed before the loop filters.

FIG.11illustrates a video streaming client device1100that implements grouping of SEI messages. The video streaming server1100includes the video decoder1000, NN post filters1110, a SEI parser1120, and a NAL de-multiplexer1130. The NAL de-multiplexer1130receives the bitstream1095from the network1195(from a video streaming server), parses and de-multiplexes the bitstream1095into VCL NAL units for the video decoder1000(to the entropy decoder1090) and non-VCL NAL units for the SEI parser1120.

The NN post filters1110applies post filtering to decoded video content received from the video decoder1000, which may be the decoded pixel data1017or the content of the decoded picture buffer1050. The NN post filters1110are configured by NN configuration data provided by SEI parser1120, which parses the SEI messages from the video streaming server. The filtered video is then provided to a display device1160to be displayed, or outputted for another purpose.

The SEI parser1120receives SEI messages as non-VCL NAL units de-multiplexed from the bitstream1095. The SEI parser1120may receive various SEI messages, including SPO SEI messages for setting the processing orders of SEI messages in SEI processing order groups (SPOGs). Each SPOG is identified by a group identifier. The generated SEI messages may use the group identifier of a SPOG to set parameters for the SPOG. In Sections II.b, II.c, II.d, the group identifier is set by spogc_id in SPOGC SEI messages or by spoga_target_id or member_id in SPOGA SEI messages. The SEI parser1120may reference a storage for SPO group definitions840and SEI buffer845when receiving SEI messages to construct the NN configuration data for the NN post filters1110.

Each SPOG may be related to a corresponding video picture so the parameters assigned to a SPOG using the group identifier may be applicable to only that specific picture or a particular set of pictures. For example, SEI messages in a SPOG related to a particular picture may be active for only that particular picture, and the SEI messages of the SPOG may carry NN configuration data that is valid for only that particular picture.

FIG.12conceptually illustrates a process1200for using grouping of SEI messages by a video streaming client. In some embodiments, one or more processing units (e.g., a processor) of a computing device implementing the decoder1000performs the process1200by executing instructions stored in a computer readable medium. In some embodiments, an electronic apparatus implementing the decoder1000performs the process1200.

The client reconstructs (at block1210) a current picture based on coded video received in a first set of network data packets (which may be VCL NAL units). The client receives (at block1220) a second set of network data packets (which may be non-VCL NAL units or SEI messages.)

The client receives (at block1230) a particular network data packet comprising a group identifier identifying a group. The group comprises the second set of network data packets. The identified group includes the second set of network data packets. The particular network data packet (e.g., SPO SEI message, SPOGA SEI message, SPOGC SEI message) may assign processing order to the network data packets in the second set of network data packets. The particular network data packet may be a SPOGC SEI message that associates persistence, purpose, grouping type, or other characteristics with the group identifier. The particular network data packet may be a SPOGA SEI message that activates or de-activates a processing function that uses the configuration data in the second set of network data packets. The activated/deactivated function may be a neural network post filter to be applied to the current picture and the generated configuration data is for configuring the neural network post filter. The group may be one of a plurality of groups that are associated with the current picture, the plurality of groups comprising multiple sets of network data packets for supporting respective multiple processing functions, which may be multiple neural network post filters to be applied to the current picture.

In some embodiments, the group identifier is a first group identifier for a first group comprising one or more network data packets and at least a second (nested) group associated with a second group identifier, with the second group including one or more network data packets.

The client outputs (at block1240) (e.g., displays) the reconstructed current picture by using a set configuration data transmitted by network data packets in the group identified by the group identifier.

VII. Example Electronic System

FIG.13conceptually illustrates an electronic system1300with which some embodiments of the present disclosure are implemented. The electronic system1300may be a computer (e.g., a desktop computer, personal computer, tablet computer, etc.), phone, PDA, or any other sort of electronic device. Such an electronic system includes various types of computer readable media and interfaces for various other types of computer readable media. Electronic system1300includes a bus1305, processing unit(s)1310, a graphics-processing unit (GPU)1315, a system memory1320, a network1325, a read-only memory1330, a permanent storage device1335, input devices1340, and output devices1345.

The bus1305collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of the electronic system1300. For instance, the bus1305communicatively connects the processing unit(s)1310with the GPU1315, the read-only memory1330, the system memory1320, and the permanent storage device1335.

From these various memory units, the processing unit(s)1310retrieves instructions to execute and data to process in order to execute the processes of the present disclosure. The processing unit(s) may be a single processor or a multi-core processor in different embodiments. Some instructions are passed to and executed by the GPU1315. The GPU1315can offload various computations or complement the image processing provided by the processing unit(s)1310.

The read-only-memory (ROM)1330stores static data and instructions that are used by the processing unit(s)1310and other modules of the electronic system. The permanent storage device1335, on the other hand, is a read-and-write memory device. This device is a non-volatile memory unit that stores instructions and data even when the electronic system1300is off. Some embodiments of the present disclosure use a mass-storage device (such as a magnetic or optical disk and its corresponding disk drive) as the permanent storage device1335.

Other embodiments use a removable storage device (such as a floppy disk, flash memory device, etc., and its corresponding disk drive) as the permanent storage device. Like the permanent storage device1335, the system memory1320is a read-and-write memory device. However, unlike storage device1335, the system memory1320is a volatile read-and-write memory, such a random access memory. The system memory1320stores some of the instructions and data that the processor uses at runtime. In some embodiments, processes in accordance with the present disclosure are stored in the system memory1320, the permanent storage device1335, and/or the read-only memory1330. For example, the various memory units include instructions for processing multimedia clips in accordance with some embodiments. From these various memory units, the processing unit(s)1310retrieves instructions to execute and data to process in order to execute the processes of some embodiments.

The bus1305also connects to the input and output devices1340and1345. The input devices1340enable the user to communicate information and select commands to the electronic system. The input devices1340include alphanumeric keyboards and pointing devices (also called “cursor control devices”), cameras (e.g., webcams), microphones or similar devices for receiving voice commands, etc. The output devices1345display images generated by the electronic system or otherwise output data. The output devices1345include printers and display devices, such as cathode ray tubes (CRT) or liquid crystal displays (LCD), as well as speakers or similar audio output devices. Some embodiments include devices such as a touchscreen that function as both input and output devices.

Finally, as shown inFIG.13, bus1305also couples electronic system1300to a network1325through a network adapter (not shown). In this manner, the computer can be a part of a network of computers (such as a local area network (“LAN”), a wide area network (“WAN”), or an Intranet, or a network of networks, such as the Internet. Any or all components of electronic system1300may be used in conjunction with the present disclosure.

While the present disclosure has been described with reference to numerous specific details, one of ordinary skill in the art will recognize that the present disclosure can be embodied in other specific forms without departing from the spirit of the present disclosure. In addition, a number of the figures (includingFIG.9andFIG.12) conceptually illustrate processes. The specific operations of these processes may not be performed in the exact order shown and described. The specific operations may not be performed in one continuous series of operations, and different specific operations may be performed in different embodiments. Furthermore, the process could be implemented using several sub-processes, or as part of a larger macro process. Thus, one of ordinary skill in the art would understand that the present disclosure is not to be limited by the foregoing illustrative details, but rather is to be defined by the appended claims.

Additional Notes