QUALITY OF SERVICE ENHANCEMENTS FOR VIDEO AND EXTENDED REALITY APPLICATIONS OVER WIRELESS NETWORKS

Disclosed herein are a system and method for efficiently routing high-priority frames through a wireless network based on markings of the frames by a data link layer, a network layer, or an application layer in a five-layer protocol stack. When a real-time application, such as a video or gaming application, is to be sent to one or more stations in the wireless network, the access point determines the strategy for routing the high-priority frames based on the markings received from the data link, network, or application layer. Strategies include discarding non-I frames when there is congestion in the wireless network, replicating I-frames to assure reliable delivery to the stations, scheduling I-frames with preference, and moving I-frames ahead in a buffer in the access point.

TECHNICAL FIELD

Embodiments presented in this disclosure generally relate to the prioritized routing of information in a wireless network. More specifically, embodiments disclosed herein prioritized routing of video and extended reality data in a wireless network.

BACKGROUND

Real-time applications, such as video and extended reality applications, rely on the reliable transmission of certain critical portions of the application data. When the critical portions of the application data reach a wireless network, the wireless network may not have knowledge of the dependencies of other portions of the application data on the critical portions and thus may not assure that the critical portions are protected from loss or delay. For example, in a video application, the critical portions are the intra-coded frames (I-frames) or base frames in a group of pictures (GOP). It is important to ensure that the I-frames or base frames are not lost or delayed because other frames, such as P-frames and B-frames or enhancement frames, depend on them. Losing or delaying an I-frame or base frame leads to poor video quality or an application that does not work.

Similar issues arise in other applications, such as AR/VR systems in which polygons that draw the contours of an object are critical to the other polygons inscribed in the object.

In all these situations, the loss or delay of certain critical information has a greater impact than losing other non-critical information. Thus, the wireless network needs an indication of the critical information to prevent the delay or loss of critical information and deliver the expected performance of a real-time application.

DESCRIPTION OF EXAMPLE EMBODIMENTS

Overview

One embodiment presented in this disclosure is a method of improving video throughput in a wireless network. The method includes receiving an indication from a layer in a network protocol stack regarding video frames of a video application to be transmitted over the wireless network, converting the indication to a quality of service (QOS) value and a traffic id (TID) value at a data link layer, and processing the video frames in the wireless network according to the QoS and TID values.

Another embodiment presented in this disclosure is an access point for improving video transmission in a wireless network. The access point includes a processor and a memory that is coupled to the processor and has loaded therein a program which, when executed by the processor, is configured to receive an indication from a layer in a network protocol stack regarding video frames of a video application to be transmitted over the wireless network, convert the indication to a quality of service (QoS) value and a traffic id (TID) value at a data link layer; and process the video frames in the wireless network according to the QoS and TID values.

Yet another embodiment presented in this disclosure is a non-transitory computer-readable medium encoding instructions, which, when executed by a processor of an access point coupled to a wireless medium, cause the access point to: receive an indication from a layer in a network protocol stack regarding video frames of a video application to be transmitted over a wireless network, convert the indication to a quality of service (QOS) value and a traffic id (TID) value at a data link layer, and process the video frames in the wireless network according to the QoS and TID values.

EXAMPLE EMBODIMENTS

A system and method for informing the AP of a wireless network of high-priority traffic and processing the high-priority traffic in the wireless network are described herein. Information at the application or network layers is provided to the AP regarding the traffic priority. In one embodiment, the application layer (following the 5-layer model) marks traffic using the RTP. In another embodiment, the network layer marks the traffic using the differentiated services facility. Based on the markings, the AP determines a strategy for routing the traffic through the wireless network to one or more stations (STAs) in the network. Strategies include discarding non-I frames when there is congestion in the wireless network, replicating transmission of I-frames to assure reliable delivery, scheduling I-frames with preference, and moving I-frames ahead in a buffer in the AP.

FIG.1Adepicts a five-layer protocol stack. The lowest layer of the five-layer stack100is the physical layer110, which transfers data in the physical medium. The data transfers may occur according to orthogonal frequency division multiplexing (OFDM)110aor multiple-in, multiple-out (MIMO) orthogonal frequency division multiple access (OFDMA)110bor other PHY technologies110c, such as OFDMA and MU (multiuser)-MIMO. Above the physical layer110is the data link layer108, which includes a logical link layer108aand a medium access (MAC) sublayer108bwhen the physical medium is a broadcast medium, such as early versions of Ethernet, radio, or other electromagnetic frequencies such as terahertz, infra-red or light. Above the data link layer108is the network layer106, which implements the internet protocol, such as IP. Above the network layer106is the transport layer104, which implements either the user datagram protocol (UDP) or transport control protocol (TCP). The application layer102resides above the transport layer104and implements applications such as the World Wide Web (WWW). As information travels down the protocol stack, lower levels in the stack have access to information from the higher levels. An access point operating at the data link layer108thus has access to information provided by the network layer106and the application layer102.

FIG.1Bdepicts frames in a video stream according to one video coding layer standard. The stream includes a GOP where the GOP includes I-frames120and non-I frames such as P-(predictive) frames126and B-(Bidirectional) frames124,126. A GOP is typically bound by I-frames120and is usually15-60frames long. Frames are marked with sequence numbers so that lost frames can be detected.

I-frames120are self-contained compressed still pictures. The P-frames126and B-frames122and124code interframe differences, where the differences for a P-frame126include only previous frames, and the differences for B-frames122and124may include both previous and future frames. To form interframe differences, frames are decomposed into macroblocks, and the differences between macroblocks are determined and encoded. Thus, the P-frames126and B-frames critically depend on the I-frames. Unless these I-frames are reliably transferred in a wireless network, the B-frames and P-frames cannot be constructed, making the entire video stream supplied to a STA in the wireless network unreliable.

In another video coding layer standard, a GOP includes a base layer and enhancement layers. Inter-layer predictions are used for the enhancement layers relative to the base layer.

FIG.2Adepicts a representative architecture of an AP. The AP220includes a processing element222and several ports or connection facilities, such as a WAN port224, USB port226, RS-232port228, LAN port230, and Bluetooth232. Also included are a clocking system234and an 8×8 radio front-end236with a transmitter and receiver, which are coupled to eight external antennas. Auxiliary modules include a temperature sensing module240, a power module242connected to a DC power source246, a power over Ethernet (POE) module244, and LED driver258. Processing element222includes a CPU248and memory250, a peripheral control interconnect express (PCle) bus controller252for connecting to the 8×8 radio front-end236, and an I/O controller254, all coupled to each other via bus256. Memory250may include one or more buffers for traffic entering or exiting AP220.

FIG.2Bdepicts a network of APs and stations (STAs). In the figure, an external network262, such as the Internet, is coupled to an AP264, such as AP220. The AP264is coupled to stations STA 1268, STA 2270, STA 3272, and STA N274via a wireless medium266. According to the figure, server276connected to the external network262or one of the stations, STA 1, can be a source for video, which is available to the other STAs270,272, and274. For example, if STA 2270connects to an online TV service, server276delivers the video stream over the external network262to the AP264and then to STA 2270over wireless medium266. In another example, if STA 1268is a video source such as a DVD player, STAs270,272, and274can watch the video stream from STA1's source via the AP264and the wireless medium266.

FIG.3Adepicts a protocol stack that includes RTP and the RTP header. The RTP302improves the operation of the multimedia application in application layer102, such as a video or extended reality application via socket interface304. In the protocol stack, a link-level frame includes a link header306, an IP header308, an RTP header310, and an RTP payload312. The RTP header310includes sections314,316,318,320,322, and324.

Section314includes the version number, a P-field, an X-field, a CC-field, an M-field, a payload type, and a sequence number. The P-field indicates that the packet has been padded. The X-field indicates that an extension header is present. The CC field tells how many contributing sources are present. The M-field is an application-specific marker bit that can be used to mark the start of a video frame or the start of a word in the audio channel. The payload type field indicates which encoding algorithm is being used. The sequence number is a counter that is incremented on each RTP packet that is sent.

Section316includes a timestamp. The stream's source produces the timestamp to note when the first sample in the packet was made.

Section318includes a synchronization source identifier. The synchronization source identifier tells which stream the packet belongs to.

Section320includes a contributing source identifier. The contributing source identifier is used when mixers are present in the audio.

Section322includes a profile-specific extension header ID and an extension header length.

Section324includes a number of extension headers, such as extension header326. An extension header326includes the following fields: ID, L, S, E, I, D, P, and S. The P bits are priority bits, and if the D bit is set to 1, the value 00 for the P bits is the highest drop priority and the value 11 for the P bits is the lowest drop priority.

FIG.3Bdepicts a protocol stack for differentiated services and the internet protocol header for differentiated services. Differentiated services operates at the network layer106of five-layer protocol stack100. When the network layer106is IP, the IP header includes sections332,334,336,338,340, and342.

Section332includes a version number, IHL, a differentiated services (DS) field, and a total length. The IHL field indicates the size of the header. The DS field (includes the DSCP field), which in turn includes a pools field328and an unused field330. Pools field328specifies three types of pools: 1, 2, and 3. The total length indicates the size of the head and data. Values in pool 2 are independent of the DSCP values used for QoS classification. In some embodiments, the DSCP field is an index to a table in which elements select a particular packet-handling mechanism.

Pool 1 consists of32recommended codepoints to be assigned as standard. Pool 2 consists of 16 code points reserved for experimental or local use. Pool 3 includes 16 code points initially available for experimental or local use but is usable to extend pool 1 if pool 1 is exhausted.

Section334includes an identification field, DF, MF, and a fragment offset. The DF field tells routers not to fragment the packet. MF indicates that additional fragments are expected. The fragment offset indicates the position of the current fragment with respect to all of the fragments of a fragmented packet.

Section336includes a time to live, a protocol field, and a header checksum.

The time-to-live field is a counter used to limit packet lifetime. The protocol field indicates the transport process to which the packet belongs. The header checksum is used to protect the header from errors.

Section338includes a source address. Section340includes a destination address, and section342includes an options field. The source address is the IP address of the source network interface, and the destination address is the IP address of the destination network interface.

FIGS.3C-3Edepict link-level frames for identifying a quality of service associated with a traffic type and frames for negotiating TID-to-Link mappings.

FIG.3Cdepicts a MAC header that carries the QoS field. The MPDU includes a MAC header344and a data portion346. The MAC header344includes a quality of service (QOS) field, which includes a traffic identifier (TID) field348. The TID field348includes a user priority field that identifies the traffic type. The user priority field values range from 0 to 7, with each priority number mapped to an access category. Priorities 1 and 2 are assigned to background activity (i.e., background or non-time-sensitive traffic). Priorities 0 and 3 are assigned to best-effort traffic. Priorities 4 and 5 are assigned to video traffic, and priorities 6 and 7 are assigned to voice traffic. Thus, the user priority numbers are a priority metric that refers to a traffic category.

In addition, a TID-to-link mapping element can be used in a TID-to-link Mapping Request frame sent by a multi-link device (MLD) to negotiate a mapping with another MLD. This mapping element determines the assignment of traffic categories to active links of the AP MLD and non-AP MLD. By default, all TIDs are mapped to all of the links. An MLD receiving a TID-to-link mapping can respond with a response frame indicating whether it accepts or rejects the requested mapping. (Re) Association Request and Response frames can also be used for the negotiation.

FIG.3Ddepicts a request frame350that includes a TID-to-Link Mapping element352in its elements field. TID-to-Link Mapping element352includes a TID-to-Link Mapping control field and optional Link mappings354of TID numbers 0-7.

The TID-to-Link Mapping control field includes a direction field, a default link mapping, a link mapping size, and a link mapping presence indicator.

The direction field is set to 0 (Downlink) if the TID-to-Link Mapping element provides the link mapping information for the downlink frame. In one embodiment, it is set to 1 (Uplink) if the TID-to-Link Mapping element provides the link mapping information for the uplink frame. It is set to 2 (Bidirectional Link) if the TID-to-Link Mapping element provides the link mapping information for both the uplink and downlink frames. In one embodiment, the value of 3 is reserved.

The default Link Mapping field is set to 1 if the TID-to-Link Mapping element represents the default link mapping. Otherwise, it is set to 0.

The Link Mapping Presence Indicator field represents which Link Mapping of TID n field is present in the TID-to-Link Mapping element. A value of 1 in bit position n of the Link Mapping Presence Indicator field means that the Link Mapping of TID n field is present in the TID-to-Link Mapping element. Otherwise, the Link Mapping of TID n field is not present in the TID-to-Link Mapping element. When the Default Link Mapping field is set to 1, this field is reserved.

The Link Mapping354of TID n field (where n=0-7) indicates the links on which frames belonging to the TID n are sent. A value of 1 in bit position i of the Link Mapping of TID n field means that the Link associated with the link ID i is used for exchanging frames belonging to the TID n.

FIG.3Edepicts a response frame. The response frame356is similar to the request frame350but contains a status code358. Status code358is capable of indicating whether the TID-to-Link Element360, which includes Link mappings362for each TID, is a preferred mapping or not.

FIG.4depicts a flow of operations for processing video frames in a network, in an embodiment. In block402, the AP receives an indication (or marking) from the data link layer or above (such as the application layer or the network layer) that video frames of a video application (or extended reality application) are to be transmitted over the wireless network. Further details for block402are provided inFIG.5.

In block404, the video frames of the video application (or extended reality application) are converted from the indication (i.e., marking) in block402to a QoS value and a TID value at the data link layer. In one embodiment, the source stack converts RTP to DSCP, and the AP converts DSCP to a QoS and TID value. In another embodiment, the AP inspects the RTP's D and P bits and uses D and P or D and P bits and DSCP to select the TID value. In block406, the wireless network processes (how it should be queued and transmitted) the video frames according to the QoS and TID values. Further details for block406are provided inFIG.6.

FIG.5depicts a flow of operations for marking traffic, in an embodiment. There are four options for marking the traffic to indicate they are video frames of a video application.

In option 1, the application layer uses the RTP to mark or indicate the video traffic in block502. In particular, the application layer indicates that the video frames are of high priority by using the D bit and P bits in the extension header326. For example, option 1 applies when the source of the video stream is server276connected to the external network262.

In option 2, the network layer uses the DSCP field to indicate that the traffic is video traffic in block504. The DSCP field comprises a six-bit field that includes management pools 1, 2, and 3, where a management pool is a set of codepoints to be managed according to a specific policy according to RFC 2474, Section 6. In particular, pool 1 consists of 32 recommended code points; pool 2 consists of 16 codepoints that are reserved for experimental or local use and are independent of the DSCP values used for QoS classification; and pool 3 consists of 16 codepoints that are available for experimental or local use but which should be used to extend pool 1. In some embodiments, pool 2 is selected to affect the frames' processing at the link layer level. In some embodiments, the network layer sets the DSCP field according to the markings of the application layer. For example, option 2 applies when the source of the video stream is server276connected to the external network262.

In option 3, the video traffic is marked by a group policy or mobile device management (MDM) policy in block506, where a group policy or MDM policy is a mechanism that delivers a collection of settings to a set of users. For example, an application (e.g., WebEx) on each station in a group, such as the smartphones of a set of employees, sends packets to the smartphone's operating system network stack and group policy via a group policy object (GPO) indicates that the application has high priority. In one embodiment, the GPO includes DSCP values for packets, such as independent and discardable.

In option 4, the video traffic is marked at the link layer in block508. In one embodiment, the AP receives a management frame that describes how the AP can detect I-frames and non-I frames. In another embodiment, the AP receives an SCS transmission from a non-AP station classifying the traffic. As an example, option 4 applies when the source of the video stream is STA 1268via the wireless medium266.

FIG.6depicts a flow of operations for processing video frames in a wireless network, in an embodiment. There are four options for processing based on the QoS and TID values derived from the markings at the layers above the data link layer. In option 1, the AP determines whether there is congestion in the network in block602. If so, the AP causes non I-frames to be discarded in block604. In option 2, the AP replicates the transmission of the I-frames for reliable delivery in block606. In option3, the AP schedules I-frame traffic with a routing preference in block608, which is further described in reference toFIG.7. In option4, the AP causes the I-frame to move ahead in the buffer in the AP in block610.

FIG.7depicts a flow of operations for scheduling a video frame with a routing preference, in an embodiment. In block702, the AP matches one of three conditions. A flag, based on the RTP header or the DSCP field, is added to the TID value in one condition. The flag indicates that the frame has higher priority. For example, an I-frame and a B-frame in a buffer of the AP are labeled as ‘I-frame’ in block704, and non-I frames or redundant B-frames are labeled ‘Discardable frames’ in block706. Thus, the AP scheduler attempts to provide slots in priority (for a given TID) to STAs displaying the presence of high-importance frames.

In another condition where an I-frame is found in an up-link transmission, the AP reschedules the STA in block708for further uplink transmissions with high priority until non-I frames are received from the STA, whereupon the AP schedules the STA at lower or normal priority until a new I frame is received from the STA. In another condition where an I-frame is not found in the up-link transmission, the AP demotes the frame in block710by changing the scheduling priority of the STA in the AP's scheduler.

FIG.8Adepicts request and response frames in an SCS. In SCS, a transmission (i.e., an SCS request or response) includes an augmented QoS characteristics element that describes the flow structure (i.e., I-frames and discardable frames). In SCS, a non-AP STA sends a request to specify traffic classes for classification and the priority to assign to matching frames. The AP may accept or reject the traffic class specified by the non-AP STA. If the AP accepts the service, the AP processes the MAC level frames matching the classification in the SCS descriptor.

InFIG.8A, the SCS request frame802includes a category field, an action field, a dialog token, a request type, and an SCS descriptor list804. The SCS descriptor list804includes an SCS ID field, a length field, an optional intra-access category priority element806, zero or more traffic classification (TCLAS) elements, optional TCLAS processing elements, and zero or more QoS Characteristics Elements (QCe)807. The intra-access category priority element806includes a user priority field, an alternate priority field, a drop eligibility field, and a reserved field. The QCe fields807allow the station to describe the traffic characteristics and QoS expectations of traffic flows that belong to the SCS stream. Using QCe fields to identify requirements of I-frames and non-I frames separately, such as relative drop eligibility, the AP can look at other packet parameters to identify the I and non-I flows and perform the dropping of frames accordingly.

The SCS response frame808, sent by the AP back to the requesting STA, includes a category field, an action field, a dialog token, and an SCS status list. The SCS status list includes the SCSID and SCS status.

FIG.8Bdepicts a flow of operations for processing a stream using SCS, in an embodiment. In block852, the AP receives an SCS transmission from an STA classifying an uplink or downlink video stream. If the case is a downlink video stream, as determined in block854, then the presence of an I-frame in the stream is determined in block856. If an I-frame is present in the stream, the I-frame is sent according to the priority identified in the SCS transmission of block860and according to the congestion in the wireless network. If a non-I frame is detected, the AP may drop the non-I frame from its buffer according to the priority identified in the SCS transmission of block858and congestion in the network.

If the case is an uplink video stream (from the STA to the AP) as determined in block854, then the presence of an I-frame in the stream is determined in block862. If an I-frame is present in the stream, the AP schedules the buffer for the expected I-frames from the STA in block866. If a non-I frame is present in the stream, the AP schedules the buffer to drop the frame if there is congestion but otherwise allows them in block864.

FIG.9depicts the markings of the I-frames, B-frames, and P-frames by the RTP or DSCP, in an embodiment. For the I-frame902, the RTP extension sets the D-flag, and DSCP sets the experimental bit. For the B-Frame904and906, the RTP extension sets the D-flag, and DSCP does not set the experimental bit. For the P-frame908, the RTP extension sets the D-flag, and DSCP does not set the experimental bit.