Systems and methods for tracking and calculating granular network performance metrics based on user traffic

A system described herein may provide for the tracking and/or calculating of performance metrics associated with a network by marking traffic and determining performance characteristics of the marked traffic. Such performance characteristics or metrics may include throughput, latency, jitter, and/or other metrics. The marking may be performed on “user” traffic, which may be traffic that is generated or sent via the network by an application or service (e.g., a voice call service, a content streaming service, etc.), as opposed to “synthetic” or “test” traffic, which is traffic that is generated or sent for the purposes of testing performance of the network (e.g., traffic related to a “speed test” or the like).

BACKGROUND

Networks may make use of Service Level Agreements (“SLAs”) to provide a given level of performance (e.g., throughput, latency, jitter, and/or other metrics) for network traffic. Various techniques may be used to determine whether particular SLAs are met, such as sending artificial “test” packets through the network.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Embodiments described herein provide for the tracking and/or calculating of performance metrics associated with a network by marking traffic and determining performance characteristics of the marked traffic. Such performance characteristics or metrics may include throughput, latency, jitter, and/or other suitable metrics. The marking may be performed on “user” traffic, which may be traffic that is generated or sent via the network by an application or service (e.g., a voice call service, a content streaming service, etc.), as opposed to “synthetic” or “test” traffic, which is traffic that is generated or sent for the purposes of testing performance of the network (e.g., traffic related to a “speed test” or the like).

For example, some networks or network devices may provide different levels of performance to synthetic traffic than to user traffic, in order to artificially boost performance metrics associated with such networks or devices. For example, such networks or network devices may examine header information or other attributes of synthetic traffic to identify that the traffic is synthetic test traffic, and may accordingly modify queue weights or other treatment of the traffic in order to cause the traffic to exhibit higher performance values. On the other hand, no such preferential treatment may be given to user traffic, which may result in user traffic exhibiting different performance values than synthetic test traffic. Thus, while synthetic test traffic may meet one or more SLAs (e.g., minimum latency or throughput values), user traffic may not meet such SLAs. As such, evaluating actual user traffic may provide a more accurate view of the actual performance of a given network, portion of the network, network devices, etc. Further, evaluating actual user traffic may conserve network resources that would be otherwise consumed by sending synthetic test packets. While discussed here in the context of user traffic, similar concepts may apply to other types of traffic that is sent within a network, such as control plane traffic, Precision Time Protocol (“PTP”) traffic, and/or other traffic.

As shown inFIG. 1, a set of network components101-1and101-2(sometimes referred to individually as “network component101” or in plural as “network components101”) may be deployed within network100. For example, network components may be, may include, or may be implemented by computing devices, Virtualized Network Functions (“VNFs”), MECs, routers, hubs, switches, and/or other types of network devices. Network components101may be associated with network interfaces (e.g., hardware interfaces) via which network components101may be communicatively coupled to each other and/or to one or more other devices and/or networks. Accordingly, network components101may communicate traffic to and/or from each other, such as user traffic received from and/or destined for a User Equipment (“UE”), such as a mobile telephone, laptop computer, tablet computer, or the like.

Network devices101may be associated with one or more SLAs for certain types of traffic (e.g., traffic associated with a particular application or class of applications, such as voice call applications or content streaming applications), certain Quality of Service (“QoS”) indicators (e.g., QoS Class Identifier (“QCI”) values), certain users (e.g., where different users may be associated with different levels of service, such as enterprise users, emergency or first responder users, or the like), certain network slices (e.g., where a network may be implemented by multiple instances or “slices”), different geographic locations (e.g., as indicated as Tracking Area Identity (“TAI”)), and/or other differentiating attributes of traffic. In one example, different entities may manage, own, operate, etc. network component101-1, network component101-2, and/or a hardware interface (e.g., fiber cable or other type of interface) via which network component101-1and network component101-2may communicate. In such situations, network component101-1and/or network component101-2may each be associated with one or more SLAs.

Embodiments described herein may calculate and determine the performance of traffic handled by network components101by marking user traffic and identifying the performance of marked traffic. For example, in some embodiments, network component101-1may be associated with Network Performance Component (“NPC”) Marker (“NPC-M”)103, and network component101-2may be associated with NPC Determiner (“NPC-D”)105.

For example, NPC-M103may be implemented by the same device or system as network component101-1, and NPC-D105may be implemented by the same device or system as network component101-2. In some embodiments, for instance, network component101-1and NPC-M103may share the same set of network interfaces, and/or NPC-M103may receive some or all user traffic egressed from network component101-1. In some embodiments, NPC-M103may include a packet analyzer or packet capture functionality, and/or traffic outputted by network component101-1may otherwise traverse NPC-M103before being sent to network component101-2. In some embodiments, NPC-M103may be, and/or may be implemented by, an application installed at network component101-1. In some embodiments, NPC-M103may be pre-loaded or pre-provisioned when network component101-1is instantiated (e.g., in situations where network component101-1includes or is implemented by a virtual machine or other set of configurable resources).

Similarly, network component101-2and NPC-D105may share the same set of network interfaces, and/or NPC-D105may receive some or all ingress user traffic destined for, or handled by, network component101-2. In some embodiments, NPC-D105may include a packet analyzer or packet capture functionality, and/or traffic outputted to network component101-2may otherwise traverse NPC-D105before being provided to network component101-2(e.g., to other components of network component101-2). In some embodiments, NPC-D105may be, and/or may be implemented by, an application installed at network component101-2. In some embodiments, NPC-D105may be pre-loaded or pre-provisioned when network component101-2is instantiated (e.g., in situations where network component101-2includes or is implemented by a virtual machine or other set of configurable resources).

In accordance with some embodiments, as shown inFIG. 1, NPC-M103may receive (at102) SLA information from SLA repository107. SLA repository107may maintain information regarding SLAs (e.g., thresholds of performance metrics) that are to be provided by network component101-1. For instance, SLA repository107may be, may include, and/or may be implemented by a Unified Data Management function (“UDM”) of a wireless network, a Home Subscriber Server (“HSS”), a Policy Charging and Rules Function (“PCRF”), a Policy Control Function (“PCF”), and/or some other suitable device or system. As noted above, the SLAs may include different performance metrics for different traffic applications, users, network slices, etc.

As further shown, network component101-1may receive (at104) traffic to be sent to (or toward) network component101-2. NPC-M103may identify (at106) a proportion (e.g., 1%, 10%, 75%, or some other proportion) of traffic associated with varying SLAs, such that this proportion can be marked in accordance with some embodiments. For example, if the SLA information (received at102) indicates three different SLAs (e.g., first, second, and third maximum latency values) for three different application types, NPC-M103may mark the particular proportion (e.g., 1%) of traffic for the first application type, 1% of traffic for the second application type, and 1% of traffic for the third application type. In some embodiments, NPC-M103may mark different proportions for different SLAs, traffic types, etc. For example, as discussed below, network component101or some other device or system may determine that performance metrics associated with a particular SLA do not meet thresholds associated with the particular SLA, and that traffic associated with the particular SLA should accordingly be marked at a greater proportion than traffic associated with other SLAs.

In some embodiments, marking (at106) the traffic may include generating a unique value (and/or selecting a value from a pool). For example, NPC-M103may generate a random or pseudorandom value, and/or may otherwise generate or determine a unique value (referred to herein as a “mark value”). NPC-M103may mark the identified traffic by placing the mark value in the traffic, such as by placing the mark value in header information associated with packets of which the identified traffic is comprised.

In some embodiments, the mark value may include, and/or may otherwise be based on, a timestamp or other indicator of a time at which the traffic was received by NPC-M103(e.g., output from network component101-1). For example, in some embodiments, the mark value may be, or may include, a value generated by performing a cryptographic hash of a timestamp that corresponds to a time at which NPC-M103received the traffic. As discussed below, this encrypted value may be decrypted (e.g., by NPC-D105) in order to determine the timestamp. For example, NPC-M103and NPC-D105may utilize a shared key cryptography technique, an asymmetric key pair cryptography technique, and/or some other suitable technique to securely communicate the timestamp via the mark values. In this manner, even if network component101-1and/or some other device or system were to identify the mark values within the traffic (e.g., by inspecting headers, performing deep packet inspection (“DPI”), etc.), such mark values would be unlikely or impossible to detect as being associated with the tracking or calculating of performance metrics.

In some embodiments, NPC-M103may mark the traffic by placing the mark value in a “shim” header, a Multi-Protocol Label Switching (“MPLS”) label, an Internet Protocol (“IP”) header (e.g., an IPv4 header or an IPv6 header), an “options” field of a header (e.g., an “options” field of an IP header), placing the mark value within traffic payloads, and/or otherwise including the mark value in the traffic. In some embodiments, NPC-M103may maintain correlation information, correlating particular mark values with particular timestamps. For example, in embodiments where the mark values do not include timestamps (or values derived from timestamps), such correlation information may be used to indicate times at which traffic, with particular mark values that correspond to particular timestamps, was sent by network component101-1.

In some embodiments, as discussed below, NPC-M103may provide the timestamp (or value derived from the timestamp, such as a hashed and/or encrypted value, as discussed above) directly to NPC-D105by including the timestamp in the mark, appending the timestamp to the mark, etc. (e.g., in header information of marked traffic). In some embodiments, NPC-M103may provide the timestamp to another device or system, along with correlation information. For example, NPC-M103may provide the timestamp and the mark value to the other device or system. As discussed below, this other device or system may provide the correlation information to NPC-D105, and/or may receive timestamp information from NPC-D105, based on which NPC-D105and/or the other device or system may determine a difference between the timestamps applied by NPC-M103and NPC-D105. This difference may indicate an amount of time that traffic took to reach network component101-2once output by network component101-1, which may be referred to as “one-way delay,” “one-way latency,” or the like.

As further shown inFIG. 1, NPC-M103may provide (at108) mark values associated with the marked traffic to NPC-D105. For example, NPC-M103and NPC-D105may communicate via an application programming interface (“API”) or some other suitable communication pathway. Additionally, or alternatively, NPC-M103may provide (at108) the mark values to another device or system which may, in turn, provide the mark values to NPC-D105. As discussed above, NPC-M103may, in some embodiments, provide (at108) timestamp information associated with marked traffic to NPC-D105and/or to another device or system.

Network component101-1may further output (at110) traffic to network component101-2. As noted above, the traffic may include user traffic associated with one or more services or applications provided by network component101-1and/or network component101-2, as opposed to synthetic test traffic. NPC-D105may identify which of the traffic (received at110) is marked traffic. For example, NPC-D105may identify header information of the received traffic to identify which of the received traffic includes mark values (provided at108). NPC-D105may further identify a time at which the marked traffic was received. As discussed above, NPC-D105may determine a difference, for some or all of the marked traffic, between the time that the traffic was output from network component101-1and the time that the traffic was received by network component101-2(e.g., the one-way delay of the traffic).

In some embodiments, as discussed below, one or more other devices or systems may determine the one-way delay of the traffic, based on timestamps generated and/or provided by NPC-M103and/or NPC-D105. As further discussed below, by aggregating one-way delay performance metrics for multiple packets that are associated with multiple SLAs, other performance metrics (e.g., round-trip delay, jitter, or other metrics) may be able to be determined on a per-SLA basis.

For example, in order to determine round-trip delay, some embodiments may include the deployment of multiple instances of NPC-M103and NPC-D105. For example, as shown inFIG. 2, a first instance of NPC-M103(i.e., NPC-M103-1) and a first instance of NPC-D105(e.g., NPC-D105-1) may be associated with (e.g., installed, provisioned, and/or otherwise associated with) a first network component (e.g., MEC201), and NPC-M103-2and NPC-D105-2may be associated with a second network component (e.g., Next Generation Node B (“gNB”)203).

As discussed below, gNB203may send and/or receive traffic via an air interface to one or more UEs, such as a wireless telephone. Further, gNB203may be communicatively coupled to MEC201, and may send and/or receive traffic, associated with one or more UEs connected to gNB203, to and/or from MEC201. MEC201may, for example, perform processing, calculations, computations, etc. on traffic received from one or more UEs (e.g., via gNB203), and may provide processed traffic to such UEs (e.g., via gNB203). Additionally, or alternatively, MEC201may generate traffic and/or receive traffic from some other source, and may provide such traffic to UEs via gNB203.

In some embodiments, MEC201may be associated with one or more SLAs, such as threshold one-way delay times (e.g., for traffic sent from MEC201to gNB203). While not shown here, NPC-M103-1and/or NPC-M103-2may receive SLA information (e.g., from SLA repository107and/or some other device or system), based on which NPC-M103-1and/or NPC-M103-2may identify packets to mark in a manner similar to that described above (e.g., a proportion of traffic associated with one or more particular SLAs).

As shown, NPC-M103-1may provide (at202) mark values to NPC-D105-2, and NPC-M103-2may provide (at204) mark values to NPC-D105-1. In some embodiments, NPC-M103-1and/or NPC-M103-2may provide (at202and204) the mark values in addition to timestamps and/or values derived from timestamps. Additionally, or alternatively, as discussed above, NPC-M103-1and/or NPC-M103-2may provide the mark values, timestamps, etc. to one or more other devices or systems. MEC201may thus output (at206) traffic marked by NPC-M103-1to gNB203, and gNB203may output (at208) traffic marked by NPC-M103-2to MEC201. As similarly discussed above, NPC-D105-1may determine performance metrics (e.g., one-way delay) of marked traffic received from gNB203, and NPC-D105-2may determine performance metrics of marked traffic received from MEC201.

Additionally, or alternatively, as shown inFIG. 3, NPC-D105-1and/or NPC-D105-2may communicate with NPC Aggregator (“NPC-A”)301in order to facilitate the determination of aggregated performance metrics, such as jitter, round-trip delay, and/or other metrics. For example, as shown, MEC201and gNB203may send and/or receive (at302) traffic, including marked traffic. The communication pathway between MEC201and gNB203may be referred to as a “haul” of traffic (e.g., a transport of traffic within the network), such as a “mid-haul” or “x-haul” of the traffic. The traffic (e.g., the marked traffic) may be traffic associated with one or more SLAs, as discussed above. Further, gNB203may wirelessly send and/or receive (at304) traffic to and/or from one or more UEs. This traffic may also be associated with one or more SLAs, which may be the same SLAs associated with the traffic sent (at302) between MEC201and gNB203, may be different SLAs that correspond to the SLAs associated with the traffic sent between MEC201and gNB203, and/or may be associated with different SLAs that are independent of SLAs associated with the traffic sent between MEC201and gNB203.

As further shown, NPC-D105-1may provide (at306) uplink performance metrics to NPC-A301. For example, NPC-D105-1may determine one-way delay of packets sent from gNB203(e.g., as marked by NPC-M103-2) to MEC201in a manner similar to that described above. Additionally, or alternatively, NPC-D105-1may provide timestamps and mark values of marked traffic to NPC-A301, based on which NPC-A301may determine the uplink performance metrics (e.g., further based on timestamps and mark values provided by NPC-M103-2). Similarly, NPC-D105-2may provide (at308) downlink performance metrics to NPC-A301, such as one-way delay of packets sent from MEC201(e.g., as marked by NPC-M103-1) to gNB203.

In some embodiments, gNB203may further provide (at310) RAN performance metrics, which may be based on performance of traffic sent to and/or received from one or more UEs by gNB203. Such RAN performance metrics may include one-way delay, round-trip delay, jitter, and/or other suitable performance metrics. In some embodiments, gNB203may determine or receive such metrics on a per-SLA basis, a per-UE basis, a per-application basis, and/or on some other basis.

NPC-A301may further receive (at312) SLA information from SLA repository107. The SLA information may indicate one or more SLAs associated with MEC201and/or gNB203(e.g., threshold values for uplink one-way delay, downlink one-way delay, round-trip delay, jitter, packet loss, and/or other performance metrics). In some embodiments, the SLA information (received at312) may include mapping or correlation information that correlates SLAs of traffic sent (at302) between MEC201and gNB203to SLAs of traffic sent (at304) between gNB203and one or more UEs. For example, gNB203may output traffic to a given UE according to an SLA that includes or relates to a particular QCI value, while MEC201may output traffic to gNB203(e.g., traffic destined for the given UE) according to an SLA that uses some other type of QoS indicator or type of SLA. For example, the SLA for traffic sent to gNB203by MEC201(e.g., for the particular UE) may be expressed in terms of a threshold one-way delay, may specify a threshold proportion of traffic that meets the threshold (e.g., at least 90% of traffic from MEC201to gNB203meets the threshold one-way delay), and/or may be specified in some other manner.

Based on the information provided (at306-312) by NPC-D105-1, NPC-D105-2, gNB203, and/or SLA repository107, NPC-A301may determine (at314) aggregated performance metrics on a per-SLA or some other granular basis.FIGS. 4A-4Cillustrate example data structures that may reflect the determination of aggregated performance metrics by NPC-A301. As shown inFIG. 4Afor example, data structure402may include information that may be used (e.g., by NPC-A301) to determine one-way delay performance metrics, on a per-SLA basis, per-UE basis, and/or some other basis. For example, data structure402may include mark values associated with traffic marked by NPC-M103, SLAs associated with marked traffic, an identifier of a particular NPC-M103that marked the traffic, an identifier of a particular NPC-D105that received the traffic, a time at which the traffic was marked and/or output (“Time out”), and a time at which the traffic was received (“Time in”).

In some embodiments, each row of data structure402may be associated with a particular marked packet. In some embodiments, different arrangements are possibly for data structure402. As shown, for example, a particular packet may have a mark value of “a123456.” As noted above, the mark value may be generated by a particular NPC-D105that performs the marking of the packet. In some embodiments, as also discussed above, the mark value may encode a timestamp (e.g., a time at which the packet was marked) and/or some other value. In this example, the mark value may begin with the letter “a,” which may be an indicator of a particular SLA associated with the packet. For example, the particular SLA associated with this packet may be an SLA that is associated with a 10 millisecond (“ms”) one-way delay of traffic from NPC-M103-1to NPC-D105-2(where this SLA is denoted in the figure as “Under 10 ms”). On the other hand, as also illustrated in the figure, the prefix “b” (e.g., as denoted by the mark value “b789012”) may indicate an SLA of “Under 30 ms.”

Additionally, or alternatively, NPC-A301may determine the SLA associated with a packet based on an identifier of a particular NPC-M103or network component101from which the packet is received, and/or an identifier of a particular NPC-D105or network component101to which the packet is destined. In some embodiments, such identifiers may include IP addresses or other identifiers that may be present in the packet (e.g., in header information of the packet). For example, SLA information (received from SLA repository107) may indicate SLAs associated with particular UEs and/or network components101.

In some embodiments, the SLA represented in data structure402may be, or may correspond to, some other type of SLA. For example, the SLA represented in data structure402may, in some embodiments, be associated with a particular QCI, a particular network slice, a particular TAI, and/or some other differentiating characteristic associated with SLAs.

The timestamps (e.g., “Time out” and “Time in”) may be expressed in terms of milliseconds after a reference time (sometimes referred to as an “epoch”). In this figure, the last five digits of such timestamps are presented, and the preceding “xx” on each of these timestamps refers to one or more preceding digits of the timestamps. In some embodiments, other representations of time are possible. The timestamps may be received from respective instances of NPC-M103and/or NPC-D105, which may send and/or receive packets having the respective mark values. As noted above, in some embodiments, the timestamps may be encoded into the mark values, may be included in header information, and/or may be provided by NPC-M103and/or NPC-D105to NPC-A301via an API or other communication pathway.

As shown inFIG. 4B, data structure404may indicate a correlation between particular packets and particular UEs associated with the packets. For example, data structure404may include mark values of packets, and identifiers of UEs associated with such packets (e.g., UEs to which the packets were sent and/or from which the packets were received). While shown in this figure in the format “UE_A” and “UE_B,” the UE identifiers may include any suitable identifiers such as International Mobile Station Equipment Identity (“IMEI”) values, International Mobile Subscriber Identity (“MR”) values, and/or other suitable identifiers. NPC-A301may identify the UE identifiers based on header information of the packets or other suitable information.

In some embodiments, NPC-A301may receive SLA information (e.g., from SLA repository107) that indicates particular SLAs associated with particular UEs. In some embodiments, such information may be used in generating data structure402(e.g., identifying a particular SLA associated with a particular packet), and/or to generate other information as discussed below.

As shown inFIG. 4C, data structure406may indicate per-SLA performance metrics associated with traffic sent between network components101. As shown, for instance, traffic associated with the “Under 10 ms” SLA may exhibit a 9 ms uplink (“UL”) delay, which may be based on one-way delay metrics over time of traffic sent from a UE to a first network component101via a second network component101. Referring to the example ofFIG. 3, the UL delay may be based on evaluating one-way delay of traffic sent from gNB203to MEC201over a particular time period. The value reflected in data structure406(i.e., 9 ms in this example) may be based on an average one-way delay over the particular time period, a median one-way delay over the particular time period, a minimum one-way delay over the particular time period, a maximum one-way delay over the particular time period, or some other value that reflects or is derived from one-way delay values over the particular time period. Similarly, the “DL delay” value may be based on one-way delay values for traffic from the second network component101toward the UE via the first network component101(e.g., from MEC201to gNB203, referring to the example shown inFIG. 3).

The round-trip (“RT”) delay values for traffic associated with a particular SLA may, in some embodiments, be based on the UL delay values and the DL delay values for such traffic. For example, in some embodiments, the RT delay values may be based on adding the UL delay values to the DL delay values, and/or may be based on other types of operations.

The jitter values for traffic associated with a particular SLA may, in some embodiments, be based on differences between individual one-way delay times of such traffic. In some embodiments, jitter may be separately reflected in terms of UL jitter and DL jitter, or may be reflected as a combined value that is based on UL jitter and DL jitter.

The packet loss values for traffic associated with a particular SLA may, in some embodiments, be based on a proportion of marked packets that were not received by their intended destination. For example, a packet may be considered as “lost” if NPC-M103marks a packet and a corresponding NPC-D105(e.g., NPC-D105associated with network component101to which the packet is destined) does not indicate that the packet is received within a threshold amount of time (e.g., one second, ten seconds, or some other threshold).

In some embodiments, one instance of data structure406may be maintained for each pair of network components101. Thus, referring to the example ofFIG. 3, NPC-A301may maintain one instance of data structure406for traffic sent between MEC201and gNB203. Further, while data structure406is described in the context of maintaining performance information for traffic on a per-SLA basis, such information may be maintained or presented on another basis, such as on a per-UE basis or some other basis.

As discussed above, performance metrics of traffic sent between network components101may be utilized in combination with RAN performance metrics, to determine overall performance metrics between a particular network component and a UE connected to a RAN. The examples illustrated inFIGS. 5A-5Ccontinues the example information reflected inFIGS. 4A-4C.

For example, as shown inFIG. 5A, data structure502may reflect RAN performance information associated with a RAN via which one or more UEs are connected (UE_A and UE_B in this example). The RAN performance information may be granularly presented based on differing QCI values associated with the RAN. While QCI values are discussed in the context of this example, in practice, other differentiating parameters for QoS, SLA, TAI, etc. may be used.

In this example, data structure502may reflect that the UL delay for traffic according to QCI 1 for UE_A is 95 ms. For example, the average, median, etc. one-way delay for traffic sent to the RAN from the UE may be 95 ms. Further, the DL delay for traffic according to QCI 1 for UE_A is indicated in data structure502as 92 ms, indicating that the average, median, etc. one-way delay for traffic sent to the UE from the RAN is 92 ms. The RT delay for traffic associated with this UE and QCI may be indicated as 182 ms. For example, the RT delay value may be calculated in a manner different from adding the UL delay value to the DL delay value (e.g., the RAN may perform different testing methodologies for testing RT delay than UL and/or DL delay). As further shown, data structure502may include jitter and packet loss values on a per-QCI and per-UE basis. As noted above, jitter and/or packet loss may, in some embodiments, be separately reflected as UL and DL values. Data structure502may, in some embodiments, be received by NPC-A301from one or more elements of a RAN, such as from gNB203(as shown in the example ofFIG. 3).

As shown inFIG. 5B, data structure504may include mappings of different parameters to SLAs. For example, data structure504may relate RAN QCI values to SLAs associated with network components101or pairs of network components101(e.g., SLAs associated with MEC201and/or gNB203). In this example, data structure504may indicate that QCI 1 is associated with the “Under 30 ms” SLA, and that QCI 3 is associated with the “Under 10 ms” SLA. In some embodiments, data structure504may include different mappings, or a different granularity of mappings. For example, in some embodiments, the same QCI may be associated with different SLAs, depending on which UE the traffic is associated. As another example, different network slices or TAIs may be associated with different SLAs.

As shown inFIG. 5C, NPC-A301may generate or maintain aggregated performance information based on RAN metrics (e.g., as reflected in data structure502) and determined performance metrics between network components101(e.g., as reflected in data structure406). For example, NPC-A301may use the mapping information provided in data structure504to determine which RAN metrics (e.g., as indicated in data structure502) correspond to which network component101metrics (e.g., as indicated in data structure406). As an example, NPC-A301may aggregate, combine, etc. the UL delay values reflected in data structure406to the UL delay values reflected in data structure502to determine an overall UL delay values associated with a particular SLA/QCI combination for a particular UE. Continuing the example ofFIG. 3, the UL delay value for traffic associated with the QCI 3/Under 10 ms QCI/SLA pair for UE_A may be 51 ms, which may reflect an average, median, etc. one-way delay of traffic sent from UE_A via a RAN associated with gNB203to MEC201. Similarly, the DL delay, RT delay, jitter, and packet loss values reflected in data structure506may be based on performance metrics associated with this traffic path (e.g., between MEC201and a UE connected to a RAN implemented by gNB203).

FIG. 6illustrates an example process600for marking traffic associated with particular SLAs and identifying performance metrics based on marked traffic. In some embodiments, some or all of process600may be performed by a NPC (e.g., NPC-M103and NPC-D105). In some embodiments, one or more other devices may perform some or all of process600in concert with, and/or in lieu of, NPC-M103and/or NPC-D105.

As shown, process600may include receiving (at602) SLA information. For example, NPC-M103may receive (e.g., from SLA repository107or some other source) information regarding one or more SLAs. As noted above, NPC-M103may be associated with a particular network component101, which may be associated with a set of SLAs. Additionally, or alternatively, NPC-M103may receive SLA information for one or more users, UEs, network slices, TAIs, and/or other granular SLA information. The SLA information may indicate threshold performance values, such as latency, jitter, packet loss, and/or other values for other performance metrics.

Process600may further include marking (at604) a proportion of traffic associated with the indicated SLAs. For example, NPC-M103may identify egress traffic (e.g., traffic that is output by, is queued to be output by, and/or is otherwise destined to be transmitted to another network component101) associated with network component101. As discussed above, NPC-M103may mark a portion of traffic associated with one or more different SLAs. In some embodiments, NPC-M103may mark all traffic associated with one or more SLAs. In some embodiments, NPC-M103may mark different portions of traffic associated with different SLAs. In some embodiments, NPC-M103may receive an instruction (e.g., from NPC-A301or some other source) to modify a portion of traffic based on one or more trigger events detected by NPC-A301. For example, NPC-A301and/or NPC-M103may determine that a particular SLA is not met, and NPC-M103may accordingly increase an amount of traffic associated with the particular SLA that is marked. In this manner, traffic associated with this particular SLA may be able to be evaluated more closely to verify whether the SLA is being met or not.

As discussed above, marking (at604) the traffic may include adding mark values to header information (e.g., shim headers, MPLS labels, etc.). In some embodiments, the mark values may include an index or other identifier associated with the particular SLA with which the traffic is associated.

Process600may additionally include outputting (at606) mark values and timestamps of output traffic. For example, NPC-M103may output the mark values to NPC-A301, NPC-D105(e.g., where NPC-D105is associated with a particular network component101to which the traffic is destined), and/or to some other device or system. Further NPC-M103may output timestamps corresponding to a time at which the traffic was output by NPC-M103(and/or by a particular network component101with which NPC-M103is associated). As discussed above, the timestamps may be encoded or otherwise included in the mark values.

Process600may also include receiving (at608) the mark values. For example, NPC-D105may receive the mark values from NPC-M103, NPC-A301, and/or some other device or system.

Process600may further include identifying (at610) traffic based on the mark values. For example, NPC-D105(e.g., a particular network component101with which NPC-D105is associated) may receive traffic with the mark values (received at608). NPC-D105may identify traffic having the mark values, and may determine a time at which the traffic is received.

Process600may additionally include outputting (at612) the mark values and timestamps of received traffic. For example, NPC-D105may output (e.g., to NPC-A301) information indicating which packets were received at which time. Based on this information, as discussed above, NPC-A301may determine one-way delay values of marked traffic, and/or may determine additional performance metrics on a per-SLA basis or some other granular basis.

FIG. 7illustrates an example process700for generating aggregated performance metrics. In some embodiments, some or all of process700may be performed by NPC-A301. In some embodiments, one or more other devices may perform some or all of process700in concert with, and/or in lieu of, NPC-A301.

As shown, process700may include receiving (at702) timestamps associated with marked traffic. For example, NPC-A301may receive (e.g., from NPC-M103and NPC-D105) timestamps indicating times at which particular traffic (e.g., marked traffic) was output by a network component101associated with NPC-M103and was received by another network component101associated with NPC-D105.

Process700may further include determining (at704) one-way delay and/or other performance metrics based on the timestamps of the marked traffic. For example, as discussed above, NPC-A301may determine amounts of time that traffic took to reach NPC-D105from NPC-M103, and may further calculate aggregated metrics such as jitter, packet loss, or the like.

Process700may additionally include receiving (at706) RAN performance information. For example, NPC-A301may receive (e.g., from a base station of a wireless network, such as a gNB, an evolved Node B (“eNB”), or some other type of base station) performance metrics associated with traffic carried by the RAN. The RAN performance information may include identifiers of particular UEs with which the RAN traffic is associated, network slices with which the RAN traffic is associated, TAIs with which the RAN traffic is associated, QCI values associated with the RAN traffic, and/or other parameters.

Process700may also include generating (at708) aggregated performance metrics. For example, as discussed above, NPC-A301may determine performance metrics based on the RAN performance information as well as the determined (at704) performance metrics associated with the traffic between NPC-M103and NPC-D105. As noted above, the same network components101that implement NPC-M103and NPC-D105, respectively, may also implement instances of NPC-D105and NPC-M103, respectively, in order to facilitate the determination of two-way performance metrics associated with such network components.

Process700may further include identifying (at710) that one or more SLAs are not met. For example, based on the performance metrics (e.g., determined at704and/or710), NPC-A301may determine that parameters of one or more performance metrics (e.g., latency, jitter, packet loss, etc.) are not met.

Process700may additionally include performing (at712) remedial one or more actions based on identifying that the one or more SLAs are not met. For example, NPC-A301may generate an alert indicating that the one or more SLAs are not met, may request or provision additional resources at a particular network component101that implements NPC-M103and/or NPC-D105, may instruct network component101to reduce a bitrate of traffic output from network component101, may instruct NPC-M103to mark a higher proportion of packets, and/or may perform other suitable remedial measures. In some embodiments, NPC-A301may utilize machine learning and/or other suitable techniques to determine an appropriate remedial action to take, based on the SLAs that were not met and/or other characteristics of the traffic (e.g., by how wide of a margin the SLAs were not met, a particular UE or network component101with which the traffic is associated, an application type associated with the traffic, and/or other characteristics).

For example, in an example where the traffic is voice call traffic that is not meeting a minimum DL delay SLA from MEC201to gNB203, NPC-A301may determine that the appropriate remedial measure to take is to instruct MEC201to allocate additional resources (e.g., processor resources, memory resources, etc.) to voice call traffic, increase a queue weight associated with voice call traffic, and/or otherwise take measures to reduce the DL delay of voice call traffic from MEC201to gNB203. As another example, if NPC-A301determines that traffic between two components of an O-RAN architecture are not meeting one or more SLAs, NPC-A301may output an alert or instruction to one or more controllers associated with the O-RAN indicating that the traffic between these two particular components is not meeting the one or more SLAs. Additionally, or alternatively, NPC-A301may provide performance metrics (e.g., as determined at704and/or708) to one or more controllers associated with the O-RAN, based on which the one or more controllers may make modifications to the O-RAN in situations where one or more SLAs are not met.

FIG. 8illustrates an example environment800, in which one or more embodiments may be implemented. In some embodiments, environment800may correspond to a Fifth Generation (“5G”) network, and/or may include elements of a 5G network. In some embodiments, environment800may correspond to a 5G Non-Standalone (“NSA”) architecture, in which a 5G radio access technology (“RAT”) may be used in conjunction with one or more other RATs (e.g., a Long-Term Evolution (“LTE”) RAT), and/or in which elements of a 5G core network may be implemented by, may be communicatively coupled with, and/or may include elements of another type of core network (e.g., an evolved packet core (“EPC”)). As shown, environment800may include UE801, RAN810(which may include one or more gNBs203), RAN812(which may include one or more one or more evolved eNBs813), and various network functions such as Access and Mobility Management Function (“AMF”)815, Mobility Management Entity (“MME”)816, Serving Gateway (“SGW”)817, Session Management Function (“SMF”)/Packet Data Network (“PDN”) Gateway (“PGW”)-Control plane function (“PGW-C”)820, PCF/PCRF825, Application Function (“AF”)830, User Plane Function (“UPF”)/PGW-User plane function (“PGW-U”)835, HSS/UDM840, and Authentication Server Function (“AUSF”)845. Environment800may also include one or more networks, such as Data Network (“DN”)850.

The quantity of devices and/or networks, illustrated inFIG. 8, is provided for explanatory purposes only. In practice, environment800may include additional devices and/or networks, fewer devices and/or networks, different devices and/or networks, or differently arranged devices and/or networks than illustrated inFIG. 8. For example, while not shown, environment800may include devices that facilitate or enable communication between various components shown in environment800, such as routers, modems, gateways, switches, hubs, etc. Alternatively, or additionally, one or more of the devices of environment800may perform one or more network functions described as being performed by another one or more of the devices of environment800. Devices of environment800may interconnect with each other and/or other devices via wired connections, wireless connections, or a combination of wired and wireless connections. In some implementations, one or more devices of environment800may be physically integrated in, and/or may be physically attached to, one or more other devices of environment800.

UE801may include a computation and communication device, such as a wireless mobile communication device that is capable of communicating with RAN810, RAN812, and/or DN850. UE801may be, or may include, a radiotelephone, a personal communications system (“PCS”) terminal (e.g., a device that combines a cellular radiotelephone with data processing and data communications capabilities), a personal digital assistant (“PDA”) (e.g., a device that may include a radiotelephone, a pager, Internet/intranet access, etc.), a smart phone, a laptop computer, a tablet computer, a camera, a personal gaming system, an IoT device (e.g., a sensor, a smart home appliance, or the like), a wearable device, an Internet of Things (“IoT”) device, a Mobile-to-Mobile (“M2M”) device, or another type of mobile computation and communication device. UE801may send traffic to and/or receive traffic (e.g., user plane traffic) from DN850via RAN810, RAN812, and/or UPF/PGW-U835.

RAN810may be, or may include, a 5G RAN that includes one or more base stations (e.g., one or more gNBs203), via which UE801may communicate with one or more other elements of environment800. UE801may communicate with RAN810via an air interface (e.g., as provided by gNB203). For instance, RAN810may receive traffic (e.g., voice call traffic, data traffic, messaging traffic, signaling traffic, etc.) from UE801via the air interface, and may communicate the traffic to UPF/PGW-U835, and/or one or more other devices or networks. Similarly, RAN810may receive traffic intended for UE801(e.g., from UPF/PGW-U835, AMF815, and/or one or more other devices or networks) and may communicate the traffic to UE801via the air interface.

RAN812may be, or may include, a LTE RAN that includes one or more base stations (e.g., one or more eNBs813), via which UE801may communicate with one or more other elements of environment800. UE801may communicate with RAN812via an air interface (e.g., as provided by eNB813). For instance, RAN810may receive traffic (e.g., voice call traffic, data traffic, messaging traffic, signaling traffic, etc.) from UE801via the air interface, and may communicate the traffic to UPF/PGW-U835, and/or one or more other devices or networks. Similarly, RAN810may receive traffic intended for UE801(e.g., from UPF/PGW-U835, SGW817, and/or one or more other devices or networks) and may communicate the traffic to UE801via the air interface.

AMF815may include one or more devices, systems, Virtualized Network Functions (“VNFs”), etc., that perform operations to register UE801with the 5G network, to establish bearer channels associated with a session with UE801, to hand off UE801from the 5G network to another network, to hand off UE801from the other network to the 5G network, manage mobility of UE801between RANs810and/or gNBs203, and/or to perform other operations. In some embodiments, the 5G network may include multiple AMFs815, which communicate with each other via the N14 interface (denoted inFIG. 8by the line marked “N14” originating and terminating at AMF815).

MME816may include one or more devices, systems, VNFs, etc., that perform operations to register UE801with the EPC, to establish bearer channels associated with a session with UE801, to hand off UE801from the EPC to another network, to hand off UE801from another network to the EPC, manage mobility of UE801between RANs812and/or eNBs813, and/or to perform other operations.

SGW817may include one or more devices, systems, VNFs, etc., that aggregate traffic received from one or more eNBs813and send the aggregated traffic to an external network or device via UPF/PGW-U835. Additionally, SGW817may aggregate traffic received from one or more UPF/PGW-Us835and may send the aggregated traffic to one or more eNBs813. SGW817may operate as an anchor for the user plane during inter-eNB handovers and as an anchor for mobility between different telecommunication networks or RANs (e.g., RANs810and812).

SMF/PGW-C820may include one or more devices, systems, VNFs, etc., that gather, process, store, and/or provide information in a manner described herein. SMF/PGW-C820may, for example, facilitate in the establishment of communication sessions on behalf of UE801. In some embodiments, the establishment of communications sessions may be performed in accordance with one or more policies provided by PCF/PCRF825.

PCF/PCRF825may include one or more devices, systems, VNFs, etc., that aggregate information to and from the 5G network and/or other sources. PCF/PCRF825may receive information regarding policies and/or subscriptions from one or more sources, such as subscriber databases and/or from one or more users (such as, for example, an administrator associated with PCF/PCRF825).

AF830may include one or more devices, systems, VNFs, etc., that receive, store, and/or provide information that may be used in determining parameters (e.g., quality of service parameters, charging parameters, or the like) for certain applications.

UPF/PGW-U835may include one or more devices, systems, VNFs, etc., that receive, store, and/or provide data (e.g., user plane data). For example, UPF/PGW-U835may receive user plane data (e.g., voice call traffic, data traffic, etc.), destined for UE801, from DN850, and may forward the user plane data toward UE801(e.g., via RAN810, SMF/PGW-C820, and/or one or more other devices). In some embodiments, multiple UPFs835may be deployed (e.g., in different geographical locations), and the delivery of content to UE801may be coordinated via the N9 interface (e.g., as denoted inFIG. 8by the line marked “N9” originating and terminating at UPF/PGW-U835). Similarly, UPF/PGW-U835may receive traffic from UE801(e.g., via RAN810, SMF/PGW-C820, and/or one or more other devices), and may forward the traffic toward DN850. In some embodiments, UPF/PGW-U835may communicate (e.g., via the N4 interface) with SMF/PGW-C820, regarding user plane data processed by UPF/PGW-U835.

HSS/UDM840and AUSF845may include one or more devices, systems, VNFs, etc., that manage, update, and/or store, in one or more memory devices associated with AUSF845and/or HSS/UDM840, profile information associated with a subscriber. AUSF845and/or HSS/UDM840may perform authentication, authorization, and/or accounting operations associated with the subscriber and/or a communication session with UE801.

DN850may include one or more wired and/or wireless networks. For example, DN850may include an Internet Protocol (“IP”)-based PDN, a wide area network (“WAN”) such as the Internet, a private enterprise network, and/or one or more other networks. UE801may communicate, through DN850, with data servers, other UEs801, and/or to other servers or applications that are coupled to DN850. DN850may be connected to one or more other networks, such as a public switched telephone network (“PSTN”), a public land mobile network (“PLMN”), and/or another network. DN850may be connected to one or more devices, such as content providers, applications, web servers, and/or other devices, with which UE801may communicate.

FIG. 9illustrates an example Distributed Unit (“DU”) network900, which may be included in and/or implemented by one or more RANs (e.g., RAN810). In some embodiments, a particular RAN may include one DU network900. In some embodiments, a particular RAN may include multiple DU networks900. In some embodiments, DU network900may correspond to a particular gNB203of a 5G RAN (e.g., RAN810). In some embodiments, DU network900may correspond to multiple gNBs203. In some embodiments, DU network900may correspond to one or more other types of base stations of one or more other types of RANs. As shown, DU network900may include Central Unit (“CU”)905, one or more Distributed Units (“DUs”)903-1through903-N (referred to individually as “DU903,” or collectively as “DUs903”), and one or more Remote Units (“RUs”)901-1through901-M (referred to individually as “RU901,” or collectively as “RUs901”).

CU905may communicate with a core of a wireless network (e.g., may communicate with one or more of the devices or systems described above with respect toFIG. 8, such as AMF815and/or UPF/PGW-U835). In the uplink direction (e.g., for traffic from UEs801to a core network), CU905may aggregate traffic from DUs903, and forward the aggregated traffic to the core network. In some embodiments, CU905may receive traffic according to a given protocol (e.g., Radio Link Control (“RLC”)) from DUs903, and may perform higher-layer processing (e.g., may aggregate/process RLC packets and generate Packet Data Convergence Protocol (“PDCP”) packets based on the RLC packets) on the traffic received from DUs903.

In accordance with some embodiments, CU905may receive downlink traffic (e.g., traffic from the core network) for a particular UE801, and may determine which DU(s)903should receive the downlink traffic. DU903may include one or more devices that transmit traffic between a core network (e.g., via CU905) and UE801(e.g., via a respective RU901). DU903may, for example, receive traffic from RU901at a first layer (e.g., physical (“PHY”) layer traffic, or lower PHY layer traffic), and may process/aggregate the traffic to a second layer (e.g., upper PHY and/or RLC). DU903may receive traffic from CU905at the second layer, may process the traffic to the first layer, and provide the processed traffic to a respective RU901for transmission to UE801.

RU901may include hardware circuitry (e.g., one or more RF transceivers, antennas, radios, and/or other suitable hardware) to communicate wirelessly (e.g., via an RF interface) with one or more UEs801, one or more other DUs903(e.g., via RUs901associated with DUs903), and/or any other suitable type of device. In the uplink direction, RU901may receive traffic from UE801and/or another DU903via the RF interface and may provide the traffic to DU903. In the downlink direction, RU901may receive traffic from DU903, and may provide the traffic to UE801and/or another DU903.

RUs901may, in some embodiments, be communicatively coupled to one or more MECs201. For example, RU901-1may be communicatively coupled to MEC201-1, RU901-M may be communicatively coupled to MEC201-M, DU903-1may be communicatively coupled to MEC201-2, DU1103-N may be communicatively coupled to MEC201-N, CU1105may be communicatively coupled to MEC201-3, and so on. MECs201may include hardware resources (e.g., configurable or provisionable hardware resources) that may be configured to provide services and/or otherwise process traffic to and/or from UE801, via a respective RU901.

For example, RU901-1may route some traffic, from UE801, to MEC201-1instead of to a core network (e.g., via DU903and CU905). MEC201-1may process the traffic, perform one or more computations based on the received traffic, and may provide traffic to UE801via RU901-1. In this manner, ultra-low latency services may be provided to UE801, as traffic does not need to traverse DU903, CU905, and an intervening backhaul network between DU network900and the core network.

In some embodiments, some or all of the elements of O-RAN environment1000may be implemented by one or more configurable or provisionable resources, such as virtual machines, cloud computing systems, physical servers, and/or other types of configurable or provisionable resources. In some embodiments, some or all of O-RAN environment1000may be implemented by, and/or communicatively coupled to, one or more MECs201.

Non-Real Time RIC1001and Near-Real Time RIC1003may receive performance information (and/or other types of information) from one or more sources, and may configure other elements of O-RAN environment1000based on such performance or other information. For example, Near-Real Time RIC1003may receive performance information, via one or more E2 interfaces, from O-eNB1005, O-CU-CP1007, and/or O-CU-UP1009, and may modify parameters associated with O-eNB1005, O-CU-CP1007, and/or O-CU-UP1009based on such performance information. Similarly, Non-Real Time MC1001may receive performance information associated with O-eNB1005, O-CU-CP1007, O-CU-UP1009, and/or one or more other elements of O-RAN environment1000and may utilize machine learning and/or other higher level computing or processing to determine modifications to the configuration of O-eNB1005, O-CU-CP1007, O-CU-UP1009, and/or other elements of O-RAN environment1000. In some embodiments, Non-Real Time RIC1001may generate machine learning models based on performance information associated with O-RAN environment1000or other sources, and may provide such models to Near-Real Time RIC1003for implementation.

O-eNB1005may perform functions similar to those described above with respect to eNB813. For example, O-eNB1005may facilitate wireless communications between801and a core network. O-CU-CP1007may perform control plane signaling to coordinate the aggregation and/or distribution of traffic via one or more DUs903, which may include and/or be implemented by one or more O-DUs1011, and O-CU-UP1009may perform the aggregation and/or distribution of traffic via such DUs903(e.g., O-DUs1011). O-DU1011may be communicatively coupled to one or more RUs901, which may include and/or may be implemented by one or more O-RUs1013. In some embodiments, O-Cloud1015may include or be implemented by one or more MECs201, which may provide services, and may be communicatively coupled, to O-CU-CP1007, O-CU-UP1009, O-DU1011, and/or O-RU1013(e.g., via an O1 and/or O2 interface).

In some embodiments, one or more elements of O-RAN environment1000may include an instance of NPC-M103and/or NPC-D105, in order to determine performance metrics of traffic sent between elements of O-RAN environment1000(e.g., via one or more of the illustrated interfaces, such as the O1 interfaces, O2 interfaces, E1 interfaces, E2 interfaces, F1-c interfaces, F1-u interfaces, etc.). In some embodiments, Non-Real Time RIC1001, Near-Real Time RIC1003, network component1015, and/or one or more other elements of O-RAN environment1000may include and/or may implement NPC-A301, in order to determine aggregated performance metrics and modify parameters of elements of O-RAN environment1000based on such aggregated performance metrics.

FIG. 11illustrates example components of device1100. One or more of the devices described above may include one or more devices1100. Device1100may include bus1110, processor1120, memory1130, input component1140, output component1150, and communication interface1160. In another implementation, device1100may include additional, fewer, different, or differently arranged components.

Bus1110may include one or more communication paths that permit communication among the components of device1100. Processor1120may include a processor, microprocessor, or processing logic that may interpret and execute instructions. Memory1130may include any type of dynamic storage device that may store information and instructions for execution by processor1120, and/or any type of non-volatile storage device that may store information for use by processor1120.

Input component1140may include a mechanism that permits an operator to input information to device1100, such as a keyboard, a keypad, a button, a switch, etc. Output component1150may include a mechanism that outputs information to the operator, such as a display, a speaker, one or more light emitting diodes (“LEDs”), etc.

Communication interface1160may include any transceiver-like mechanism that enables device1100to communicate with other devices and/or systems. For example, communication interface1160may include an Ethernet interface, an optical interface, a coaxial interface, or the like. Communication interface1160may include a wireless communication device, such as an infrared (“IR”) receiver, a Bluetooth® radio, or the like. The wireless communication device may be coupled to an external device, such as a remote control, a wireless keyboard, a mobile telephone, etc. In some embodiments, device1100may include more than one communication interface1160. For instance, device1100may include an optical interface and an Ethernet interface.