Patent ID: 12200534

While the disclosure is susceptible to various modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and will be described in detail herein. However, it should be understood that the disclosure is not intended to be limited to the particular forms disclosed. Rather, the intention is to cover all modifications, equivalents and alternatives falling within the scope of the disclosure as defined by the appended claims.

DETAILED DESCRIPTION

FIG.1is a schematic100that shows a plurality of microflows140aggregated from multiple sites110A,110B,110C. Each site110A,110B,110C is capable of transmitting and receiving data via a network device, or devices, such as, but not limited to, a router or the like. Each site may be transmitting and/or receiving an application to and/or from a destination site. For example, the first site110A may be utilizing a first application120A. The second site110B may be utilizing a second application120B, a third application120C, and a fourth application120D. The third site110C may be utilizing a fifth application120E. The number of sites and applications are shown for illustrative purposes and may be varied as would be appreciated by one of ordinary skill in the art having the benefit of this disclosure.

Each site110A,110B,110C communicates with a net cloud services gateway (NCSG)130that aggregates the connections with each site110A,110B,110C and converts each connection into microflows140as shown inFIG.1. The NCSG130does link aggregation with the multiple sites110A,110B,110C. Each application120A-120E may be split across multiple microflows140by the NCSG130based on both a Quality of Experience (QOE) score as well as priority of the application120A-120E as discussed herein. The QOE score may be determined based on measurements of the network as discussed herein. Each application120A-120E may also be able to flow through multiple paths along a network as shown inFIG.2.

FIG.2is a schematic200that shows a plurality of paths205between an ingress NCSG (I-NCSG)130I and an egress NCSG (E-NCSG)130E of a network201, which may be, but is not limited to a cloud. A plurality of paths205may be bound together as a single pipe at the logical level. The I-NCSG130I and the E-NCSG130E enable optimal communication between multiple sites110A,110B,11C, which may be are collectively referred to as a source location220, and a destination230, which also could be multiple sites as would be appreciated by one of ordinary skill in the art having the benefit of this disclosure. As discussed above, each site110A,110B,110C may utilize one or more applications120A-120E. The I-NCSG130I determines the paths205to utilize for each application120A-120E due to policies determined by QOE scores and/or application priority as discussed herein. The E-NCSG130E will reconstruct the packets to deliver the application to the destination230as the flows reach the E-NCSG130E. One benefit is the I-NCSG130I provides multiple paths because it can utilize multiple clouds, such as but not limited to Amazon Web Services, Google Cloud, Microsoft Azure, Megaport, and Equinix, it can utilize multiple cellular providers, such as but not limited to AT&T and Verizon, and it can utilize multiple telephone company providers.

FIG.3is a schematic that shows an embodiment of a plan300for capturing data at each hop along the paths between a starting location220and destination location230on a cloud and/or wireless network. The hops may include various numbers and configurations of network communication elements (NCEs)105,205and NCXs205A,205B along the paths. The NCE may be a router, mobile device, or the like. The NCX may be a server, net cloud services gateway, or the like. A path between the starting location220and destination location230may include an unknown carrier network310. Data may be collected from a NCX205A prior to entrance of the unknown carrier network310and from a NCX205B upon exiting the unknown carrier network310.

Various data221is collected by the system at the starting location220. The data221may be, but is not limited to, packet data, capture user ID, role ID, flow start time, application ID, start time, device type, source IP, and the like. Various data106A is collected by the system at a first NCE105A. The data106A from the first NCE105A may be packet data, capture user ID, flow start time, application ID, start time, bandwidth, latency, device type, and source IP. Various data206A is collected by the system at a first NCX205A. The data206A from the first NCX205A may be packet data, capture user ID, flow start time, application ID, start time, bandwidth, latency, device type, source IP, and the like. Additionally, the data206A from the first NCX205A may include proximity headend NCX data, location, NCX type, version, bandwidth from NCE, and latency from NCE.

Various data206B is collected by the system at a second NCX205B. The data206B from the second NCX205B may be packet data, capture user ID, flow start time, application ID, start time, bandwidth, latency, device type, and source IP. Additionally, the data206B from the second NCX205B may include proximity NCX headend data, NCX location, NCX type, version, bandwidth from NCE, and latency form NCE. Various data106B is collected by the system at a second NCE105B. The data106B from the second NCE105B may be packet data, capture user ID, flow start time, application ID, start time, bandwidth, latency, device type, and source IP. Various data231is collected by the system at the destination location230. The data231may be packet data, capture user ID, flow start time, application ID, start time, bandwidth, latency, device type, and source IP.

The configuration shown inFIG.3may be varied as would be appreciated by one of ordinary skill in the art having the benefit of this disclosure. For example, the number and location of each NCSG and network device may be varied. The data collected at each hop is used to determine a QOE score. The system uses the QOE score to implement the traffic steering policies over the cloud or wireless network. A controller NCSG automatically generates traffic steering polices based on the QOE score and/or application priority as discussed herein. These traffic steering policies are dynamically generated due to the collection of data from each hop along the wireless network and used to determine how to optimally steer traffic through the wireless network.

A QOE score may be determine using one or more of the following formulas:QOE Score at Source=0QOE Score at a NCE directly connected to the source (SQOE)=available bandwidth/time take to transmit (in seconds)QOE Score at the NCX connected to the source site NCE (SNCXQOE)=(SQOE+(available bandwidth from NCE to NCE/Time taken to transmit from NCE to NCX))/2QOE Score at destination NCX (DNCEQOE)=(DNCXQOE+(available bandwidth from Carrier Network (Middle Mile) to NCX/Time taken from source NCX to destination NCX))/2QOE Score at the destination NCE (DNCXQOE)=(DNCXQOE+(available bandwidth from Destination NCX to Destination NCE/Time taken from destination NCX to destination NCE))/2

The QOE score at the destination NCE may be used for validation to a service level agreement (SLA) from end to end and may be reported to a customer as an SLA violation. The QOE score and the destination NCE may be used to determine end to end SLA creep as the packet moves through the network. The bandwidth QOE score may be measured at a per carrier network level and may be compared to the bandwidth SLA provided by the carrier to identify potential SLA violations between each hop. A historic average bandwidth QOE score may be used to select path of traffic movement based on the bandwidth QOE score for each path. An application SLA for bandwidth may be set by the customer at the network level by setting the priority of an application at the network level, which may be converted to the Application QOE score at a per site level.

The customer may also designate a cost threshold that is not to be exceeded, which may be used to determine the QOE score. For example, a customer may want to limit and/or prevent traffic on a particular cellular, or other wireless network, to ensure that the cost of the transmission does not exceed a threshold amount or level. The QOE score may take into account the customer's cost limit and steer traffic based on automated polices generated due to the QOE score. In other words, the customer may designate to sacrifice potential optimization due to a larger desire to limit the costs of the communication over the wireless network.

Traffic steering policies may be implemented at individual NCEs and/or NCSGs. A potential QOE score at a hop may be use a historic QOE score of the site-to-site path taken and variance from that QOE score. In this instance, 7-day aggregate QOE scores will handle (i.e., minimize) network variations due to unknown dedicated internet access (DIA) traffic. The traffic steering policies may take into account anomalies due to seasonal changes (e.g., holidays) or other changes or events. The traffic steering policies may set a tolerable variance that is within +/−3 limits. One or more of the following may be used to determine a historic QOE score at a hop.Average Time of Day Bandwidth at Hop—Last 7 Days (ATB7)Average Time of Day Bandwidth Variance at Hop—Last 7 Days (AVTB7)Average Number of Concurrent Flows at Time—Time of Day (ANCT)Variance in Concurrent Flows—Time of Day (ANCVT)Average Aggregated Packet Size—Time of Day (AAPST)Variance in Packet Size—Time of Day (AVPST)Bandwidth Measured—Time of Day (BTD)Total Number of Samples—Bandwidth (TNSB)Concurrent Flows Measured—Time of Day (CFTD)Total Number of Samples for Concurrent Flow (TNSCF)Concurrent Packet Size—Time of Day (CPSTD)Total Number of Samples for Concurrent Packet (TNSCP)Average Packet Loss at Hop—Time of Day—Last 7 days—computed for the next potential Hop (APLT)Average Packet Loss at Hop—Last 24 Hours—computed for the next potential hop (APLD)AVTB7=SUM (BTD−ATB7)/(TNSB−1)ANCVT=SUM (CFTD−ANCT)/(TNSCF−1)AVPST=SUM (SUM (CPSTD)−AAPST)/(TNSCP−1)

One or more of the following may be used to set a +/−3 variance limit measure to measure value at a hop.If (current Bandwidth−ATB7) is between −3*AVTB7 and 3*AVTB7 then 1 else 0 end AS BQOeIf (Current Flows−ANCT) is between −3*ANCVT and 3*ANCVT then 1 else 0 end AS FQOeIf (Current Concurrent Packet Size−AAPST) is between −3*AVPST and 3*AVPST then 1 else 0 end AS PQOe

One or more of the following may be used to consider current available bandwidth and size of the current concurrent packet in the forwarding paths.Packet Proportionate bandwidth (PPB)=(Packet Size*ATB7)/AAPSTIf (Packet Size>50% ATB7) Then 0 else 1 end AS PBQOEIf (Packet Size/Current concurrent packet size)>0.5 then 0 else 1 end AS PCQOe

As discussed herein, the system may include customer prioritization and SLA at an application level with 1 being the highest priority and 3 being the lowest priority. Application priority may be set at high (1), medium (2), or low (3) priority. The following may be used to apply the application priority to application packets.Application Packet Priority (APPR)=(BQOe+FQOe+PQOe+PBEOE+PCQOe)*ASLA

If loss is increasing at the site where the next hop is identifying chance of pack loss, the following may be used add the loss condition.If APLD>APLT Then APLD/(Total Packet Loss at all potential Hops) END AS APLOSS

As discussed herein, the system provides a dynamic traffic steering policy to each hop along paths between the source and destination. A rule for choosing the next hop may be a function of APPR and APLOSS. Ideally, APPR should be reverse of APLOSS. APPR may be measured per physical link. Packets may be ranked by their APP score per physical link. Packets with the lowest APP score should be sent through a link or tunnel having the highest bandwidth. In this way, the traffic steering may be optimized through a wireless network such as a cloud. The measurements discussed herein may be made at each hop at a per packet level. The measurements discussed herein may be made at each hop at a per segment level.

FIG.4is a schematic of an embodiment of a system400for steering traffic. The system400includes a source NCE105A that is communicating with a destination NCE105B over a wireless network. The source NCE105A may connect to the system400via an I-NCSG130I and the destination NCE105B may connect to the system400via a E-NCSG130E. The I-NCSG130I may be located in a first data center of a customer. The E-NCSG130E may be located in a second data center of a customer. The customer may include NCEs105connected to the system400in satellite data centers. The NCEs105in the satellite data centers may be connected to the system400by various links. For example, the NCEs105may be connected via cellular networks440A,440B.

The system includes a controller NCSG (C-NCSG)410that may be wirelessly connected to the system400. For example, the C-NCSG410may be connected via a cloud. The C-NCSG410automatically generates traffic steering policies that may be used throughout the system400to steer traffic from the source NCE105A to the destination NCE105B. The C-NCSG410continuously receives inputs420that may be used to dynamically generate traffic steering policies sent out to each element in the system400. The C-NCSG410receives measurements of the network that are used to generate current traffic steering policies. The C-NCSG410may use a QOE score to generate traffic steering policies. The QOE score may be based on measurements of bandwidth, loss, latency, and/or jitter at each hop on the system400. The bandwidth, loss, latency, and/or jitter may be measured at each hop to determine the QOE score at a per packet level or measuring at a per segment level. The C-NCSG410may determine a current QOE score at each hop along the system400. The C-NCSG410may compare the current QOE score at each hop to a historical QOE score per path. The C-NCSG410may provide automated steering policies as an embedded control packet at each hop.

The C-NCSG410may also receive customer prioritization at an application level. The automated steering policies may also be based on the customer prioritization at the application level. The system400may include customer prioritization and SLA at an application level with 1 being the highest priority and 3 being the lowest priority. Application priority may be set at high (1), medium (2), or low (3) priority.

The C-NCSG410may generate automated steering policies based on a historic average bandwidth QOE score based on a bandwidth QOE score of each path. The C-NCSG410may generate automated steering policies based on a seven-day aggregate value of bandwidth at the hop and a seven-day aggregate value of bandwidth variance at the hop. The C-NCSG410may generate automated steering policies based on an average number of concurrent flows and a variance in the average number of concurrent flows at a specified time. The C-NCSG410may generate automated steering policies based on an average aggregate packet size and a variance in the average aggregate packet size at the specified time. The C-NCSG410may generate automated steering policies based on an average bandwidth at the specified time and a total number of samples of bandwidth.

The C-NCSG410may generate automated steering policies based on concurrent flows measured at the specified time and a total number of samples for concurrent flows. The C-NCSG410may generate automated steering policies based on a concurrent packet size at the specified time and a total number of samples of concurrent packet size. The C-NCSG410may generate automated steering policies based on a seven-day aggregate value of average package loss at a hop and a twenty-four-hour aggregate value of average packet loss at the hop.

The system400may include a cloud430. The C-NCSG410can control communication over multiple clouds430, such as but not limited to Amazon Web Services430E, Google Cloud430C, Microsoft Azure430D, Megaport430A, or Equinix430B. Likewise, the C-NCSG410can control communication over multiple cellular networks such as but not limited to AT&T440A and Verizon440B. Likewise, the C-NCSG410can control communication over multiple telephone company networks450.

FIG.5is a schematic of an embodiment of a system500for steering traffic. The system500may be a simple direct connect from various NCEs105C,105D,105E to a cloud430. The NCEs105C,105D,105E are connected to a data center510via the cloud430. A C-NCSG410located on the cloud430generates automated steering policies to steer the traffic between each of the NCEs and the data center510as discussed herein.

FIG.6is a schematic of an embodiment of a system600for steering traffic. A plurality of NCEs105F,105G,105H may be wirelessly connected to a first virtual data center610via a first cellular network601or via a second cellular network602. Another NCE105I may be connected to the first virtual data center610via a wired connection603. The first virtual data center610may include a C-NCSG410that generates automated steering policies for the system600as discussed herein. The system600includes a second virtual data center620wirelessly connected to the first virtual data center610. The second virtual data center620may include a C-NCSG410that generates automated steering policies for the system600as discussed herein.

The system600includes a third virtual data center630. The third virtual data system630may be connected to the first virtual data center610by various networks640such as a cloud or cellular network as would be appreciated by one of ordinary skill in the art having the benefit of this disclosure. The third virtual data center630may also include a C-NCSG410that generates automated steering policies for the system600. The system600includes a corporate data center650that is linked to the NCEs105F-105I. The one or more C-NCSGs410generate automated steering policies to optimize communication between the corporate data center650and each of the NCEs105F-105I.

FIG.7is a schematic of an embodiment of a system700for steering traffic. Like the system600ofFIG.6, a plurality of NCEs105F,105G,105H may be wirelessly connected to a first virtual data center610via a first cellular network601or via a second cellular network602. Another NCE105I may be connected to the first virtual data center610via a wired connection603. The first virtual data center610may include a C-NCSG410that generates automated steering policies for the system600as discussed herein. The system600includes a second virtual data center620wirelessly connected to the first virtual data center610. The second virtual data center620may include a C-NCSG410that generates automated steering policies for the system700as discussed herein.

The system700includes a third virtual data center630. The third virtual data system630may be connected to the first virtual data center610by various networks640such as a cloud or cellular network as would be appreciated by one of ordinary skill in the art having the benefit of this disclosure. The third virtual data center630may also include a C-NCSG410that generates automated steering policies for the system700. The system700includes a corporate data center650that is linked to the NCEs105F-105I. The one or more C-NCSGs410generate automated steering policies to optimize communication between the corporate data center650and each of the NCEs105F-105I. The system700may be more secure than the system600ofFIG.6. System700may include increased security between each element of the system700as indicated by elements710.

FIG.8is a flow chart of an embodiment of a method800for steering traffic in a wireless network. The method800includes providing an ingress point of the wireless network and an egress point of the wireless network, with a plurality of paths between the ingress point and the egress point, wherein a plurality of hops are located along the plurality of paths between the ingress point and the egress point, at805. The method800includes binding the plurality of paths as a single pipe at a logical level, at810. The method800includes determining a quality of experience (QOE) score, at815. The method800includes providing automated steering policies, wherein the automated steering policies are based on the determined QOE score, at820. The method800includes steering traffic at each hop based on the automated steering policies, at825.

The method800may include measuring bandwidth, loss, latency, and jitter at each hop of the plurality of hops to determine the QOE score, at830. The method800may include receiving customer prioritization at an application level. At835. The automated steering policies may also be based on the customer prioritization at the application level. The method800may include measuring bandwidth, loss, latency, and jitter at each hop at a per packet level or measuring at a per segment level to determine the QOE score, at840. The method800may include determining a current QOE score at each hop, at845. The method800may include comparing the current QOE score at each hop to a historical QOE score per path, at850. The method800may include embedding a control packet at each hop, at855. The method800may include steering traffic at each hop may be based on a historic average bandwidth QOE score based on a bandwidth QOE score of each path, at860.

The method800may include providing automated steering policies based on a seven-day aggregate value of bandwidth at the hop and a seven-day aggregate value of bandwidth variance at the hop, at865. The method800may include providing automated steering policies based on an average number of concurrent flows and a variance in the average number of concurrent flows at a specified time, at870. The method800may include providing automated steering policies based on an average aggregate packet size and a variance in the average aggregate packet size at the specified time, at875. The method800may include providing automated steering policies based on an average bandwidth at the specified time and a total number of samples of bandwidth, at880. The method800may include providing automated steering policies based on concurrent flows measured at the specified time and a total number of samples for concurrent flows, at885. The method800may include providing automated steering policies based on a concurrent packet size at the specified time and a total number of samples of concurrent packet size, at890. The method800may include providing automated steering policies based on a seven-day aggregate value of average package loss at a hop and a twenty-four-hour aggregate value of average packet loss at the hop, at895.

Although this disclosure has been described in terms of certain embodiments, other embodiments that are apparent to those of ordinary skill in the art, including embodiments that do not provide all of the features and advantages set forth herein, are also within the scope of this disclosure. Accordingly, the scope of the present disclosure is defined only by reference to the appended claims and equivalents thereof.