Patent Description:
Today's <NUM>, <NUM>, and LTE networks operate using multiple data centers ("DCs") that can be distributed across clouds. These networks are centrally managed by only a few operating support systems ("OSSs") and network operations centers ("NOCs"). <NUM> technology will dramatically increase network connectivity for all sorts of devices that will need to connect to the Telco network and share the physical network resources. Current network architectures cannot scale to meet these demands.

Network slicing is a form of virtualization that allows multiple logical networks to run on top of a shared physical network infrastructure. A distributed cloud network can share network resources with various slices to allow different users, called tenants, to multiplex over a single physical infrastructure. For example, Internet of Things ("IoT") devices, mobile broadband devices, and low-latency vehicular devices will all need to share the <NUM> network. These different use cases will have different transmission characteristics and requirements. For example, the loT devices will typically have a large number of devices but very low throughput. Mobile broadband will be the opposite, with each device transmitting and receiving high bandwidth content. Network slicing can allow the physical network to be partitioned at an end-to-end level to group traffic, isolate tenant traffic, and configure network resources at a macro level.

In a traditional network, routing decisions are made based on the destination IP address of a packet. Each router looks at the destination internet protocol ("IP") address of the packet and forwards the packet to the next router based on a routing table at the router. However, in a slice-based virtual service network ("VSN"), a single IP address can be shared by packets in multiple slices where each slice can have its own path through the network. New techniques are required for routers to make forwarding decisions based on the slice to which a packet belongs rather than just a destination IP address.

Additionally, traditional switches can allow creation of a link aggregation group ("LAG"). A LAG can allow the links between switches to be bundled together into a single virtual link to distribute traffic across the links. However, in a slice-based network, packets that belong to the same slice can end up on different links, resulting in out-of-order delivery of data. Some links are longer than others, for example.

As a result, a need exists for new systems to perform slice-based routing in a VSN.

By way of introduction, the following description concerns systems and methods for slice-based routing in a VSN. The system can include multiple switches, physical or virtual, distributed over a slice-based network on one or more clusters or clouds. The switches can be programmable to perform functions at the physical layer of the network. In one non-claimed example, an orchestrator can remotely send programming commands to the switches. The switches can include programmable networking application-specific integrated circuits ("ASICs"), such as the Tofino chip, that can be programmed using the P4 language or some other protocol. Virtual switches likewise can be programmable.

Once programmed, the switches can perform stages for slice-based routing. The invention defines a method for slice-based routing as defined by claim <NUM>, a computer-readable medium according to claim <NUM> and, respectively, a system according to claim <NUM>.

Reference will now be made in detail to the present examples, including examples illustrated in the accompanying drawings.

In one example, a system includes multiple switches across one or more clusters or clouds in a VSN. An orchestrator can manage multiple slices within the network, which can be used for specific purposes or leased to tenants. For example, the slices can be reserved for particular applications, IoT devices, or customers. Each slice can be a virtual network that runs on top of a shared physical network infrastructure distributed across one or more Telco clouds. A slice can include a service chain of VNFs for performing certain network tasks.

In one example, a switch can use a slice ID for a packet for routing traffic. The switch can receive ingress packets having the same destination IP address but route them differently, to different switches, based on slice ID. The switch can be programmed to include slice paths that govern which output ports are used and what the next hop will be, based on slice ID.

The switch can also implement equal cost multi-path ("ECMP"). ECMP can include features in Layers <NUM> and <NUM> of the open systems interconnection ("OSI") model. In Layer <NUM>, the networking layer, ECMP can be modified to use a slice-based table and additional logic when distributing traffic across multiple equal cost paths. Traditionally, using ECMP routing based on destination IP address to determine which ECMP route to use can result in out-of-order data in a slice-based network. To avoid this problem, multi-path tables for ECMP ("ECMP tables" or "ECMP routing tables") can instead be slice-based and keyed on slice ID. This can allow for using ECMP for slices when the switch is using ECMP to connect with the next hop. These slice-specific ECMP actions can be programmed into the switch by the orchestrator process, in an example.

For Layer <NUM> aspects of ECMP, the switch can implement LAG, in one example, to bundle ports together in a virtual channel to another switch. However, the packets with the same slice ID can be routed on the same link between the switches. This can ensure that the slice data remains in order. Without taking slice ID into account, data could arrive out of order based on different travel times across the links. This can be due to different travel distances for the links and other factors.

<FIG> is an example flow chart for performing slice-based routing in a VSN. At stage <NUM>, the switch receives a packet. The packet can be received at one of multiple ingress ports. The packet can be part of a slice in the VSN.

At stage <NUM>, the switch determines a slice identifier for the packet. The switch can do this by inspecting the packet header. For example, a switch or slice selector can use a combination of layer <NUM> to layer <NUM> (L2-L4) headers or by performing deep packet inspection (e.g., to classify traffic based on data in the layer <NUM> (L7) header). For example, slice selection can be based simply on the source device by using the source network layer (e.g., IP) address, or can be based on the type of traffic or destination network domain by looking at the L7 header. In some embodiments, the network slice selector maintains state for mapping connections to network slices so that deep packet inspection does not need to be performed on each data message of a connection. In addition, for some connections, only certain data messages contain the L7 header information required for performing the slice selection.

When performing slice selection using deep packet inspection, the initial data message for a connection may not include the L7 header information that the slice selector needs to correctly identify the slice. For example, a connection between an endpoint device (for example, a mobile device such as a smart phone or tablet, a laptop or desktop computer, an IoT device, a self-driving automobile, a smart camera belonging to a security system) and a network domain often begins with a set of connection initiation messages, such as a TCP handshake. After completion of the handshake, the device then sends, for example, an http get message that includes the network domain. Subsequent data messages sent between the device and the network domain may not include such information.

After initial slice selection at the selector, the routers can be updated to include slice tables (one type of routing table) that correlate packet header information to slice ID. This can allow the switch to use the packet header can include address information to look up the slice ID. For example, the packet header can list source and destination media access control ("MAC") addresses, source and destination internet protocol ("IP") addresses, source and destination ports, and indicate a packet protocol. With this information, the switch can identify a slice ID in a local slice table. An example slice table is shown below, as Table <NUM>.

The agent can use source IP address ("Source IP"), destination IP address ("Dest IP"), source port, destination port, and a protocol type to find the correlated Slice ID. The protocol type can be represented as a number in one example, with different numbers correlating to different packet types, such as ICMP, ARP, and TCP.

The slice table can be programmed into the switch by an orchestrator process, in an example. For example, an orchestrator can remotely program the switches to perform slice-specific actions. In one example, the switches include programmable networking application-specific integrated circuits ("ASICs"), such as the TOFINO chip, that can be programmed using the P4 language or some other protocol.

At stage <NUM>, the switch determines a next hop for the packet based on the slice ID. A next hop can be another switch in the slice path. To determine the next hop, the switch can access a routing table using the slice ID. The routing table can return information regarding the slice path for that slice ID, including the next hop. The routing table can be located locally on the switch. In one non-claimed example, the orchestrator programs the switch to include the routing table. The orchestrator can contact the switch to update the routing table each time the orchestrator changes a slice path that impacts routing by the switch. For example, if a first slice path includes a next hop of switch <NUM> and a new slice path includes a next hop of switch <NUM>, the orchestrator can send that information to the switch so that future traffic for that slice can be routed to switch <NUM>.

An example routing table is shown below as Table <NUM>.

An example routing table is shown below in Table <NUM>. Using the slice ID and a destination IP, the next hop can be determined. In one example, the routing table can also provide which egress interface to use. In another example, the routing table can be the same as the slice table, such as the slice table of Table <NUM>, in an example. In that example, one or more additional columns can represent the next hop or egress interface. Alternatively, the Dest IP or Dest Port of Table <NUM> can be used to determine the egress interface. Using a routing table can also include looking up information in an ECMP routing table or one or more LAG tables, as will be discussed below.

At stage <NUM>, the switch selects an egress port for the packet based on the slice identifier. For example, the slice can have an assigned egress interface in the routing table that corresponds to the next hop. Selecting an egress port can be achieved by selecting an egress interface, in an example. An egress interface can have packet queues and other mechanisms, such as load balancing algorithms, that govern when the packet is actually sent to the next hop from an egress port at the egress interface.

In one example, the switch can support ECMP. Traditional ECMP routing can allow traffic with the same source and destination to be transmitted across multiple paths of equal cost. This can allow for load balancing traffic and increasing bandwidth by fully utilizing otherwise unused bandwidth on links to the same destination (next hop). When ECMP is used, next-hop paths of equal cost are identified based on routing metric calculations and hash algorithms. That is, routes of equal cost can have the same preference and metric values, and the same cost to the network. The ECMP process can identify a set of routes, each of which is a legitimate equal cost next hop towards the destination. The identified routes are referred to as an ECMP set. Because it addresses only the next hop destination, ECMP can be used with various routing protocols. In one example, ECMP can operate with a Layer <NUM> networking protocol of the OSI model. Layer <NUM> can be referred to as the network layer of a networking model.

In one example, the information accessed from the slice table can be used with an ECMP routing table, an example of which is shown below in Table <NUM>. ECMP can allow use of multiple paths between two switches and determine how to distribute traffic across those paths when the paths are of equal metrics.

As shown in Table <NUM>, each row can represent an ECMP set. The ECMP Members can represent next hops that are based on a subnet corresponding to a destination IP address. For example, for subnet <NUM>. <NUM> (and mask <NUM>. <NUM>), six different possible next hops (ECMP members) are available in the ECMP set. The second ECMP set has four next hops, and the third ECMP set has two next hops.

In one example, the slice table (e.g., Table <NUM>) and ECMP routing table (e.g., Table <NUM>) can be used to determine the next hop. For example, for a packet with a Slice ID of <NUM>, the destination IP is <NUM>. This corresponds to the first ECMP set, which has a subnet of <NUM>.

An ECMP set can be formed when the ECMP routing table, such as in Table <NUM>, contains multiple next-hop addresses for use in reaching the same destination with equal cost. Routes of equal cost have the same (or similar within a threshold) preference and metric values. If there is an ECMP set for the active route, a load balancing algorithm can be used to choose one of the next-hop addresses in the ECMP set to install in the forwarding table.

The ECMP can use load balancing to select between the multiple links in the ECMP set. In one example, slice-based load balancing can be used to distribute the packets across the multiple links. Packets having the same slice ID can be guaranteed to take the same link. For example, a table correlating slice ID to the links can be used. Alternatively, an additional column on Table <NUM> can indicate Slice ID, such that slice is used to select between the links in the ECMP set.

In one example, a hash that includes the slice ID can be used to choose which ECMP path corresponds to the packet. This approach can also allow for distribution of different slices across different links while the same slice operates on the same link. The hash can include source and destination IP and source and destination ports.

The hash can be used as an ECMP ID in one example, to select which link applies to the packet. For example, the ECMP table can include a column for ECMP ID and another column for egress interface or port, similar to Table <NUM>, which is discussed below. Additionally, a hash seed can be used for load balancing purposes. The hash seed can cause more slices to use one link than another, for example. This can allow one or more links to be preserved for high priority slices.

In one example, an ECMP operation can be modified at the switch to accommodate slice-based routing. Table <NUM>, below, includes pseudocode related to sliced-based use of an ECMP table, such of that of Table <NUM>, in an example.

Turning to the pseudocode of Table <NUM>, when a packet is received by the switch, the switch can determine the slice ID, as discussed for stage <NUM>. In this example, the agent can retrieve the slice ID from a slice table (e.g., Table <NUM>) based on source IP address ("sip"), destination IP address ("dip"), source port ("sport"), destination port ("dport"), and protocol type ("proto"). Then, the agent can get the next hop using the slice ID from the routing table, such as Table <NUM> or Table <NUM>, as described for stage <NUM>. This can include looking up a next hop address in the routing table based on, for example, slice ID and destination IP. The routing table can include a column for slice ID and another column for next hop.

In this example, at stage <NUM>, the switch can determine if the next hop is represented in the ECMP routing table (e.g., Table <NUM>), which can be used to determine which link (e.g., ECMP Member) to take to the next hop. For example, if the Subnet exists in Table <NUM> for the Dest IP of Table <NUM>, then the agent can use ECMP to determine which of the links (ECMP Members) to take to the next hop.

ECMP can be modified to use Slice ID to make this link determination. In one example, the agent retrieves or generates a hash. The hash (hash_id) can be created (create_hash) using slice ID, the packet destination IP address, and the ECMP member count, in an example. This hash ID can then be used to select which of the ECMP members applies to the packet.

The load balancing algorithm can determine which link to use for the slice when multiple links are available having the same (or similar within a threshold) preference and metric values. If there is an ECMP set for the active route, a load balancing algorithm can be used to choose one of the next-hop addresses in the ECMP to use for the packet, based on the slice ID. The load balancing algorithm can also preferentially treat one or more slices relative to others. For example, a first link can be reserved for a preferred slice when performance on that link falls below a threshold. For example, <NUM> calls can be the only slice using a particular link when multiple links are available to the next hop and the slice for <NUM> is prioritized. Slice prioritization can be determined at the switch or orchestrator based on service level agreement ("SLA") requirements for a slice. For example, each slice can have guaranteed performance characteristics, such as threshold levels of latency, throughput, and bandwidth. SLA records can explicitly assign prioritization parameters, in an example, that are used by the switch for load balancing purpose.

In one example, the orchestrator supplies the switch with a slice-based load balancing algorithm to use in ECMP routing. This can be based on slice priorities and slice routes, in an example.

For the purposes of this disclosure, selecting an egress interface can be the same as selecting an egress port.

Selecting an egress port at stage <NUM> can further include ensuring that the slice routes properly according to Layer <NUM> ("L2") of the OSI model, which can be referred to as the data link layer. In one example, the switch can implement a LAG using, for example, the link aggregation control protocol ("LACP"). A LAG can group together links between switches by bundling ports into a channel. LACP can be used with load balancing and ECMP, in an example. An example L2 forwarding table is shown below in Table <NUM>. Additionally, an example LAG table is shown in Table <NUM>.

In one example, the LAG can be modified to utilize a slice ID to select the port through which the packet will travel. This can include routing all packets that belong to the same slice down the same link.

Conventional approaches that do not use slice ID for LAG can result in out-of-order data delivery when the two different links are use, which can result in errors in data transmission. This is because the packets for a single slice could be distributed on different links, which can have different distances and travel times to the next hop (switch). For example, a first link can be one kilometer long while a second link is ten kilometers long. Incorporating slice ID into choosing the LAG link can ensure that the switch keeps the packets for the slice together. For example, slice ID can be used along with packet information (source IP address, destination IP address, source port, and destination port) to create a hash regarding which link to choose. The hash can be keyed on the overall number of available links and used as an index to select the link such that slices are distributed across the links.

Continuing with the pseudocode from Table <NUM>, if the next hop is not included in the ECMP routing table, then the agent can retrieve a lag interface for the next hop (e.g., getLAGIntf), such as the level <NUM> ("L2") forwarding table exemplified in Table <NUM> or the LAG table exemplified in Table <NUM>. In one example, the LAG ID is determined based on L2 information, such as destination MAC address ("Dest MAC") shown in Table <NUM>. The destination for the packet (information about the next hop) can be matched against Dest MAC to determine a LAG ID. Then, from the LAG table, the agent can retrieve the corresponding LAG Member Count and LAG member ports.

The agent can then determine which of the LAG Member Ports to use based on slice ID, ensuring that packets from the same slice stay together. This can include creating a hash that is used to select the LAG Member Port. The hash can be based on slice ID, destination IP address ("P->ip_id"), and the LAG member count, in an example that uses Table <NUM>. The egress interface can be determined by selecting the LAG Member Port by using the hash as an index or selector.

Then, the agent can send the packet P to the egress interface that was chosen through slice-based selection using ECMP or LAG.

ECMP and LAG can be used together for load balancing purposes, to increase bandwidth and throughput on the switch. By incorporating slice ID into the lookup for ECMP and LAG, the switch can keep data together for a slice and also select links based on slice prioritization. For example, ECMP can use a first link for a high priority slice and ensure that the congestion on that link remains low as part of load balancing. In this way, different slices can be balanced across the ports. Aggregation of links can likewise be based on slice prioritization, such as by forming a LAG that is primarily or solely used by a high priority slice.

At stage <NUM>, the switch sends the packet out into the network, to the next hop, using the egress interface (i.e., egress port) selected based on one or more of the routing table, ECMP, and LAG. The egress process can be governed by a policing algorithm that determines which packets are output through the egress ports first. In one example, the switch uses a FIFO algorithm, in which packets are routed to the egress queues in the order in which they are received in the slice-based pool. Alternatively, a slice round robin ("SRR") algorithm can be used, in which one packet from each slice is forwarded at a time. For example, the switch can take a packet from each slice-based ingress queue and route it to the appropriate interface into an egress queue for the same slice. In another example, a slice-weighted round robin algorithm ("SWRR") is used. In SWRR, packets can be forwarded based on a weight of each slice.

<FIG> is an example illustration of a sequence diagram for slice-based routing in a VSN. At stage <NUM>, an orchestrator sends slice path information to a switch. The switch can be programmable, such that the slice path information is used to update a slice table and other tables on the switch. For example, the slice table can indicate which egress interface to utilize for which slice. The slice table can also be used to determine a slice ID in some examples based on packet header information, such as source and destination IP addresses, MAC address, and ports.

At stage <NUM>, the switch can use the slice path information to update a multi-path table, such as an ECMP routing table or LAG table. In general, these can be referred to as routing tables. In one example, the routing table is slice-based, such as Table <NUM>. In one example, ECMP and LAG can be modified to operate based on slice ID. For example, a hash based on the slice ID for the packet can be used as an index to select the ECMP or LAG member to use as the egress interface. In another example, the ECMP routing table can be modified to also operate based on slice ID. This can allow traffic with the same source and destination to be transmitted across multiple paths of equal metrics, based on slice ID. For example, different slice IDs can correspond to the different paths such that slice data can be kept together on the same link of a LAG.

At stage <NUM>, policing and load balancing algorithms can be sent to the switch from the orchestrator. This can occur as part of stage <NUM> in an example. The load balancing algorithm can be used in conjunction with the multi-path table to determine which packets to place on which links (egress ports), based on slice ID. The policing algorithm can be used to determine the order of packets output into the network in times of congestion.

At stage <NUM>, the switch can receive ingress packets. These packets can be received on multiple ingress ports from multiple source locations. For each packet, the switch can determine the slice ID at stage <NUM>. In one example, the switch reads the slice ID from the packet header. An upstream switch or slice selector can write the slice ID into the packet header for use by other switches in the network. Alternatively, the switch can determine the slice ID by looking it up in a slice table maintained locally at the switch. To do this, the switch can use packet header information that will be unique to the slice, such as the combination of source and destination IP addresses, MAC addresses, and ports.

Based on the slice ID, the switch can then retrieve the next hop at stage <NUM>. A routing table can include next hop information for each slice. In one example, switch can check for ECMP routing. This can be a separate lookup at an ECMP table in one example. Alternatively, an ECMP process on the switch can update the routing table based on ECMP, allowing the switch to lookup the correct egress interface based on slice ID in the routing table. The egress interface can dynamically change based on load balancing for ECMP or based on changed characteristics of different links in a LAG. For example, if a particular link is not performing well, a load balancing algorithm can move a prioritized slice onto a different link. When the next hop is retrieved at stage <NUM>, the routing table can include updated information such that the packet for that slice is routed to the different link.

At stage <NUM>, the packet is sent into the network to the next hop. The order with which packets are sent into the network from the switch can be governed by a policing algorithm, in an example. The policing algorithm can be used to prioritize egress packets that belong to prioritized slices. This prioritization can escalate based on congestion. Congestion can be detected at the switch or detected downstream and reported back to the switch.

The next hop can be another switch where similar stages are performed. For example, the next switch can determine slice ID at stage <NUM>. Based on the slice ID, that switch determine a new next hop at stage <NUM>. The packets can continue through the network in this way until reaching a final destination, which can be slice-specific. For example, a slice can end at a call center, a database, a server, or other hardware that is used to carry out the purpose of the slice.

The switches can also be programmed by the orchestrator to execute an agent that reports telemetric data back to the orchestrator at stage <NUM>. The telemetric data can include any performance data related to latency, bandwidth, throughput, round-trip time, or any other performance metric of an SLA. The telemetric data can be used by the orchestrator or a related process to determine how to alleviate network congestion, in an example. This can include changing load balancing or policing algorithms at one or more switches, in an example. It can also include re-routing a slice in a network.

When the orchestrator makes a change, such as changing a slice path, that change can be reported to impacted switches at stage <NUM>. In the case of a new slice path, the switch can be programmed to indicate the correct next hop for use in that new slice path. In one example, the orchestrator can send a message to the switch that causes the switch to update one or more routing tables based on the new slice path. This can include determining whether the next hop is part of the existing ECMP and updating the routing tables accordingly. For example, if a first slice originally had a next hop at switch <NUM> in a first ECMP set but the updated next hop at switch <NUM> is not part of that ECMP set, then the ECMP routing table can be updated. The first set can be load balanced differently according to the additional available bandwidth. Additionally, if a second ECMP set connects to switch <NUM>, the ECMP routing table can be updated to include the slice in that set.

<FIG> an example illustration of system components used in routing packets without LAG. In this example, a host <NUM>, such as a server, sends packets to a first switch <NUM>. The first switch <NUM> can include multiple links <NUM> to a second switch <NUM> that is in the path to a destination host <NUM>. In this example, the first and second switches <NUM>, <NUM> are connected using four <NUM> Gbps interfaces, which comprise the links <NUM>. However, a network operating system ("OS") on the switches <NUM>, <NUM> can disable three of those <NUM> Gbps interfaces based on Layer <NUM> loops. A switching loop can occur in a network when there is more than one Layer <NUM> path between two nodes, such as the first and second switches <NUM>, <NUM>. These loops can create broadcast storms based on the switches <NUM>, <NUM> forwarding broadcasts at all the ports and Layer <NUM> packet headers not supporting a time to live ("TTL") value. Because having multiple links <NUM> can be attractive for redundancy reasons, loops are typically eliminated using a shortest-path bridging ("SPB") protocol. The downside of this approach is that the redundant links <NUM> remain inactive, indicated by x's in <FIG>. This means the overall bandwidth of the switch is less than optimal.

Bandwidth can be increased by using a LAG, such as with ECMP, as shown in <FIG> is an example illustration of system components for slice-based routing in a VSN. As shown in <FIG>, packets <NUM> coming into the first switch <NUM> can be routed evenly over the links <NUM> using a switch-to-switch LAG <NUM>. A load balancing algorithm on the first switch <NUM> can distribute the packets <NUM> across all of the links <NUM> between the switches <NUM>, <NUM>.

In one example, the packets <NUM> are distributed on the links <NUM> according to slice ID. In this example, the links <NUM> are bundled together into a single virtual link, but slice ID is used to maintain packets from the same slice on the same link. The link used for a particular slice can change in an example, based on load balancing. However, first and second packets with the same slice ID can be kept on the same link within the virtual channel. This can ensure that packets for a slice do not arrive out-of-order at the second switch <NUM>. This allows the second switch <NUM> to forward the packets <NUM> to the destination host <NUM> without introducing errors in a flow for a slice.

<FIG> is an example illustration of system components used for routing packets in different slice paths in a VSN. A first slice <NUM> can span switches R1, R3, R5, and R6. A second slice <NUM> can span switches R1, R2, R4, and R6. Both slices <NUM>, <NUM> span from San Francisco <NUM> to New York City <NUM> in this example, but take different routes across that span. The switches in this example can be routers. These switches can each calculate packet rate and timing information for each slice <NUM>, <NUM>, in an example.

Slice paths, routes through the VSN, can be changed based on an orchestration process. For example, prior to the illustration in <FIG>, both the first and second slices <NUM>, <NUM> can take the same slice path through the network, shown as the path of the first slice <NUM>. In that example, the second slice <NUM> originally has a slice path from San Francisco <NUM> to New York City <NUM> that traverses switches R3 and R5 (as currently shown for the first slice <NUM> in <FIG>). Based on congestion, the slice path for the second slice <NUM> can have been changed to use switches R2, R4 (as currently shown in <FIG>) instead of switch R3, R5. To do this, the orchestrator can update routing tables at switch R1 to change the next hop to switch R2 instead of switch R2. The routing tables at switches R2, R3, R4, and R5 can also be updated to reflect the new slice path. The orchestrator can also instantiate VNFs needed by the second slice at switches R2 and R4.

Updating the routing tables can include updating slice-based tables for ECMP and LAG. When the next hop changes, the ECMP sets can also change relative to which slices are in which sets. For example, at a first switch R1, a first ECMP set can include both the first and second slices <NUM>, <NUM>. However, when the first slice <NUM> is re-routed to follow a slice path along switches R2, R4, the first switch R1 can no longer include the first slice <NUM> in the first ECMP set. This is because the next hop is now switch R2 instead of switch R3. Therefore, the first switch R1 can remove the first slice <NUM> from the first ECMP set and instead place it in a second ECMP set that includes bundled links to switch R2. In one example, the switch is programmed to change its ECMP sets each time it receives new slice path information.

As previously mentioned, the orchestrator can change slice paths based on congestion. To detect congestion, the orchestrator can use telemetric data, such as the packet rate and timing information from the various switches. Using this data, the orchestrator can attempt to aggregate throughput for each slice <NUM>, <NUM>. However, switches R3 and R5 can have duplicate packet rate data for the first slice <NUM> going from San Francisco <NUM> to New York <NUM>, but switch R5 will also include non-duplicative and relevant packet rate data for the first slice <NUM> going from Miami <NUM> to New York <NUM>. Therefore, while the data from switch R3 should be ignored, the Miami <NUM> data at switch R5 should be included in calculating throughput for the first slice <NUM>, in an example. This is because the first slice <NUM> has two different illustrated flows, one from San Francisco <NUM> and another from New York City <NUM>.

Although an orchestrator is referred to as an example, a separate process that communicates with the orchestrator, such as a monitoring module, can perform the steps attributed to the orchestrator in the examples herein.

<FIG> is an example diagram of system components in a VSN <NUM>. The VSN <NUM> can be a distributed Telco cloud network with one or more clouds <NUM>, <NUM>. Slices <NUM>, <NUM>, <NUM> can be distributed across these clouds <NUM>, <NUM>.

Each cloud <NUM>, <NUM> can have physical and virtual infrastructure for network function virtualization ("NFV") <NUM>. For example, physical switches <NUM>, such as routers and servers, can run VMs <NUM> or microservices that provide VNF functionality. A slice can include a first VNF that executes on an edge cloud <NUM>. The VNF can utilize one or more vCPUs, which can be one or more VMs <NUM> in an example. However, the edge cloud <NUM> can execute numerous VNFs, often for multiple tenants where the VNFs are part of various slices. The slices can be kept separate from a functional perspective, with VNFs from different slices not aware of the existence of each other even when they rely on VMs <NUM> operating on shared physical hardware <NUM>.

A first VNF in the slice path can communicate with a second VNF, which can be located in a different cloud <NUM>. For example, the second VNF can include one or more VMs <NUM> operating on physical hardware <NUM> in a core cloud <NUM>. The second VNF can communicate with yet another VNF in the slice path. One or more of these VNFs can act as an egress to the internet <NUM>, in an example.

One or more user devices <NUM> can connect to a slice in the VNF <NUM> using, for example, a <NUM> data connection. The user devices <NUM> can be any physical processor-enabled device capable of connecting to a Telco network. Examples include cars, phones, laptops, tablets, IoT devices, virtual reality devices, and others. Cell towers <NUM> or other transceivers can send and receive transmissions with these user devices <NUM>. At the ingress point to edge clouds <NUM>, slice selectors <NUM> can receive data sent from the user devices <NUM> and determine which slice applies. The slice selectors <NUM> can operate as VMs <NUM> in the edge cloud or can run on different hardware connected to the edge cloud <NUM>. The slice selectors can use information in the packet headers to determine which slice the packets belong to, in an example.

In one example, a slice selector <NUM> initially processes the packets and assigns them to one of the network slices of the VSN. The slice selector <NUM> can also handle service chaining operations to ensure that the packets processed by the correct set of network services for the assigned slice. In various examples, the slice selector <NUM> can be implemented by a VM, a software forwarding element (e.g., a flow-based forwarding element) operating within a VM or within virtualization software of a host computer, a set of modules executing outside of a forwarding element (e.g., between a VM and a port of a forwarding element) within virtualization software of a host computer, among others.

In some cases, many slice selectors <NUM> are configured for a VSN. In a telecommunications service provider example, a network slice selector can be configured for each cell tower, base station, or other aspect of the access network. The telecommunications service provider access network can include edge clouds for each cell tower and configure at least one slice selector <NUM> at each such edge cloud. In other examples (e.g., for SD-WAN traffic entirely contained within a set of connected datacenters), distributed network slice selectors are configured such that the network slice selection for a data message sent from a VM occurs at the same host computer as the source of the data message (though outside of the source VM).

Slice selection can be based on information in the packet header for a packet. For example, a switch or slice selector can use a combination of layer <NUM> to layer <NUM> (L2-L4) headers or by performing deep packet inspection (e.g., to classify traffic based on data in the layer <NUM> (L7) header. For example, slice selection can be based simply on the source device by using the source network layer (e.g., IP) address, or can be based on the type of traffic or destination network domain by looking at the L7 header. In some embodiments, the network slice selector maintains state for mapping connections to network slices so that deep packet inspection does not need to be performed on each data message of a connection. In addition, for some connections, only certain data messages contain the L7 header information required for performing the slice selection.

To manage the distributed virtual infrastructure, a provider can run a topology <NUM> of management processes, including an orchestrator <NUM> having a monitoring module. The orchestrator <NUM> can alternately communicate with a monitoring module that runs separately on a different server or in a different virtual environment. In that example, the monitoring module can be part of the topology <NUM> that works with the orchestrator <NUM>. One example framework for these processes is VCLOUD NFV by VMWARE, which can use VSPHERE for network virtualization and VREALIZE for virtual analytics. An example orchestrator is CLOUDIFY.

The orchestrator <NUM> can be responsible for managing slices and VNFs, in an example. This can include provisioning new slices or re-provisioning existing slices based on performance metrics and network load. The orchestrator <NUM> can run on one or more physical servers located in one or more core clouds <NUM>, <NUM> or separate from the clouds. The orchestrator <NUM> can provide tools for keeping track of which clouds and VNFs are included in each slice. The orchestrator <NUM> can further track slice performance for individual tenants <NUM>, <NUM>, and provide a management console. The orchestrator <NUM> can also receive performance metrics and load information and determine when the monitoring module should find a new slice path.

In this example, a first tenant <NUM> has multiple slices <NUM>, <NUM>. Each slice <NUM>, <NUM> can be defined by a slice record that indicates VNF requirements for that slice. VNFs <NUM>, <NUM> can each provide different functionality in the service chain.

In addition, an SLA can specify various threshold performance requirements for the slices. These performance requirements can include latency, round-trip time, bandwidth, and others. These can serve as per-slice QoS requirements, in an example.

The orchestrator <NUM> can rely on the monitoring module to receive telemetric information from the switches <NUM>, <NUM> and determine if the SLA is satisfied. In one example, the monitoring module provides the switches <NUM>, <NUM> with an agent <NUM>. The switches <NUM>, <NUM> can be programmed to execute the agent <NUM>. The monitoring module can also supply user-defined policing algorithms that the switch uses to move packets from ingress ports <NUM> to egress ports <NUM>, and from egress ports <NUM> to the next hop in the physical network <NUM>. The monitoring module can also supply slice path information that the switches <NUM>, <NUM> use to determine next hops and which egress interfaces (e.g., ports) to use for those next hops.

The orchestrator <NUM> can also change settings in the slice selectors <NUM> and switches <NUM>, <NUM> to ensure traffic routes correctly down a slice path.

Claim 1:
A method for slice-based routing, comprising:
receiving a first packet (<NUM>) at a switch (<NUM>);
determining, at the switch (<NUM>), a slice identifier for the first packet based on header information of the first packet, wherein the slice identifier uniquely identifies a slice in a virtual service network ,VSN, and wherein slices represent different virtual networks on a shared physical network infrastructure;
determining, at the switch (<NUM>), a next hop (<NUM>) for the first packet; the method being characterised by selecting, by the switch, an egress interface from a plurality of interfaces based on the slice identifier, wherein the slice identifier is used to create a hash, and the hash is used to select the egress interface; and
sending the first packet from the egress interface to the next hop (<NUM>).