Patent Description:
A computer network is a collection of interconnected computing devices that can exchange data and share resources. In a packet-based network, such as an Ethernet network, the computing devices communicate data by dividing the data into variable-length blocks called packets, which are individually routed across the network from a source device to a destination device. The destination device extracts the data from the packets and assembles the data into its original form.

Certain devices, referred to as routers, maintain routing information representative of a topology of the network. The routers exchange routing information so as to maintain an accurate representation of available routes through the network. A "route" can generally be defined as a path between two locations on the network. Upon receiving an incoming data packet, a router examines information within the packet, often referred to as a "key," to select an appropriate next hop device to which to forward the packet in accordance with the routing information.

Routers may include one or more packet processors interconnected by an internal switch fabric. Packet processors exchange data with other external devices via interface cards. The switch fabric provides an internal interconnect mechanism for forwarding data within the router between the packet processors for ultimate transmission over a network. In some examples, a router or switching device may employ a distributed, multi-stage switch fabric architecture, in which network packets traverse multiple stages of the switch fabric located in distributed packet processors of the router to travel from an ingress point of the switch fabric to an egress point of the switch fabric. <CIT> relates to receive packet steering for virtual networks. <CIT> relates to secure forwarding of tenant workloads in virtual networks.

In general, the disclosure describes techniques for programming a forwarding plane of a network device with routes and forwarding nexthops that include metadata to cause the forwarding plane to load balance or otherwise direct packet flows to particular central processing unit (CPU) cores among a plurality of CPU cores. In some examples, a network device may include a set of one or more CPUs, each CPU of the set of CPUs including a set of one or more CPU cores. The network device may receive and forward network traffic (e.g., packets) that corresponds to any one or more of a set of services. The network device may use the plurality of CPU cores across the set of CPUs to process the network traffic. In some examples, the network device may route a packet to a particular CPU or CPU core based on one or more applications and/or services (e.g., messaging applications, email applications, video streaming applications, Internet Protocol Security (IPsec), and Carrier Grade Network Address Translation (CG-NAT) services) associated with the packet. In some examples, the packet may include a packet header which identifies one or more applications and/or services associated with the packet.

IPsec represents a communication protocol which allows an encryption and a decryption of one or more data packets for transmission between two or more devices within a network. For example, a first network device may process one or more packets associated with an IPsec "session" between the first network device and a second network device. In some examples, processing the one or more packets includes encrypting one or more packets for forwarding to the second network device and decrypting one or more packets received from the second network device, however this is not required. The first network device may, in some cases, act as an intermediary device to facilitate an IPsec session between two devices. The first network device may route packets associated with an IPsec session to at least one of a plurality of CPU cores for processing.

A network device may include a control plane for controlling one or more functions of the network device, a forwarding plane for receiving and forwarding network traffic, and a service plane for processing network traffic in order to perform one or more services. The control plane may also apply services in some cases. In some examples, the forwarding plane includes a forwarding path having one or more forwarding path elements (e.g., forwarding nexthops), where the one or more forwarding path elements are configured to route incoming packets through the forwarding path and through the service plane, in some examples. Additional description of forwarding nexthops and other forwarding path structures is found in <CIT>. The service plane may include a set of service cards, each service card of the set of service cards including a CPU having a set of CPU cores. The control unit may configure the forwarding path to route the packet to one of the set of service cards in the service plane, and route the packet to one of the CPU cores in the respective service card for processing.

The techniques described herein provide one or more technical improvements having at least one practical application. For example, it may be beneficial for the network device to route each packet associated with a particular application and/or service to the same CPU core for processing. Additionally, it may be beneficial for the control plane of the network device to program the forwarding plane of the network device to more efficiently distribute a plurality of packets across the plurality of CPU cores of the network device as compared with techniques in which the control plane does not program the forwarding plane in such a manner. In some examples, the control plane may program the forwarding plane such that the forwarding plane routes each packet associated with an IPsec session to the same CPU core or set of CPU cores. In some examples, the control plane may program the forwarding plane such that the forwarding plane routes each packet associated with a service and/or and application to the same CPU core or set of CPU cores.

In some examples, a network device includes a control unit including processing circuitry in communication with a memory, where the processing circuitry is configured to execute one or more processes; and a forwarding unit including an interface card, a packet processor, and a forwarding unit memory, where the one or more processes of the control unit are configured to configure the forwarding unit memory of the forwarding unit with one or more forwarding path elements, where the one or more forwarding path elements map a packet flow to a central processing unit (CPU) core of a plurality of CPU cores for processing, where the forwarding unit is configured to receive, via the interface card, a packet of the packet flow, where the forwarding unit further comprises a plurality of output queues for each of the plurality of CPU cores, where each of the plurality of output queues for a given CPU core is associated with a different priority level where packets in higher priority queues are configured to be forwarded before packets in lower priority queues, and where the packet processor of the forwarding unit is configured to execute the one or more forwarding path elements in the forwarding unit memory to steer the packet to the CPU core by adding the packet to one of the plurality of output queues for the CPU core.

In some examples, a method includes executing, by processing circuitry of a control unit, one or more processes, where the processing circuitry is in communication with a memory; configuring, by the one or more processes of the control unit, a forwarding unit memory of a forwarding unit with one or more forwarding path elements, where the forwarding unit includies an interface card, a packet processor, and the forwarding unit memory, and where the one or more forwarding path elements map a packet flow to a central processing unit (CPU) core of a plurality of CPU cores for processing, and where the forwarding unit further comprises a plurality of output queues (<NUM>, <NUM>) for each of the plurality of CPU cores, where each of the plurality of output queues for a given CPU core is associated with a different priority level where packets in higher priority queues are configured to be forwarded before packets in lower priority queues; receiving, by the forwarding unit via the interface card, a packet of the packet flow; and executing, by the packet processor, the one or more forwarding path elements in the forwarding unit memory to steer the packet to the CPU core by adding the packet to the output queue for the CPU core.

In some examples, a computer-readable medium includes instructions for causing one or more programmable processors of a network device to: execute one or more processes of a control plane; configure a forwarding unit memory of a forwarding unit with one or more forwarding path elements, where the forwarding unit includes an interface card, a packet processor, and the forwarding unit memory, and where the forwarding path elements map a packet flow to a central processing unit (CPU) core of a plurality of CPU cores, and wherein the forwarding unit further comprises a plurality of output queues for each of the plurality of CPU cores, wherein each of the plurality of output queues for a given CPU core is associated with a different priority level where packets in higher priority queues are configured to be forwarded before packets in lower priority queues; receive a packet of the packet flow; and execute the one or more forwarding path elements in the forwarding unit memory to steer the packet to the CPU core for processing by the CPU core by adding the packet to the output queue for the CPU core.

Like reference characters refer to like elements throughout the text and figures.

<FIG> is a block diagram illustrating an example network system <NUM> including a router <NUM> for load-balancing network traffic across a plurality of Central Processing Unit (CPU) cores, in accordance with one or more techniques of this disclosure. Router <NUM> may include, in some examples, a control plane, a forwarding plane and a service plane The example network system <NUM> of <FIG> provides packet-based network services to subscriber devices <NUM>. That is, network system <NUM> provides authentication and establishment of network access for subscriber devices <NUM> such that a subscriber device may begin exchanging data packets with public network <NUM>, which may be an internal or external packet-based network such as the Internet.

In the example of <FIG>, network system <NUM> includes access network <NUM> that provides connectivity to public network <NUM> via service provider software-defined wide area network <NUM> (hereinafter, "SD-WAN <NUM>") and router <NUM>. SD-WAN <NUM> and public network <NUM> provide packet-based services that are available for request and use by subscriber devices <NUM>. As examples, SD-WAN <NUM> and/or public network <NUM> may provide bulk data delivery, voice over Internet protocol (VoIP), Internet Protocol television (IPTV), Short Messaging Service (SMS), Wireless Application Protocol (WAP) service, or customer-specific application services. Public network <NUM> may include, for instance, a local area network (LAN), a wide area network (WAN), the Internet, a virtual LAN (VLAN), an enterprise LAN, a layer <NUM> virtual private network (VPN), an Internet Protocol (IP) intranet operated by the service provider that operates access network <NUM>, an enterprise IP network, or some combination thereof. In various examples, public network <NUM> is connected to a public WAN, the Internet, or to other networks. Public network <NUM> executes one or more packet data protocols (PDPs), such as IP (IPv4 and/or IPv6), X. <NUM> or Point-to-Point Protocol (PPP), to enable packet-based transport of public network <NUM> services.

In general, subscriber devices <NUM> connect to gateway router <NUM> via access network <NUM> to receive connectivity to subscriber services for applications hosted by public network <NUM> or data center <NUM>. A subscriber may represent, for instance, an enterprise, a residential subscriber, or a mobile subscriber. Subscriber devices <NUM> may be, for example, personal computers, laptop computers or other types of computing devices positioned behind customer equipment (CE) <NUM>, which may provide local routing and switching functions. Each of subscriber devices <NUM> may run a variety of software applications, such as word processing and other office support software, web browsing software, software to support voice calls, video games, video conferencing, and email, among others. For example, subscriber device <NUM> may be a variety of network-enabled devices, referred generally to as "Internet-of-Things" (IoT) devices, such as cameras, sensors (S), televisions, appliances, etc. In addition, subscriber devices <NUM> may include mobile devices that access the data services of Network system <NUM> via a radio access network (RAN) <NUM>. Example mobile subscriber devices include mobile telephones, laptop or desktop computers having, e.g., a cellular wireless card, wireless-capable netbooks, video game devices, pagers, smart phones, personal data assistants (PDAs) or the like.

A network service provider operates, or in some cases leases, elements of access network <NUM> to provide packet transport between subscriber devices <NUM> and router <NUM>. Access network <NUM> represents a network that aggregates data traffic from one or more of subscriber devices <NUM> for transport to/from SD-WAN <NUM> of the service provider. Access network <NUM> includes network nodes that execute communication protocols to transport control and user data to facilitate communication between subscriber devices <NUM> and router <NUM>. Access network <NUM> may include a broadband access network, a wireless LAN, a public switched telephone network (PSTN), a customer premises equipment (CPE) network, or other type of access network, and may include or otherwise provide connectivity for cellular access networks, such as a radio access network (RAN) (not shown). Examples include networks conforming to a Universal Mobile Telecommunications System (UMTS) architecture, an evolution of UMTS referred to as Long Term Evolution (LTE), mobile IP standardized by the Internet Engineering Task Force (IETF), as well as other standards proposed by the <NUM>rd Generation Partnership Project (3GPP), <NUM>rd Generation Partnership Project <NUM> (3GGP/<NUM>) and the WiMAX forum.

Router <NUM> may be a customer edge (CE) router, a provider edge (PE) router, or other network device between access network <NUM> and SD-WAN <NUM>. SD-WAN <NUM> offers packet-based connectivity to subscriber devices <NUM> attached to access network <NUM> for accessing public network <NUM> (e.g., the Internet). SD-WAN <NUM> may represent a public network that is owned and operated by a service provider to interconnect a plurality of networks, which may include access network <NUM>. In some examples, SD-WAN <NUM> may implement Multi-Protocol Label Switching (MPLS) forwarding and in such instances may be referred to as an MPLS network or MPLS backbone. In some instances, SD-WAN <NUM> represents a plurality of interconnected autonomous systems, such as the Internet, that offers services from one or more service providers. Public network <NUM> may represent the Internet. Public network <NUM> may represent an edge network coupled to SD-WAN <NUM> via a transit network <NUM> and one or more network devices, e.g., a customer edge device such as customer edge switch or router. Public network <NUM> may include a data center. Router <NUM> may exchange packets with service nodes 10A-10N (collectively, "service nodes <NUM>") via virtual network <NUM>, and router <NUM> may forward packets to public network <NUM> via transit network <NUM>.

In examples of network system <NUM> that include a wireline/broadband access network, router <NUM> may represent a Broadband Network Gateway (BNG), Broadband Remote Access Server (BRAS), MPLS PE router, core router or gateway, or Cable Modem Termination System (CMTS). In examples of network system <NUM> that include a cellular access network as access network <NUM>, router <NUM> may represent a mobile gateway, for example, a Gateway General Packet Radio Service (GPRS) Serving Node (GGSN), an Access Gateway (aGW), or a Packet Data Network (PDN) Gateway (PGW). In other examples, the functionality described with respect to router <NUM> may be implemented in a switch, service card or another network element or component. In some examples, router <NUM> may itself be a service node.

A network service provider that administers at least parts of network system <NUM> typically offers services to subscribers associated with devices, e.g., subscriber devices <NUM>, that access Network system <NUM>. Services offered may include, for example, traditional Internet access, VoIP, video and multimedia services, and security services such as Internet Protocol Security (IPsec). As described above with respect to SD-WAN <NUM>, SD-WAN <NUM> may support multiple types of access network infrastructures that connect to service provider network access gateways to provide access to the offered services. In some instances, the network system may include subscriber devices <NUM> that attach to multiple different access networks <NUM> having varying architectures.

In general, any one or more of subscriber devices <NUM> may request authorization and data services by sending a session request to a gateway device such as Router <NUM> or router <NUM>. In turn, router <NUM> may access a central server (not shown) such as an Authentication, Authorization and Accounting (AAA) server to authenticate the one of subscriber devices <NUM> requesting network access. Once authenticated, any of subscriber devices <NUM> may send subscriber data traffic toward SD-WAN <NUM> to access and receive services provided by public network <NUM>, and such packets may traverse router <NUM> as part of at least one packet flow. In some examples, Router <NUM> may forward all authenticated subscriber traffic to public network <NUM>, and router <NUM> may apply services <NUM> and/or steer particular subscriber traffic to a data center <NUM> if the subscriber traffic requires services on service nodes <NUM>. Applications (e.g., service applications) to be applied to the subscriber traffic may be hosted on service nodes <NUM>.

For example, when forwarding subscriber traffic, router <NUM> may direct individual subscriber packet flows through services <NUM> executing on a set service cards installed within router <NUM>. In some examples, service cards may be referred to herein as "field-replaceable units (FRUs). " Each service card of the set of service cards installed within router <NUM> may include a Central Processing Unit (CPU) including a set of CPU cores in the service plane for deep packet processing of network traffic. In some examples, the set of service cards may be a part of a "service plane" of router <NUM>. Router <NUM> may also include a control plane and a forwarding plane. The control plane may include one or more virtual machines (VMs) executed by processors, where the one or more VMs are configured to program the forwarding plane to route network traffic such as packets to one or more CPU cores of the service plane for processing. Subsequently, the forwarding plane of router <NUM> may forward the network traffic to a respective destination device (e.g., one of subscriber devices <NUM>).

Network system <NUM> may include a data center <NUM> having a cluster of service nodes <NUM> that provide an execution environment for the mostly virtualized network services. In some examples, each of service nodes <NUM> represents a service instance. Each of service nodes <NUM> may apply one or more services to traffic flows. As such, router <NUM> may steer subscriber packet flows through defined sets of services provided by service nodes <NUM>. That is, in some examples, each subscriber packet flow may be forwarded through a particular ordered combination of services provided by service nodes <NUM>, each ordered set being referred to herein as a "service chain. " As examples, services <NUM> and/or service nodes <NUM> may apply stateful firewall (SFW) and security services, deep packet inspection (DPI), carrier grade network address translation (CGNAT), traffic destination function (TDF) services, media (voice/video) optimization, Internet Protocol security (IPSec)/virtual private network (VPN) services, hypertext transfer protocol (HTTP) filtering, counting, accounting, charging, and/or load balancing of packet flows, or other types of services applied to network traffic.

In the example of <FIG>, subscriber packet flows may be directed along a service chain that includes any of services <NUM> and/or services applied by service nodes <NUM>. Once processed at a terminal node of the service chain, i.e., the last service to be applied to packets flowing along a particular service path, the traffic may be directed to public network <NUM>.

Whereas a "service chain" defines one or more services to be applied in a particular order to provide a composite service for application to packet flows bound to the service chain, a "service tunnel" or "service path" refers to a logical and/or physical path taken by packet flows processed by a service chain along with the forwarding state for forwarding packet flows according to the service chain ordering. Each service chain may be associated with a respective service tunnel, and packet flows associated with each subscriber device <NUM> flow along service tunnels in accordance with a service profile associated with the respective subscriber. For example, a given subscriber may be associated with a particular service profile, which in turn is mapped to a service tunnel associated with a particular service chain. Similarly, another subscriber may be associated with a different service profile, which in turn is mapped to a service tunnel associated with a different service chain. In some examples, after router <NUM> has authenticated and established access sessions for the subscribers, router <NUM> or router <NUM> may direct packet flows for the subscribers along the appropriate service tunnels, thereby causing data center <NUM> to apply the requisite ordered services for the given subscriber.

In some examples, service nodes <NUM> may implement service chains using internally configured forwarding state that directs packets of the packet flow along the service chains for processing according to the identified set of service nodes <NUM>. Such forwarding state may specify tunnel interfaces for tunneling between service nodes <NUM> using network tunnels such as IP or Generic Route Encapsulation (GRE) tunnels, Network Virtualization using GRE (NVGRE), or by using VLANs, Virtual Extensible LANs (VXLANs), MPLS techniques, and so forth. In some instances, real or virtual switches, routers or other network elements that interconnect service nodes <NUM> may be configured to direct the packet flow to the service nodes <NUM> according to service chains.

Although illustrated as part of data center <NUM>, service nodes <NUM> may be network devices coupled by one or more switches or virtual switches of SD-WAN <NUM>. In one example, each of service nodes <NUM> may run as VMs in a virtual compute environment. Moreover, the compute environment may include a scalable cluster of general computing devices, such as x86 processor-based servers. As another example, service nodes <NUM> may include a combination of general purpose computing devices and special purpose appliances. As virtualized network services, individual network services provided by service nodes <NUM> can scale just as in a modern data center through the allocation of virtualized memory, processor utilization, storage and network policies, as well as horizontally by adding additional load-balanced VMs. In other examples, service nodes <NUM> may be gateway devices or other routers. In further examples, the functionality described with respect to each of service nodes <NUM> may be implemented in a switch, service card, or another network element or component.

In accordance with techniques described herein, one or more processes executing on a control plane of router <NUM> may generate data including instructions for configuring one or more forwarding path elements of a forwarding path which represents a part of the forwarding plane of router <NUM>. In some examples, the processes may generate the data for configuring the one or more forwarding path elements based on user input. In some examples, the processes may automatically generate the data for configuring the one or more forwarding path elements based on resource data (e.g., current CPU availability and/or current CPU usage) within router <NUM>. Subsequently, the control plane processes of router <NUM> may configure the one or more forwarding path elements based on the instructions in order to allow the forwarding path to steer, based on the packet header, the packet to a particular CPU core of a plurality of CPU cores for processing. Additionally, in some cases, the control plane processes may save, to a memory, a current configuration of the plurality of forwarding path elements after the processing circuitry configures the one or more forwarding path elements based on the instructions.

In some examples, the plurality of forwarding path elements of router <NUM> include a first nexthop element and a second nexthop element, and router <NUM> is configured to identify, in the instructions for configuring the one or more forwarding path elements, an indication of one or more services corresponding to each service card of a set of service cards located in a service plane of router <NUM>. Router <NUM> may configure the first nexthop element to forward a packet to a service card of the set of service cards based on a service associated with the packet, which is identified in a packet header. In this way, when the control plane of router <NUM> configures the first nexthop element, the forwarding unit may read the packet header of the packet to identify a service, and the first nexthop element may forward the packet to a service card of the set of service cards based on the service identified by the packet header.

Each service card of the set of service cards may include a set of CPU cores, and router <NUM> may be configured to steer network traffic to specific CPU cores based on services and/or applications associated with the respective network traffic. Additionally, the control plane of router <NUM> may program the second nexthop element to steer a packet to a particular CPU core of the set of CPU cores based on one or more services and/or applications identified by the respective packet header. In this way, the control plane of router <NUM> may configure the forwarding path of router <NUM> in order designate one or more CPU cores of the plurality of CPU cores for processing packets associated with each respective service of a set of services. In some examples, the control plane may configure the first nexthop element and the second nexthop element to steer packets associated with a first IPsec session to a first CPU core, steer packets associated with a second IPsec session to a second CPU core, steer packets associated with a third IPsec session to a third CPU core, and so on. IPsec represents a communication protocol which allows an encryption and a decryption of one or more data packets for transmission between two or more devices within a network by creating a secure tunnel between two or more endpoints. It may be beneficial for one CPU core to process all packets associated with one IPsec session in order to improve an efficiency as compared with techniques in which more than one CPU core is used to process packets associated with one IPsec session. The packet header may identify an IPsec session associated with the packet.

<FIG> is a block diagram illustrating an example network device <NUM> configured to program one or more forwarding path elements to route a packet to one or more CPU cores of a plurality of CPU cores 130A-130N, in accordance with the techniques of this disclosure. While network device <NUM> may be any network device configured to perform the techniques described herein, network device <NUM> may be an example of Router <NUM> of <FIG> or Router <NUM> of <FIG>. Network device <NUM> may be described herein within the context of Network system <NUM> of <FIG>. Moreover, while described with respect to a particular network device, e.g., a router, the techniques may be implemented by any network device, such as a client device, a Layer <NUM> (L3) or L2/L3 switch, or server.

In this example, network device <NUM> is divided into three logical or physical "planes" to include a control plane <NUM> that performs control operations for the device, a forwarding plane <NUM> for forwarding transit network traffic and a service plane <NUM> for application of one or more network services <NUM> to transit packet flows that are forwarded by the router. That is, network device <NUM> implements three separate functionalities (e.g., the routing/control functionalities, forwarding data functionalities, and network service functionalities), either logically, e.g., as separate software instances executing on the same set of hardware components, or physically, e.g., as separate physical dedicated hardware components that either statically implement the functionality in hardware or dynamically execute software or a computer program to implement the functionality. In this example, a high-speed internal switch fabric <NUM> couples control plane <NUM>, service plane <NUM>, and forwarding plane <NUM> to deliver data units and control messages among the units. Switch fabric <NUM> may represent an internal switch fabric or crossbar, bus, or link.

Control plane <NUM> includes control unit <NUM> having processing circuitry <NUM>, which executes device management services, subscriber authentication and control plane routing functionality of network device <NUM>. Additionally, control unit <NUM> includes VMs 90A-90N (collectively, "VMs <NUM>") and routing engine <NUM> which are executed by processing circuitry <NUM>. Each of VMs <NUM> may be an example of a control plane process. Routing engine <NUM> includes routing information <NUM> and CPU information <NUM>. Forwarding plane <NUM> includes forwarding unit <NUM> which receives and outputs network traffic (e.g., packets) via interface cards 114A-114N (collectively, "IFCs <NUM>"). For example, IFCs <NUM> receive network traffic via inbound links 116A-116N (collectively, "inbound links <NUM>") and output network traffic via outbound links 118A-118N (collectively, "outbound links <NUM>"). Additionally, forwarding unit <NUM> includes packet processor <NUM> and forwarding path <NUM>. Forwarding path <NUM> includes forwarding path elements 124A-124N (collectively, "forwarding path elements <NUM>"). Service plane <NUM> includes service units 126A-126N (collectively, "service units <NUM>"). In some examples, a service unit (e.g., service unit 126A) represents a service card that may be added to and/or removed from network device <NUM>. In some examples, service units <NUM> may be referred to herein as "field-replaceable units (FRUs). " Service unit 126A includes, for example, microprocessor <NUM> which is configured to execute hypervisor <NUM> and services <NUM>.

Although illustrated and described herein primarily with respect to insertable service cards, the techniques may apply to directing network packets to other types of service units, including such as real or virtual servers. Service nodes <NUM> of <FIG> that are external to router <NUM> and router <NUM> may represent examples of service units, for instance.

In the example of <FIG>, processing circuitry <NUM> which executes device management services, subscriber authentication and control plane routing functionality of network device <NUM>. Processing circuitry <NUM> may include, for example, microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate array (FPGAs), or equivalent discrete or integrated logic circuitry, or a combination of any of the foregoing devices or circuitry. Accordingly, the terms "processor" or "controller," as used herein, may refer to any one or more of the foregoing structures or any other structure operable to perform techniques described herein. Executables, such as VMs 90A-90N (collectively, "VMs <NUM>") and routing engine <NUM> including routing information <NUM> and CPU information <NUM>, may be operable by processing circuitry <NUM> to perform various actions, operations, or functions of network device <NUM>. For example, processing circuitry <NUM> of network device <NUM> may retrieve and execute instructions stored by various data stores that cause processing circuitry <NUM> to perform the operations of VMs <NUM> and routing engine <NUM>.

One or more storage components within network device <NUM> may store information for processing during operation of network device <NUM> (e.g., network device <NUM> may store data accessed by VMs <NUM>, routing engine <NUM>, and services <NUM> during execution at network device <NUM>). In some examples, the storage component is a temporary memory, meaning that a primary purpose of the storage component is not long-term storage. Storage components on network device <NUM> may be configured for short-term storage of information as volatile memory and therefore not retain stored contents if powered off. Examples of volatile memories include random access memories (RAM), dynamic random access memories (DRAM), static random access memories (SRAM), and other forms of volatile memories known in the art.

Storage components, in some examples, also include one or more computer-readable storage media. Storage components in some examples include one or more computer-readable storage mediums. Storage components may be configured to store larger amounts of information than typically stored by volatile memory. Storage components may further be configured for long-term storage of information as non-volatile memory space and retain information after power on/off cycles. Examples of non-volatile memories include magnetic hard discs, optical discs, floppy discs, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories. Storage components may store program instructions and/or information (e.g., data) associated with VMs <NUM>, routing engine <NUM>, and services <NUM>. The storage components may include a memory configured to store data or other information associated with VMs <NUM>, routing engine <NUM>, and services <NUM>.

In general, control unit <NUM> includes a routing engine <NUM> configured to communicate with a forwarding unit <NUM> and, in some cases, other forwarding units of network devices not illustrated in <FIG>. Routing engine <NUM> may, in some cases, represent control plane management of packet forwarding throughout network device <NUM>. For example, Network device <NUM> includes interface cards 114A-114N (collectively, "IFCs <NUM>") that receive packets via inbound links and send packets via outbound links. IFCs <NUM> typically have one or more physical network interface ports. In some examples, after receiving a packet via IFCs <NUM>, network device <NUM> uses forwarding unit <NUM> to forward the packet to a next destination based on operations performed by routing engine <NUM>. In this way, routing engine <NUM> may provide control plane functionality to network device <NUM>. Routing engine <NUM> may include information corresponding to one or both of service plane <NUM> and forwarding plane <NUM>. Routing engine <NUM> may include routing information <NUM> which indicates a current configuration of one or more forwarding path elements of packet processor <NUM>, for example. Additionally, routing engine <NUM> includes CPU information <NUM>, which indicates respective current utilizations of one or more CPU cores of CPU cores <NUM>.

Routing engine <NUM> may provide an operating environment for various protocols (not illustrated in <FIG>) that execute at different layers of a network stack. Routing engine <NUM> may be responsible for the maintenance of routing information <NUM> to reflect the current configuration of packet processor <NUM>. In particular, routing protocols periodically update routing information <NUM> to accurately reflect the current configuration of Forwarding plane <NUM> based on metadata generated by network device VMs <NUM>, for example. The protocols may be software processes executing on processing circuitry <NUM>. In this way, routing engine <NUM> may occupy a group of compute nodes in processing circuitry <NUM> such that the group of compute nodes are not available to execute VMs such as VMs <NUM>. For example, routing engine <NUM> may include bridge port extension protocols, such as IEEE <NUM>. Routing engine <NUM> may also include network protocols that operate at a network layer of the network stack. In the example of <FIG>, network protocols may include one or more control and routing protocols such as border gateway protocol (BGP), interior gateway protocol (IGP), label distribution protocol (LDP) and/or resource reservation protocol (RSVP). In some examples, the IGP may include the open shortest path first (OSPF) protocol or the intermediate system-to-intermediate system (IS-IS) protocol. Routing engine <NUM> also may include one or more daemons that include user-level processes that run network management software, execute routing protocols to communicate with peer routers or switches, maintain and update one or more routing tables, and create one or more forwarding tables for installation to forwarding unit <NUM>, among other functions.

Routing information <NUM> may include, for example, route data that describes various routes within network device <NUM> and within network system <NUM>, and corresponding next hop data. For example, routing information <NUM> may include information indicating a packet destination element of packet processor <NUM> following a specific nexthop element of packet processor <NUM>, where the destination element depends on a packet header of the packet. Network device <NUM> updates routing information <NUM> based on metadata generated by VMs <NUM> for programming packet processor <NUM>. Based on routing information <NUM>, routing engine <NUM>, executing on processing circuitry <NUM>, may generate forwarding information (not illustrated in <FIG>) and output the forwarding information to forwarding unit <NUM> in order to program one or more forwarding path elements <NUM> of packet processor <NUM>. The forwarding information, in some examples, associates one or more CPU cores pf CPU cores <NUM> with specific services associated with incoming packets. For example, Routing engine <NUM> may generate the forwarding information based on metadata generated by VMs <NUM> and output the forwarding information in order to program packet processor <NUM> to steer a packet associated with a particular service to one or more CPU cores of CPU cores <NUM> for processing. In some examples, the processing of the packet by the one or more CPU cores may include full packet encryption and/or full packet decryption. Forwarding unit <NUM> may identify the service associated with the packet by reading the packet header of the packet.

Forwarding plane <NUM>, in this example includes forwarding unit <NUM> configured to perform packet forwarding functionality. In the example of network device <NUM>, forwarding plane <NUM> includes forwarding unit <NUM> that provides high-speed forwarding of network traffic received by IFCs <NUM> via inbound links <NUM> and output via outbound links <NUM>. Forwarding unit <NUM> may include packet processor <NUM> which is coupled to IFCs <NUM>. Packet processor <NUM> may represent one or more packet forwarding engines ("PFEs") including, for example, a dense port concentrator (DPC), modular port concentrator (MPC), flexible physical interface card (PIC) concentrator (FPC), or another line card, for example, that is insertable within a chassis or combination of chassis of network device <NUM>.

In one example, forwarding path <NUM> arranges forwarding path elements <NUM> as next hop data that can be chained together as a series of "hops" along an internal packet forwarding path for the network device. In many instances, forwarding path elements <NUM> perform lookup operations within internal memory of forwarding unit <NUM>, where the lookup may be performed against a tree (or trie) search, a table (or index) search. Other example operations that may be specified with the next hops include filter determination and application, or a rate limiter determination and application. Lookup operations locate, within a lookup data structure (e.g., a lookup tree), an item that matches packet contents or another property of the packet or packet flow, such as the inbound interface of the packet. The result of packet processing in accordance with the operations defined by the next hop forwarding structure within forwarding path <NUM> determines the manner in which a packet is forwarded or otherwise processed by forwarding unit <NUM> from its input interface on one of IFCs <NUM> to its output interface on one of IFCs <NUM>.

Service plane <NUM> of network device <NUM> includes a plurality of service units <NUM> that may be, as examples, removable service cards, which are configured to apply network services to packets flowing through forwarding plane <NUM>. Service units <NUM> may include FRUs in some examples. That is, when forwarding packets, forwarding units <NUM> may steer packets to service plane <NUM> for application of one or more network services <NUM> by service units <NUM>. In this example, each of service units <NUM> includes a microprocessor configured to execute a hypervisor to provide an operating environment for a plurality of network services. For example, service unit <NUM> includes microprocessor <NUM> configured to execute hypervisor <NUM> to provide an operating environment for network services <NUM>. As examples, service units <NUM> may apply firewall and security services, carrier grade network address translation (CG-NAT), media optimization (voice/video), IPSec/VPN services, deep packet inspection (DPI), HTTP filtering, counting, accounting, charging, and load balancing of packet flows or other types of services applied to network traffic. Each of services <NUM> may be implemented, for example, as virtual machines or containers executed by hypervisor <NUM> and microprocessor <NUM>. In some examples, service plane <NUM> may be configured to execute services <NUM> more efficiently as compared with forwarding plane <NUM>.

In some examples, forwarding unit <NUM> of network device <NUM> is configured to receive one or more packets, each packet of the one or more packets being associated with one or more services. For example, each packet of the one or more packets may be associated with one or more network services <NUM> executed by microprocessor <NUM> of service unit 126A. In one example, IFC 114A of forwarding unit <NUM> may receive a packet via inbound link 116A. Subsequently, the packet may travel through forwarding path <NUM>. In some examples, the packet may include a header. Forwarding unit <NUM> may read the header of the packet. The header of the packet may include information which indicates a number of details associated with the packet such as, for example, a device which network device <NUM> receives the packet from, a device which network device <NUM> is to forward the packet to, and one or more services (e.g., one or more of network services <NUM> and/or one or more other services not illustrated in <FIG>) associated with the packet. Forwarding unit may identify the packet header, and process the packet header in order to determine the information indicated by the packet header. In other words, forwarding unit <NUM> may process the packet header in order to determine the one or more services associated with the packet.

Packet processor <NUM> processes the packet according to forwarding path elements <NUM> along forwarding path <NUM>. Forwarding path elements <NUM> represent a logical flow which forms a set of "paths" for packets being processed according to forwarding path <NUM>. A packet may travel through the paths formed by forwarding path elements <NUM> based on data included in the header of the packet, in some cases. Control plane <NUM> (e.g., VMs <NUM> and/or routing engine <NUM>) may generate forwarding path elements <NUM> to map the packet based on the packet header. In some examples, forwarding path elements <NUM> may cause forwarding unit <NUM> to steer the packet to one or more other components of network device <NUM> for processing, such as service units <NUM>. For example, in order to apply a service to a packet, a forwarding path element of forwarding path elements <NUM> may map, based on the packet header of a packet, the packet to a service unit of service units <NUM> for processing by a CPU core located on the respective service unit.

In accordance with techniques described herein, forwarding path elements <NUM> map packet flows to respective, particular CPU cores <NUM> and/or to respective, particular service units <NUM>. For example, one of forwarding path elements <NUM> may map a first packet flow, corresponding to a first service, to CPU core 130B. As such packet processor <NUM> processes packets of the first packet flow with the forwarding path element <NUM> to direct the packets to CPU core 130B for processing. In some examples, the forwarding path elements <NUM> map packet flows between a source and destination, in the both the uplink and downlink direction, to the same one of CPU cores <NUM> to facilitate processing by that CPU core <NUM> of the packet flows in both the uplink and downlink direction.

Control plane <NUM> generates and downloads, to forwarding unit <NUM>, forwarding path elements <NUM> that include metadata to map packet flows to particular CPU cores <NUM> and/or particular service units <NUM>. The metadata may, for instance, parameterize forwarding next hops of forwarding path elements <NUM> with identifiers for CPU cores <NUM> and/or service units <NUM> such that the forwarding path elements <NUM>, when executed, cause packet processor <NUM> to steer matching packets to a particular CPU core <NUM>. Metadata may include, for instance, link identifier, tunnel (e.g., IPSec) identifier, or n-tuple data for matching to a packet. Metadata may also include, for instance, data identifying a lookup table for determining a service unit, data identifying a lookup table for determining a CPU core, or other data to map matching packets to a particular service unit <NUM> or CPU core <NUM>.

Forwarding path elements <NUM> may, in some cases, include a first nexthop including a first steering logic and a second nexthop including a second steering logic. When the packet arrives at the forwarding unit <NUM>, in some cases, packet processor <NUM> may execute the first steering logic to map the packet to a service unit (e.g., service unit 126A) of service units <NUM> based on the one or more services associated with the packet. Forwarding unit <NUM> may read the packet header to determine the one or more services associated with the packet. In some examples, not every service unit of service units <NUM> are configured to apply the same set of services as each other service unit of service units <NUM>. In other words, network services <NUM> may include a set of services that is at least partially different than a set of services associated with at least one other service unit of service units <NUM>. As such, it may be beneficial for forwarding unit <NUM> to steer the packet to a service unit of service units <NUM> that is configured to apply one or more services associated with the packet. The first steering logic of the first nexthop element may be configured to identify the one or more services associated with the packet based on the packet header and map the packet to a service unit (e.g., service unit 126A) of service units <NUM> corresponding to the one or more services. As such, the first steering logic may cause forwarding unit <NUM> to steer the packet to service unit 126A via switch fabric <NUM>.

As discussed above, service unit 126A may include CPU cores <NUM>. It may be beneficial for the packet associated with a service to be processed using the same CPU cores of CPU cores <NUM> as other packets associated with the same service. The second steering logic of the second nexthop element may map the packet to a CPU core of CPU cores <NUM>. For example, the second steering logic may map the packet to CPU core 130B for processing the packet. After the packet travels through forwarding path elements <NUM>, forwarding unit <NUM> may steer the packet to service unit 126A and CPU core 130B may process the packet in order to execute one or more services of network services <NUM>, where CPU core 130B processes the packet apart from control plane <NUM> and forwarding plane <NUM>. Subsequently, service unit 126A may steer the packet back to forwarding unit <NUM> and forwarding unit <NUM> may forward the packet via an outbound link (e.g., outbound link 118A) of outbound links <NUM>.

Control unit <NUM> may be configured to configure forwarding unit <NUM> in order to steer packets arriving at inbound links <NUM> to service units <NUM> for processing. For example, VMs <NUM> may be configured to generate at least one of forwarding path elements <NUM> in order to cause routing engine <NUM> to output the at least one of forwarding path elements <NUM> to forwarding unit <NUM>. For example, VMs <NUM> may generate the at least one of forwarding path elements <NUM> in order to cause routing engine <NUM> to output a first nexthop element of forwarding path elements <NUM> for steering packets to service units <NUM> based on one or more services based on a respective packet. For example, routing engine <NUM> may output instructions to associate each service unit of service units <NUM> with one or more services. Additionally, in some cases, Control unit <NUM> may be configured to program forwarding unit <NUM> in order to route packets arriving at inbound links <NUM> to for processing by one or more specific CPU cores of a specific service unit for processing.

In some examples, each service unit of service units <NUM> may represent a service card which may be added to network device <NUM> and/or removed from network device <NUM>. Responsive to a service unit <NUM> being added to network device <NUM> or responsive to a service unit <NUM> being removed from network device <NUM>, control unit <NUM> may re-program forwarding path elements <NUM> based on the addition or removal of the respective control unit. For example, if forwarding path elements <NUM> is currently programmed to cause forwarding unit <NUM> to route a packet associated with a particular service to service unit 126N, and service unit 126N is subsequently removed from network device <NUM>, control unit <NUM> may automatically reconfigure forwarding path elements <NUM> in order to route incoming packets associated with the service to another service unit of service units <NUM> without interrupting any services provided by network device <NUM>. In a similar way, control unit <NUM> may automatically re-configure forwarding path elements <NUM> in order to steer incoming packets associated with the service to a specific CPU core or group of CPU cores within another service unit of service units <NUM>.

The network services <NUM> executed by microprocessor <NUM> may include, in some examples, IPsec. IPsec is a communication protocol which allows an encryption and a decryption of one or more data packets for transmission between two or more devices (e.g., router <NUM>, service nodes <NUM>, subscriber devices <NUM>, and router <NUM> of <FIG> and network device <NUM> of <FIG>) within a network (e.g., network system <NUM> of <FIG>). For example, network device <NUM> device may process one or more packets associated with an IPsec session between a first device and a second device. In order to process a packet associated with the IPsec session, a CPU core, such as CPU core 130B of microprocessor <NUM> may execute an IPsec service of network services <NUM>. Processing the packet according to the IPsec service may involve encrypting data and/or decrypting data within a payload of the packet for forwarding to a destination device of the first device and the second device, but this is not required. Processing the packet according to the IPsec service may include other actions in addition to or alternatively to decrypting and decrypting.

Control unit <NUM> may configure forwarding path elements <NUM> to route packets associated with different IPsec sessions to respective CPU cores. Control unit <NUM> may configure forwarding path elements <NUM> to steer packets associated with a first IPsec session to service unit 126A for processing by CPU core 130A, steer packets associated with a second IPsec session to service unit 126A for processing by CPU core 130C, steer packets associated with a third IPsec session to service unit 126A for processing by CPU core 130N, and steer packets associated with a fourth IPsec session to service unit 126N for processing by a CPU core located on service unit 126N, as an example. It may be beneficial for one CPU core to process all packets associated with one IPsec session in order to improve an efficiency as compared with techniques in which more than one CPU core is used to process packets associated with one IPsec session. One CPU core, such as CPU core 130B, may be configured to process packets associated with more than one IPsec session. For example, CPU core 130B may process packets associated with a fifth IPsec session and process packets associated with a sixth IPsec session.

The term "session," "packet flow," "traffic flow," or simply "flow" may refer to a set of packets originating from a particular source device or endpoint and sent to a particular destination device or endpoint. A single flow of packets may be identified by a <NUM>-tuple hash: <source network address, destination network address, source port, destination port, protocol>, for example. This <NUM>-tuple hash generally identifies a packet flow to which a received packet corresponds. An n-tuple refers to any n items drawn from the <NUM>-tuple. For example, a <NUM>-tuple for a packet may refer to the combination of <source network address, destination network address> or <source network address, source port> for the packet. The <NUM>-tuple hash of a packet may be located in the packet header of packet <NUM>.

Traditional networks and cloud-based networks, such as network system <NUM>, may use information other than a five-tuple hash of a packet to determine a destination of the packet. One or more service models may implement load-balancing solutions in order to apply services to incoming network traffic. One or more techniques described herein may use application identification (e.g., "app-id") load distribution and one or more techniques described herein may use tunnel-based load distribution or session-based load distribution. Additionally, or alternatively, one or more techniques described herein may use some stateless load distribution.

It may be beneficial to increase an efficiency of the use of CPU cores and the use bandwidth available in each service unit of service units <NUM> by implementing load-balancing, as compared with techniques which do not use load-balancing between CPU cores and service units. As the compute power increases, packet distribution help to achieve optimal usage of compute power and bandwidth available in network system <NUM>.

Network device <NUM> may implement CPU core load-balancing using route-based next-hops. CPU core load-balancing may be applied to different services (e.g., network services <NUM>) and applications running in network system <NUM>. Route-based next-hops may co-exist with firewall-based next-hops and routes. One or more techniques may use route metadata to decide that network traffic on a particular route (e.g., session) should be steered to a given CPU core or set of CPU cores identified by a CPU core id. Network device <NUM> may implement a CPU distribution profile such as in Virtual Network Function (VNF) microservices models and containerized microservices models. Network device <NUM> may support symmetric route-based load balancing for uplink and downlink network traffic.

<FIG> is a conceptual diagram illustrating an example forwarding path 122A, in accordance with one or more techniques of this disclosure. Forwarding path 122A may be an example of forwarding path <NUM> of <FIG>. Forwarding path 122A includes routing table <NUM>, feature list <NUM>, first nexthop element <NUM>, service table <NUM>, second nexthop element 154A, core table <NUM>, and queues <NUM>. First nexthop element <NUM> includes service unit steering logic <NUM> and second nexthop element 154A includes CPU core steering logic 156A. Control plane <NUM> may, in some cases, output instructions to configure routing table <NUM>, feature list <NUM>, first nexthop element <NUM>, and second nexthop element 154A. In the example of <FIG>, forwarding path 122A may route packets to service unit 126A for processing, but this is not required. Forwarding path 122A may route packets to any one or more of service units <NUM> of <FIG>.

Routing table <NUM> represents a forwarding path element of the set of forwarding path elements of forwarding path 122A. Routing table <NUM> is a data table which contains a list of routes to a set of packet destinations within a network, such as network system <NUM>. In some examples, when a packet (e.g., packet <NUM>) arrives at forwarding path 122A, routing table <NUM> may select one or more routes of the list of routes based on a packet header of packet <NUM>. For example, the packet header of packet <NUM> may include data indicative of a destination device which packet <NUM> is bound for, a device in which packet <NUM> originates at, one or more other devices in which packet <NUM> has been to or is bound for, or any combination thereof. Forwarding unit <NUM> of <FIG> may, in some cases, process the packet header of packet <NUM> in order to obtain the information included in the packet header. Routing table <NUM> may select the one or more routes of the list of routes based on the information included by the packet header. Subsequently, a logic of forwarding path 122A proceeds to feature list <NUM>. Feature list <NUM> may represent a routing table nexthop (RTNH) feature list that is associated with packets which have a set of certain identifiers in their respective packet headers. For example, packet <NUM> may include a link identification code, a core identification code, and a tunnel identification (Tid) code. In this example, packet processor <NUM> may advance the logic of forwarding path 112A from routing table <NUM> to feature list <NUM>, which is associated with the link identification code, the core identification code, and the Tid identification code.

As seen in <FIG>, the logic of forwarding path 122A proceeds to first nexthop element <NUM> from feature list <NUM>. Packet processor <NUM> may execute first nexthop element <NUM> to map the packet flow to service unit 126A of service units <NUM> using service unit steering logic <NUM>. In some examples, control plane <NUM> may generate first nexthop element <NUM> in order to steer packet <NUM> and other packets of the packet flow associated with packet <NUM> to a CPU core of a plurality of CPU cores. In this way, control plane <NUM> may generate first nexthop element <NUM> in order to map packet <NUM> to service unit 126A, which includes the CPU core of the plurality of CPU cores. Subsequently, the logic of forwarding path 122A proceeds to service table <NUM>. The advance of forwarding path 122A from first nexthop element <NUM> to service table <NUM> may represent a mapping of packet <NUM> to service unit <NUM> of service units <NUM>.

The logic of forwarding path 122A may subsequently proceed to second nexthop element 154A which includes CPU core steering logic 156A. Packet processor <NUM> may execute second nexthop element 154A, which includes CPU core steering logic 156A, to map packet <NUM> to CPU core 130B of CPU cores <NUM> which are located on service unit 126A. This mapping of packet <NUM> to CPU core 130B may be represented by an advance of forwarding path 122A from second nexthop element 154A to core table <NUM>. In some examples, control plane <NUM> may generate second nexthop element 154A in order to map packets associated with the packet flow of packet <NUM> to the same CPU core. In the example of <FIG>, this CPU core is CPU core 130B. Since CPU core 130B is located on service unit 126A, control plane <NUM> may generate first nexthop element <NUM> and second nexthop element 154A to map packet <NUM> to CPU core 130B, allowing forwarding unit <NUM> to steer packet <NUM> to CPU core 130B for processing. In one or more other examples not illustrated in <FIG>, control plane <NUM> may generate a first nexthop element and a second nexthop element to map packet <NUM> to a CPU core located on another service unit, such as service unit 126N. In one or more such examples, control plane <NUM> may generate the respective first nexthop element to map packet <NUM> to service unit 126N and generate the respective second nexthop element to map packet <NUM> to the correct CPU core of a set of CPU cores located on service unit 126N.

Although first nexthop element <NUM>, which corresponds to service unit 126A, is illustrated in <FIG>, forwarding path 122A may also include a respective nexthop element corresponding to each other service unit of service units <NUM> not illustrated in <FIG>. Additionally, although second nexthop element 154A, which corresponds to CPU core 130B, is illustrated in <FIG>, forwarding path 122A may also include a respective nexthop element corresponding to each other CPU core of CPU cores <NUM>. In this way, it may be possible for control plane <NUM> to generate nexthop elements in order to map packet <NUM> or other packets to any CPU core located on any service unit of service plane <NUM>.

The logic of forwarding path 122A proceeds to queues <NUM>. Queue 164A may receive packet <NUM> from core table <NUM>. In some examples, queues 164A-164N may represent queues that are each associated with a priority level. For example, queue 164A may correspond to a first priority level and queue 164B may correspond to a second priority level, where the first priority level is higher than the second priority level. As such, packet <NUM> may represent a "high priority" packet, which is forwarded by queues <NUM> before packets in queues 164B-164N that correspond to priority levels lower than the first priority level. Queue 164A may steer packet <NUM> to service unit 126A so that CPU core 130B may process packet <NUM> based on first nexthop element <NUM> and second nexthop element 154A mapping packet <NUM> to CPU core 130B of service unit 126A. In some examples, CPU core 130B may process packet <NUM> in order to administer services, such as one or more of network services <NUM>.

Service unit 126A may steer packet 196A to routing table <NUM> after processing by CPU core 130B. Routing table <NUM> represents a forwarding path element of the set of forwarding path elements of forwarding path 122A. Routing table <NUM> is a data table which contains a list of routes to a set of packet destinations within a network, such as network system <NUM>. Routing table <NUM> may parform a route lookup for packet <NUM>, and routing table <NUM> may forward packet <NUM> via WAN interface <NUM>. In some examples, WAN interface <NUM> may represent an IFC of IFCs <NUM> of <FIG>.

Control plane <NUM> may generate first nexthop element <NUM> and second nexthop element 154A in order to map packet <NUM> to a CPU core or set of CPU cores which process all or nearly all packets associated with a service associated with packet <NUM>. For example, if packet <NUM> is associated with an IPsec session, Control plane <NUM> may configure first nexthop element <NUM> and second nexthop element 154A in order to steer packet <NUM> to CPU core 130B, which processes all packets associated with the IPsec session associated with packet <NUM>. In some cases, control plane <NUM> may configure first nexthop element <NUM> and second nexthop element 154A in order to route packets associated with another IPsec session different than the IPsec session associated with packet <NUM> to a CPU core other than CPU core 130B. In some cases, control plane <NUM> may configure first nexthop element <NUM> and second nexthop element 154A in order to route packets associated with another IPsec session different than the IPsec session associated with packet <NUM> to CPU core 130B. It may be more efficient for one CPU core to process packets associated with one service as compared with techniques in which more than one CPU core processes packets associated with one service.

The techniques of this disclosure are not meant to be limited to IPsec services. Control plane <NUM> may output instructions to configure first nexthop element <NUM> and second nexthop element 154A in order to steer packet <NUM> to a particular CPU core (e.g., CPU core 130B) of service unit 126B for processing based on packet <NUM> being associated with CG-NAT, media optimization (voice/video), VPN services, DPI services, HTTP filtering services, counting services, accounting services, charging services, load balancing services, or any combination thereof. For example, if packet <NUM> is associated with a VPN service, control plane <NUM> may output instructions to configure first nexthop element <NUM> and second nexthop element 154A in order to route packet <NUM> to CPU core 130B for processing with other packets associated with the VPN service.

<FIG> is a conceptual diagram illustrating another example forwarding path 122B, in accordance with one or more techniques of this disclosure. Forwarding path 122B may be an example of forwarding path <NUM> of <FIG>. Forwarding path 122B includes routing table <NUM>, feature list <NUM>, first nexthop element <NUM>, service table <NUM>, second nexthop element 154B, core table <NUM>, and queue <NUM>. First nexthop element <NUM> includes service unit steering logic <NUM> and second nexthop element 154B includes hash logic <NUM> and CPU core steering logic 156A. Control plane <NUM> may, in some cases, output instructions to configure routing table <NUM>, feature list <NUM>, first nexthop element <NUM>, and second nexthop element 154B. In the example of <FIG>, forwarding path 122B may route packets to service unit 126A for processing, but this is not required. Forwarding path 122B may route packets to any one or more of service units <NUM> of <FIG>.

Forwarding path 122B may be substantially the same as forwarding path 122A of <FIG>, except that second nexthop element 154B of <FIG> includes hash logic <NUM> in addition to CPU core steering logic 156B, whereas second nexthop element 154A includes CPU core steering logic 156A and does not include a hash logic. In some examples, it may be beneficial for bi-directional traffic of each IPsec session of a set of IPsec sessions to be processed by a common CPU core. In some examples, control plane <NUM> may program forwarding path 122B based on a set of metadata and may forward network traffic to one or more CPU cores provisioned by control plane <NUM>. For example, control plane <NUM> may program first nexthop element <NUM> and second nexthop <NUM> to steer packet <NUM> for processing by CPU core 130B of service unit 126A.

The term "session," "packet flow," "traffic flow," or simply "flow" refers to a set of packets originating from a particular source device or endpoint and sent to a particular destination device or endpoint. A single flow of packets may be identified by a <NUM>-tuple hash: <source network address, destination network address, source port, destination port, protocol>, for example. This <NUM>-tuple hash generally identifies a packet flow to which a received packet corresponds. An n-tuple refers to any n items drawn from the <NUM>-tuple. For example, a <NUM>-tuple for a packet may refer to the combination of <source network address, destination network address> or <source network address, source port> for the packet. The <NUM>-tuple hash of packet <NUM> may be located in the packet header of packet <NUM>.

In the example of <FIG>, control plane <NUM> may program forwarding path 122B in order to steer packet <NUM> to a group of CPU cores located in service plane <NUM>, such as two or more of CPU cores <NUM>. Second nexthop 154B may select, based on a <NUM>-tuple hash or a <NUM>-tuple hash of packet <NUM>, one of the two or more of CPU cores <NUM> in which to steer packet <NUM>. In the example of <FIG>, on the other hand, control plane <NUM> may program forwarding path 122B to steer packet <NUM> to a particular CPU core, such as CPU core 130B.

In some examples, to steer packet <NUM> to CPU core 130B, packet processor <NUM> of forwarding unit <NUM> is configured to execute the second nexthop element 154B to select CPU core 130B from CPU cores <NUM> of a selected service unit 126B. In some examples, to select CPU core 130B from the set of CPU cores <NUM> of the selected service unit 126A, packet processor <NUM> of forwarding unit <NUM> is configured to execute second nexthop element 154B to apply, using hash logic <NUM>, a hash function to one or more elements of a packet header of packet <NUM> to generate a hash index that maps to CPU core 130B. In this way, second nexthop element 154B may map packet <NUM> to CPU core 130B for processing on the <NUM>-tuple hash of packet <NUM>.

<FIG> is a flow diagram illustrating an example operation for steering a packet to a service plane <NUM> for processing, in accordance with one or more techniques of this disclosure. <FIG> is described with respect to control plane <NUM>, forwarding plane <NUM>, and service plane <NUM> of network device <NUM> of <FIG>. However, the techniques of <FIG> may be performed by different components of network device <NUM> or by additional or alternative devices.

In some examples, control unit <NUM> of <FIG> includes processing circuitry <NUM> in communication with a memory, where the processing circuitry is configured to execute one or more processes. The one or more processes may, in some cases, include VMs <NUM>. Forwarding unit <NUM> of <FIG> includes IFCs <NUM>, packet processor <NUM>, and a forwarding unit memory. In some examples, the one or more processes of control unit <NUM> are configured for execution by processing circuitry <NUM> in order to configure the forwarding unit memory of forwarding unit <NUM> with one or more forwarding path elements <NUM> The one or more forwarding path elements <NUM> may map a packet flow to a CPU core of a plurality of CPU cores located in service plane <NUM>. After the one or more processes of control unit <NUM> configure the forwarding unit memory with one or more forwarding path elements <NUM>, forwarding unit <NUM> may receive, via one of IFCs <NUM>, a packet. Additionally, packet processor <NUM> of forwarding unit <NUM> is configured to execute the one or more forwarding path elements <NUM> in the forwarding unit memory to steer the packet to the CPU core of the plurality of CPU cores located in service plane <NUM>.

As seen in <FIG>, the one or more processes of control unit <NUM> may select a service unit of a plurality of service units <NUM> (<NUM>) located in service plane <NUM>. In some examples, control unit <NUM> may select service unit 126A with which to process the packet flow which forwarding path elements <NUM> map to the CPU core of the plurality of CPU cores, where service unit 126A includes the CPU core. The one or more processes of control unit <NUM> may select, from a set of CPU cores <NUM> of the selected service unit 126A, CPU core 130B (<NUM>). The one or more processes may generate a first nexthop element and a second nexthop element (<NUM>). In some examples, the first nexthop element map the packet flow to the selected service unit 126A and the second nexthop element may map the packet flow to the selected CPU core 130B of CPU cores <NUM> of the selected service unit 126A.

In some examples, to generate the first nexthop element and generate the second nexthop element, the one or more processes of control unit <NUM> are configured to identify, based on a packet header of a packet arriving at forwarding unit <NUM>, a service associated with the packet, and generate, based on the service, the first nexthop element and the second nexthop element. Additionally, or alternatively, the one or more processes may associate each service of a plurality of services with one or more CPU cores of the plurality of CPU cores located in service units <NUM> of service plane <NUM>. In some examples, the plurality of services may include network services <NUM> of service unit 126A and one or more other services executing on service units 126B-126N. The one or more processes of control unit <NUM> may generate, based on the one or more CPU cores associated with each service of the plurality of services, the first nexthop element and the second nexthop element. In some examples, the one or more processes may generate the first nexthop element and the second nexthop element in order to balance a processing load across the plurality of CPU cores located on service units <NUM> of service plane <NUM>. In some cases, the one or more processes may generate forwarding path elements in addition to the first nexthop element and the second nexthop element.

Forwarding unit <NUM> may store the first nexthop element and the second nexthop element (<NUM>) generated by the one or more processes of control unit <NUM>. Forwarding unit <NUM> may receive, via one of IFCs <NUM>, a packet (<NUM>) of the packet flow which forwarding path elements <NUM> steer to the CPU core of the plurality of CPU cores located on service units <NUM> of service plane <NUM>. Subsequently, packet processor <NUM> of forwarding unit <NUM> may execute forwarding path elements <NUM> in order to steer the packet to the CPU core (<NUM>) of the plurality of CPU cores. For example, to steer the packet to the CPU core, packet processor <NUM> may be configured to execute the first nexthop element to map the packet flow to the selected service unit 126A and packet processor <NUM> may be configured to execute the second nexthop element to map the packet flow to the selected CPU core 130B of the set of CPU cores <NUM> located on the selected service unit 126A. Additionally, or alternatively, to steer the packet to CPU core 130B, packet processor <NUM> of forwarding unit <NUM> is configured to execute the second nexthop element to select CPU core 130B from CPU cores <NUM> of the selected service unit 126A. In some examples, to select CPU core 130B from CPU cores <NUM> of the selected service unit 126A, packet processor <NUM> of forwarding unit <NUM> is configured to execute the second nexthop element to apply a hash function to one or more elements of a packet header of the packet to generate a hash index that maps to CPU core 130B from the CPU cores <NUM> of the selected service unit 126A.

The selected service unit 126A may receive the packet <NUM> (<NUM>). Subsequently, CPU core 130B may process the packet (<NUM>) in order to apply a service of services <NUM> to the packet. For example, the packet flow of the packet may correspond to an IPsec session. And CPU core 130B may process the packet in order to encrypt or decrypt data in the payload of the packet, but this is not required. CPU core 130B may process the packet in order to apply any respective one of services <NUM> that is associated with the packet flow. Service unit 126A may send the packet to forwarding unit <NUM> (<NUM>) and forwarding unit <NUM> may receive the packet (<NUM>). Forwarding unit <NUM> may forward the packet (<NUM>) via one of IFCs <NUM>.

If implemented in hardware, this disclosure may be directed to an apparatus such as a processor or an integrated circuit device, such as an integrated circuit chip or chipset. Alternatively or additionally, if implemented in software or firmware, the techniques may be realized at least in part by a computer-readable data storage medium including instructions that, when executed, cause a processor to perform one or more of the methods described above. For example, the computer-readable data storage medium may store such instructions for execution by a processor.

A computer-readable medium may form part of a computer program product, which may include packaging materials. A computer-readable medium may include a computer data storage medium such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), Flash memory, magnetic or optical data storage media, and the like. In some examples, an article of manufacture may include one or more computer-readable storage media. Additionally, or alternatively, a computer-readble medium may include transient media such as carrier signals and transmission media.

In some examples, the computer-readable storage media may include non-transitory media. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier wave or a propagated signal. In certain examples, a non-transitory storage medium may store data that can, over time, change (e.g., in RAM or cache).

The code or instructions may be software and/or firmware executed by processing circuitry including one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. In addition, in some aspects, functionality described in this disclosure may be provided within software modules or hardware modules.

Claim 1:
A network device comprising:
a control unit (<NUM>) comprising processing circuitry (<NUM>) in communication with a memory, wherein the processing circuitry is configured to execute one or more processes; and
a forwarding unit (<NUM>) comprising an interface card (<NUM>), a packet processor (<NUM>), and a forwarding unit memory,
wherein the one or more processes of the control unit are configured to configure the forwarding unit memory of the forwarding unit with one or more forwarding path elements,
wherein the one or more forwarding path elements map a packet flow to a central processing unit, CPU, core of a plurality of CPU cores (<NUM>) for processing,
wherein the forwarding unit is configured to receive, via the interface card, a packet of the packet flow,
wherein the forwarding unit further comprises a plurality of output queues (<NUM>, <NUM>) for each of the plurality of CPU cores, wherein each of the plurality of output queues for a given CPU core is associated with a different priority level where packets in higher priority queues are configured to be forwarded before packets in lower priority queues, and
wherein the packet processor of the forwarding unit is configured to execute the one or more forwarding path elements in the forwarding unit memory to steer the packet to the CPU core by adding the packet to one of the plurality of output queues for the CPU core.