Patent Description:
The communications industry is rapidly changing to adjust to emerging technologies and ever increasing customer demand. This customer demand for new applications and increased performance of existing applications is driving communications network and system providers to employ networks and systems having greater speed and capacity (e.g., greater bandwidth). In trying to achieve these goals, a common approach taken by many communications providers is to use packet switching technology in packet switching networks of various topologies. <CIT> describes a method of supporting in-band Operation, Administration and Maintenance (OAM) in a Multi-Protocol Label Switching-Transport Profile (MPLS-TP) network. The method may include generating a merged OAM packet by merging a plurality of OAM packets received from a plurality of leaf nodes, and transmitting the merged OAM packet to a root node through a Label-Switched Path (LSP).

Disclosed are, inter alia, methods, apparatus, computer-storage media, mechanisms, and means associated with coordinated offloaded recording of In-Situ Operations, Administration, and Maintenance (IOAM) data to packets traversing network nodes.

In one embodiment, a first network node receives a packet via a network. The first network node adds first IOAM data and second IOAM data to the particular packet, with the first IOAM data related to the first network node and the second IOAM data related to a second network node. The particular packet is sent into the network from the first network node.

In one embodiment, adding of second IOAM data to the particular packet is performed in response to the first network node determining an IOAM processing capability or an IOAM processing state of the second network node indicates that the second network node would not add this second IOAM data to the particular packet. In one embodiment, the first network node receives one or more control plane messages identifying an IOAM processing capability or state of the second network node. In one embodiment, the IOAM processing state includes, but is not limited to, suspension of adding IOAM data to packets. In one embodiment, the second network node suspends adding IOAM data to packets in response to a resource utilization characteristic of the second network node.

In one embodiment, the first and second network nodes are neighbors in the network, such as, but not limited to, the second network node being a nexthop neighbor of the first network node. In one embodiment, the first IOAM data includes proof of transit (PoT) information identifying the first network node, and the second IOAM includes PoT information identifying the second network node. In one embodiment, a node or apparatus in the network includes one or more processing elements and memory and one or more interfaces sending and receiving packets.

The appended claims set forth the features of one or more embodiments with particularity. The embodiment(s), together with its advantages, may be best understood from the following detailed description taken in conjunction with the accompanying drawings of which:.

Disclosed are, inter alia, methods, apparatus, computer-storage media, mechanisms, and means associated with coordinated offloaded recording of In-Situ Operations, Administration, and Maintenance (IOAM) data to packets traversing network nodes. Embodiments described herein include various elements and limitations, with no one element or limitation contemplated as being a critical element or limitation. Each of the claims individually recites an aspect of the embodiment in its entirety. Moreover, some embodiments described may include, but are not limited to, inter alia, systems, networks, integrated circuit chips, embedded processing elements, ASICs, methods, and computer-readable media containing instructions. One or multiple systems, devices, components, etc., may comprise one or more embodiments, which may include some elements or limitations of a claim being performed by the same or different systems, devices, components, etc. A processing element may be a general processor, task-specific processor, a core of one or more processors, or other co-located, resource-sharing implementation for performing the corresponding processing. The embodiments described hereinafter embody various aspects and configurations, with the figures illustrating exemplary and non-limiting configurations. Computer-readable media and means for performing methods and processing block operations (e.g., a processor and memory or other apparatus configured to perform such operations) are disclosed and are in keeping with the extensible scope of the embodiments. The term "apparatus" is used consistently herein with its common definition of an appliance or device.

The steps, connections, and processing of signals and information illustrated in the figures, including, but not limited to, any block and flow diagrams and message sequence charts, may typically be performed in the same or in a different serial or parallel ordering and/or by different components and/or processes, threads, etc., and/or over different connections and be combined with other functions in other embodiments, unless this disables the embodiment or a sequence is explicitly or implicitly required (e.g., for a sequence of read the value, process said read value - the value must be obtained prior to processing it, although some of the associated processing may be performed prior to, concurrently with, and/or after the read operation). Also, nothing described or referenced in this document is admitted as prior art to this application unless explicitly so stated.

The term "one embodiment" is used herein to reference a particular embodiment, wherein each reference to "one embodiment" may refer to a different embodiment, and the use of the term repeatedly herein in describing associated features, elements and/or limitations does not establish a cumulative set of associated features, elements and/or limitations that each and every embodiment must include, although an embodiment typically may include all these features, elements and/or limitations. In addition, the terms "first," "second," etc., are typically used herein to denote different units (e.g., a first element, a second element). The use of these terms herein does not necessarily connote an ordering such as one unit or event occurring or coming before another, but rather provides a mechanism to distinguish between particular units. Moreover, the phrases "based on x" and "in response to x" are used to indicate a minimum set of items "x" from which something is derived or caused, wherein "x" is extensible and does not necessarily describe a complete list of items on which the operation is performed, etc. Additionally, the phrase "coupled to" is used to indicate some level of direct or indirect connection between two elements or devices, with the coupling device or devices modifying or not modifying the coupled signal or communicated information. Moreover, the term "or" is used herein to identify a selection of one or more, including all, of the conjunctive items. Additionally, the transitional term "comprising," which is synonymous with "including," "containing," or "characterized by," is inclusive or open-ended and does not exclude additional, unrecited elements or method steps. Finally, the term "particular machine," when recited in a method claim for performing steps, refers to a particular machine within the <NUM> USC § <NUM> machine statutory class.

As used herein, a "data packet" refers to a standard packet communicating information (such as a customer data packet), with a probe packet (e.g., test packet) not being included in the definition of a data packet.

As used herein, "operations data" refers to operations, administration, maintenance (OAM) and/or provisioning (OAM-P) information (e.g., including operational and telemetry information), such as, but not limited to, in-band OAM data, or more specifically, In-Situ OAM Data. In one embodiment, the operations data is raw data, processed data, and/or data resulting from processing of other information. As used herein, "offloading" between network nodes is used to connote moving of a task (e.g., IOAM-related processing) from one entity (e.g., network node, process, network processor) to another entity.

In one embodiment, the operations data is related to data-plane and/or control-plane processing in the network (e.g., in a portion of, or the entire network). In one embodiment, the operations data is related to communication (including, but not limited to, verifying and/or discovering a path taken and/or performance measurement data or results) and/or other processing of packet(s) in a network. In one embodiment, the operations data is related to process(es), hardware, link(s), and/or other resources of one or more elements in the network (e.g., node(s), router(s), packet switching device(s), network management or other control system(s), host(s), server(s), apparatus, application processor(s), service devices(s), application processor(s), transmission and/or communications equipment). In one embodiment, operations data includes information related to the communication of a packet through a network, other protocol layer processing, and/or same layer processing.

In one embodiment, the operations data encompasses data related to one or more underlay protocols/networks. In one embodiment, the operations data encompasses data related to one or more overlay protocols/networks.

A typical use of IOAM is collects real-time operations data by embedding the operations data within actual data traffic. Such collected in-band telemetry data allows a network to instantly react to network events. IOAM data is typically inserted in packet headers of one or more protocols at one or more protocol layers. In one embodiment, IOAM data is recorded in a header of a network layer protocol (e.g., a Hop-by-Hop option will collect path and/or performance data from network elements at the network layer). In one embodiment, IOAM data is recorded in a header of a network layer protocol (e.g., Segment Routing (SRv6), Network Service Header (NSH), Generic Routing Encapsulation (GRE) to collect operations data from service nodes. Processing of the IOAM provides operations capabilities and improved network operations not available prior to IOAM.

In one embodiment, not all nodes record IOAM data in packets. For example, a network node might not be IOAM capable, or a capable node may suspend recording IOAM data due to current resource utilization (e.g., the processing load of the node is currently high, data structure or memory resources are low). Unaccounted for intermittent behavior in a network (e.g., such as related to adding and suspending adding IOAM data to a packets) provides great network operations problems.

In one embodiment, network nodes coordinate recording of In-Situ Operations, Administration, and Maintenance (IOAM) data in packets traversing the network nodes, including a node adding IOAM data of another node to packets on behalf of the another node. After receiving a particular packet, a network node adds first IOAM data and second IOAM data to the particular packet, with the first IOAM data related to the first network node and the second IOAM data related to a second network node. The packet is then sent from the first network node. The coordinated offloading of the adding of IOAM data to packets allows a node to free up resources currently used for IOAM operations to be used for other packet processing operations, while still having IOAM data related to the node recorded in packets. The coordinated offloading may include control plane communication (e.g., via a routing or other protocol). In one embodiment, IOAM data is offload added to a packet by an upstream node in the path taken by a packet. In one embodiment, IOAM data is offload added to a packet by a downstream node in the path taken by a packet.

<FIG> illustrates a network <NUM> (e.g., an aggregation of one or more networks of one or more different entities) operating using multiple protocol layers in processing packets (e.g., using overlay and underlay protocols/networks) according to one embodiment. As shown, network <NUM> includes client networks <NUM> and <NUM> (which are the same network in one embodiment) communicatively coupled to a provider network <NUM>. In one embodiment, network <NUM> uses Segment Routing (SR), Multiprotocol Label Switching (MPLS), tunnels, Ethernet VPN (EVPN), Provider Backbone Bridging EVPN (PBB-EVPN), Internet Protocol version <NUM> and/or <NUM> (IP), and/or other encapsulating and/or packet forwarding technology.

In one embodiment, provider network <NUM> includes provider edge nodes <NUM> and <NUM>, and a network <NUM> of network nodes, gateways, service functions, hosts (e.g., end nodes), network management, operations support systems, etc. In one embodiment, provider edge nodes <NUM> and <NUM> process packets received from networks <NUM> and <NUM>, which may include encapsulating or otherwise processing these packets into Segment Routing packets such as by adding a SR header (and possibly another IP header) to these packets according to a data plane ascertained Segment Routing policy, and subsequently decapsulating or removing a Segment Routing header (and possibly another IP header) and forwarding the native (e.g., IP) packets into network <NUM> and <NUM>. In one embodiment, edge nodes <NUM> and <NUM> perform ingress and egress processing of packets, including adding and extracting operations data fields and operations data to packets.

<FIG> illustrates a process according to one embodiment associated with coordinated offloaded recording of IOAM data to packets traversing network nodes. Processing begins with process block <NUM>. In process block <NUM>, network nodes in the networks typically continuously advertise/exchange routing, forwarding, IOAM, capability and state information (e.g., including operations capabilities and current operations processing state), etc., via one or more routing, label distribution, discovery, signaling and/or other control-plane protocols. In process block <NUM>, the network nodes continuously update their network information, such as, but not limited to, Routing Information Bases (RIBs), Forwarding Information Bases (FIBS), forwarding tables, IOAM processing tables, packet/network processing and/or control data structures, operations data structures, etc. Processing of the flow diagram of <FIG> is complete as indicated by process block <NUM>.

<FIG> illustrates a process performed by an IOAM-capable network node according to one embodiment associated with coordinated offloaded recording of IOAM data to packets traversing network nodes. Processing begins with process block <NUM>. In process block <NUM>, the network node advertises to other node(s) its IOAM capability and/or current IOAM state (e.g., performing full IOAM processing, performing a subset of IOAM processing - possibly identifying the subset, or suspending all IOAM processing). If the network node changes its IOAM state as determined in process block <NUM>, then the IOAM-capable network node advertises its current IOAM state in process block <NUM>. The processing loop returns to process block <NUM>.

As used herein, "advertising" IOAM capability or state refers to communication of the IOAM capability or state to other network node(s), with the network possibly distributing this information throughout the network. A routing or other protocol used in one embodiment to advertise the IOAM capability and/or state includes, but is not limited to, Routing Information Protocols(RIP), Interior Gateway Protocol (IGRP), Open Shortest Path First (OSPF), Exterior Gateway Protocol (EGP), Enhanced interior gateway routing protocol (EIGRP), Border Gateway Protocol (BGP), and/or Intermediate System-to-Intermediate System (IS-IS). One embodiment advertises its IOAM capability and/or state using Link Local IGP Opaque LSA, BGP Extended Communities, Extended ARP, an IPv6 Neighbor Discovery option, and/or Router Capability in IGP.

<FIG> illustrates a process performed by an IOAM-capable network node according to one embodiment associated with coordinated offloaded recording of IOAM data to packets traversing network nodes. Processing begins with process block <NUM>. The processing loop remains at process block <NUM> until the network node receives an advertisement of an IOAM capability and/or state of another one or more network nodes; then in process block <NUM>, the network node updates its IOAM processing data structure(s) to reflect the received IOAM advertisement of the other network node(s). The processing loop returns to process block <NUM>.

<FIG> and their discussion herein provide a description of various network nodes according to one embodiment.

<FIG> illustrates one embodiment of a packet switching device <NUM> (e.g., router, node, switching, appliance, gateway) according to one embodiment. As shown, packet switching device <NUM> includes multiple line cards <NUM> and <NUM>, each with one or more network interfaces for sending and receiving packets over communications links (e.g., possibly part of a link aggregation group), and with one or more processing elements that are used in one embodiment associated with coordinated offloaded recording of IOAM data to packets traversing network nodes. Packet switching device <NUM> also has a control plane with one or more processing elements (e.g., Route Processor(s)) <NUM> for managing the control plane and/or control plane processing of packets associated with coordinated offloaded recording of IOAM data to packets traversing network nodes. Packet switching device <NUM> also includes other cards <NUM> (e.g., service cards, blades) which include processing elements that are used in one embodiment to process (e.g., forward/send, drop, manipulate, change, modify, receive, create, duplicate, perform operations data processing functionality, apply a service according to one or more service functions) packets associated with coordinated offloaded recording of IOAM data to packets traversing network nodes, and some hardware-based communication mechanism <NUM> (e.g., bus, switching fabric, and/or matrix, etc.) for allowing its different entities <NUM>, <NUM>, <NUM> and <NUM> to communicate. Line cards <NUM> and <NUM> typically perform the actions of being both an ingress and egress line card, in regards to multiple other particular packets and/or packet streams being received by, or sent from, packet switching device <NUM>. In one embodiment, operations data processing and storage functions are implemented on line cards <NUM>, <NUM>.

<FIG> is a block diagram of an apparatus <NUM> (e.g., host, router, node, destination, or portion thereof) used in one embodiment associated with coordinated offloaded recording of IOAM data to packets traversing network nodes. In one embodiment, apparatus <NUM> performs one or more processes, or portions thereof, corresponding to one of the flow diagrams illustrated or otherwise described herein, and/or illustrated in another diagram or otherwise described herein.

In one embodiment, apparatus <NUM> includes one or more processor(s) <NUM> (typically with on-chip memory), memory <NUM> (possibly shared memory), storage device(s) <NUM>, specialized component(s) <NUM> (e.g. optimized hardware such as for performing lookup, packet processing (including Segment Routing processing) and/or service function operations; associative memory; binary and/or ternary content-addressable memory; Application Specific Integrated Circuit(s), cryptographic hash hardware, etc.), and interface(s) <NUM> for communicating information (e.g., sending and receiving packets, user-interfaces, displaying information, etc.), which are typically communicatively coupled via one or more communications mechanisms <NUM> (e.g., bus, links, switching fabric, matrix), with the communications paths typically tailored to meet the needs of a particular application.

Various embodiments of apparatus <NUM> may include more or fewer elements. The operation of apparatus <NUM> is typically controlled by processor(s) <NUM> using memory <NUM> and storage device(s) <NUM> to perform one or more tasks or processes. Memory <NUM> is one type of computer-readable/computer-storage medium, and typically comprises random access memory (RAM), read only memory (ROM), flash memory, integrated circuits, and/or other memory components. Memory <NUM> typically stores computer-executable instructions to be executed by processor(s) <NUM> and/or data which is manipulated by processor(s) <NUM> for implementing functionality in accordance with an embodiment. Storage device(s) <NUM> are another type of computer-readable medium, and typically comprise solid state storage media, disk drives, diskettes, networked services, tape drives, and other storage devices. Storage device(s) <NUM> typically store computer-executable instructions to be executed by processor(s) <NUM> and/or data which is manipulated by processor(s) <NUM> for implementing functionality in accordance with an embodiment.

<FIG> illustrates a network <NUM> operating according to one embodiment. As shown, network <NUM> includes nodes N1 (<NUM>), R1 (<NUM>), R2 (<NUM>), R3 (<NUM>), R4 (<NUM>), and N2 (<NUM>). Also shown, and according to one embodiment, are IOAM processing tables <NUM>, that include R1's IOAM processing table <NUM>, R2's IOAM processing table <NUM>, and R3's IOAM processing table <NUM>. As R4 (<NUM>) is not IOAM capable, R2 (<NUM>) does not receive an IOAM capable advertisement from R4 (<NUM>), thus, R4's (<NUM>) entry in R2's IOAM processing table <NUM> is set to NULL.

In one embodiment, IOAM-capable nodes R1 (<NUM>), R2 (<NUM>) and R3 (<NUM>) advertise their IOAM capabilities, with the respective neighboring node storing this information in their respective network neighboring IOAM processing table <NUM>-<NUM>. As reflected in IOAM processing tables <NUM>-<NUM>, each of IOAM-capable nodes R1 (<NUM>), R2 (<NUM>) and R3 (<NUM>) have advertised that they are IOAM capable and either expressly or implicitly, that they are in a state of performing IOAM operations, including adding IOAM data (e.g., a node data list entry) related to itself.

<FIG> illustrates the path (shown on the left-side, going from top of the page down) taken by packet (<NUM>) as it traverses network <NUM> (of <FIG>), including according to the current IOAM state reflected in IOAM processing table <NUM>-<NUM>. As each of nodes R1 (<NUM>), R2 (<NUM>) and R3 (<NUM>) are in a state of performing IOAM processing, no IOAM offloading is performed as shown in <FIG>.

As shown, packet <NUM> includes an outer IPv6 outer header, with an IOAM Type-Length-Value (TLV) in an IP extension header. The TLV includes a node data list (storing data related to individual nodes); and an Incomplete Flag that when False identifies that there has been no offloading of adding IOAM data to packet <NUM>, and when True identifies that there has been offloading of adding IOAM data to packet <NUM>. Note, packet <NUM> is denoted also using reference numbers <NUM>-<NUM> for description purposes, but is still considered as being a same packet <NUM> traversing network <NUM>.

As shown, N1 (<NUM>) sends packet <NUM>, which is received by R1 (<NUM>).

R1 (<NUM>) processes received packet <NUM>, including by adding IOAM data "R1" to the IOAM TLV and sending packet <NUM>, which is received by R2 (<NUM>).

R2 (<NUM>) processes received packet <NUM>, including by adding IOAM data "R2" to the IOAM TLV and sending packet <NUM>, which is received by R3 (<NUM>).

R3 (<NUM>) processes received packet <NUM>, including by adding IOAM data "R3" to the IOAM TLV and sending packet <NUM>, which is received by N2 (<NUM>).

<FIG> illustrates network <NUM> (same network as shown in <FIG>) operating according to one embodiment. However, as shown in <FIG>, node R2 (<NUM>) signals (<NUM>, <NUM>) to its neighboring nodes R1 (<NUM>) and R3 (<NUM>) that it is going to suspend adding IOAM data to packets (e.g., due to a resource limitation). In response to the received advertisements (<NUM>, <NUM>), nodes R1 (<NUM>) and R3 (<NUM>) update the entry for R2 (<NUM>) in their respective IOAM processing table <NUM>, <NUM> to reflect that R2's IOAM processing should be offloaded to nodes R1 (<NUM>) and R3 (<NUM>).

<FIG> illustrates the path (shown on the left-side, going from top of the page down) taken by packet (<NUM>) as it traverses network <NUM> (of <FIG>), including according to the current IOAM state reflected in IOAM processing table <NUM>-<NUM>.

R1 (<NUM>) processes received packet <NUM>, including by:.

R2 (<NUM>) processes received packet <NUM> (without performing IOAM processing as this processing has been offloaded to another node), including by sending packet <NUM>, which is received by R3 (<NUM>).

<FIG> illustrates a process performed in one embodiment by a particular IOAM-capable network node currently in an IOAM state to perform IOAM processing of received packets.

Processing begins with process block <NUM>. In process block <NUM>, a particular packet is received by the particular node. As determined in process block <NUM>, if the received packet does not include an IOAM data field required according to a corresponding policy for processing of the received packet, then in process block <NUM>, this IOAM data field is added to the packet.

In process block <NUM>, the particular adds IOAM data related to itself to an IOAM data field.

In process block <NUM>, the particular node performs an egress lookup operation on a destination address in an egress forwarding information base (FIB), and identifies a nexthop neighbor (e.g., neighboring node in the network) and an egress port of the particular node.

In process block <NUM>, the particular node performs a lookup operation based on the nexthop neighbor in a corresponding IOAM processing table. In one embodiment, the data of the IOAM processing table is added to leafs of the egress FIB.

As determined in process block <NUM>, if the particular node is to perform offload IOAM processing (e.g., record IOAM data in the received packet) on behalf of the nexthop neighboring node, then in process block <NUM>: IOAM data related to the nexthop node is added to the IOAM data field by the particular node on behalf of the nexthop node; and the particular node signals in the IOAM data field (e.g., sets or clears a flag, adds some value) to identifying that offload IOAM processing was performed on the received packet. In one embodiment, the particular node sets one or more IOAM details to NULL that the nexthop node would have added to the IOAM field and/or adding/updating other information (e.g., setting an IOAM value to a TTL of the particular packet in furtherance of identifying the node that offload added IOAM data on behalf of the nexthop node).

In process block <NUM>, the packet (with the added and/or updated IOAM data) is sent from the egress port of the particular device.

Processing of the flow diagram of <FIG> is complete as indicated by process block <NUM>.

In summary, in one embodiment, network nodes coordinate recording of In-Situ Operations, Administration, and Maintenance (IOAM) data in packets traversing the network nodes, including a node adding IOAM data of another node to packets on behalf of the another node. After receiving a particular packet, a network node adds first IOAM data and second IOAM data to the particular packet, with the first IOAM data related to the first network node and the second IOAM data related to a second network node. The packet is then sent from the first network node. The coordinated offloading of the adding of IOAM data to packets allows a node to free up resources currently used for IOAM operations to be used for other packet processing operations, while still having IOAM data related to the node recorded in packets. The coordinated offloading may include control plane communication (e.g., via a routing or other protocol).

Claim 1:
A method, comprising:
receiving a particular packet, by a first network node (<NUM>) in a network (<NUM>);
adding, by the first network node, first In-Situ Operations, Administration, and Maintenance, IOAM data and second IOAM data to the particular packet (<NUM>), with the first IOAM data related to the first network node (<NUM>) and the second IOAM data related to a second network node (<NUM>); and
sending the particular packet (<NUM>) from the first network node (<NUM>);
wherein said adding second IOAM data to the particular packet (<NUM>) is performed in response to at least one of: (i) receiving, by the first network node (<NUM>), notification (<NUM>) of an IOAM processing state that the second network node (<NUM>) is suspending adding IOAM data to packets, and (ii) the first network node (<NUM>) determining IOAM processing capability or IOAM processing state of the second network node (<NUM>) indicating that the second network node (<NUM>) would not add said second IOAM data to the particular packet (<NUM>).