Hardware-friendly mechanisms for in-band OAM processing

In one illustrative example, a network node (e.g. a router or switch) may receive a data packet and timestamp a copy of the data packet. The node may also compute a signature for the copy and insert the signature in a header of the copy. The node may send the copy to a controller for correlation with one or more other timestamped data packet copies of the data packet from one or more other network nodes having the same signature and for the computation of delay. The original data packet may be forwarded to a next network node without any timestamp or other metadata added to it. The processing of the data packets may be performed as part of a function for punting the timestamped data packet copy and forwarding, or as a function for forwarding and punting the timestamped data packet copy.

TECHNICAL FIELD

The present disclosure relates generally to In-band Operations, Administration and Maintenance (iOAM) processing in communication networks.

BACKGROUND

“In-band” or “in-situ” Operations, Administration and Maintenance (iOAM) is one of the requirements in networks such as Fifth Generation (5G) networks. One of the challenges associated with at least some iOAM mechanisms is the difficulty of implementing them in device hardware (e.g. router or switch hardware) without performance penalties.

DESCRIPTION OF EXAMPLE EMBODIMENTS

Overview

Hardware-friendly mechanisms for In-band Operations, Administration and Maintenance (iOAM) processing are described herein.

In one illustrative example, a network node (e.g. a router or switch) may receive a data packet and, if an indicator is identified in a header of the data packet, the network node may obtain a copy of the data packet and timestamp the data packet copy with a timestamp to produce a timestamped data packet copy. The network node may also compute a signature for the timestamped data packet copy and insert the signature in a header of the timestamped data packet copy. The network node may send the timestamped data packet copy to a controller for correlation with one or more other timestamped data packet copies of the data packet from one or more other network nodes having the same signature, to produce correlated timestamped data packet copies. The original data packet may be forwarded to a next network node without any timestamp or other metadata added to it. In some implementations, the processing of the data packets may be performed as part of a function for punting the timestamped data packet copy and forwarding, or as a function for forwarding and punting the timestamped data packet copy. At the controller, a delay associated with the data packet may be computed based on timestamps of the correlated timestamped data packet copies.

In some implementations, the signature may be computed based on at least a portion of a payload of the data packet. The signature may be or based on a cyclical redundancy check (CRC) or a hash of the at least portion of the payload of the data packet. In some implementations, the signature may be inserted as segment ID (SID) information in a segment routing header (SRH) of an SRv6 packet.

In another illustrative example, a controller (e.g. a software-defined network or “SDN” controller or an analytics function such as a network data analytics function or “NWDAF”) may receive from a network node a timestamped data packet copy of a data packet that is forwarded from the network node to a next network node. The controller may obtain a signature from a header of the timestamped data packet copy. The controller may correlate the timestamped data packet copy with one or more other received timestamped copies of the data packet from one or more other network nodes having the same signature as the timestamped data packet copy, to produce correlated timestamped data packet copies. The controller may compute a delay of the data packet between network nodes based on timestamps of the correlated timestamped data packet copies. In some implementations, the processing of the data packets at the network node may be performed as part of a function for punting the timestamped data packet copy and forwarding, or as a function for forwarding and punting the timestamped data packet copy, where the original data packet is forwarded to the next network node without any timestamp or other metadata being added to it.

More detailed and alternative techniques and implementations are provided herein as described below.

Example Embodiments

“In-band” or “In-situ” Operations, Administration and Maintenance (iOAM) is one of the requirements in networks such as Fifth Generation (5G) networks. One of the challenges associated with at least some iOAM mechanisms is the difficulty of implementing them in device hardware without performance penalties.

One way to enable iOAM in a segment routing (SR) for IPv6 (SRv6) based network is to carry data in fields of SR header (SRH) type-length-values (TLVs). However, some hardware implementations may be unable to perform such computationally-extensive TLV manipulation in device hardware without adversely affecting performance (e.g. it may not be achievable at the line rate). Thus, enabling iOAM TLVs for a given data packet may potentially adversely affect the timing of the very stream being monitored and troubleshooted.

According to at least some implementations of the present disclosure, what is proposed is a controller-based approach to iOAM which is fundamentally different than current techniques for iOAM, as it does not need to insert any metadata or TLV in the actual customer packets. The techniques may use real customer traffic to telemeter iOAM data in “copied” data packets to the controller. As is apparent, much complexity for iOAM processing is moved from the devices (e.g. routers, switches) to the controller. By moving complexity from the devices to the controller, device hardware implementation may be easier for certain types (or wider varieties) of device hardware.

Further, note that one of the current limitations in SR Multiprotocol Label Switching (MPLS) based networks is that the data-plane is not designed to carry metadata (TLVs) in the data packets. Therefore, current iOAM mechanisms may not be applicable to SR-MPLS. Advantageously, as techniques of the present disclosure do not require data packets to carry metadata, these techniques may be applied to SR-MPLS networks for iOAM processing.

In some implementations, techniques of the present disclosure may be applied to a 5G network. In a 5G network, SR policies must satisfy relatively strict service level agreement (SLA) criteria with respect to delay. Here, the techniques are suitable to provide both end-to-end and segment-by-segment delay measurement. Notably, segment-by-segment delay measurement allows customers to isolate issues with respect to a specific segment in the network. Further note that the present techniques may also provide proof-of-transit with respect to customer traffic.

To better illustrate,FIG. 1Ais an illustrative representation of a network architecture100A of a 5G mobile network. The mobile network is configured to facilitate communications for a user equipment (UE)102. In general, network architecture100aincludes common control network functions (CCNF)105and a plurality of slice-specific core network functions106. UE102may obtain access to the mobile network via an access network (AN)104, which may be a radio access network (RAN). In the present disclosure, the UEs operating in the 5G mobile network may be any suitable type of devices, such as cellular telephones, smart phones, tablet devices, IoT devices, and machine-to-machine (M2M) communication devices, to name but a few.

CCNF105includes a plurality of network functions (NFs) which commonly support all sessions for UE102. UE102may be connected to and served by a single CCNF105at a time, although multiple sessions of UE102may be served by different slice-specific core network functions106. CCNF105may include, for example, an access and mobility management function (AMF) and a network slice selection function (NSSF). UE-level mobility management, authentication, and network slice instance selection are examples of common functionalities provided by CCNF105.

Slice-specific core network functions of network slices106are separated into control plane (CP) NFs108and user plane (UP) NFs110. In general, the user plane carries user traffic while the control plane carries network signaling. CP NFs108are shown inFIG. 1Aas CP NF1through CP NF n, and UP NFs110are shown inFIG. 1Aas UP NF1through UP NF n. CP NFs108may include, for example, a session management function (SMF), whereas UP NFs110may include, for example, a user plane function (UPF).

FIG. 1Bis an illustrative representation of a more detailed network architecture100B of the 5G mobile network ofFIG. 1A. As provided in 3GPP standards for 5G (e.g. 3GPP 23.501 and 23.502), network architecture100bfor the 5G mobile network which is operated by a mobile network operator (MNO) may include an authentication server function (AUSF)116, a unified data management (UDM)118(having a unified data repository or UDR), an AMF112, a policy control function (PCF)114, an SMF120, and a UPF122. One or more application functions, such as an application function (AF)124, of a data network (DN)111may connect to the 5G mobile network via PCF114or UPF122. A plurality of interfaces and/or reference points N1-N8, N10-N13, N15, and N22shown inFIG. 1B(as well as others) may define the communications and/or protocols between each of the entities, as described in the relevant (evolving) standards documents.

UPF122is part of the user plane and all other NFs (i.e. AMF112, SMF120, PCF114, AUSF116, and UDM118) are part of the control plane. Separation of user and control planes guarantees that each plane resource can be scaled independently; it also allows UPFs to be deployed separately from CP functions in a distributed fashion. The NFs in the CP are modularized functions; for example, AMF and SMF are independent functions allowing for independent evolution and scaling. As specifically illustrated inFIG. 1B, NFs such as SMF120and UPF122ofFIG. 1Bmay be provided as specific instances in a first network slice (e.g. network slice 1) of a plurality of network slice instances made available in the mobile network.

Also as shown inFIG. 1B, a network exposure function (NEF)150may be provided in the mobile network. In general, NEF150may be configured to receive information from other NFs (e.g. based on exposed capabilities of other NFs) and store the received information as structured data. The storage operations may be performed with use of a standardized interface to a data storage network function. The stored information may be re-exposed by NEF150to other NFs and used for other purposes (e.g. analytics). One example use of NEF150is to assist in the establishment of an AS-initiated communication with a UE/IoT device, from an application server (AS)180through an API (e.g. where no other existing data connection exists).

As shown in a network node arrangement180ofFIG. 1C, an analytics function190may also be provided in the 5G mobile network. Analytics function190may be a network data analytics function (NWDAF) as described in the relevant (evolving) standards documents, such as TS 23.503 and TS 29.520. Analytics function190may be used for data collection and data analytics in a centralized manner. Analytics function190may receive activity data and local analytics from NFs170or an AF172, and/or access data from one or more data repositories178. Resulting analytics data may be generated and sent or otherwise provided by analytics function190to the NFs and/or AF172. Data for OAM may be provided to and from network nodes or OAM-processing nodes176.

In the present disclosure, implementations may make use of segment routing (SR) for communications in the network. The SR may be SR for IPv6 (SRv6). To illustrate a few SRv6 examples,FIGS. 2A-2Dare block diagrams of network nodes which are configured to route packets using SRv6.

With reference first toFIG. 2A, a network200awhich includes a plurality of nodes202(e.g. routers, servers, base stations, gateways, CP or UP entities, etc.) is shown. In this example, the plurality of nodes202includes nodes210,212,214,216,218,220,222,224, and226which are designated as nodes A, B, C, D, E, F, G, H, and Z, respectively. Here, node210(i.e. node A) is considered to be a source node and node226(i.e. node Z) is considered to be a destination node. Nodes212,214,216,218,220,222, and226which correspond to nodes B, C, D, E, F, and G are part of an SR domain (i.e. nodes that are SRv6-capable nodes/SRv6-configured nodes). The source node (node210or A) and the destination node (node226or Z) are not part of or outside of the SR domain (e.g. they may or may not be SRv6-configured nodes, such as “regular” IPv6 nodes).

A basic data format of an SR-IPv6 packet260for use in SRv6 routing is also shown inFIG. 2A. As illustrated, the data format of SR-IPv6 packet260includes an IPv6 header262and a payload264. For SRv6 routing of IPv6 packet260, the data format of IPv6 packet260further includes an SR header270or “SRH” (i.e. an extension header for SR as defined by RFC2460). SR header270may include an ordered list of segments272which defines a network path250along which the SR-IPv6 packet260will be communicated in network200a. In the example ofFIG. 2A, the ordered list of segments272includes node214(“node C”), node220(“node F”), and node224(“node H”) in network path250. A segment is or includes an instruction (e.g. forwarding, servicing, application-specific, etc.) to be applied to the SR-IPv6 packet260. Thus, an SR-IPv6 packet (e.g. SR-IPv6 packet260) may be communicated in network200afrom a source node (e.g. node210or A) to a destination node (e.g. a node226or Z) along a desired or predetermined network path250. The source node (e.g. node210or A) may operate to choose this network path250and encode it in the SR header270as the ordered list of segments272. The rest of network200amay operate to execute the encoded instructions without any further per-flow state.

FIG. 2Bis an illustrative representation of a network200bwhich is similar to network200aofFIG. 2A. Here, nodes212,214,216,218,220,222, and226which correspond to nodes B, C, D, E, F, and G are shown to be part of an SR domain280. The source node (node210or A) and the destination node (node226or Z) are not part of or outside of the SR domain280(e.g. they may or may not be SRv6-configured nodes). In the example ofFIG. 2B, node212or B may be considered as an ingress node of the SR domain280and node222or G may be considered as an egress node of the SR domain280.

Note that an SR header may be inserted in an IPv6 packet at a source node or at an ingress node, or even encapsulated at the ingress node, as a few examples. In the example shown inFIG. 2B, an SR header of an IPv6 packet is inserted at the source node (node210or A) to produce an SR-IPv6 packet290b. In this case, the source node (node210or A) which is SRv6-capable may originate the SR-IPv6 packet290b. Here, the SR header of SR-IPv6 packet290bincludes an ordered list of segments (SL) designating nodes B, D, G, and Z to define network path250. Initially, a source address (SA) of SR-IPv6 packet290bis designated as node A and a destination address (DA) of SR-IPv6 packet290bis designated as node B (i.e. the first node in the SL). When SR-IPv6 packet290bis communicated to the ingress node (i.e. node212or B), the DA is modified by the ingress node to include the next or second node in the SL (i.e. node D), as indicated in SR-IPv6 packet292b. When SR-IPv6 packet292bis communicated to the node D (via node C), the DA is modified by node D to include the next or third node in the SL (i.e. node G), as indicated in SR-IPv6 packet294b. When SR-IPv6 packet294bis further communicated to the node G (via node F), the DA is modified by node G to include the next or fourth node in the SL (i.e. node Z which is the destination node), as indicated in SR-IPv6 packet296b.

In the example ofFIG. 2C, an SR header of an IPv6 packet290cis inserted at the ingress node (node212or B) to produce an SR-IPv6 packet292c. Here, the SR header of SR-IPv6 packet292cincludes an ordered list of segments (SL) designating nodes D, G, and Z to define network path250. In this case, the source node, which may or may not be SRv6-configured, may originate the IPv6 packet290cwithout any SR header. When SR-IPv6 packet292cis communicated to node D (via node C), the DA is modified by node D to include the next or second node in the SL (i.e. node G), as indicated in SR-IPv6 packet294c. When SR-IPv6 packet294cis further communicated to the node G (via node F), the DA is modified by node G to include the next or third node in the SL (i.e. node Z, which is the destination node) and the SR header is removed, as indicated in IPv6 packet296c. Here, similar to the source node, the destination node may or may not be SRv6-configured.

In the example ofFIG. 2D, the source node, which may or may not be SRv6-configured, originates an IPv6 packet290dwithout any SR header. The ingress node (node212or B) operates to encapsulate IPv6 packet290dwith a new, outer IPv6 header followed by an SR header, to produce an SR-IPv6 packet292d. The SL of the SR header includes nodes D and G, but does not include the destination node (node226or Z). When SR-IPv6 packet292dis communicated to node D (via node C), the DA is modified by node D to include the next or second node in the SL (i.e. node G), as indicated in SR-IPv6 packet294d. When SR-IPv6 packet294dis further communicated to the node G (via node F), the SR-IPv6 packet294dis decapsulated by node G, which is represented by SR-IPv6 packet296d. Here, similar to the source node, the destination node may or may not be SRv6-configured.

Note that the current state of the art for SRv6 is further described in various standards-related documents, including Internet Engineering Task Force (IETF) documents, and Internet-Drafts, such as “Segment Routing Architecture” identified by “draft-ietf-spring-segment-routing-14,” C. Filsfils et al., Dec. 20, 2017; “IPv6 Segment Routing Header (SRH)” identified by “draft-ietf-6man-segment-routing-header-07,” C. Filsfils et al., Jul. 20, 2017; and “SRv6 Network Programming” identified by “draft-filsfils-spring-srv6-network-programming-04,” C. Filsfils et al., Mar. 4, 2018, each of which are hereby incorporated by reference as though fully set forth herein.

FIG. 3is an illustrative representation of a network node arrangement300aof an SR network302for describing an example of metadata or iOAM processing. The SR network302includes a plurality of switches304designated as S1, S2, S3, and S4, and a controller306(e.g. an analytics server). In this example, an incoming data packet320to the SR network302may be an SRv6 packet. The incoming data packet320may be received and stamped at S1(switch 1 metadata), forwarded from S1to be received and again stamped at S2(switch 2 metadata), forwarded from S2to be received and again stamped at S3(switch 3 metadata), forwarded from S4to be received at the controller for processing. An outgoing data packet322may leave the SR network302. As indicated inFIG. 3, a packet310received at the controller306may include at least switch 1 metadata, switch 2 metadata, and switch 3 metadata in its header (e.g. SRH). If such metadata is carried in SRH TLVs of a customer data packet, the existing hardware in the switches S1, S2, and S3may be unable to perform computationally-extensive TLV manipulation without adversely affecting performance associated with the packet (e.g. it may not be achievable at the line rate). Thus, such iOAM processing may potentially adversely affect the timing of the very stream being monitored and troubleshooted.

FIG. 4Ais an illustrative representation of a network node arrangement400aof an SR network402for describing an example of metadata or iOAM processing according to some implementations of the present disclosure. The SR network402includes a plurality of switches404designated as S1, S2, S3, and S4, and a controller406(e.g. an analytics server). In this example, an incoming data packet420to the SR network402may be an SRv6 packet. The incoming data packet420may be received at S1, where a stamped copy of the packet (e.g. stamped with a timestamp or other metadata) may be sent (e.g. telemetered) to controller406from S1with a signature inserted with the packet copy. The original data packet (e.g. without any S1stamping of iOAM data) may be forwarded from S1to be received at S2, where a stamped copy of the packet (e.g. stamped with a timestamp or other metadata) may be sent (e.g. telemetered) to controller406from S2with a signature inserted with the packet copy. The original data packet (e.g. without any S1and/or S2stamping of iOAM data) may be forwarded from S2to be received at S3, again where a stamped copy of the packet (e.g. stamped with a timestamp or other metadata) may be sent (e.g. telemetered) to controller406from S3with a signature inserted with the packet copy. The original data packet (e.g. without any S1, S2, and/or S3stamping of iOAM data) may be forwarded from S3to be received at S4, again where a stamped copy of the packet (e.g. stamped with a timestamp or other metadata) may be sent (e.g. telemetered) to controller406from S4with a signature inserted with the packet copy. An outgoing data packet422(e.g. without any S1, S2, S3, and/or S4stamping of iOAM data) may leave the SR network402.

InFIG. 4A, processing associated with producing a stamped copy of the original data packet (e.g. stamped with a timestamp or other metadata) for communication to the controller may involve a function for “punting a stamped copy and forwarding.” As an alternative, the processing associated with producing the stamped copy of the original data packet may involve a function for “forwarding and punting a stamped copy” instead of the “punting a stamped copy and forwarding.” The latter technique makes it easier for device hardware to implement. The function to “forward and punt a time-stamped copy of the packet” processes the packet without head-of-line blocking in switching the packet, and hence is more efficient and hardware-friendly.

Regarding the above, a network node may process a packet at one of a plurality of different switching method levels. “Punting” may be characterized or viewed as an action by an interface's device driver of sending a packet “down” to the next lower switching level for communication. As one specific example, a list of switching method levels or paths in order from fastest to slowest may be as follows: (1) a distributed express-forwarding of a packet with use of a hardware application-specific integrated circuit (ASIC); (2) a (non-distributed) express forwarding of a packet with use of the hardware ASIC; (3) a fast switching; and (4) a process switching. The first two switching method levels (“fast paths”) make use of hardware ASICs in the devices, but others may make use of a slower path in software associated with a route processor (RP) CPU.

At the controller406ofFIG. 4A, some or all of the copied data packets with iOAM stampings and signatures may be received, and these copied data packets and/or metadata may be grouped together or correlated based on the signatures (e.g. analytics is performed on the received copies having the same signature).

To better illustrate with reference to a network node arrangement400bofFIG. 4B, a plurality of stamped data packet copies460of various data packets may be telemetered or otherwise sent to controller406. These data packet copies460may be received at controller406in a scattered or arbitrary fashion. Each one of the stamped data packet copies460includes a signature, where each signature is uniquely associated with a corresponding original data packet. Controller406may perform processing and sorting of the data packet copies460in a plurality of logical or physical groupings490of data packet copies based on the signatures. Here, each stamped data packet copy may be correlated with one or more other stamped data packet copies of the data packet from one or more other network nodes having the same signature. InFIG. 4B, the plurality of groupings490include a grouping472for “signature 1” data packets and a grouping474for “signature 2” data packets. In some implementations, controller406may use the sending network node's address (e.g. IP address) or other identification to perform such analysis and computation.

At the controller406, one or more delay measurements associated with the data packet may be computed based on timestamps of the correlated timestamped data packet copies. As illustrated in a network node arrangement400cofFIG. 4C, controller406may compute an end-to-end delay measurement450between ingress and egress nodes, and/or one or more segment delay measurements452,454,456between network nodes.

FIGS. 5A, 5B, and 5Care illustrative representations of stamped copies500a,500b, and500c, respectively, of an original data packet which has traversed a plurality of routers or switches (e.g. S1, S2, and S3ofFIGS. 4A-4C), where the stamped copies may be forwarded to a controller for iOAM processing, according to some implementations. InFIG. 5A, the message format500aof the copied data packet includes a header502a(e.g. a local or punt header) and a payload504awhich may include payload data. Header502amay include router- or switch-added data510a(i.e. data that are added to or inserted in the header502aby the router or switch) which may be referred to as iOAM data. More specifically inFIG. 5A, header502amay include a timestamp “1” (TS1) (and/or other iOAM data) which is stamped at the router/switch S1. InFIG. 5B, the message format500bof the copied data packet includes a header502b(e.g. a local or punt header) and a payload504bwhich may include payload data. Header502bmay include router- or switch-added data510b(i.e. data that are added to or inserted in the header502bby the router or switch) which may be referred to as iOAM data. More specifically inFIG. 5B, header502bmay include a timestamp “2” (TS2) (and/or other iOAM data) which is stamped at the router/switch S2. InFIG. 5C, the message format500cof the copied data packet includes a header502c(e.g. a local or punt header) and a payload504cwhich may include payload data. Header502cmay include router- or switch-added data510c(i.e. data that are added to or inserted in the header502cby the router or switch) which may be referred to as iOAM data. More specifically inFIG. 5C, header502cmay include a timestamp “3” (TS3) (and/or other iOAM data) which is stamped at the router/switch S3.

Also as indicated inFIGS. 5A, 5B, and 5C, data that are used for a (unique) signature corresponding to the (original) data packet may include or be based on at least a portion of the payload data. Each signature may be inserted in the corresponding header502a,502b, or502c(i.e. the local or punt header). The signatures associated with these copied data packets for the same data packet are the same. In some implementations, each signature may be or based on a cyclical redundancy check (CRC) or a hash of the at least portion of the payload. These data packet copies with iOAM stampings and signature data may be received at the controller, where the data packet copies and/or metadata (e.g. timestamps) are grouped together or correlated based on the signatures (e.g. analytics is performed on the received copies having the same signatures).

FIGS. 6A, 6B, and 6Care illustrative representations of stamped copies600a,600b, and600c, respectively, of an original data packet which has traversed a plurality of routers or switches (e.g. S1, S2, and S3ofFIGS. 4A-4C), where the stamped copies may be forwarded to a controller for iOAM processing, according to some implementations. The stamped copies of the data packet ofFIGS. 6A-6Care substantially the same as those described in relation toFIGS. 5A-5C, except that the data that are used for a (unique) signature corresponding to the (original) data packet may be included as segment ID (SID) information in an SRH header.

More specifically, inFIG. 6A, a message format600aof the copied data packet includes a header602a1(e.g. a local or punt header), a header602a2(e.g. an SRH), and a payload604awhich may include payload data. Header602a1may include the router- or switch-added data510a(i.e. data that are added to or inserted in the header602a1by the router or switch) which may be referred to as iOAM data, and header602a2which is the SRH may include SID information corresponding to the signature. More specifically inFIG. 6A, header602a1may include a timestamp “1” (TS1) (and/or other iOAM data) which is stamped at the router/switch S1. InFIG. 6B, the message format600bof the copied data packet includes a header602b1(e.g. a local or punt header), a header602b2(e.g. an SRH), and a payload604bwhich may include payload data. Header602b1may include the router- or switch-added data510b(i.e. data that are added to or inserted in the header602b1by the router or switch) which may be referred to as iOAM data, and header602b2which is the SRH may include SID information corresponding to the signature. More specifically inFIG. 6B, header602b1may include a timestamp “2” (TS2) (and/or other iOAM data) which is stamped at the router/switch S2. InFIG. 6C, the message format600cof the copied data packet includes a header602c1(e.g. a local or punt header), a header602c2(e.g. an SRH), and a payload604cwhich may include payload data. Header602c1may include the router- or switch-added data510c(i.e. data that are added to or inserted in the header602c1by the router or switch) which may be referred to as iOAM data, and header602c2which is the SRH may include the SID information corresponding to the signature. More specifically inFIG. 6C, header602c1may include a timestamp “3” (TS3) (and/or other iOAM data) which is stamped at the router/switch S3. The signatures associated with these copied data packets may be the same. Again, these data packet copies with iOAM stampings and signature data may be received at the controller, where the data packet copies and/or metadata (e.g. timestamps) may be grouped together or correlated based on the signatures (e.g. analytics is performed on the received copies having the same signatures).

FIG. 7Ais a flowchart700afor describing a method of metadata or iOAM processing according to some implementations of the present disclosure. The method ofFIG. 7Amay be performed at a network node, such as a router or switch (e.g. S1, S2, and/or S3ofFIG. 4B). The network node may include one or more processors, one or more memories coupled to the one or more processors, and one or more network/communication interfaces or ports. The method may be embodied as a computer program product including a non-transitory computer readable medium and instructions stored in the computer readable medium, where the instructions are executable on one or more processors of the network node for performing the steps of the method.

Beginning at a start block ofFIG. 7A, the network node may receive a data packet (step704ofFIG. 7A). The network node may identify whether an indicator is included in a header of the packet (step706ofFIG. 7A). This indicator may be, for example, a bit indicator (e.g. a predefined bit such as the ‘O’ bit) or a label indicator (e.g. a predefined label such as a predefined MPLS label). In alternative implementations, the indicator (and the detection thereof) is not needed, where the appropriate processing may be inferred based on other data. If the indicator is not identified in the header, then the network node may perform normal processing for forwarding the data packet to a next network node (step708ofFIG. 7A). If the indicator is identified in the header, then the network node may still perform the processing for forwarding the data packet at step708. Here, however, the network node may perform a function to punt a stamped copy of the data packet and forward (strep710aofFIG. 7A), or alternatively perform a function to forward and punt a stamped copy of the data packet (step710bofFIG. 7A). The network node may compute a signature corresponding to the original data packet and insert the signature into a header of the stamped data packet copy (step712ofFIG. 7A). The signature may be used by the controller to group together or correlate the data packet copies of the original data packet for processing of the metadata (e.g. timestamps).

FIG. 7Bis a flowchart700bfor describing a method of metadata or iOAM processing according to some implementations of the present disclosure. The method ofFIG. 7Amay be performed at a network node, such as a router or switch (e.g. S1, S2, and/or S3of FIG.4B). In some implementations, the steps inFIG. 7Bmay be performed as part of steps710and712ofFIG. 7A(i.e. performed as part of the punting-related steps).

InFIG. 7B, a network node (e.g. a router or switch) may receive a data packet and produce or otherwise obtain a copy of the data packet (step720ofFIG. 7B). The network node may timestamp the data packet copy with a timestamp (step722ofFIG. 7B). The network node may also compute a signature for the timestamped data packet copy and insert the signature in a header of the timestamped data packet copy (step724ofFIG. 7B). The network node may send the timestamped data packet copy to a controller for correlation with one or more other timestamped data packet copies of the data packet from one or more other network nodes having the same signature (step726ofFIG. 7B). In these steps, the processing of the data packets may be performed as part of a function for punting the timestamped data packet copy and forwarding, or as a function for forwarding and punting the timestamped data packet copy. The original data packet may be forwarded to a next network node without any timestamp or other metadata added to it. At the controller, a delay associated with the data packet may be computed based on timestamps of the correlated timestamped data packet copies (controller processing step728ofFIG. 7B).

In some implementations of step724, the signature may be computed based on at least a portion of a payload of the data packet. The signature may be or based on a CRC or a hash of the at least portion of the payload of the data packet. In some implementations, the signature may be inserted as SID information in an SRH of an SRv6 packet.

FIG. 8is a flowchart for describing a method of metadata or iOAM processing according to some implementations of the present disclosure. The method ofFIG. 8may be performed at a controller (e.g. the controller ofFIGS. 4A-4C), such as an SDN controller or other analytics function, such as an NWDAF. The controller may include one or more processors, one or more memories coupled to the one or more processors, and one or more network/communication interfaces or ports. The method may be embodied as a computer program product including a non-transitory computer readable medium and instructions stored in the computer readable medium, where the instructions are executable on one or more processors of controller for performing the steps of the method.

Beginning at a start block ofFIG. 8, the controller may receive from a network node a timestamped copy of a data packet that is forwarded from the network node to a next network node (step804ofFIG. 8). The controller may obtain a signature from a header of the timestamped data packet copy (step806ofFIG. 8). The controller may group together or otherwise correlate the timestamped data packet copy with one or more other received timestamped copies of the data packet from one or more other network nodes having the same signature as the timestamped data packet copy (step808ofFIG. 8). The controller may perform analytics with respect to the timestamps and/or other metadata (step810ofFIG. 8). Here, the controller may compute a delay of the data packet between network nodes based on timestamps of the correlated timestamped data packet copies. The controller may also use the sending network node's address (e.g. IP address) or other identification to perform such analysis and computation.

With respect to the data packet copies received from the network nodes (e.g. step804), the processing of data packets at the network nodes may involve a function for punting the timestamped data packet copy and forwarding, or as a function for forwarding and punting the timestamped data packet copy, as described earlier, where the original data packet is forwarded to the next network node without any timestamp or other metadata being added to it.

In some implementations ofFIG. 8, the signature used for correlation may be computed based on at least a portion of a payload of the data packet. The signature may be or based on a CRC or a hash of the at least portion of the payload of the data packet. In some implementations, the signature may be inserted as SID information in an SRH of an SRv6 packet. In other implementations ofFIG. 8, a characteristic of the copied data packet other than a signature is used for correlation at the controller.

According to some implementations described herein, a controller (e.g. an SDN controller) may initiate analysis with respect to in-band segment-by-segment or end-to-end delay encountered by customer traffic associated with a flow or steered via an SR policy. During observation of customer traffic as it travels through an SR network, techniques of the present disclosure need not insert or carry any OAM metadata in the packet.

In some implementations, the technique may rely on the “O-bit” in SRH to instruct “punt a time-stamped copy of the packet and forward” or “forward and punt a time-stamped copy of the packet” behavior. In SR-MPLS implementations, the use of a “sampling label” or “GAL/GAch labels” to instruct a “punt a time-stamped copy of the packet and forward” or “forward and punt a time-stamped copy of the packet” behavior. The punted copy of the packets may be timestamped in hardware. The punted packets may be processed in the “slow path” and telemetered to the SDN controller. The end-to-end or segment-by-segment delays may be computed at the SDN controller. A proof-of-transit may also be established at the SDN controller.

Specific techniques are now described further below (e.g. elaborating or providing alternatives to the operation described in relation toFIGS. 4, 5A-5C, 6A-6C, 7, and 8), in relation to specific processing at the different nodes as well as the controller.

FIG. 9Ais a network node arrangement900afor describing metadata or iOAM processing in relation to an ingress node in a communication network according to some implementations. InFIG. 9A, the communication network may include a plurality of network nodes902(e.g. routers, switches) which may be in communication with a controller904. Network nodes902may include (in order from left to right) a customer edge (CE) router910, a provider edge (PE) router920, a router922, a PE router924, and a CE router912. PE router920may be referred to as the ingress node.

In-band OAM may be enabled for a particular flow or a policy. Matching to these flows may be based on incoming flow identifiers or based on steering into an SR policy. Incoming packets may be matched and “sampled” for in-band OAM. InFIG. 9A, the controller904may initiate measurement for in-band segment-by-segment or end-to-end delay encountered by customer traffic associated with a flow or steered via an SR policy (indicated at an instruction process “1” inFIG. 9A). Here, the ingress router (i.e. PE router920) may receive an incoming data packet930. For the packets that are internally marked for sampling, the O-bit may be set in the inserted SRH (to instruct sampling at the egress node), as indicated in an outgoing data packet932a. The data-plane may follow a “punt a time-stamped copy of the packet and forward” or a “forward and punt a time-stamped copy of the packet” behavior. No metadata is or needs to be carried in the forwarded packet (forwarded packet is not manipulated), as shown in the outgoing data packet932a. Controller904may receive a timestamped copy of some of these sampled packets from PE router920(i.e. the ingress), PE router924(i.e. the egress), and each segment node (e.g. router922). The controller904may correlate copied data packets from other network nodes based on signatures uniquely corresponding to the original data packet. The controller904may key or sort the copied data packets based on the signature and perform analytics (e.g. compute delay measurements).

FIG. 9Bis a network node diagram for describing metadata or iOAM processing in relation to the ingress node ofFIG. 9A, which includes driving a packet signature in the slow path according to some implementations. A data packet copy932b-copy (“punted copy”) may be obtained. A hardware timestamp944(T×1) may also be obtained (e.g. from a line card or “LC”940) and provided in a punt header of the data packet copy. The slow path may take data packet copy932b-copy with hardware timestamp (T×1), where a CPU942may compute a signature of the packet and insert it in the punt header with the hardware timestamp (T×1), as shown in a data packet copy934b-copy.

In some implementations, the signature may be derived based on at least a portion of the payload. For example, the signature may be derived from four (4) bytes of data at the tail end of the payload (or other offset and/or chunk-size of data). In some other implementations, the signature may be or include a CRC or a type of hash or secure hash algorithm (SHA). Both the ingress and egress nodes may use the same logic or function in computing the signature. The signature of the packet is used by the controller to correlate copy of the same packet independently sent by the ingress and the egress nodes. No metadata needs to be added to the original data packet.

FIG. 9Cis a network node diagram for describing metadata or iOAM processing in relation to an ingress node, with use of a SID-based signature according to some implementations. This procedure may be complementary to driving the signature from the packet payload itself. This procedure may use SID information, and may be very hardware-efficient as it does not need to employ any TLV manipulation. The SID function may accessible in the hardware and operation to drive the signature may be performed when the SID is being accessed by the data-plane.

Again, a data packet copy932c-copy (“punted copy”) may be obtained. The hardware timestamp944(T×1) may also be obtained (e.g. from the LC940) and provided in a punt header of the copy. Here, the fast path may insert (signature=BSID|Sequence Number) in the argument of SRH.SID[SL]—48 bits. This may use SRH encoding without the .REDUCE function. The slow path may receive the punted copy with the hardware timestamp (T×1) and signature, where CPU942may export the copy (i.e. a data packet copy934c-copy) to controller904in accordance with streaming telemetry or, alternatively, netflow export may be used. Note that since the information is encoded in the argument of the SID, the SID value may still be used for any other purpose (e.g., SRH reversal, etc.)

As described previously, in SRv6 Network Programming (draft-filsfils-spring-srv6-network-programming-04), the SRH.Flags.O-bit may be used to instruct a “punt a time-stamped copy of the packet and forward” behavior. This feature may be extended to include a “forward and punt a time-stamped copy of the packet” behavior as it is easier for the hardware to implement. The “forward and punt a time-stamped copy of the packet” processes the packet without head-of-line blocking in switching the packet, and hence is more efficient and hardware-friendly. Thus, the SRH.Flags.O-bit may be used to instruct a “Punt a Timestamped Copy and Forward” behavior or a “Forward and Punt a Timestamped Copy” behavior.

FIG. 10Ais a network node arrangement1000afor describing metadata or iOAM processing in relation to an egress node in a communication network according to some implementations. PE router924may be referred to as the egress node. At the egress node, data packet932amay arrive with the SRH.Flags.O=1. If SRH.Flags.O-bit=1, the egress node may implement a “Punt a Timestamped Copy and Forward” behavior or a “Forward and Punt a Timestamped Copy” behavior with respect to this data packet932a. A hardware-based timestamp may be inserted in a punt header of the punted copy of the packet, and the timestamped punted copy may be sent to the controller together with the inserted signature. With respect to the original data packet, the egress router may operate to remove the SRH, and the resulting packet may be forwarded to the CE router912as a data packet1032aas shown.

FIG. 10Bis a network node diagram for describing metadata or iOAM processing in relation to an egress node, with driving a packet signature in the slow path according to some implementations. The procedure is the same or substantially the same as observed by the ingress node (and segment nodes). A data packet copy1032b-copy (“punted copy”) may be obtained. A hardware timestamp1044(R×2) may also be obtained (e.g. from a line card or “LC”1040) and provided in a punt header of the data packet copy. The slow path may take data packet copy1032b-copy with hardware timestamp (R×2), where a CPU1042may compute a signature of the packet and insert it in the punt header with the hardware timestamp (R×2), as shown in a data packet copy1034b-copy. The same logic used by the ingress/segment nodes is applied to compute the signature of the packet (e.g. apply the same CRC). Note that the diagram ofFIG. 10Bassumes the “Forward and Punt a Timestamped Copy” behavior; however, the procedure is equally applicable to a “Punt a Timestamped Copy and Forward” behavior.

FIG. 10Cis a network node diagram for describing metadata or iOAM processing in relation to an egress node, with use of a SID-based signature according to some implementations. This procedure may be complementary to driving the signature from the packet payload itself. This procedure may use SID information, and may be very hardware-efficient as it does not need to employ any TLV manipulation. The SID function may accessible in the hardware and operation to drive the signature may be performed when the SID is being accessed by the data-plane.

Here, the procedure may be the same or substantially the same as observed by the ingress node (and segment nodes) perFIG. 9C. Again, a data packet copy1032c-copy (“punted copy”) may be obtained. The hardware timestamp1044(T×1) may also be obtained (e.g. from the LC1040) and provided in a punt header of the copy. Here, the fast path may insert (signature=BSID|Sequence Number) in the argument of SRH.SID[SL]—48 bits. The policy ID and sequence number may already be encoded in the packet at argument of the first SID. The slow path may receive the punted copy with the hardware timestamp (T×1) and signature, where CPU1042may export the copy (i.e. a data packet copy1034c-copy) to controller904in accordance with streaming telemetry or, alternatively, NetFlow export may be used. Note that since the information is encoded in the argument of the SID, the SID value may still be used for any other purpose (e.g., SRH reversal, etc.)

Regarding processing at segment end nodes, packets may arrive with the SRH.Flags.O=1. As the SRH.Flags.O bit=1, a segment end node may instruct a “Punt a Timestamped Copy and Forward” behavior or a “Forward and Punt a Timestamped Copy” behavior. The processing at the segment end node is the same or very similar to the processing at the egress node and hence it is not repeated here.

FIG. 11is a table1100of example data associated with the correlating of stamped packets at a controller according to some implementations. The controller may receive timed-stamped punted copies of the same packet (as well as different packets) from ingress, egress and each segment node. Each copy of the original data packet may be associated with a packet signature (e.g. packet copy 1 is associated with signature 1; packet copy 2 is associated with signature 2; and packet copy 3 is associated with signature 3). For copies of the same original data packet, these signatures will be the same. The controller may process and sort the packet copies based on the signatures (and e.g. the sending node's IP address or other identification). Other relevant information is shown in table1100, including a timestamp associated with each node and computed time differentials or delays as indicated. As is apparent, the controller may correlate the time stamps to identify segment-by-segment delays or end-to-end delay. The controller may also perform a proof-of-transit check using the copy of the same packet received from ingress, egress and the transit nodes. Note further that the controller may be configured to handle packet loss. In fact, it may detect losses in a sampled stream (e.g. especially when sequence number is used in SRH). The controller may be configured to handle out-of-order packets, as sampled packets may be keyed on signature. For example, T2 may be greater than T4 per the table.

The techniques described above may be applicable to other networks and/or network protocols. For example, the techniques may be applied to SR-MPLS-based networks. One of the current limitations of SR-MPLS data-plane is that it cannot carry metadata (TLVs) in the data packets. Therefore, at least some current “In-situ OAM” mechanisms are not applicable to SR-MPLS. As the SDN-based procedure proposed herein does not require data packets to carry any meta-data, the procedure may advantageously be applied to such SR-MPLS-based networks.

For SR-MPLS, iOAM processing may be instructed with use of an indicator according to some implementations as follows. For one, an indicator may be provided in the most significant byte (MSB) in a time-to-live (TTL) field of MPLS (VPN) label. On the other hand, an indicator may be provided using a Generic Associate Label (Gal) or Generic Associated Channel (GAch) Labels with a control word. Further, an indicator may be provided using a local label assigned by an egress node. Finally, an indicator may be provided with use of a newly defined, special-purpose label.

As is apparent, an iOAM “sampling” label may be used in SR-MPLS for the same or similar purpose as the SRH.Flags.O bit in SRv6. This is an SDN-based approach where an SDN controller may co-ordinate a sampling label between ingress and egress nodes of an SR-MPLS policy. The sampling label may instruct the egress node that, if the traffic arrives with the sampling label, it is marked in-band OAM using a “Punt a Timestamped Copy and Forward” behavior or a “Forward and Punt a Timestamped Copy” behavior.

FIG. 12Ais a network node diagram1200afor describing metadata or iOAM processing associated with an SR-MPLS based network. The work flow may be as follows. The controller904(e.g. SDN controller) learns that customers would like to enable iOAM on customer traffic with respect to a given policy. The controller904may solicit a sampling label L1from egress node (i.e. PE router924) (i.e. the end point of the SR MPLS policy). The egress node may assign a label (e.g. “L1”) for sampling. Here, “L1” is a local label at the egress node; the egress understands the semantics that it is to use for sampling. The controller904may communicate this label L1to the ingress node (i.e. PE router920) as the “sampling” label. For the packets that are internally marked for sampling, the sampling label L1is pushed (e.g. to convey sampling to the egress). Thus, inFIG. 12A, an incoming data packet1202is communicated to PE router920(i.e. the ingress node), which communicates an MPLS data packet1204ahaving sampling label L1to network node922, which communicates an MPLS data packet1206ahaving sampling label L1to PE router924(i.e. the egress node), which communicates an outgoing data packet1208(non-MPLS) to CE router912. If L1is identified at the egress node (i.e. PE router924), the network node may implement a “Punt a Timestamped Copy and Forward” behavior or a “Forward and Punt a Timestamped Copy” behavior. No metadata needs to be inserted or carried in the forwarded packet (the signature is derived from the payload).

FIG. 12Bis a network node diagram1200bfor describing metadata or iOAM processing in an SR-MPLS network. Here, the MSB in the TTL field of MPLS (VPN) label may be used for the same purpose as the SRH.Flags.O bit for SRv6. For packets that are internally marked for sampling, the TTL for the bottom most label is set with MSB=1. InFIG. 12B, incoming data packet1202is communicated to PE router920(i.e. the ingress node), which communicates an MPLS data packet1204bhaving the predefined TTL setting (i.e. MSB=1) to network node922, which communicates an MPLS data packet1206ahaving the predefined TTL setting to PE router924(i.e. the egress node), which communicates an outgoing data packet1208(non-MPLS) to CE router912. If the predefined TTL setting is identified the egress node (i.e. PE router924), the network node may implement a “Punt a Timestamped Copy and Forward” behavior or a “Forward and Punt a Timestamped Copy” behavior. No metadata needs to be inserted or carried in the forwarded packet (the signature is derived from the payload).

FIG. 13illustrates a block diagram of a network element or node1300(e.g. a router or switch) configured to perform operations described above, for example, in connection withFIG. 7and associated figures. The network node1300includes one or more control processors1310, memory1320, a bus1330and a network processor unit1340. The control processor1310may be a microprocessor or microcontroller. The network processor unit1340may include one or more ASICs, linecards, etc., and facilitates network communications between the node1300and other network nodes.

There are a plurality of network ports1342at which the node1300receives packets and from which the node1300sends packets into the network. The processor1310executes instructions associated with software stored in memory1320. Specifically, the memory1320stores instructions for control logic1350that, when executed by the processor1310, causes the processor1310to perform various operations on behalf of the node1300as described herein. The memory1320also stores configuration information1360received from a network controller to configure the network node according to desired network functions. It should be noted that in some embodiments, the control logic1350may be implemented in the form of firmware implemented by one or more ASICs as part of the network processor unit1340.

The memory1320may include read only memory (ROM) of any type now known or hereinafter developed, random access memory (RAM) of any type now known or hereinafter developed, magnetic disk storage media devices, tamper-proof storage, optical storage media devices, flash memory devices, electrical, optical, or other physical/tangible memory storage devices. In general, the memory1320may comprise one or more tangible (non-transitory) computer readable storage media (e.g., a memory device) encoded with software comprising computer executable instructions and when the software is executed (by the processor1310) it is operable to perform certain network node operations described herein.

FIG. 14illustrates a block diagram of a computing/control entity1400that may perform the functions of the controller (e.g. SDN) or other analytics entity as described herein. The computing/control entity1400includes one or more processors1410, memory1420, a bus1430and a network interface unit1440, such as one or more network interface cards that enable network connectivity. The memory1420stores instructions for control and management logic1450, that when executed by the processor1410, cause the processor to perform the software defined network controller operations described herein.

The memory1420may include ROM of any type now known or hereinafter developed, RAM of any type now known or hereinafter developed, magnetic disk storage media devices, tamper-proof storage, optical storage media devices, flash memory devices, electrical, optical, or other physical/tangible memory storage devices. In general, the memory1420may comprise one or more tangible (non-transitory) computer readable storage media (e.g., a memory device) encoded with software comprising computer executable instructions and when the software is executed (by the processor1410) it is operable to perform the network controller operations described herein.

Some implementations of the present disclosure have been shown in the figures to apply to a 5G network. Here, network nodes (e.g. routers, switches) of the present disclosure may be employed along the N3 interface or N6 interface of the 5G network, as examples. Other implementations may be readily applied to other types networks, such as other SR-type networks, as well as other mobile networks including 4G, Long Term Evolution (LTE) based networks having a control and user plane separation (CUPS) architecture, as one ordinarily skilled in the art will readily appreciate. In 4G/LTE with CUPS, the user plane function may be a gateway—user plane (GW-U). As other examples, the SMF may instead be a GW—control plane (GW-C), the AMF may instead be a mobility management entity (MME), the PCF may instead be a policy and control rules function (PCRF). The SMF and GW-C may be more generally referred to as a CP entity for session management.

Note that, although in some implementations of the present disclosure, one or more (or all) of the components, functions, and/or techniques described in relation to the figures may be employed together for operation in a cooperative manner, each one of the components, functions, and/or techniques may indeed be employed separately and individually, to facilitate or provide one or more advantages of the present disclosure.

It will also be understood that, although the terms “first,” “second,” etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used to distinguish one element from another. For example, a first data packet could be termed a second data packet, and similarly, a second data packet could be termed a first data packet, without changing the meaning of the description, so long as all occurrences of the “first data packet” are renamed consistently and all occurrences of the “second data packet” are renamed consistently. The first data packet and the second data packet are both data packets, but they are not the same data packet.