Patent Description:
A mobile core network (e.g., fifth generation (<NUM>)) telcocloud infrastructure is distributed across multiple locations. The workloads may be virtual and may migrate across different hardware resources. Moreover, a control plane function may be remote from a data plane/user plane function. The control plane is responsible for instantiating state entries on the forwarding plane (data plane or user plane) based on the control messages.

In-Situ Operations Administration and Management (iOAM) is an inband telemetry data collection technique. iOAM allows a network/service operator to collect real-time telemetry data by embedding the data inband within actual traffic. With more interest in service level agreement (SLA)-based service offerings, selecting the best path for control plane and data plane traffic is becoming inevitable for service providers. In a <NUM> mobile core network, because most of workloads are virtual, it can be challenging to maintain out-of-band OAM connectivity.

<CIT> is directed to a technique for allocating data or bearers among multiple paths in a multi-radio access technology (multi-RAT) wireless network that includes a cellular base station and a wireless local area network (WLAN) access point that are connected to a user device, the technique including: providing a primary data path between the base station and the user device, providing a secondary data path between the base station and the user device via the WLAN access point, determining a transmission delay for the secondary data path between the base station and the user device via the WLAN access point, and allocating data, for forwarding to the user device based on at least the transmission delay for the secondary path, to either the primary data path or the secondary data path. <NPL> is directed to requirements for In-situ OAM.

Techniques are presented herein for best path selection between a control plane entity (node) and a user plane entity (node), such as may be useful in a mobile core network that supports communications for wireless user equipment. In accordance with an example embodiment, a control plane entity is configured to establish at least a first path and a second path through a network for communications between the control plane entity and a user plane entity. The first path and the second path being disjoint from each other, wherein the first path is designated as a primary path and the second path is designated as a standby/warm path. The control plane entity generates first messages which include entropy that causes the first messages to traverse the first path, first inband metadata that includes timestamp and sequence information, control plane message information, and a flag to indicate that the first messages are sent via the primary path and carry real control plane message information. The control plane entity generates second messages which include entropy that causes the first messages to traverse the second path, second inband metadata that includes timestamp and sequence information, dummy control plane message information, and a flag to indicate that the second messages are sent via the warm path and carry dummy control plane message information to be ignored by the user plane entity. The control plane entity sends the first messages and the second messages into the network.

The user plane entity is configured to obtain the first inband metadata from the first messages and second the inband metadata from the second messages, sends the first inband metadata and the second inband metadata to a performance analysis server. The performance analysis server is configured to analyze performance of the first path and of the second path based on the first inband metadata and the second inband metadata, and determine whether to toggle a state of the first path and of the second path such that the first path is the warm path and the second path is the primary path.

In a modern mobile core network (e.g., <NUM>), there is disaggregation of the control plane and the data plane. Control plane and data plane entities may be running in different parts of the network. Therefore, they need to communicate with each other over the network using some form of inband mechanism.

Currently, there is only one path from the control plane entity to the user plane entity. This may not be sufficient in all cases. For example, if something goes wrong on the path, the control plane entity may not be able to update the data plane entity in time to account for the failure. Any latency or packet loss issues in the network will have a direct impact on the control plane and the user plane communication. This in turn causes delay in instantiating the state and trees on the user plane entity, which can impact service level agreement (SLA) requirements.

In-Situ Operations Administration and Management (iOAM) is an inband telemetry data collection technique. iOAM allows a network/service operator to collect real-time telemetry data by embedding the data inband within actual traffic. Such collected inband telemetry data allows a network/service operator to instantly react to any network events. The telemetry data collected by iOAM can be done on different layers. For example, a Hop-by-Hop option collects the path and/or performance data from each network element at the network layer.

Presented herein are techniques that leverage iOAM for use in a mobile core network in order to create relevant third generation 3rd Generation Partnership Project (3GPP) control plane interfaces between virtual control plane (vCP) and virtual user plane (vUP) elements that can be used to instruct the virtual entities to perform various OAM functions, collect telemetry and other data from the virtual entities and signal maintenance messages between control plane and user plane entities. One 3GPP control plane interface can be used to perform holistic OAM functionalities and identify the better performing path and accordingly use primary/warm sessions for control plane exchanges.

More specifically, techniques are provided for multipath awareness and performance analysis using iOAM to toggle the between two paths/sessions between the control plane entity and the user plane entity. The two paths are referred to herein as a primary path/session and a warm path/session. The warm path is not a backup session.

Best path selection is made using real data and in-situ in-band performance analysis. Multiple (two or more) connections are made between vCP and the vUP entities. One path is designated as the primary path or session and another of the paths is designated as the warm (standby) path. Based on performance measurements of the paths, a non-selected path is maintained as a warm (standby) path all the while iOAM is used to continue measuring performance of the paths, such as delay/latency, jitter, packet drops, etc. iOAM messages are optimized and scalability of the mobile core (e.g., <NUM>) network architecture is achieved. Again, iOAM is used on parallel links (primary and warm paths) for performance-based (delay) path selection. The warm path is maintained with active performance measurement.

Reference is first made to <FIG> shows a network environment <NUM> that includes a virtual control plane (vCP) entity <NUM> and a virtual user plane function (vUPF) <NUM> that are running on physical devices separate from each other and which communicate over an Internet Protocol (IP) or Multi-Protocol Label Switching (MPLS) network <NUM>. In one example, the vCP entity <NUM> may be a Packet Data Network (PDN) Gateway Control Plane (PGW-C) function or a Serving Gateway Control Plane (SGW-C) function. Similarly, the vUPF <NUM> may be a PDN Gateway User Plane (PGW-U) function for <NUM> Control and User Plane Separation (CUPS) or a Serving Gateway User Plane (SGW-U) function for <NUM> CUPS.

The vCP entity <NUM> and vUPF <NUM> use multipath aware (bidirectional) control plane sessions for communication. The multiple paths between the vCP entity <NUM> and the vUPF <NUM> may be manually defined by the system operator, configured using a dual-planar network design or achieved using a centralized intelligence like a Path Computation Element (PCE) <NUM>. Each of the (multipath) sessions will traverse over diverse paths, such as by leveraging software-defined networking (SDN) intelligence provided by the PCE <NUM>.

Multipath control plane communication is inserted with an iOAM header that carries telemetry information, such as packet sequencing and/or timestamps. For a delay aware vCP-vUP path selection, time stamping is a useful component to be included in the iOAM header. To this end, <FIG> shows a packet <NUM> sent by the vCP <NUM> on the primary path <NUM> to vUPF <NUM>. The packet <NUM> includes an Internet Protocol (IP) header <NUM>, a User Datagram Protocol (UDP) header <NUM>, an iOAM header <NUM> and a control plane message payload <NUM>. Similarly, a packet <NUM> is sent by the vUPF <NUM> on the warm path <NUM> to vCP <NUM>. The packet <NUM> includes an IP header <NUM>, a UDP header <NUM>, an iOAM header <NUM> and a control plane message payload <NUM>. UDP is only an example of a transport type.

In one example, the control plane message carried in the packets <NUM> and <NUM> is based on the Packet Forwarding Control Protocol (PFCP). PFCP is a 3GPP protocol used on the Sx/N4 interface between the control plane function and the user plane function, specified in 3GPP Technical Specification (TS) <NUM>.

Some properties of PFCP are noted here. One Sx association is established between a vCP entity <NUM> and a vUPF <NUM> before being able to establish Sx sessions on the vUPF <NUM>. The Sx association may be established by the vCP entity <NUM> or by the vUPF <NUM>. An Sx session is established in the vUPF <NUM> to provision rules instructing the vUPF <NUM> how to process certain traffic. An Sx session may correspond to an individual PDN connection, Traffic Detection Function (TDF) session or a standalone session not tied to any PDN connection/TDF session.

While PFCP is referred to as the control plane communications between the vCP <NUM> and the vUPF <NUM>, this is only an example. The techniques presented herein are applicable to any communications between a control plane entity and a user plane entity in a mobile core network.

In one embodiment, native iOAM is used in which the vCP entity <NUM> and the vUPF <NUM> insert the iOAM header directly in the control plane session packets. In another embodiment, as shown in <FIG>, an OAM virtual network function (vNF) is provided that runs as close as possible to the vCP <NUM> and vUPF <NUM>. This is shown by the OAM vNF <NUM> proximate the vCP entity <NUM> and the OAM vNF <NUM> proximate the vUPF entity <NUM>. The OAM vNFs <NUM> and <NUM> and exchange packets with iOAM with the above telemetry information.

Again, according to the embodiments presented herein, multiple sessions are created based on multipath awareness. One session/path is used as the primary path/session and the other is used as the warm path/session. Performance is monitored on the primary path/session and on the warm path/session. Based on analysis of the performance of the primary session and warm session, it may be determined to toggle between the primary path and the warm path.

Using iOAM has advantages over out-of-band probes. A probe is useful for performing measurements along a path, but it is not true inband because a transit node does not process a probe packet the same way as a data packet because the parameters (source address, destination address, source port, destination port, etc.) of the probe packet are not the same as a data packet communicated between the control plane entity and a user plane entity. Moreover, the transit nodes could be configured to deprioritize probe packets and therefore the performance analytics of probe packets are not an accurate representation of the control session between the control plane entity and the user plane entity.

Reference is now made to <FIG> shows a network environment <NUM> that includes a first control plane entity, vCP1, at <NUM>(<NUM>) and two user plane entities vUPF1 at <NUM>(<NUM>) and vUPF2 at <NUM>(<NUM>). <FIG> also shows the IP/MPLS network <NUM> over which the vCP1 <NUM>(<NUM>) communicates with the vUPF1 <NUM>(<NUM>) and vUPF2 <NUM>(<NUM>). The IP/MPLS network <NUM> includes transit network elements R1, R2, R3, R4, R5, R6, C1, C2 and C3, for example,.

Performance analysis server <NUM> is an entity (server computer, for example) that has communication connectivity with the vCP1 <NUM>(<NUM>), vUPF <NUM><NUM>(<NUM>) and vUPF2 <NUM>(<NUM>). It is to be understood that a connection arrow between the vCP1 <NUM>(<NUM>) and the performance analysis server <NUM> is not shown in <FIG> to simplify the diagram, but there is communication connectivity between the vCP1 <NUM>(<NUM>) and the performance analysis server <NUM>. The performance analysis server <NUM> may also run OAM related operations. Each of the virtual entities (vCP1 <NUM>(<NUM>), vUPF1 <NUM>(<NUM>) and vUPF2 <NUM>(<NUM>)) upon receiving the iOAM data will extract the iOAM data and forward it to the performance analysis server <NUM> for analytics purpose.

For simplicity, communication between vCP1 <NUM>(<NUM>) and vUPF1 <NUM>(<NUM>) is described, as an example. It should be understood that similar techniques apply to communications between vCP1 <NUM>(<NUM>) and vUPF2 <NUM>(<NUM>). vCP1 <NUM>(<NUM>) is instructed to create two disjoint paths/sessions. These instructions may originate from a PCE, such as the PCE <NUM> shown in <FIG>. The two paths/sessions are shown at <NUM> and <NUM>. First path/session <NUM> traverses R1-C1-R4-vUPF1 and second path/session <NUM> traverses R2-C2-R5-vUPF1.

The paths/sessions are guaranteed to be diverse/disjointed by using different source/destination port information. For example, Path/Session <NUM> (corresponding to path/session <NUM>) = src_udp=x1; dst_udp = y1. Path/Session <NUM> may be designated, by default, as the primary path/session. Path/Session <NUM> may be designated the warm path/session. Path/Session <NUM> (corresponding to path/session <NUM>) = src_udp = x1, dst_udp = y2. Thus, since x1 and y1 are used as source and destination ports for packets along Path/Session <NUM>, these packets are guaranteed to travel a path diverse from packets with source and destination ports of x1 and y2, respectively.

The primary path/session <NUM> carries real/actual control plane messages, while the warm path/session <NUM> carries dummy control plane messages. Both sessions will carry iOAM data that indicates the performance related and other metrics of the path/session. The performance analysis server <NUM> uses the iOAM data to determine which path (primary or warm) has better performance, and sends a command/trigger to the vCP1 <NUM>(<NUM>) to toggle to the other path/session for use as the primary path.

<FIG> shows that, in one embodiment, the performance analysis server <NUM> may be a process running on a separate entity (server) that receives the per-path iOAM telemetry reports from the vCP1 <NUM>(<NUM>) and vUPF1 <NUM>(<NUM>). In another embodiment, there is a performance analysis process running on the user plane entities. For example, vUPF1 <NUM>(<NUM>) is configured to execute a performance analysis process <NUM>. Instead of the vCP1 <NUM>(<NUM>) and vUPF1 <NUM>(<NUM>) sending the iOAM data to the performance analysis server <NUM>, the performance analysis process <NUM> may obtain the iOAM data in packets it receives from the vCP1 <NUM>(<NUM>) and perhaps the iOAM data in packets the vCP1 <NUM>(<NUM>) receives from vUPF1 <NUM>(<NUM>), and the performance analysis process <NUM> computes the performance analytics on the primary path and warm path between the vCP1 <NUM>(<NUM>) and vUPF1 <NUM>(<NUM>).

Reference is now made to <FIG> for a further description of a process <NUM> for the exchange of messages between the vCP1 <NUM>(<NUM>) and the vUPF1 <NUM>(<NUM>). Reference is also made to <FIG> for purposes of the description of <FIG>. In one form, vCP1 <NUM>(<NUM>) sends control plane messages over UDP to vUPF1 <NUM>(<NUM>). vUPF <NUM>(<NUM>) creates a state entry for the UDP session. Since UDP is connectionless, it relies on acknowledgement (ACK) messages. For every message sent from vCP1 <NUM>(<NUM>), an ACK message is expected from the vUPF1 <NUM>(<NUM>). In the case in which the control plane messages are PFCP messages, the PFCP messages may include session establishment, session modification and session deletion messages.

As shown in <FIG>, vCP1 <NUM>(<NUM>) sends a first message (primary path/session control plane message) <NUM> to the vUPF1 <NUM>(<NUM>), and the vUPF1 <NUM>(<NUM>) responds with ACK <NUM>. The first message <NUM> includes an IP header <NUM>, transport (e.g., UDP) header <NUM>, an iOAM header <NUM> and a control plane message payload <NUM>. The UDP header <NUM> includes source port and destination port information set to values associated with a first path through the network to the vUPF <NUM>(<NUM>). The iOAM header <NUM> includes iOAM data such as timestamp information, sequence information, and perhaps other iOAM data. The control plane message payload <NUM> includes real control plane message information and sequence information. In one form, a primary/warm (P/W) flag is included to indicate whether the message is a message for primary path or a message for the warm path. In the case of message <NUM>, the P/W flag is set to P. Moreover, the P/W flag can be included in the iOAM header <NUM> or in the control plane message payload <NUM>.

The ACK <NUM> also includes an IP header <NUM>, UDP header <NUM>, iOAM header <NUM> and control plane message payload <NUM>. The iOAM header <NUM> includes a timestamp, sequence information, and perhaps other iOAM data. The control plane message payload <NUM> includes a control plane ACK and sequence information. Like message <NUM>, the ACK <NUM> includes a P/W flag (set to P) in either the iOAM header <NUM> or in the control plane message payload <NUM>.

As indicated in <FIG>, additional control plane message exchanges over the primary path occur in a similar manner.

vCP1 <NUM>(<NUM>) also sends a second message (warm path/session control plane message) <NUM> to the vUPF1 <NUM>(<NUM>), and the vUPF1 <NUM>(<NUM>) responds with ACK <NUM>. The second message <NUM> includes an IP header <NUM>, UDP header <NUM>, an iOAM header <NUM> and a control plane message payload <NUM>. The UDP header <NUM> includes source port and destination port information set to values associated with a second path (disjoint to the first path) through the network to the vUPF <NUM>(<NUM>). The iOAM header <NUM> includes iOAM data such as timestamp information, sequence information, and perhaps other iOAM data. The control plane message payload <NUM> includes dummy/null control plane message information and sequence information. A primary/warm (P/W) flag is included to indicate whether the message is a message for primary path or a message for the warm path. In the case of message <NUM>, the P/W flag is set to W. The P/W flag can be included in the iOAM header <NUM> or in the control plane message payload <NUM>.

The ACK <NUM> also includes an IP header <NUM>, UDP header <NUM>, iOAM header <NUM> and control plane message payload <NUM>. The iOAM header <NUM> includes a timestamp, sequence information, and perhaps other iOAM data. The control plane message payload <NUM> includes a control plane ACK and sequence information. Like message <NUM>, the ACK <NUM> includes a P/W flag (set to W) in either the iOAM header <NUM> or in the control plane message payload <NUM>.

As indicated in <FIG>, additional control plane message exchanges over the warm path occur in a similar manner.

The vUPF1 <NUM>(<NUM>) reads the state of the P/W flag in the control plane messages it receives (messages <NUM> and <NUM>) and knows when the P/W flag is set to P, then it should expect real or actual control plane message information and when the W is set, then it should expect dummy/null control plane message information.

As explained above, when creating the primary path/session and the warm path/session, different entropy in the packets will be used so that they take different paths through the network between the control plane entity and the user plane entity. Also, the user plane entity knows which session to consider primary and which to consider warm (based on the state of the P/W flag) so that it knows via which session to expect the dummy control plane message and the actual control plane message.

The dummy control plane message information included in message <NUM> fools other nodes in the network <NUM> (<FIG>) to treat the message as an actual control plane message, but the vUPF1 <NUM>(<NUM>) knows it is a dummy message and will ignore it but send the iOAM data contained in the message to the performance analysis server. The dummy control plane message information may be a new message type (e.g., a new PFCP message type) to indicate that it is a "null" message. Both the control plane entities and user plane entities would be configured to know how to interpret that.

The iOAM data carried in messages shown in <FIG> may be in a new iOAM message type. Am iOAM trace type or flag field may be used, or a new flag with details in the trace option data to indicate the primary path/warm path flag.

Thus, as depicted in <FIG>, the control plane entity generates first messages (messages <NUM>) which include entropy that causes the first messages to traverse a first path in a network that is disjoint from a second path in the network. The first messages include first inband metadata that includes timestamp and sequence information, control plane message information, and a flag to indicate that the first messages are sent via the primary path and carry real control plane message information. The control plane entity also generates second messages (messages <NUM>) which include entropy that causes the first messages to traverse the second path. The second messages include second inband metadata that includes timestamp and sequence information, dummy control plane message information, and a flag to indicate that the second messages are sent via the warm path and carry dummy control plane message information to be ignored by the user plane entity.

Entropy are key values derived from different features of a packet in a packet flow, and may include labels (e.g., MPLS labels), Layer <NUM> header (source IP address and destination IP address) information, transport (e.g., User Datagram Protocol) source port, transport destination port, etc..

Reference is now made to <FIG> illustrates a flow chart for a method <NUM> performed at a control plane entity, e.g., vCP <NUM>(<NUM>) shown in <FIG>, according to an example embodiment. At <NUM>, the control plane entity establishes at least a first path and a second path through a network for communications between the control plane entity and a user plane entity. The first path and the second path are disjoint from each other. In one example, the first path is designated as a primary path and the second path is designated as a warm path, though this is arbitrary. Further, in one example, the first path and the second path are established based on commands obtained from a path computation entity or a network controller entity.

At <NUM>, the control plane entity generates first messages which include entropy that causes the first messages to traverse the first path, first inband metadata that includes timestamp and sequence information, control plane message information, and a flag to indicate that the first messages are sent via the primary path and carry real control plane message information. Similarly, at <NUM>, the control plane entity generates second messages which include entropy that causes the second messages to traverse the second path, second inband metadata that includes timestamp and sequence information, dummy control plane message information, and a flag to indicate that the second messages are sent via the warm path and carry dummy control plane message information to be ignored by the user plane entity.

At <NUM>, the control plane entity provides/sends the first messages and the second messages into the network.

As described above, in one form, the control plane message information in the first messages and second messages is in accordance with a Packet Forwarding Control Protocol (PFCP) of the <NUM>rd Generation Partnership Project. The first inband metadata and the second inband metadata is formatted in accordance with one of: the in-situ Operations, Administration and Management (iOAM) standard, Inband Flow Analytics (IFA) standard and Inband Network Telemetry (INT) standard. In addition, the flag of the first messages is included either as part of the first inband metadata or as part of the control plane message information, and the flag of the second messages is included either as part of the second inband metadata or as part of the dummy control plane message information. Further still, the entropy in the first messages may comprise transport (e.g., UDP) source port and transport (e.g., UDP) destination port information.

As explained above in connection with <FIG>, the user plane entity obtains the first inband metadata from the first messages (sent over the first path) and the second inband metadata from the second messages (sent over the second path). The user plane entity provides the first inband metadata and the second inband metadata to a performance analysis process, e.g. a process running on the performance analysis server <NUM> or the performance analysis process <NUM> running on the vUPF1 <NUM>(<NUM>), as shown in <FIG>.

The control plane entity <NUM>(<NUM>) will also send to the performance analysis process the inband metadata in ACK messages received from the vUPF1 <NUM>(<NUM>) for both the primary path and the warm path. Thus, the performance analysis process can also use the inband metadata contained in the messages sent from the user plane entity to the control plane entity for purposes of analyzing the performance of the first path and the second path.

Turning now to <FIG>, a flow chart is shown for a method <NUM> by which iOAM data is analyzed for best path selection. The method <NUM> may be performed by a process that may run on the performance analysis server <NUM> or executed by the performance analysis process <NUM>, both shown in <FIG>. At <NUM>, the performance analysis process obtains first inband metadata included in first messages sent between a control plane entity and a user plane entity through a first path in a network. The first metadata includes timestamp and sequence information for the first messages sent through the first path. The first messages further include a flag to indicate that the first messages are sent via a primary path and carry real control plane message information. At <NUM>, the performance analysis process obtains second inband metadata included in second messages sent between a control plane entity and a user plane entity through a second path in the network. The first path and the second path are disjoint from each other. The second metadata includes timestamp and sequence information for the second messages sent through the second path. The second messages further include a flag to indicate that the second messages are sent via a warm path and carry dummy control plane message information.

At <NUM>, the performance analysis process analyzes performance of the first path and of the second path based on the first inband metadata and the second inband metadata. For example, the performance analysis process analyzes performance of the first path and the second path by computing one or more of latency, jitter and packet loss for the first path based on the first inband metadata and computing one or more of latency, jitter and packet loss for the second path based on the second inband metadata. The performance analysis process compares one or more of latency, jitter and packet loss for the first path with corresponding one or more of latency, jitter and packet loss for the second path. The latency, jitter or packet loss on the second path may, at some point in time, become better than the latency, jitter or packet loss on the first path. When that happens, the performance analysis process may determine to toggle a state of the first path and of the second path such the first path is/becomes the warm path and the second path is/becomes the primary path. In other words, at <NUM>, based on the analyzing step <NUM>, the performance analysis process determines whether (or not) to toggle a state of the first path and of the second path such that the first path is the warm path and the second path is the primary path.

At <NUM>, when it is determined that performance of the second path is better than performance of the first path, the performance analysis process provides a command/trigger at least to the control plane entity to cause the control plane entity to generate the first messages to include the dummy control plane message information and to set the flag of the first messages to indicate that the first messages are sent via the warm path, and to cause the control plane entity to generate the second messages to include the real control plane message information and to set the flag of the second messages to indicate that the second messages are sent via the primary path.

Thus, the performance analysis process (running on the performance analysis server or in an performance analysis process running on the user plane entity) can control the control plane entity to toggle to the other path and make what had been previously the warm path (the second path), now the primary path, and what had been previously the primary path (the first path), now the warm path. It is to be understood that at some time later, it is possible that the inband metadata may reveal that the paths should be toggled back, such that the first path becomes the primary path and the second path becomes the warm path. The session/path that has better performance (e.g., least delay, least jitter and/or least number of packet drops) will be marked as the primary/active session and will be actively used for control plane communication, whereas the other session/path is marked as the warm path/session.

<FIG> illustrates this in a graphical manner where the performance analysis server <NUM>, for example, obtains the per-path iOAM telemetry reporting data from the vUPF1 <NUM>(<NUM>) (and from the vCP1 <NUM>(<NUM>)). The performance analysis server <NUM> will issue a command/trigger to the vCP1 <NUM>(<NUM>), as shown at <NUM>, and optionally to the vUPF1 <NUM>(<NUM>) as shown at <NUM>, to cause the vCP1 <NUM>(<NUM>) to toggle the content of the messages sent through the network on the respective first and second paths <NUM> and <NUM>.

In the case in which the vUPF1 <NUM>(<NUM>) runs the performance analysis process <NUM> (as shown in <FIG>), the control plane messages or iOAM may be used to signal back to the vCP1 <NUM>(<NUM>) about degradation of performance of the primary path/session and thus the need to toggle such that what had been the warm path/session now becomes the primary path/session.

For example, the first path/session <NUM> over path R1-C1-R4 may better performing and accordingly will be marked as the primary/active session while the second path/session <NUM> over path R2-C2-R5 will be marked as the warm session. The active session will be actively used for control plane communication while the warm session will still continue to use simple/dummy control plane message information with iOAM header inserted for analytics.

Performance of any path may change based on various factors. Thus, at some point in time, the second path/session <NUM> may be determined to perform better than the first path/session <NUM>, and as such, the second path/session <NUM> will be marked as the primary/active session while the first path/session <NUM> one will be marked as the warm path/session.

When switching the warm path to the primary path (and vice versa), changes are made to the payload content of the messages to make this indication (using the aforementioned primary/warm path flag) and the control plane message (e.g., PFCP message) is changed to carry actual PFCP message information (for the primary path), not the dummy message which had been sent when the path was used as the warm path. No changes are made to the entropy/header so that messages/packets always take the same path through the network <NUM> they had been taking.

Sequencing at the control plane (PFCP) layer. When a path/session is upgraded from warm path to primary/active path, it may be useful to keep track of the sequencing so that the control plane messages in the warm path can pick up where the control plane messages left off in the primary path right before the switch to be sure control plane message information is not missed. For example, the user plane entity could receive control plane dummy message sequence numbers <NUM>, <NUM> and then real control plane message number <NUM> on the primary path after the switch. The user plane entity would then know that it missed the real control plane message <NUM>, and would need to request it again from the control plane entity.

Sequencing at the iOAM layer. The iOAM metadata will have all the details about the sequencing, timestamp, etc., which is used for measuring packet loss, delay and jitter. Thus, sequencing analysis at the iOAM layer is more relevant for packet loss, delay and jitter computation.

<FIG> illustrates a hardware block diagram of a computing device <NUM> that may perform functions of the control plane entity, user plane entity and performance analysis server described above in connection with <FIG>. It should be appreciated that <FIG> provides only an illustration of one embodiment and does not imply any limitations with regard to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made.

As depicted, the device <NUM> includes a bus <NUM>, which provides communications between computer processor(s) <NUM>, memory <NUM>, persistent storage <NUM>, communications unit <NUM>, and input/output (I/O) interface(s) <NUM>. Bus <NUM> can be implemented with any architecture designed for passing data and/or control information between processors (such as microprocessors, communications and network processors, etc.), system memory, peripheral devices, and any other hardware components within a system. For example, bus <NUM> can be implemented with one or more buses.

Memory <NUM> and persistent storage <NUM> are computer readable storage media. In the depicted embodiment, memory <NUM> includes random access memory (RAM) <NUM> and cache memory <NUM>. In general, memory <NUM> can include any suitable volatile or non-volatile computer readable storage media. Instructions for the control logic <NUM> that controls and performs operations of the control plane entity, user plane entity and performance analysis process, may be stored in memory <NUM> or persistent storage <NUM> for execution by processor(s) <NUM>. When the processor(s) <NUM> execute the control logic for the control plane entity, the processor(s) <NUM> are caused to perform the control plane entity described above in connection with <FIG>. When the processor(s) <NUM> execute the control logic for the user plane entity, the processor(s) <NUM> are caused to perform the user plane entity described above in connection with <FIG>. When the processor(s) <NUM> execute the control logic for performance analysis server <NUM> or performance analysis process <NUM> shown in <FIG>), the processor(s) <NUM> are caused to perform the performance analysis process described above in connection with <FIG>.

One or more programs may be stored in persistent storage <NUM> for execution by one or more of the respective computer processors <NUM> via one or more memories of memory <NUM>. The persistent storage <NUM> may be a magnetic hard disk drive, a solid state hard drive, a semiconductor storage device, read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, or any other computer readable storage media that is capable of storing program instructions or digital information.

There is provided a computer readable storage medium carrying computer executable instructions that, when executed on one or more processors, cause any of the methods hereindescribed to be carried out.

Communications unit <NUM>, in these examples, provides for communications with other data processing systems or devices. In these examples, communications unit <NUM> includes one or more network interface cards. Communications unit <NUM> may provide communications through the use of either or both physical and wireless communications links.

I/O interface(s) <NUM> allows for input and output of data with other devices that may be connected to computer device <NUM>. For example, I/O interface <NUM> may provide a connection to external devices <NUM> such as a keyboard, keypad, a touch screen, and/or some other suitable input device. External devices <NUM> can also include portable computer readable storage media such as database systems, thumb drives, portable optical or magnetic disks, and memory cards.

Software and data used to practice embodiments can be stored on such portable computer readable storage media and can be loaded onto persistent storage <NUM> via I/O interface(s) <NUM>. I/O interface(s) <NUM> may also connect to a display <NUM>. Display <NUM> provides a mechanism to display data to a user and may be, for example, a computer monitor.

The programs described herein are identified based upon the application for which they are implemented in a specific embodiment. However, it should be appreciated that any particular program nomenclature herein is used merely for convenience, and thus the embodiments should not be limited to use solely in any specific application identified and/or implied by such nomenclature.

The present embodiments may employ any number of any type of user interface (e.g., Graphical User Interface (GUI), command-line, prompt, etc.) for obtaining or providing information (e.g., data relating to scraping network sites), where the interface may include any information arranged in any fashion. The interface may include any number of any types of input or actuation mechanisms (e.g., buttons, icons, fields, boxes, links, etc.) disposed at any locations to enter/display information and initiate desired actions via any suitable input devices (e.g., mouse, keyboard, etc.). The interface screens may include any suitable actuators (e.g., links, tabs, etc.) to navigate between the screens in any fashion.

The environment of the present embodiments may include any number of computer or other processing systems (e.g., client or end-user systems, server systems, etc.) and databases or other repositories arranged in any desired fashion, where the present embodiments may be applied to any desired type of computing environment (e.g., cloud computing, client-server, network computing, mainframe, stand-alone systems, etc.). The computer or other processing systems employed by the present embodiments may be implemented by any number of any personal or other type of computer or processing system (e.g., desktop, laptop, PDA, mobile devices, etc.), and may include any commercially available operating system and any combination of commercially available and custom software (e.g., machine learning software, etc.). These systems may include any types of monitors and input devices (e.g., keyboard, mouse, voice recognition, etc.) to enter and/or view information.

The various functions of the computer or other processing systems may be distributed in any manner among any number of software and/or hardware modules or units, processing or computer systems and/or circuitry, where the computer or processing systems may be disposed locally or remotely of each other and communicate via any suitable communications medium (e.g., LAN, WAN, Intranet, Internet, hardwire, modem connection, wireless, etc.). For example, the functions of the present embodiments may be distributed in any manner among the various end-user/client and server systems, and/or any other intermediary processing devices. The software and/or algorithms described above and illustrated in the flow charts may be modified in any manner that accomplishes the functions described herein. In addition, the functions in the flow charts or description may be performed in any order that accomplishes a desired operation.

The software of the present embodiments may be available on a non-transitory computer useable medium (e.g., magnetic or optical mediums, magneto-optic mediums, floppy diskettes, CD-ROM, DVD, memory devices, etc.) of a stationary or portable program product apparatus or device for use with stand-alone systems or systems connected by a network or other communications medium.

The communication network may be implemented by any number of any type of communications network (e.g., LAN, WAN, Internet, Intranet, VPN, etc.). The computer or other processing systems of the present embodiments may include any conventional or other communications devices to communicate over the network via any conventional or other protocols. The computer or other processing systems may utilize any type of connection (e.g., wired, wireless, etc.) for access to the network. Local communication media may be implemented by any suitable communication media (e.g., local area network (LAN), hardwire, wireless link, Intranet, etc.).

The system may employ any number of any conventional or other databases, data stores or storage structures (e.g., files, databases, data structures, data or other repositories, etc.) to store information (e.g., data relating to contact center interaction routing). The database system may be implemented by any number of any conventional or other databases, data stores or storage structures (e.g., files, databases, data structures, data or other repositories, etc.) to store information (e.g., data relating to contact center interaction routing). The database system may be included within or coupled to the server and/or client systems. The database systems and/or storage structures may be remote from or local to the computer or other processing systems, and may store any desired data (e.g., data relating to contact center interaction routing).

The present embodiments may employ any number of any type of user interface (e.g., Graphical User Interface (GUI), command-line, prompt, etc.) for obtaining or providing information (e.g., data relating to providing enhanced delivery options), where the interface may include any information arranged in any fashion. The interface may include any number of any types of input or actuation mechanisms (e.g., buttons, icons, fields, boxes, links, etc.) disposed at any locations to enter/display information and initiate desired actions via any suitable input devices (e.g., mouse, keyboard, etc.). The interface screens may include any suitable actuators (e.g., links, tabs, etc.) to navigate between the screens in any fashion.

The embodiments presented may be in various forms, such as a system, a method, and/or a computer program product at any possible technical detail level of integration. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of presented herein.

Computer readable program instructions for carrying out operations of the present embodiments may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuitry, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Python, C++, or the like, and procedural programming languages, such as the "C" programming language or similar programming languages. In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects presented herein.

Aspects of the present embodiments are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to the embodiments.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures.

In summary, a method is provided comprising: at a control plane entity, establishing at least a first path and a second path through a network for communications between the control plane entity and a user plane entity, the first path and the second path being disjoint from each other, wherein the first path is designated as a primary path and the second path is designated as a warm path; generating first messages which include entropy that causes the first messages to traverse the first path, first inband metadata that includes timestamp and sequence information, control plane message information, and a flag to indicate that the first messages are sent via the primary path and carry real control plane message information; generating second messages which include entropy that causes the second messages to traverse the second path, second inband metadata that includes timestamp and sequence information, dummy control plane message information, and a flag to indicate that the second messages are sent via the warm path and carry dummy control plane message information to be ignored by the user plane entity; and sending the first messages and the second messages into the network.

Similarly, an apparatus is provided comprising a communication interface, and a processor coupled to the communication interface, wherein the processor is configured to perform operations including: at a control plane entity, establishing at least a first path and a second path through a network for communications between the control plane entity and a user plane entity, the first path and the second path being disjoint from each other, wherein the first path is designated as a primary path and the second path is designated as a warm path; generating first messages which include entropy that causes the first messages to traverse the first path, first inband metadata that includes timestamp and sequence information, control plane message information, and a flag to indicate that the first messages are sent via the primary path and carry real control plane message information; generating second messages which include entropy that causes the second messages to traverse the second path, second inband metadata that includes timestamp and sequence information, dummy control plane message information, and a flag to indicate that the second messages are sent via the warm path and carry dummy control plane message information to be ignored by the user plane entity; and sending the first messages and the second messages into the network.

Further, one or more non-transitory computer readable media are provided, encoded with instructions that, when executed by a processor, cause the processor to perform operations including: at a control plane entity, establishing at least a first path and a second path through a network for communications between the control plane entity and a user plane entity, the first path and the second path being disjoint from each other, wherein the first path is designated as a primary path and the second path is designated as a warm path; generating first messages which include entropy that causes the first messages to traverse the first path, first inband metadata that includes timestamp and sequence information, control plane message information, and a flag to indicate that the first messages are sent via the primary path and carry real control plane message information; generating second messages which include entropy that causes the second messages to traverse the second path, second inband metadata that includes timestamp and sequence information, dummy control plane message information, and a flag to indicate that the second messages are sent via the warm path and carry dummy control plane message information to be ignored by the user plane entity; and sending the first messages and the second messages into the network.

In addition, a method is provided comprising: obtaining first inband metadata included in first messages sent between a control plane entity and a user plane entity through a first path in a network, the first inband metadata including timestamp and sequence information for the first messages sent through the first path, the first messages further including a flag to indicate that the first messages are sent via a primary path and carry real control plane message information; obtaining second inband metadata included in second messages sent between a control plane entity and a user plane entity through a second path in the network, wherein the first path and the second path are disjoint from each other, and wherein the second inband metadata including timestamp and sequence information for the second messages sent through the second path, the second messages further including a flag to indicate that the second messages are sent via a warm path and carry dummy control plane message information; analyzing performance of the first path and of the second path based on the first inband metadata on and the second inband metadata; based on the analyzing, determining whether to toggle a state of the first path and of the second path such that the first path is the warm path and the second path is the primary path; and when it is determined that performance of the second path is better than performance of the first path, providing a command at least to the control plane entity to cause the control plane entity to generate the first messages to include the dummy control plane message information and to set the flag of the first messages to indicate that the first messages are sent via the warm path, and to cause the control plane entity to generate the second messages to include the real control plane message information and to set the flag of the second messages to indicate that the second messages are sent via the primary path.

Further, an apparatus is provided comprising a communication interface, and a processor coupled to the communication interface, wherein the processor is configured to perform operations including: obtaining first inband metadata included in first messages sent between a control plane entity and a user plane entity through a first path in a network, the first inband metadata including timestamp and sequence information for the first messages sent through the first path, the first messages further including a flag to indicate that the first messages are sent via a primary path and carry real control plane message information; obtaining second inband metadata included in second messages sent between a control plane entity and a user plane entity through a second path in the network, wherein the first path and the second path are disjoint from each other, and wherein the second inband metadata including timestamp and sequence information for the second messages sent through the second path, the second messages further including a flag to indicate that the second messages are sent via a warm path and carry dummy control plane message information; analyzing performance of the first path and of the second path based on the first inband metadata on and the second inband metadata; based on the analyzing, determining whether to toggle a state of the first path and of the second path such that the first path is the warm path and the second path is the primary path; and when it is determined that performance of the second path is better than performance of the first path, providing a command at least to the control plane entity to cause the control plane entity to generate the first messages to include the dummy control plane message information and to set the flag of the first messages to indicate that the first messages are sent via the warm path, and to cause the control plane entity to generate the second messages to include the real control plane message information and to set the flag of the second messages to indicate that the second messages are sent via the primary path.

Still further, one or more non-transitory computer readable media are provided, encoded with instructions that, when executed by a processor, cause the processor to perform operations including: obtaining first inband metadata included in first messages sent between a control plane entity and a user plane entity through a first path in a network, the first inband metadata including timestamp and sequence information for the first messages sent through the first path, the first messages further including a flag to indicate that the first messages are sent via a primary path and carry real control plane message information; obtaining second inband metadata included in second messages sent between a control plane entity and a user plane entity through a second path in the network, wherein the first path and the second path are disjoint from each other, and wherein the second inband metadata including timestamp and sequence information for the second messages sent through the second path, the second messages further including a flag to indicate that the second messages are sent via a warm path and carry dummy control plane message information; analyzing performance of the first path and of the second path based on the first inband metadata on and the second inband metadata; based on the analyzing, determining whether to toggle a state of the first path and of the second path such that the first path is the warm path and the second path is the primary path; and when it is determined that performance of the second path is better than performance of the first path, providing a command at least to the control plane entity to cause the control plane entity to generate the first messages to include the dummy control plane message information and to set the flag of the first messages to indicate that the first messages are sent via the warm path, and to cause the control plane entity to generate the second messages to include the real control plane message information and to set the flag of the second messages to indicate that the second messages are sent via the primary path.

In still another form, a system is provided comprising: a control plane entity configured to: establish at least a first path and a second path through a network for communications between the control plane entity and a user plane entity, the first path and the second path being disjoint from each other, wherein the first path is designated as a primary path and the second path is designated as a warm path; generate first messages which include entropy that causes the first messages to traverse the first path, first inband metadata that includes timestamp and sequence information, control plane message information, and a flag to indicate that the first messages are sent via the primary path and carry real control plane message information; generate second messages which include entropy that causes the first messages to traverse the second path, second inband metadata that includes timestamp and sequence information, dummy control plane message information, and a flag to indicate that the second messages are sent via the warm path and carry dummy control plane message information to be ignored by the user plane entity; send the first messages and the second messages into the network; a user plane entity configured to: obtain the first inband metadata from the first messages and second the inband metadata from the second messages; and send the first inband metadata and the second inband metadata to a performance analysis server; the performance analysis server configured to: analyze performance of the first path and of the second path based on the first inband metadata and the second inband metadata; and determine whether to toggle a state of the first path and of the second path such that the first path is the warm path and the second path is the primary path.

Claim 1:
A method comprising:
at a control plane entity (<NUM>), establishing (<NUM>) at least a first path and a second path through a network for communications between the control plane entity (<NUM>) and a user plane entity, the first path and the second path being disjoint from each other, wherein the first path is designated as a primary path and the second path is designated as a warm path;
generating (<NUM>) first messages which include entropy that causes the first messages to traverse the first path, first inband metadata that includes timestamp and sequence information, control plane message information, and a flag to indicate that the first messages are sent via the primary path and carry real control plane message information;
generating (<NUM>) second messages which include entropy that causes the second messages to traverse the second path, second inband metadata that includes timestamp and sequence information, dummy control plane message information, and a flag to indicate that the second messages are sent via the warm path and carry dummy control plane message information to be ignored by the user plane entity; and
sending (<NUM>) the first messages and the second messages into the network,
wherein the entropy of the first messages and the entropy of the second messages each comprise one or more of: respective MPLS labels; a respective source IP address information and a respective destination IP address information; a respective transport source port; and a respective transportation destination port.