Telemetry aggregation and network autonomic scaling

A method implemented by a central controller is provided. The method includes receiving traffic monitoring information from a client, and sending a traffic monitor request to a network controller, the traffic monitor request requesting the network controller to monitor one or more traffic parameters based on the traffic monitoring information. The method further includes receiving a traffic report from the network controller, and responsive to determining that a monitored traffic parameter in the traffic report does not satisfy a condition specified by the traffic monitoring information, initiating a scaling operation based on data contained in the traffic report.

Not applicable.

REFERENCE TO A MICROFICHE APPENDIX

Not applicable.

BACKGROUND

Abstraction and Control of Traffic Engineered (TE) Networks (ACTN) refers to a set of virtual network operations for controlling and managing large-scale multi-domain, multi-layer, and multi-vendor TE networks, so as to facilitate network programmability, automation, and efficient resource sharing. Various ACTN techniques may be employed for operating a network such as a Multiprotocol Label Switching (MPLS) TE (MPLS-TE) network or a transport layer (Layer 1/0) network, e.g., to provide connectivity and virtual network services for customers of the network. Services provided by the network may be tuned to meet certain requirements (e.g., traffic patterns, quality, reliability, etc.) of applications hosted by the network's customers. Moreover, data models (e.g., a representation of objects) may be used to configure or model a variety of network devices, protocol instances, and network services. For example, a YANG data model may describe how customers or end-to-end orchestrators can request and/or instantiate a generic virtual network service. Another YANG data model may describe a connection between certain YANG model classifications to ACTN interfaces. In particular, it may describe mapping a customer service model into the Customer Network Controller (CNC)—Multi-Domain Service Coordinator (MSDC) interface (CMI), where the customer service model may be known as the YANG model on the ACTN CMI.

SUMMARY

In one embodiment, the disclosure includes a method implemented by a central controller. The method includes receiving traffic monitoring information from a client; sending a traffic monitor request to a network controller, the traffic monitor request requesting the network controller to monitor one or more traffic parameters based on the traffic monitoring information; receiving a traffic report from the network controller; and responsive to determining that a monitored traffic parameter in the traffic report does not satisfy a condition specified by the traffic monitoring information, initiating a scaling operation based on data contained in the traffic report.

Optionally, in any of the preceding aspects, the traffic monitoring information may specify one or more telemetry parameters associated with at least one of a virtual network (VN) or a tunnel that part of the VN. Optionally, in any of the preceding aspects, the one or more telemetry parameters comprise at least one of utilized bandwidth, packet loss, packet delay, or jitter. Optionally, in any of the preceding aspects, determining that the monitored traffic parameter in the traffic report does not satisfy the condition comprises determining that the utilized bandwidth is such that bandwidth utilized by the VN or the tunnel exceeds a predetermined threshold. Optionally, in any of the preceding aspects, initiating the scaling operation comprises automatically requesting the network controller to increase bandwidth for the VN or the tunnel by a predetermined percentage. Optionally, in any of the preceding aspects, the central controller may comprise a multi-domain service coordinator (MSDC), wherein the client comprises a customer network controller (CNC), and wherein the network controller comprises at least one provisioning network controller (PNC). Optionally, in any of the preceding aspects, the method may further include sending a plurality of traffic monitor request to a plurality of provisioning network controllers (PNCs) in charge of a plurality of domains, respectively, receiving a plurality of traffic reports from the plurality of PNCs, respectively, and aggregating telemetry data from the plurality of traffic reports, the aggregated telemetry data comprising virtual network (VN) data and end-to-end traffic engineered (TE) data, where the VN data is associated with a VN extending across at least domains selected from the plurality of domains and wherein the end-to-end TE data is associated with tunnels that form part of the VN.

In another embodiment, the disclosure includes a central controller comprising a non-transitory memory storage comprising instructions, and one or more processors in communication with the memory. The one or more processors are configured to execute the instructions to: receive traffic monitoring information from a client; send a traffic monitor request to a network controller, the traffic monitor request requesting the network controller to monitor one or more traffic parameters based on the traffic monitoring information; receive a traffic report from the network controller; and responsive to determining that a monitored traffic parameter in the traffic report does not satisfy a condition specified by the traffic monitoring information, initiate a scaling operation based on data contained in the traffic report.

Optionally, in any of the preceding aspects, the traffic monitoring information specify one or more telemetry parameters associated with at least one of a virtual network (VN) or a tunnel that part of the VN. Optionally, in any of the preceding aspects, the one or more telemetry parameters comprise at least one of utilized bandwidth, packet loss, packet delay, or jitter. Optionally, in any of the preceding aspects, determining that the monitored traffic parameter in the traffic report does not satisfy the condition comprises determining that the utilized bandwidth is such that bandwidth utilized by the VN or the tunnel exceeds a predetermined threshold. Optionally, in any of the preceding aspects, initiating the scaling operation comprises automatically requesting the network controller to increase bandwidth for the VN or the tunnel by a predetermined percentage. Optionally, in any of the preceding aspects, the central controller comprises a multi-domain service coordinator (MSDC), wherein the client comprises a customer network controller (CNC), and wherein the network controller comprises at least one provisioning network controller (PNC). Optionally, in any of the preceding aspects, the one or more processors may further execute the instructions to: send a plurality of traffic monitor request to a plurality of provisioning network controllers (PNCs) in charge of a plurality of domains, respectively; receive a plurality of traffic reports from the plurality of PNCs, respectively; and aggregate telemetry data from the plurality of traffic reports, the aggregated telemetry data comprising virtual network (VN) data and end-to-end traffic engineered (TE) data, where the VN data is associated with a VN extending across at least domains selected from the plurality of domains and wherein the end-to-end TE data is associated with tunnels that form part of the VN.

In yet another embodiment, the disclosure includes non-transitory computer readable medium storing computer instructions, that when executed by one or more processors implemented in a central controller, cause the one or more processors to perform the steps of receiving traffic monitoring information from a client; sending a traffic monitor request to a network controller, the traffic monitor request requesting the network controller to monitor one or more traffic parameters based on the traffic monitoring information; receiving a traffic report from the network controller; and responsive to determining that a monitored traffic parameter in the traffic report does not satisfy a condition specified by the traffic monitoring information, initiating a scaling operation based on data contained in the traffic report.

Optionally, in any of the preceding aspects, the traffic monitoring information may specify one or more telemetry parameters associated with at least one of a virtual network (VN) or a tunnel that part of the VN. Optionally, in any of the preceding aspects, the one or more telemetry parameters comprise at least one of utilized bandwidth, packet loss, packet delay, or jitter. Optionally, in any of the preceding aspects, determining that the monitored traffic parameter in the traffic report does not satisfy the condition may comprise determining that the utilized bandwidth is such that bandwidth utilized by the VN or the tunnel exceeds a predetermined threshold. Optionally, in any of the preceding aspects, initiating the scaling operation may comprise automatically requesting the network controller to increase bandwidth for the VN or the tunnel by a predetermined percentage. Optionally, in any of the preceding aspects, the central controller comprises a multi-domain service coordinator (MSDC), wherein the client comprises a customer network controller (CNC), and wherein the network controller comprises at least one provisioning network controller (PNC).

DETAILED DESCRIPTION

Disclosed herein are embodiments of telemetry-based data models that describe key performance indicator (KPI) data for traffic engineered (TE) tunnels and Abstraction and Control of TE Network (ACTN) virtual networks (VNs). The telemetry-based data models facilitate the aggregation and abstraction of performance telemetry data such as end-to-end (E2E) TE-tunnel level data and/or a Virtual Network level. In addition, the telemetry-based data models provide a mechanism for allowing clients to subscribe to key performance monitoring data and to define autonomic network scaling capabilities. Furthermore, one or more telemetry-based data models disclosed herein may augment existing data models to enhance data monitoring capabilities at the tunnel level and/or the VN level. These and other features are detailed below.

FIG. 1is a diagram of a system100comprising a communications network102. The system100may be configured to support packet transport and transport services among network elements using the network102. For example, the system100may transport data traffic for services between clients124and126and a service provider122(such as a core data center, for example). Examples of such services may include Internet service, virtual private network (VPN) services, value added service (VAS) services, Internet Protocol Television (IPTV) services, content delivery network (CDN) services, Internet of things (IoT) services, data analytics applications, and Internet Protocol Multimedia services. The system100may comprise a network102, network elements108,110,112,114,116,118,120,128, and130, service provider122, and clients124and126. The system100may be configured as shown or in any other suitable manner.

The network102may include a network infrastructure that comprises a plurality of integrated network nodes104, and it may be configured to support transporting both optical data and packet switching data. Moreover, the network102may implement the network configurations to configure flow paths or virtual connections between client124, client126, and service provider122via the integrated network nodes104. In some aspects, the network102may be a backbone network which connects a cloud computing system of the service provider122to clients124and126. The network102may also connect a cloud computing system of the service provider122to other systems such as an external network or Internet, other cloud computing systems, data centers, and any other entity desiring access to the service provider122.

The integrated network nodes104may comprise reconfigurable hybrid switches configured for packet switching and optical switching. In an embodiment, one or more integrated network nodes104may comprise at least one packet switch, optical data unit (ODU) cross-connect, and reconfigurable optical add-drop multiplex (ROADM), and/or router. The integrated network nodes104may be coupled to each other and to other network elements using virtual links150and physical links152. For example, virtual links150may be logical paths between integrated network nodes104and physical links152may be optical fibers that form an optical wavelength division multiplexing (WDM) network topology. The integrated network nodes104may be coupled to each other using any suitable virtual links150or physical links152as would be appreciated by one of ordinary skill in the art upon viewing this disclosure. The integrated network nodes104may consider the network elements108-120as dummy terminals (DTs) that represent service and/or data traffic origination points and destination points.

Network elements108-120,128, and130may include clients, servers, broadband remote access servers (BRAS), switches, routers, service router/provider edge (SR/PE) routers, digital subscriber line access multiplexer (DSLAM) optical line terminal (OTL), gateways, home gateways (HGWs), service providers, PE network nodes, customers edge (CE) network nodes, an Internet Protocol (IP) router, and/or an IP multimedia subsystem (IMS) core. Clients124and126may include user devices in residential and business environments. For example, client126may reside in a residential environment and may communicate data with the network102via network elements120and108, while client124may reside in a business environment and may communicate data with the network102via network element110.

Examples of service provider122may include a network operator, a cloud provider, an Internet service provider, an IPTV service provider, an IMS core, a private network, an IoT service provider, a CDN, etc. In one embodiment, the service provider122may be a core data center that pools computing or storage resources to serve multiple clients124and126that request services from the service provider122. For example, the service provider122may use a multi-tenant model where fine-grained resources may be dynamically assigned to a client specified implementation and reassigned to other implementations according to consumer demand. Moreover, the service provider122may automatically control and optimize resource use by leveraging a metering capability at some level of abstraction appropriate to the type of resource (e.g., storage, processing, bandwidth, and active user accounts).

In some implementations, the service provider122may include a cloud computing system configured to provide cloud-based services to requesting clients124and126, e.g., via one or more models such as Service (IaaS) model, a Platform as a Service (PaaS) model, a Software as a Service (SaaS) model, etc. The cloud computing system, cloud computing, or cloud services may refer to a group of servers, storage elements, computers, laptops, cell phones, and/or any other types of network devices connected together by an IP network in order to share network resources (e.g., servers, processors, memory, applications, virtual machines, services, etc.) stored at one or more data centers of the service provider122.

The service provider122may comprise a cloud infrastructure or platform managed as a large resource pool for multiple clusters, which may each include a cluster of hardware units or servers for data analytics and data monitoring. Data analytics may generally include the use of various computing models to evaluate raw data, while data monitoring may generally include monitoring traffic in the network102to facilitate automatic discovery of traffic imbalance and/or initiate network optimization procedures, thus helping the service provider122to use the network102efficiently and save capital expenditures/operating expenses. A large-scale platform typically demands many resources to effectively process and analyze large quantities of data such as “big data,” which may range from a few terabytes (TBs) of data to many petabytes (PBs) of data or greater. Data analytics and performance monitoring may demand different types of resources, such as computing resources, networking resources, storage resources, and hardware resources, e.g., central processing units (CPUs), control boards, etc.

In some cases, the service provider122may provide services subject to service level agreement (SLA) requirements, such as service availability, latency, latency jitter, packet loss rate, bit error rate (BER), etc. A transport network may satisfy service availability and BER requirements by providing different protection and restoration mechanisms. However, there are no such mechanisms for satisfying other performance parameters. In order to provide high quality services according to customer SLAs, one possible solution may involve monitoring SLA-related performance parameters, and dynamically provisioning and optimizing services based on the performance monitoring results. Yet relatively large quantities of raw performance information may be generated as a result of monitoring such parameters, particularly in large-scale infrastructures such as previously discussed.

FIG. 2depicts an ACTN system200according to an embodiment of the disclosure. In some implementations, the network102inFIG. 1may serve as a backbone network to couple entities such as clients124and126and service provider122to the system200. The system200comprises a software-defined networking (SDN) controller such as a multi-domain service coordinator (MDSC)202, which may be configured to communicate with a Customer Network Controller (CNC)204and a provisioning Network Controllers (PNC)206. For example, the CNC204may be communicatively coupled to the MDSC202via a CNC-MDSC Interface (CMI)208, while the PNC206may be communicatively coupled to the MDSC202via a MDSC-PNC Interface (MPI)210. Moreover, the CNC204may include or be coupled to one or more applications212via a northbound interface214, which may be used to communicate application demand and requests for network resources, topology, or services. WhileFIG. 2only depicts one CNC204, PNC206, and application212, the ACTN system200may comprise a plurality of CNCs and/or PNCs in other implementations.

The MDSC202may be configured to perform various ACTN functionalities such as a multi-domain coordination function, a virtualization/abstraction function, a customer mapping function, and a virtual service coordination function. The multi-domain coordination function may involve overseeing specific aspects of different domains and building a single abstracted end-to-end network topology in order to coordinate end-to-end path computation and path/service provisioning. The virtualization/abstraction function may be used to provide an abstracted view of underlying network resources towards customers (e.g., CNC204). It may include converting customer resource requests into virtual network paths based on the global network-wide abstracted topology and the creation of an abstracted view of network slices allocated to each customer. The customer mapping function may be performed to map customer VN setup commands into network provisioning requests to the PNC206according to a static or dynamic policy. The customer mapping function may also provide mapping and translation of customer virtual network slices into physical network resources. The virtual service coordination function may be used to incorporate customer service-related knowledge into virtual network operations in order to seamlessly operate virtual networks while meeting customer service requirements.

In general, information carried over the CMI208may relate to traffic monitoring and control strategy (e.g., service and/or traffic parameters to be monitored), while information carried over the MPI210may relate to traffic monitoring parameters and results. For example, the CNC204may use the CMI208to send the MDSC202requests for network resources or SLA-based services, while the MDSC202and the PNC206may use the MPI210to exchange communications associated with such requests. The PNC206is configured to manage a network216, which may be substantially similar to the network102ofFIG. 1. For example, the PNC206may be in charge of configuring network elements, monitoring the physical topology of the network216and passing it, either raw or abstracted, to the MDSC202via the MPI210. The PNC206may also implement ACTN functions such as the multi-domain coordination and virtualization/abstraction functions described above. In some embodiments, the MDSC202and/or the PNC206may be implemented by a network provider substantially similar to the service provider122ofFIG. 1. Moreover, the CNC204may comprise different types of customers or clients (e.g., clients124or126) such as residential users, mobile users, enterprises, governments, utilities, etc.

FIG. 3depicts a flow diagram300for dynamic service control according to an embodiment of the disclosure. At step1, the CNC204sends traffic monitoring and traffic optimization strategy information to the MDSC202(e.g., via CMI208). For instance, such information may include SLA requirements (e.g., service availability, latency, latency jitter, packet loss rate, BER, etc.) so that the MDSC202may determine which performance parameters should be monitored and identify a service optimization strategy. At step2, the MDSC202sends the corresponding path traffic monitoring request to the PNC206(e.g., via PMI210). The request traffic monitoring from the MDSC202may include performance monitoring parameters such as delay, jitter, packet loss, bit error rate, monitoring cycle (e.g., 15 minutes or 24 hours), etc.

In turn, the PNC206begins performance monitoring in the underlying physical networks, collects the results of related path, and translates the performance results of the physical topology to the performance information of the abstract topology. The PNC206may collect performance results using any suitable protocols such as the Path Computation Element Protocol (PCEP) and/or databases, e.g., a Traffic Engineering Database (TED) and Label Switched Path Database (LSP-DB). At step3, the PNC206may report the performance results to the MDSC202. Based on the traffic optimization information obtained from the CNC204at step1, the MDSC202determines whether the service should be adjusted (e.g., to meet SLA requirements such as packet delay), or if a new connection should be created. If so, the MDSC202sends the traffic monitoring results to the CNC204at step4, e.g., to indicate to the CNC204that the service needs adjustment. The CNC204may then confirm whether the service can be optimized. If so, the CNC204sends a service adjustment request to the MDSC202at step5. At step6, the MDSC202may convert the service adjustment request into a path modification or creation request, which is then sent to the PNC206to complete the service optimization. At step7, the PNC206may return the optimization results to the MDSC202, which may then pass the optimization results to the CNC204at step8.

FIG. 4depicts an ACTN system400according to an embodiment of the disclosure. In some implementations, the network102inFIG. 1may serve as a backbone network to couple entities to the system400. The system400comprises the CNC204, the MDSC202, and a plurality of PNCs406A,406B, and406C, which may be substantially similar to the PNC206inFIG. 2. The PNCs406A,406B, and406C are configured to manage first, second, and third domains408A,408B, and408C, respectively. The first, second, and third domains408A,408B, and408C may generally be similar to the network216inFIG. 2.

The system400inFIG. 4is based on an example in which a virtual network is created to balance load between two tunnels. The virtual network comprises a first tunnel established between a first endpoint410and a third endpoint412, and a second tunnel established between the first endpoint410and a second endpoint414. The first tunnel includes a first link416between node418and node420; a second link422between node420and node424; a third link426between node424and node428; a fourth link430between node428and node432; a fifth link434between node432and node436; and a sixth link438between node436and node440. The second tunnel includes a first link442between node418and node444; a second link446between node444and node448; a third link450between node448and node452; and a fourth link454between node452and node456.

Generally speaking, the CNC404may instantiate the virtual network using a YANG module such as described in the Internet Engineering Task Force (IETF) working document entitled, “A Yang Data Model for ACTN VN Operation” (hereinafter, the “ACTN VN Model”), which is incorporated herein in its entirety. The ACTN VN Model provides characteristics of Access Points (APs) that describe a customer's end point characteristics; characteristics of Virtual Network Access Points (VNAP) that describe how an AP is partitioned for multiple VNs sharing the AP and its reference to a Link Termination Point (LTP) of a Provider Edge (PE) Node; and characteristics of a customer's VNs in terms of VN Members (e.g., links416,422,426,430,434,438,442,446,450,454) comprising a VN, multi-source and/or multi-destination characteristics of VN Member, the VN's reference to TE-topology's Abstract Node.

Interfaces, tunnels, and Label Switched Paths (LSPs) associated with the virtual network inFIG. 4may be defined using a YANG module such as described in the IETF working document entitled, A YANG Data Model for Traffic Engineering Tunnels and Interfaces” (hereinafter, the “TE-Tunnel Model”), which is incorporated herein in its entirety. The TE-Tunnel Model describes data models for TE Tunnels, LSPs and TE interfaces that cover data applicable to generic or device-independent, device-specific, Multiprotocol Label Switching (MPLS) technology specific, and Segment Routing (SR) TE technology.

As previously discussed with respect toFIG. 3, the CNC204may send traffic monitoring and optimization information to the MDSC202, which may use such information to ensure that the PNCs406A-406C monitor and report appropriate traffic parameters. The traffic monitoring and optimization information may comprise performance monitoring strategies, including traffic monitoring objects (e.g., services to be monitored), monitoring parameters (e.g., transmitted and received bytes per unit time), traffic monitoring cycles, traffic monitoring thresholds, Quality of Service (QoS) parameters, SLA requirements (e.g., BER, delay, jitter, packet loss rate, throughput, etc.), and the like. The traffic monitoring and optimization information may also comprise a policy that indicates if and when an existing service should be modified (e.g., upon traffic exceeding a threshold).

The MDSC202may use the traffic monitoring and optimization information to determine traffic information to be monitored by the PNCs406A-406C and reported back to the MDSC202. This way, traffic monitoring and optimization information may be used to monitor aggregated telemetry across all three domains408A-408C, as E2E KPIs may differ from KPIs measured locally at each individual domain408A,408B,408C.

In some aspects, the MDSC202may be configured to periodically send traffic monitoring results to the CNC204. In other aspects, the MDSC202may be configured to send traffic monitoring results to the CNC204responsive to an event (e.g., if a measurement report from PNC406A,406B, or406C indicates that traffic exceeds a threshold). Traffic monitoring results may include monitoring results of service performance, performance monitoring parameters, and indications of services that have been influenced. Traffic monitoring results may also include service optimization results based on performance. As previously mentioned, performance monitoring in a large-scale network can result in relatively large amounts of performance information being generated.

In an embodiment, one or more data models may be defined to aggregate telemetry data so as to reduce the overall amount of performance information exchanged within the system400. One or more data models may also be defined to provide applications (e.g., applications212) and/or clients (e.g., CRC202) with a subscription mechanism for specifying autonomic network scaling capabilities. In some embodiments, such data models may be implemented with existing data models (e.g., the ACTN VN Model, the TE-Tunnel Model, etc.) so as to enhance ACTN interfaces such as the CMI and PMI.

FIGS. 5A and 5Bdepict a module500implementing an ACTN TE-Telemetry Model according to an embodiment of the disclosure. In some embodiments, the ACTN VN Model may be extended based on the disclosed ACTN TE-Telemetry Model to enhance VN monitoring capability, and thereby facilitate proactive re-optimization and reconfiguration of VNs based on the performance monitoring data collected via the ACTN VN Telemetry YANG model. Moreover, the ACTN TE-Telemetry Model may enable an autonomic TE scaling intent configuration mechanism on the VN level.

In general, a customer such as the CNC204(or application212) may use the module500to specify Key Performance Indicator (KPI) telemetry data to which the customer subscribes. As shown inFIG. 5, for example, the module500may be used to indicate whether to monitor one-way delays from one endpoint to another endpoint (e.g., from EP1to EP3) and/or two-way delays between two endpoints. If so, the CNC204may specify a minimum and/or maximum one-way delay; a minimum and/or maximum two-way delay; a one-way delay variation (e.g., the difference in end-to-end one-way delay in forward or reverse direction); and/or a two-way packet delay variation (e.g., average delay variation in forward and reverse directions). Moreover, the module500may also be used to indicate whether to monitor one-way packet loss (e.g., in one direction), two-way packet loss (e.g., in forward and reverse directions), and/or utilized bandwidth (e.g., bandwidth utilized over a specific source and destination).

InFIG. 5, “te-ref?” may refer to the name of a tunnel such as the first tunnel (e.g., Tunnel1); “vn-ref?” may refer to an identifier identifying the virtual network associated with the tunnel; and “vn-member-ref?” may refer to a VN member (e.g., link416) corresponding to the tunnel. As shown at the bottom of the module500, one or more grouping operations may be performed in some embodiments. For example, the CNC204may use a grouping operation (e.g., AND, OR, MIN, MAX, etc.) to monitor multiple performance metrics.

In an embodiment, the module500may allow customers such as the CNC204to scaling criteria such that the MDSC204may automatically react to a certain set of variations in monitored parameters. For example, the CNC204may specify scaling criteria such that when a PNC (e.g., PNC406A) sends the MDSC202traffic performance results indicating that bandwidth provisioned for a tunnel (e.g., tunnel1) exceeds a threshold specified by the CNC202, the MDSC202may automatically request the PNC to increase the total bandwidth for that tunnel by a certain amount such as a percentage of the bandwidth that was originally provisioned.

As shown inFIG. 5B, the module500may allow the CNC204to specify a threshold time before a scaling in or scaling out operation takes places. For instance, the threshold time may be such that the MDSC202is to wait a certain amount of time before initiating a scaling in or scaling out operation. The module500may also allow the CNC204to specify a cooldown time such that when a monitored parameter (e.g., bandwidth utilization or latency) meets a certain threshold specified by the CNC204, the MDSC202is to wait a certain amount of time, e.g., to avoid frequent scaling in/out operations due to hysteresis. In some embodiments, the grouping operation inFIG. 5Amay be employed to trigger scaling in/out operations according to the criteria set forth inFIG. 5B. For example, an AND operation may be used such that a scaling-in operation is triggered when at least two monitored parameters meet conditions specified by the CNC204, e.g., if two-way delay in a VN exceeds a certain time (e.g., 50 milliseconds) and two-way packet loss in the VN exceeds a certain threshold (e.g., greater than one percent).

Summing upFIGS. 5A and 5B, the CNC204may use the module500(e.g., at step1inFIG. 3) to send the MDSC202traffic monitoring information such as a list of traffic parameters to be monitored at the TE level, as well as scaling criteria that set forth conditions for triggering scaling in/out operations. In turn, the MDSC202may request the PNCs406A-406C (e.g., at step2inFIG. 3) to monitor traffic parameters based on the traffic monitoring information received from the CNC204. For example, the traffic parameters to be monitored may include bandwidth utilization of links of a particular tunnel (e.g., tunnel1) extending through all three domains408A,408B, and408C. In some embodiments, traffic monitoring results from the PNCs406A-406C may be automatically reported to the MDSC202in real-time so that accurate performance data may be readily available to the MDSC202.

After receiving traffic monitoring results from the PNCs406A-406C (e.g., at step3inFIG. 3), the MDSC202may aggregate the traffic results and calculate the bandwidth utilized by all links (e.g., links416,422,426,430,434, and438) for that particular tunnel. If the bandwidth utilized exceeds a threshold specified by the CNC204, the MDSC202may automatically increase the tunnel's bandwidth by an amount/percentage specified by the CNC204. In other words, the MDSC202may carry out the automatic increase without having to explicitly inform the CNC204(e.g., such as at step4inFIG. 3) or receive a separate request from the CNC204(e.g., such as at step5inFIG. 3). Nevertheless, while the MDSC202need not explicitly notify the CNC202that the bandwidth utilized exceeds the specified threshold, the interface between the MDSC202and CNC204may be such that traffic monitoring results from the PNCs406A-406C may be automatically streamed to the CNC204or instantly available upon demand.

FIG. 6depicts a module600implementing a TE KPI Model according to an embodiment of the disclosure. Generally speaking, the TE KPI Model provides similar functionality as the ACTN TE-Telemetry Model, except the TE KPI Model defines mechanisms for the aggregation/abstraction of performance telemetry data at the TE-tunnel level rather than the VN level. In an embodiment, the TE-Tunnel Model may be extended based on the disclosed TE KPI Model to enhance TE performance monitoring capability, and thereby facilitate proactive re-optimization and reconfiguration of TE tunnels or VNs based on the performance monitoring data collected via the disclosed TE KPI model. Moreover, the TE KPI Model may enable an autonomic TE scaling intent configuration mechanism on the TE-tunnel level.

Accordingly, the TE KPI and ACTN TE-Telemetry Models disclosed herein may be used to monitor one or more different types of telemetry data at both the TE-tunnel level (e.g., endpoint to endpoint) and the VN level (e.g., E2E tunnel1and ETE tunnel2). In some embodiments, a data model representing a combination of the TE KPI Model and the ACTN TE-Telemetry Model may be provided.

FIG. 7depicts a block diagram700illustrating the relationship between the ACTN TE-Telemetry Model and the ACTN VN Model according to an embodiment of the disclosure. For example, the ACTN TE-Telemetry Model may augment the basic ACTN VN Model to enhance VN monitoring capabilities, such as previously discussed.

FIG. 8depicts a block diagram800illustrating the relationship between the TE KPI Model and the TE-Tunnel Model according to an embodiment of the disclosure. For example, the TE KPI Model may augment the basic TE-Tunnel Model to enhance TE performance monitoring capabilities, such as previously discussed.

FIG. 9depicts an example of a YANG Data Tree according to an embodiment of the disclosure, whileFIGS. 10 and 11depict examples of YANG coding for implementing the respective TE KPI Models and ACTN TE-Telemetry disclosed herein.

In some embodiments, notifications may be enabled in the disclosed embodiments using mechanisms such as described in IETF working documents entitled, “Subscribing to YANG datastore push updates” and “Subscribing to Event Notifications,” both of which are incorporated herein in their entirety. For example, such mechanisms may be employed to subscribe to notifications on a per client (e.g., CNC204) basis; specify subtree filters or xpath filters so that only interested contents will be sent; and specify either periodic or on-demand notifications.

FIG. 12depicts an example of push subscription schema according to an embodiment of the disclosure. For example, the schema inFIG. 12may be used by the CNC204to subscribe for telemetry information for a particular tunnel (e.g., E2E tunnel1), where “utilized bandwidth” is the telemetry parameter that the CNC204is interested in having monitored in this example.

FIG. 13depicts another example of push subscription schema according to an embodiment of the disclosure. For example, the schema inFIG. 13may be used by the CNC204to subscribe for telemetry information for all VNs, where “one-way packet loss” and “utilized bandwidth” are the telemetry parameters that the CNC204is interested in having monitored in this example.

FIG. 14is a flowchart illustrating a method1400according to an embodiment of the disclosure. The operations may be performed in the order shown, or in a different order. Further, two or more of the operations may be performed concurrently instead of sequentially. The method1400commences at block1402, where a central controller (e.g., MDSC202) receives traffic monitoring information from a client (e.g., CNC204or application212). At block1404, the central controller may send a traffic monitor request to a network controller (e.g., PNC206, PNC406A, PNC406B, or PNC406C), where the traffic monitor request may request the network controller to monitor traffic parameters based on the traffic monitoring information received from the client. At block1406, the central controller may receive a traffic report from the network controller. At block1408, the central controller may determine whether traffic parameters in the traffic report satisfy at one or more conditions specified by the traffic monitoring information. At block1410, the central controller may perform a scaling operation (e.g., scale resource up or down) responsive to determining that a traffic parameter does not satisfy a condition specified by the traffic monitoring information.

FIG. 15is a diagram of a network device1500according to an embodiment of the disclosure. The network device1500is suitable for implementing the disclosed embodiments as described herein. For example, the network device1500may include one or more components in the system100inFIG. 1, the system200inFIG. 2, and/or the system400inFIG. 4. The network device1500comprises one or more ingress ports1510and a receiver unit (Rx)1520for receiving data; at least one processor, microprocessor, logic unit, or central processing unit (CPU)1530to process the data; a transmitter unit (Tx)1540and one or more egress ports1550for transmitting the data; and a memory1560for storing the data. The network device1500may also comprise optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to the one or more ingress ports1510, the receiver unit1520, the transmitter unit1540, and the one or more egress ports1550for egress or ingress of optical or electrical signals.

The at least one processor1530may be implemented by hardware and/or software. The at least one processor1530may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), or digital signal processors (DSPs). The at least one processor1530may be communicatively linked to the one or more ingress ports1510, receiver unit1520, transmitter unit1540, one or more egress ports1550, and/or memory1560.

The at least one processor1530comprises a module1570configured to implement the embodiments disclosed herein, including method1400. The inclusion of the module1570may therefore provide a substantial improvement to the functionality of the network device1500and effects a transformation of the network device1500to a different state. Alternatively, the module1570may be implemented as readable instructions stored in the memory1560and executable by the at least one processor1530. The network device1500may include any other means for implementing the embodiments disclosed herein, including method1400.

The memory1560comprises one or more disks, tape drives, or solid-state drives and may be used as an over-flow data storage device, to store programs when such programs are selected for execution, or to store instructions and data that are read during program execution. The memory1560may be volatile or non-volatile and may be read-only memory (ROM), random-access memory (RAM), ternary content-addressable memory (TCAM), or static random-access memory (SRAM).

In an embodiment, the network device1500comprises a central controller (e.g., MDSC202) comprising a non-transitory memory storage1560storing instructions, and one or more processors1530in communication with the memory1560. The one or more processors1530execute the instructions to perform one or more embodiments of the present disclosure.

In some embodiments, there is provided a method including means receiving traffic monitoring information from a client; means for sending a traffic monitor request to a network controller, the traffic monitor request requesting the network controller to monitor one or more traffic parameters based on the traffic monitoring information; means for receiving a traffic report from the network controller; and means for determining that a monitored traffic parameter in the traffic report does not satisfy a condition specified by the traffic monitoring information, and responsive thereto, means for initiating a scaling operation based on data contained in the traffic report.

In some embodiments, there is provided central controller comprising a non-transitory memory storage means comprising instructions, and one or more processor means in communication with the memory, the one or more processor means configured to execute the instructions to cause the processor means to receive traffic monitoring information from a client; send a traffic monitor request to a network controller, the traffic monitor request requesting the network controller to monitor one or more traffic parameters based on the traffic monitoring information; receive a traffic report from the network controller; and responsive to determining that a monitored traffic parameter in the traffic report does not satisfy a condition specified by the traffic monitoring information, initiate a scaling operation based on data contained in the traffic report.