Non-invasive diagnosis of configuration errors in distributed system

Techniques for non-invasive diagnosis of configuration errors in distributed system are disclosed including obtaining data packets from a network that include inflows and outflows associated with a given component of the network. The number of inflows and the number of outflows for the given component of the network are determined based on the obtained data packets. An intersection point corresponding to a predetermined number of inflows and a predetermined number of outflows is obtained and a score for the given component is determined based on a relationship between the intersection point and the determined number of inflows and outflows for the given component. Whether the score for the given component is greater than a predetermined threshold is determined, and the given component is identified as having a configuration error in response to determining that the score for the given component is greater than a predetermined threshold.

FIELD

The present application generally relates to information technology, and, more particularly, to diagnosing configuration errors in components of a network.

BACKGROUND

Most services have a distributed deployment with components, such as, e.g., servers, client devices, firewalls, routers, or other similar components, connected over a network. In some cases, service outages may occur within a network when one or more of the components is misconfigured. Such configuration errors in a network may often be hard to identify.

SUMMARY

In one embodiment of the present invention, techniques for non-invasive diagnosis of configuration errors in distributed systems are provided. An exemplary computer-implemented method can include steps of obtaining data packets from a network that includes inflows and outflows associated with a given component of the network. The number of inflows and the number of outflows for the given component of the network are determined based on the obtained data packets. An intersection point corresponding to a predetermined number of inflows and a predetermined number of outflows is obtained and a score for the given component is determined based on a relationship between the intersection point and the determined number of inflows and outflows for the given component. The method further includes determining whether the score for the given component is greater than a predetermined threshold and identifying the given component as having a configuration error in response to determining that the score for the given component is greater than the predetermined threshold.

Another embodiment of the invention or elements thereof can be implemented in the form of a computer program product tangibly embodying computer readable instructions which, when implemented, cause a computer to carry out a plurality of method steps, as described herein. Furthermore, another embodiment of the invention or elements thereof can be implemented in the form of a system including a memory and at least one processor that is coupled to the memory and configured to perform noted method steps. Yet further, another embodiment of the invention or elements thereof can be implemented in the form of means for carrying out the method steps described herein, or elements thereof; the means can include hardware module(s) or a combination of hardware and software modules, wherein the software modules are stored in a tangible computer-readable storage medium (or multiple such media).

DETAILED DESCRIPTION

Most services have a distributed deployment with components, such as, servers, client devices, firewalls, routers, or other similar components, connected over a network. In some cases, service outages may occur within a network when one or more of the components is misconfigured. Such configuration errors in a network may often be hard to identify. For example, while each component may have its own configuration, the root cause of a service outage may not be readily apparent by simply reviewing each component's configuration alone. For example, it may be impossible to know what is considered a “correct” configuration for an individual component. In some cases, the interactions between the configurations of two or more components may be the cause of the service outage while the individual components, on their own, may be considered to be configured correctly.

With reference now toFIG. 1, an embodiment of a system architecture100for non-invasive diagnosis of configuration errors in distributed system is illustrated. In some embodiments, system100includes a computing device110, and a network150.

Computing device110includes at least one processor112, memory114, at least one network interface116, a display118, an input device120, and may include any other features commonly found in a computing device. In some embodiments, computing device110may include, for example, a personal computer, workstation, laptop, tablet, smart device, smart phone, smart watch, or any other similar computing device that may be used by a user.

Processor112may include, for example, a central processing unit (CPU), a microcontroller, Field Programmable Gate Array (FPGAs), or any other form of processing circuitry that is configured to perform various operations. Processor112may be configured to execute instructions as described below. These instructions may be stored, for example, in memory114. As used herein, the term “processor” may include a single core processor, a multi-core processor, multiple processors located in a single device, or multiple processors in wired or wireless communication with each other and distributed over a network of devices, the Internet, or the cloud. Accordingly, as used herein, functions, features or instructions performed or configured to be performed by a processor may include the performance of the functions, features or instructions by a single core processor, may include performance of the functions, features or instructions collectively or collaboratively by multiple cores of a multi-core processor, or may include performance of the functions, features or instructions collectively or collaboratively by multiple processors, where each processor or core is not required to perform every function, feature or instruction individually.

Memory114may include, for example, computer readable media or computer readable storage media in the form of volatile memory, such as random-access memory (RAM) and/or cache memory or others. Memory114may include, for example, other removable/non-removable, volatile/non-volatile storage media. By way of non-limiting examples only, memory114may include a portable computer diskette, a hard disk, a random-access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

In some embodiments, computer software including instructions or code for performing the methodologies of the invention, as described herein, may be stored in associated memory114and, when ready to be utilized, loaded in part or in whole (for example, into RAM) and implemented by processor112. Such software could include, but is not limited to, firmware, resident software, microcode, and the like. The memory114may include local memory employed during actual implementation of the program code, bulk storage, and cache memories which provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during implementation.

Network interface116is configured to transmit and receive data or information to and from a network150or any other server, component, or computing device via wired or wireless connections. For example, network interface116may utilize wireless technologies and communication protocols such as Bluetooth®, WIFI (e.g., 802.11a/b/g/n), cellular networks (e.g., CDMA, GSM, M2M, and 3G/4G/4G LTE), near-field communications systems, satellite communications, via a local area network (LAN), via a wide area network (WAN), or any other form of communication that allows computing device110to transmit or receive information to or from network150including components of network150.

Display118may include any display device that is configured to display information to a user of computing device110. For example, in some embodiments, display118may include a computer monitor, television, smart television, or other similar displays. In some embodiments, display118may be integrated into or associated with computing device110, for example, as a display of a laptop, smart phone, smart watch, or other smart wearable devices, as a virtual reality headset associated with computing device110, or any other mechanism for displaying information to a user. In some embodiments, display118may include, for example, a liquid crystal display (LCD), an e-paper/e-ink display, an organic LED (OLED) display, or other similar display technologies. In some embodiments, display118may be touch-sensitive and may also function as an input device120.

Input device120may include, for example, a keyboard, a mouse, a touch-sensitive display118, a keypad, a microphone, or other similar input devices or any other input devices that may be used alone or together to provide a user with the capability to interact with computing device110.

Network150includes components152A,152B, . . . ,152N that are configured to communicate with each other to perform one or more services. Each component may include a processor, memory, and a network interface that may include similar functionality as processor112, memory114, and network interface116. In some embodiments, each component152A-N may comprise a computing device, server, or similar device or system that is configured transmit and receive data packets in the network150.

IP traffic data between components of a network may be analyzed by capturing data packets at various points in a system. For example, a variety of IP network packet capture tools may be utilized to capture packets at a firewall, router, or other component of a network. In some embodiments, an IP network packet capture tool such as, e.g., Netflow™ developed by Cisco Systems, may be utilized to capture data packets in a network. The tool may, for example, utilize a standardized packet format which is supported by most or all of the components in the network such as, e.g., a CISCO® standardized packet format. In some embodiments, for example, where different packet formats are used by the components of a network, additional or alternative tools may be utilized to capture the data packets. The IP network packet capture tool may sample data packets, construct in-memory flow statistics, and export or log the flow statistics for later review. The tool may track data packet flow properties such as, e.g., source IP, source port, destination IP, destination port, protocol, flags, and other similar properties of a particular flow.

A non-limiting list of additional or alternative IP network packet capture tools that may be utilized to capture data packets may include, for example, Argus® developed by The Argus Project, Jflow™ developed by Juniper Networks, Cflowd™ developed by the Center for Applied Internet Data Analysis, NetStream® developed by Network Data Systems (NDS), Rflow™ developed by Real Solutions, AppFlow® developed by Citrix®, Traffic Flow™ developed by MicroTik, sFlow® developed by SFlow, or other similar tools.

In some embodiments, any of the IP network packet capture tools described above may be utilized to capture IP network data packets for network150. In some embodiments, any other IP network packet capture tool may be utilized to capture IP network data packets for network150.

In some embodiments, an IP network packet capture tool may collect or obtain log data associated with network150over a period of time. In some embodiments, for example, the period of time may be a predetermined period of time. In some embodiments, the period of time may be obtained from a user of the system, e.g., via input device120, or it may be obtained in any other manner.

In some embodiments, the IP network packet capture tool may construct patterns of network traffic and flows among and between networked components. These components and their patterns and flows may be characterized based on the network characteristics.

In some embodiments, additional data about the components may be imported from alternative sources. The additional data about a component may include, for example, a date of provisioning, role, power state time series, aggregate bandwidth usage, or other similar data about a particular component. In some embodiments, the alternative sources may include, for example, one or more databases external to the network150, non-network computing devices, non-network servers, or other similar sources. In some embodiments, the alternative sources may include one or more of the components of network150themselves. For example, where the IP network data packets may be captured by the tool at routers or other similar devices in the network150, the additional data may be obtained directly from the service components themselves such as, e.g., servers, computing devices, or other similar devices that perform services.

In some embodiments, anomalous behavior may be identified, for example, based on component characteristics, based on flow characteristics, or based on similarity to past behavior, deviation from past behavior, or both, e.g., a trend.

In some embodiments, components with configuration errors leading to anomalous behavior may be identified, for example, based on a temporal trends and co-relation with other components. For example, data packets associated with interactions of a given component of the network with one or more other components of the network may be analyzed to determine if there is a trend in the co-relation of the given component with other components that indicates a potential configuration error or anomaly.

In some embodiments, closed loop verification and tracking of anomalous patterns may be used to determine whether a configuration change to a given component has fixed a configuration error. For example, the configuration fix may be performed manually to fix the given component, and the corresponding flow may be labeled for future tracking. In some embodiments, reinforced or supervised learning models may then utilize the labels to determine whether the configuration fix was successful, for example, by monitoring the labeled flows and corresponding IP network data packets related to the given component for further anomalous behavior. In some embodiments, a smoke test, e.g., fake flows or interactions, may be triggered in the given component to automate verification of the configuration fix.

In some embodiments, the network150and the components152A-152N of the network150may include particular characteristics that may be analyzed to determine whether there is a configuration error. For example, a given component of the network150, e.g., a server or other computing device, may have associated characteristics such as, e.g., number of inflows, number of outflows, total bandwidth used, and which other components are connected to the given component. A given flow or type of flow in the network150, e.g., a communication between two or more components of the network150, may include characteristics such as duration, volume of data transferred, frequency, and what flags are set.

When analyzing network packets that are captured by an IP network packet capture tool, various kinds of anomalous behavior patterns may be seen.

For example, in some cases, a communication flow may not complete, e.g., due to an invalid IP or port combination. In this case, for example, analysis of the network packets may show that only synchronization (SYN) packets are sent by a given component of network150and connection establishment or data packet transfer may or may not be dropped by a firewall. For example, the firewall may not execute or forward the packet to the next destination.

As another example, where a given flow has completed but the behavior of the flow has changed from prior instances of the flow, i.e., historical instances of the same flow or type of flow for that component, an anomaly may be detected. For example, if the given flow has a short duration or volume of data being transferred while the prior instance of the flow had a longer duration or higher volume of data being transferred, or vice versa, this may indicate that there is an anomaly for the given flow which may have been caused by a configuration error. As another example, if the timing of when the given flow occurred relative to the timing between prior instances of the flow, e.g., the frequency, has been reduced or increased, this may indicate that there is an anomaly for the given flow which may have been caused by a configuration error.

In some cases, for example, where a given flow is completed, but to an endpoint other than that defined, for example, by a policy of the network or service, this may indicate that an anomaly may be present. For example, if the network includes a satellite server in a local environment and a global satellite server that may perform the same service or function, the policy of the network may indicate that the local satellite server is preferable to the global satellite server as an endpoint. For example, the use of a local satellite server may reduce costs, reduce latency, or provide other similar advantages. If the given flow completes with an endpoint of the global satellite server instead of the local satellite server, this may indicate that an anomaly is present in the configuration of the component that is executing the flow.

In another case, for example, one component in a cluster of components may have flows that are different from other similar components in the cluster. In some embodiments, to make this determination, additional data may be required aside from the network packets. For example, if one worker node in a Hadoop® cluster is referring to a zookeeper server that the other worker nodes in the cluster do not refer to, this information may be utilized to identify a misconfiguration in the one worker node.

In an embodiment, network packet data may be utilized to assist in determining whether there are any misconfigurations in a network150. For example, in some embodiments, the number of inflows and number of outflows for components152A-152N of the network150may be obtained over a predetermined period of time and analyzed to determine whether there are potential misconfigurations for those components in the network150. In some embodiments, the predetermined period of time may be a period of from the provisioning date of the component to a current time. In some embodiments, the predetermined period of time may be a time since the last configuration change of a component.

In some embodiments, the number of inflows and number of outflows for a given component may form a pair that may be used as a data point. For example, a given component that has 150 inflows and 120 outflows may have a data point of 150, 120.

With reference now toFIGS. 2-4, for example, the data points of the components152A-152N may be plotted. For example, as illustrated inFIG. 2, the number of inflows may be plotted against the number of outflows for each component152A-152N to generate a plot200.

With reference now toFIG. 3, in some embodiments, the plot200may be divided into buckets around an intersection point202. In some embodiments, the intersection point202may be predetermined point. In some embodiments, the intersection point202may be set by a user of computing device110, e.g., via input device120. In some embodiments, for example, an intersection point202may be determined based on a manual analysis of the traffic patterns of the network over time. In some embodiments, machine learning techniques such as, e.g., clustering, may be employed to analyze the traffic patterns of the network and determine an intersection point202. In some embodiments, the intersection point202may be determined based on a defined rule. As illustrated inFIG. 3, for example, the intersection point202may be 100, 100. In some embodiments, the intersection point202may be any point on the plot200. For example, the intersection point202may be, e.g., 100, 100; 150, 150; 200, 200; 1000, 1000; 10,000, 10,000; 100, 200; 100, 300; or any other pair of inflow and outflow numbers. The number of inflows need not match the number of outflows at the intersection point202.

In some embodiments, a first bucket204may comprise all components152A-152N that have a number of inflows below the number of inflows at the intersection point202and a number of outflows below the number of outflows at the intersection point202. The first bucket204may, for example, represent those components that have a low level of activity with low numbers of inflows and outflows relative to the intersection point202.

In some embodiments, a second bucket206may comprise all components152A-152N that have a number of inflows above the number of inflows at the intersection point202and a number of outflows below the number of outflows at the intersection point202. The second bucket206may, for example, represent those components that have mismatched activity between inflows and outflows with more outflows than inflows.

In some embodiments, a third bucket208may comprise all components152A-152N that have a number of inflows below the number of inflows at the intersection point202and a number of outflows above the number of outflows at the intersection point202. The third bucket208may, for example, represent those components that have mismatched activity between inflows and outflows with more inflows than outflows.

In some embodiments, a fourth bucket210may comprise all components152A-152N that have a number of inflows above the number of inflows at the intersection point202and a number of outflows above the number of outflows at the intersection point202. The fourth bucket210may, for example, represent those components that have a high level of activity with high numbers of inflows and outflows relative to the intersection point202.

While illustrated with four buckets204-210inFIG. 3, any other number of buckets may be used without departing from the scope of the present disclosure including, for example, no buckets, one bucket, two buckets, three buckets, five buckets, or any other number of buckets.

While illustrated as a plot200inFIGS. 2-4, in some embodiments, the data points may also or alternatively be stored in a data structure and analyzed without plotting.

With reference now toFIG. 4, the Euclidean distance between the intersection point202and the data point for each component may be determined as a score for that component. For example, a data point having a longer Euclidean distance from the intersection point202will have a higher score than a data point having a shorter Euclidean distance from the intersection point202. For example, if the points (x 1, y 1) and (x 2, y 2) are in 2-dimensional space, then the Euclidean distance between them is SqRoot{(x 2−x 1){circumflex over ( )}2+(y 2−y 1){circumflex over ( )}2}.

Using Euclidean distance as the basis for a score for a component allows outlier components having large Euclidean distances from the intersection point202, e.g., high scores, to be identified, e.g., due to large mismatches in inflows and outflows, very large levels of inflows and outflows, or very low levels of inflows and outflows, while components having low scores may be disregarded, e.g., those components that have relatively matched numbers of inflows and outflows with the numbers being relatively close to the intersection point202. In some embodiments, for example, the components with the top 5 scores, e.g., 5 longest Euclidean distances, may be targeted for further analysis or action.

In some embodiments, the determined scores for each component may be compared to a predetermined threshold where, for example, scores above the predetermined threshold may indicate that a given component may have a configuration error. In some embodiments, for example, the predetermined threshold may be 100 where any distances above the predetermined threshold of 100 may be considered abnormal. Any other predetermined threshold may be used. For example, the predetermined threshold may be obtained from a user, e.g., via input device120, determined in view of an analysis of the traffic patterns in the network or in any other manner. In a non-limiting example, the components with the five largest Euclidean distances above the predetermined threshold may be considered abnormal. Any other number of components having distances above the predetermined threshold may be considered abnormal.

In some embodiments, an alert may be raised in response to a detection of abnormalities in one or more components, e.g., based on their Euclidean distances being greater than the predetermined threshold, being in the top 5 number of largest Euclidean distances, or both. Corrective action may then be taken by the system. In some embodiments, for example, the corrective action may be taken automatically by the system in response to an alert. In some embodiments, an administrator of the system may review the alert and analyze the component to determine whether the abnormality is indicative of a configuration error in that component.

In some embodiments, the analysis of the data points may be extended to an N dimensional Eigen vector. For example, where the determination of Euclidean distance is described above with respect to a two dimensional plane, i.e., inflow and outflow, other factors may also be considered including, for example, inflow, outflow, latency, bandwidth, etc., which would increase the number of dimensions. For example, inflow, outflow, latency, and bandwidth may be a four dimensional plane in which the distance between points in four dimensions, e.g., an Eigen vector, may be used to determine whether there are any abnormalities in a similar manner to the Euclidean distance in two dimensions described above.

In some embodiments, computing device110may automatically trigger corrective action for those components having scores above the predetermined threshold, the top N scores, or both, e.g., automatic configuration updates, requesting manual configuration updates, flagging the component for a configuration update, transmitting an indication of a configuration error, restarting a service, re-routing network traffic around the abnormal component to a backup service, or other similar action.

When such a corrective action has been executed, the network packets associated with that component may be monitored in the manner described above to determine whether the configuration error is still present and whether the configuration update has any effect on other components in the network150.

The above disclosed techniques identify at least some misconfigured components in a network without requiring access to configuration files or other knowledge of the component itself. These techniques are non-intrusive and do not require any additional operation or action by the components themselves to provide information to the computing device110, which preserves computing resources and efficiency in the network. For example, as described above, the network packets may be obtained by an IP network packet capture tool during the normal course of operation and may be analyzed to identify the number of inflows and outflows for each component152A-152N of the network150. The number of inflows and outflows for a particular component may be used as a paired data point and the Euclidean distance of that data point to an intersection point may be used to determine a score for that component. A score above a predetermined threshold may indicate a potential configuration error and may trigger corrective action. The disclosed techniques do not require an agent or footprint on the component or other endpoint, are easy to deploy, and are non-intrusive.

In another embodiment, each component in the network may have an associated blame value that may be attributed to the component for potential configuration errors. For example, computing device110may store a data structure or other construct, e.g., an array, which tracks blame values for each component. In some embodiments, the blame values may be stored in another location on the network.

In some embodiments, for example, the blame value for each component may be initialized to a predetermined value, e.g., 0. In some embodiments, after a corrective action is taken on a component, the blame value for that component may be reset, e.g., back to the predetermined initialization value. In some embodiments, the blame value for a component may be reset to the predetermined initialization value or decremented after a predetermined period of time in which the blame value is not incremented.

In this embodiment, IP network data packets may be obtained as described above using an IP network packet capture tool and may be analyzed to determine whether any components are only receiving or only transmitting packets in particular flows. For example, if only SYN packets are being transmitted or received in a flow between two or more components, this may indicate that there is a configuration error on one or more of the components. For example, this situation may indicate that there is no service on a particular port associated with the flow on one or more of the components.

When such a situation occurs, for example, a blame value may be assigned to both the component that received the SYN packets, and the component that transmitted the SYN packets for that flow. For example, a blame value associated with each component may be initialized or incremented. As additional IP network data packets are transmitted and received in the network, blame values may be accumulated on some components from similar flows, e.g., all clients to same server/port, to identify the components, e.g., clients, servers, firewalls, or other similar components, that have configuration errors. For example, if a given component is only transmitting SYN packets to, or receiving SYN packets from, other components without taking any other action for those SYN packets, the blame value for the given component may be incremented for each flow that only transmits or receives the SYN packets. When the blame value for the given component is incremented above a predetermined threshold, this may indicate that that the given component has a configuration error.

In some embodiments, short lived flows may be used to identify a component having a configuration error. Short lived flows may, for example, result from application connection issues between components. In some embodiments, for example, a trend of the flow or other similar flows may be identified, e.g., based on historical information about that flow or matching characteristics to other flows. The trend may be identified, for example, based on flow information collected over time by the IP network packet capture tool. In some embodiments, the trend may be identified based on other data obtained from other sources.

In an illustrative example, if the trend of the flow or of similar flows is to be long lived flow, e.g., database sessions such as login, query, read responses, etc., but the current flow of the same or similar type is short lived, this may indicate a configuration error. For example, if a component periodically connects to another component to execute some program and this process typically takes about 15 minutes, but the periodic connection between the two components has now dropped to only a few seconds, this may indicate an abnormal condition in one or both of the components. An alarm may be raised in response to such a dramatic change in the time of the periodic connection because this change is not normal. Affected components such as, e.g., servers, may see short term connections to other components, e.g., database server, port, etc. which may indicate that there is a connection issue. In an illustrative example, the username or password for a user associated with the flow may be expired or have been locked out which may cause the application connections to be refused by one or more components of the network, e.g., servers, databases, or other similar components. While short lived and long lived are relative terms described above with respect to particular times, these terms are not limited to the above described amounts of time. For example, in some embodiments, short lived may be any time that is smaller than a long lived time as may exist in the system and vice versa.

In some embodiments, supervised or unsupervised machine learning may be trained using multiple network characteristics. For example, clustering may be used to identify components having similar properties or activities. For example, a machine learning model may be trained using data including, e.g., inflows, outflows, active sessions, time since the service is up, or other similar inputs. The machine learning model may be configured to output, for example, clusters or groups of components that are exhibiting similar behavior where, for example, outlier components that are acting in an abnormal manner may receive a small or individual cluster or grouping and may be easily identifiable based on the clustering or grouping.

The techniques described herein may be used to identify mis-configurations in components of a network without directly accessing the configuration files, software, or other knowledge of the components. For example, by using IP network data packets obtained by an IP network packet capture tool, configuration issues in the components of the network may be identified in a non-invasive manner that does not tax the processing resources of the system.

In addition, other sources of data in addition to the IP network packet data may be incorporated to allow the assignment of a blame value to particular components which exhibit anomalous behavior. Some examples of additional data include scalable language application programming interfaces (SL APIs), logging and monitoring of metric data, logging, monitoring, and alerting (LMA) service alerts, or other sources of data.

FIG. 5is a flow diagram illustrating a method according to an illustrative embodiment.

At502, data packets are obtained from the network150by computing device110, for example, using one or more of the IP network packet capture tools described above. The obtained data packets may include inflows and outflows associated with a given component of the network.

At504, computing device110determines a number of inflows and a number of outflows for the given component of components152A-152N of the network150based on the obtained data packets. In some embodiments, the number of inflows and outflows may be provided by the IP network packet capture tool.

At506, computing device110obtains an intersection point corresponding to a number of inflows and a number of outflows. For example, in some embodiments, the intersection point may be predetermined and stored in computing device110or on the network150. In some embodiments, the intersection point may be obtained from a user input, e.g., via input device120.

At508, computing device110determines a score for the given component based on a relationship between the intersection point and the determined number of inflows and outflows for the given component. For example, in some embodiments, the relationship may be a Euclidean distance, Eigen vector or other similar relationship between the intersection point and a data point defined by the number of inflows, outflows, or other similar inputs (parameters) for the given component.

At510, computing device110determines whether or not the score for the given component is greater than a predetermined threshold. If the score is not greater than a predetermined threshold, the method ends, otherwise the method proceeds to step512.

At512, computing device110identifies the given component as having a configuration error in response to determining that the score for the given component is greater than a predetermined threshold.

At514, computing device110triggers a corrective action, for example, automatically, for the given component in response to the identification of the given component as having a configuration error. In some embodiments, for example, the corrective action may include an update to the configuration of the given component.

Characteristics are as follows:

Service Models are as follows:

Deployment Models are as follows:

Workloads layer90provides examples of functionality for which the cloud computing environment may be utilized. Examples of workloads and functions which may be provided from this layer include: mapping and navigation91; software development and lifecycle management92; virtual classroom education delivery93; data analytics processing94; transaction processing95; and non-invasive diagnosis of configuration errors in distributed systems96.