Patent Description:
When a message is communicated from one computing system to another over a network, that message is typically routed through several intermediary systems (also called "routers"). For any given message, it is possible to trace the route that the message has taken to get from its origin to its destination. Such as description of the route of a message is often termed a "trace route". For instance, a trace route might contain an ordered list of Internet Protocol (IP) addresses beginning at the origin IP address, through the various intermediary systems (e.g., routers, firewalls, gateways, and so forth) and ending at the destination IP address.

Like every complex system, networks have a variety of ways in which they can fail to achieve their best function. For instance, an intermediary system might experience a hardware failure, be misconfigured, having insufficient resources, fail to dynamically adapt to increased traffic, experience an unanticipated state, suffer software failure, dispatch messages along inefficient routes, and any of a variety of different mechanisms of possible failure.

Rather, this background is only provided to illustrate one exemplary technology area where some embodiments describe herein may be practiced. <CIT> describes an alert system and method to identify and characterize real-time information transmission anomalies in high-frequency global and local traceroute data. <CIT> describes how a GUI, system, and method for displaying operating performance of a packet network may include displaying first graphical representations illustrative of network communications devices operating on a packet network, where the first graphical representations may be displayed in a first mode if respective network communications devices are operating normally to communicate data packets including real-time content and non-real-time content and in a second mode if respective network communications devices are not operating normally to communicate data packets including real-time content and non-real-time content.

Embodiments disclosed herein relate to the computer-aided identification of a potential problem node in a network. As part of data collection, a computing system gathers telemetric and trace route data for communications that occur within a network. Upon detecting that a performance problem has occurred within the network, multiple communications that occurred within a window of time containing the time of the performance problem occurred are identified. As an example, the gathered telemetric and trace route data may itself be used in order to detect the performance problem in the first place.

Then, the trace route data for these multiple communications are aggregated. For instance, the aggregation may take the form of a graph in which each node of the graph represents a discrete network node. The aggregation may also aggregate the gathered telemetric data. For instance, an aggregated telemetry for a given network node may represents an aggregation of the telemetry for at least some of the communications that passed the associated network node.

The aggregated data may then be used to identify a potential problem network node within the network to allow for further investigation. For instance, the aggregated telemetry for a given node may itself reveal performance issues associated with that network node. The identity of the problem network node (and potentially observed telemetry for that network node) may then be communicated to an entity that is affected by the performance problem. This may allow the entity to understand which network node is potentially causing the problem and thus make appropriate investigation and remedial measures. Thus, network performance problems may be quickly analyzed to identify and remedy problematic network nodes that contribute to the network performance problems. Accordingly, the principles described herein advance the art of network performance troubleshooting.

In order to describe the manner in which the above-recited and other advantages and features can be obtained, a more particular description of the subject matter briefly described above will be rendered by reference to specific embodiments which are illustrated in the appended drawings in which:.

<FIG> illustrates a network environment <NUM> in which the principles described herein may operate. The network environment <NUM> includes a network <NUM> and a number of computing systems <NUM> (each being represented with a square) that can connect to the network <NUM> so as to be able to receive messages from and/or transmit message over the network <NUM>. The network <NUM> also includes a number of intermediary systems <NUM> (each being represented by a circle, also referred to as an "intermediary") through which a message may pass as it travels from one of the computing systems <NUM> to another of the computing systems <NUM>. As an example, an intermediary might be a firewall, a gateway, a router, or the like. Each of the computing systems <NUM> and the intermediaries <NUM> may be structured as described below for the computing system <NUM> of <FIG>.

The network <NUM> may be a relatively small network, such as a small local area network. On the other extreme, the network <NUM> may expand across the globe and include innumerable computing systems, such as the Internet. The network <NUM> may also include any network in-between these two extremes. As more specific examples, the network <NUM> might be a home network, an enterprise network, an Internet Service Provider (ISP) network, a transit network, a cloud provider network, combinations of the above, and so forth. The fact that the computing systems <NUM> are illustrated as including only eight computing systems <NUM> through <NUM> is thus merely for example purposes only. The ellipsis <NUM> represents that the principles described herein are not limited to the particular nature, size or complexity of the network. Furthermore, the ellipsis <NUM> represents that the number of computing systems <NUM> that are used to communicate messages over the network <NUM> may be static or dynamic, with computing systems <NUM> being added to and/or removed from the network <NUM> over time. Likewise, although only seven intermediaries <NUM> through <NUM> are illustrated as being included within the intermediaries <NUM> within the network <NUM>, the ellipsis <NUM> represents that the principles described herein are not limited to the number of intermediaries within the network <NUM>.

<FIG> illustrates a computing environment <NUM> that may be used to evaluate the network environment <NUM> of <FIG>. The computing environment <NUM> includes data gathering components <NUM>, fault detection component <NUM>, fault analysis components <NUM>, and reporting component <NUM>. Each of these components may be structured as described below for the executable component <NUM> of <FIG>. Furthermore, the computing environment <NUM> may be operated using one computing system, or a distribution of multiple computing systems. The computing environment <NUM> also includes a database <NUM> for storing gathered data. Potentially, the computing environment <NUM> includes a mapping <NUM> of network addresses to physical locations and/or entities that are responsible for maintaining the network node.

As an example, if the computing environment <NUM> is operated using the computing system <NUM> of <FIG>, the various components <NUM>, <NUM>, <NUM> and <NUM> may perform their operations in response to the processing unit <NUM> executing computer-executable instructions that are in the memory <NUM>. Furthermore, the components <NUM>, <NUM>, <NUM> and <NUM> may be instantiated in the memory in response to the processing unit <NUM> executing computer-executable instructions.

<FIG> illustrates a flowchart of a method <NUM> for computer-aided identification of a potential problem node in a network in response to detecting that a performance problem has occurred in the network. As an example only, the method <NUM> may be performed using the computing environment <NUM> of <FIG> in order to identify potential problem nodes in the network environment <NUM> of <FIG>. Accordingly, the method <NUM> of <FIG> will now be described with frequent reference to <FIG> and <FIG> as an example.

The method <NUM> includes gathering telemetric and trace route data (act <NUM>) for each of multiple real-time communications that occurred within a network. This gathering act (act <NUM>) is shown independent of the other acts within the method <NUM>, to emphasize that there is no restricted temporal dependency upon other acts within the method <NUM>. As an example only, the telemetric and trace route data may be gathered (act <NUM>) continuously, as real-time communications are performed. As an example, the real-time communication might be a telephonic communication (e.g., Voice over IP), a video call, an online conference communication, or the like.

<FIG> illustrates that the data gathering components <NUM> includes a trace route determination component <NUM>, that may gather the trace route information for each of multiple real-time communications. Thus, the trace route determination component may perform part of the gathering act (act <NUM>) of <FIG>. <FIG> illustrates an example network path <NUM> taken by a real-time communication. The example network path <NUM> has an origin node of computing system <NUM>. The communication passes through intermediaries <NUM>, <NUM>, <NUM> and <NUM> in that order, and then arrives at a destination node of computing system <NUM>. In this example, the trace rout determination component <NUM> would gather an ordered list of Internet Protocol (IP addresses) for this communication.

<FIG> illustrates that the data gathering components <NUM> also include a communications quality component <NUM>, which estimates the quality of the communication for each real-time communication for which the trace route determination component <NUM> determined the trace route. For instance, the communication quality component <NUM> might gather end-to-end telemetric data collected from the endpoints of a real-time communication. As an example, in the case of communication <NUM>, the communications quality component <NUM> may gather end-to-end telemetry from each or both of the computing systems <NUM> and <NUM>. For instance, the telemetry might include whether there was (and/or degree of) jitter, delay, packet loss, or even whether the connection was dropped (e.g., if the communication was a call, whether the call was dropped).

Other communication quality metrics might be what the average and/or median signal to noise ratio was, user feedback metrics (whether there was a user complaint, how the user rated the call quality, and so forth), whether there was and the degree of any glitching in the real-time communication, and so forth. Thus, the communication quality component <NUM> may also perform part of the gathering act (act <NUM>) of <FIG>. The telemetrics and associated trace route data for each monitored communication is stored (e.g., in the database <NUM> of <FIG>).

<FIG> illustrates an example data set <NUM> that might be gathered for multiple communications occurring within the network environment <NUM> of <FIG>. The example data set <NUM> includes trace route data (see column <NUM>) telemetric data (see column <NUM>), and time data (see column <NUM>) for each of six communications (see rows <NUM> through <NUM>) that occurred within the network environment <NUM>.

For instances, row <NUM> includes trace route data, telemetry data, and time data for the communication <NUM> of <FIG>. Of course, the principles described herein are not limited to such a simple case of six communications performed in a simple and small network, which is just provided by way of being a manageable example. In practice, the data set <NUM> might thousands, millions, or more, communications, occurring with the network environment including innumerable computing systems and intermediaries (e.g., perhaps the entirety of the Internet).

Here, the telemetric data is reduced to a score between <NUM> and <NUM>, with <NUM> indicating high communication quality, and <NUM> indicating low communication quality. Communications <NUM>, <NUM> and <NUM> indicate a relatively high communication quality (<NUM>, <NUM> and <NUM>, respectively). On the other hand, communications <NUM>, <NUM> and <NUM> indicate a relatively low communication quality (below <NUM>). The principles described herein are not limited to the structure of the telemetry data, nor how the telemetric data is generated. In one example, the telemetry data is much more descriptive about the type of any problem encountered in the communication. For instance, for communication <NUM>, rather than just have a simple telemetric summary (<NUM> on a scale of <NUM> to <NUM>), the telemetry might include an indication that the communication was dropped, that there was a certain amount of delay in the communication, that the signal-to-noise was at a certain level, that the communication had a certain number of other intermediaries between the origin and destination, that the intermediary had a particular configuration, and so forth.

The example data set <NUM> may include a time (see column <NUM>) at which the respective communication happened. The time may be represented using any format such as Universal Coordinated Time (UTC). Alternatively, the time might be a logical time (such as a monotonically increasing communication identifier). However, in the illustrated example, the times are represented as minutes in local time (the date is omitted since the times all occurred on the same day in the example).

Referring back to <FIG>, the content of dashed-lined box <NUM> may be performed each time a performance problem has been detected as occurring within the network (act <NUM>). This detection may be performed using at least some of the gathered telemetric and trace route data (which was gathered in act <NUM>). For instance, in <FIG>, the fault detection component <NUM> receives data gathered and stored in the database <NUM>. The fault detection component <NUM> may query the database <NUM> for gathered data. As previously mentioned, an example of that gathered data is the data set <NUM> of <FIG>. Using the data set <NUM> as an example, the fault detection component <NUM> may detect that the telemetrics declined suddenly between the communication associated with row <NUM> which occurred at <NUM>:<NUM> pm and the communication associated with row <NUM> which occurred at <NUM>:<NUM>. Using this information, the fault detection component <NUM> may detect that a fault occurred at <NUM>:<NUM> pm.

The fault detection component <NUM> might continuously operate so as to detect problems substantially when they have occurred. Alternatively, or in addition, the fault detection component <NUM> might detect problems in response to a query. For instance, upon starting work in the morning, an Information Technology employee might query for any problems that have occurred in the last <NUM> hours since the employee was last at work. The fault detection component <NUM> might then evaluate the database <NUM> to identify times at which problems in telemetry have occurred.

Returning to <FIG>, once a performance problem is identified as having occurred (act <NUM>), multiple real-time communications are identified which have occurred around a time that the performance problem occurred in the network (act <NUM>). For instance, if a performance problem is identified as having occurred at <NUM>:<NUM> pm (using data set <NUM>), a window of time is selected around the time of the performance problem. Then, any real-time communications that happened within this period of time may be identified. For instance, if the window of time is <NUM> minutes before to <NUM> minutes after the detected performance problem of <NUM>:<NUM> pm, all communications between <NUM>:<NUM> pm and <NUM>:<NUM> pm might be identified. In the data set <NUM> of <FIG>, the four communications represented by row <NUM> (occurring at <NUM>:<NUM> pm), row <NUM> (occurring at <NUM>:<NUM> pm), row <NUM> (occurring at <NUM>:<NUM> pm), and row <NUM> (occurring at <NUM>:<NUM> pm) may be identified. Other windows of time may also be suitable. For instance, the window of time might begin with the time of the performance problem and go until some duration after the time of the performance problem. As another example, the window of time might begin some duration prior to the time of the performance problem, and end at the time of the performance problem.

Referring to <FIG>, this communication set gathering may be performed by an input communication set gathering component <NUM>, which may be one of the fault analysis components <NUM>. The input communication set gathering component <NUM> receives the time of the performance problem from the fault detection component <NUM>, and may query the database <NUM>.

The trace route data is then aggregated (act <NUM>) using the multiple real-time communications that were identified as occurring around the time of the performance problem (act <NUM>). Referring to <FIG>, this aggregation may be performed by the aggregation component <NUM>, which may also be one of the fault analysis components <NUM>. As one example, the aggregation component <NUM> may construct an aggregated acyclic graph by aggregating the trace route data for the identified input communication set. For instance, the aggregated acyclic graph might include a graph in which the multiple input communications are superimposed such that the same computing system or intermediary is represented as a single node in the graph, and in which each graph link represents a distinct link between network nodes.

<FIG> illustrates an example directed acyclic graph that represents an aggregation of the trace route data for the four communications represented by rows <NUM> through <NUM> (the input communication set in the example). The links <NUM>, <NUM>, <NUM> and <NUM> represent an ordered path of the trace route for the communication of row <NUM> of <FIG>. The links <NUM>, <NUM>, <NUM> and <NUM> represent an ordered path of the trace route for the communication of row <NUM> of <FIG>. The links <NUM>, <NUM>, <NUM> and <NUM> represent an ordered path of the trace route for the communication of row <NUM> of <FIG>. The links <NUM>, <NUM> and <NUM> represent an ordered path of the trace route for the communication of row <NUM> of <FIG>.

Thus, node <NUM> of <FIG> is a superimposed node distinctly representing intermediary <NUM> of <FIG>, which was used for the two communications of rows <NUM> and <NUM>. Node <NUM> of <FIG> is a node that distinctly represents intermediary <NUM> of <FIG>, which was used for a single communication of row <NUM>. Node <NUM> of <FIG> is a node that distinctly represents intermediary <NUM> of <FIG>, which was used for a single communication of row <NUM>. Node <NUM> of <FIG> is a node that distinctly represents intermediary <NUM> of <FIG>, which was used for a single communication of row <NUM>. Node <NUM> of <FIG> is a superimposed node distinctly representing intermediary <NUM> of <FIG>, which was used for the two communications of rows <NUM> and <NUM>. Node <NUM> of <FIG> is a superimposed node distinctly representing intermediary <NUM> of <FIG>, which was used for all four communications of rows <NUM>, <NUM>, <NUM> and <NUM>.

In addition to aggregating the trace routes, the telemetric data may also be aggregated for each network node. This aggregation may also be performed by the aggregation component <NUM> of <FIG>. In a simple example, for each of the network nodes <NUM> through <NUM> involved in at least one communication of the input communication set, an aggregated telemetry data is obtained by averaging the telemetric data for each communication that passed through the network node.

Thus, in this simple example, for network node <NUM>, the aggregated telemetric data might be <NUM> (the average of <NUM> for row <NUM>, and <NUM> for row <NUM>). For network node <NUM> through <NUM>, the aggregated telemetric data might be the same as the single row in which the network nodes appeared (<NUM> for network node <NUM> corresponding to row <NUM>, <NUM> for network node <NUM> corresponding to row <NUM>, and <NUM> for network node <NUM> corresponding to row <NUM>). For network node <NUM>, the aggregated telemetric data might be <NUM> (the average of <NUM> for row <NUM>, and <NUM> for row <NUM>). For network node <NUM>, the aggregated telemetric data might be <NUM> (the average of <NUM> for row <NUM>, <NUM> for row <NUM>, <NUM> for row <NUM>, and <NUM> for row <NUM>). This aggregated telemetric data might be considered to be a hazard score, where lower scores indicate a greater hazard. An alternative method for obtaining a hazard score will be described further below.

Referring to <FIG>, the method <NUM> then includes identifying a potential problem network node in the network using the aggregation of the trace route data for at least some of the multiple real-time communications that were identified as occurring around the time of the performance problem (act <NUM>). Referring to <FIG>, this identification of problem nodes may be performed by the problem node identification component <NUM>, which may also be one of the fault analysis components <NUM>.

In one embodiment, this identification of problem nodes is also done by using the aggregated telemetric data. For instance, in the above example in which the telemetric data for all communications going through a network node is averaged in order to determine the aggregated telemetric score, the network nodes may be ranked by their aggregated telemetric score. Network node <NUM> has the most severe hazard score at <NUM>, then network node <NUM> at <NUM>, then network node <NUM> at <NUM>, then network node <NUM> at <NUM>, then network nodes <NUM> and <NUM> each at <NUM>. The identification may determine that each network node having aggregated telemetric data that meets a threshold is determined to be a potential problem node. For instance, each network node having an aggregated telemetric data at or below <NUM> might be determined to be a potential problem network node. Thus, the network nodes <NUM> and <NUM> are potentially contributing to the performance problem of the network, with network node <NUM> being the more likely candidate for contributing to adverse network performance.

Returning to <FIG>, once one or more problem network nodes are identified (act <NUM>), the problem network nodes may be communicated to a computing system associated with an entity that is affected by the detected performance problem (act <NUM>). For instance, this reporting may be performed by the reporting component <NUM> of <FIG>. As an example, the reporting component <NUM> may report the problem network node <NUM> to computing system <NUM> and/or <NUM> since they are endpoints in the communication represented by row <NUM> (which includes intermediary <NUM>) for which there was a telemetric data of a mere <NUM> out of <NUM>. Likewise, the reporting component may report the problem node <NUM> to computing system <NUM> and <NUM> since they are endpoints in the communications represented by row <NUM> (which includes intermediary <NUM>) for which there was telemetric data of a very low <NUM> out of <NUM>. If there are multiple potentially problem network nodes, they might be displayed in order of ranking.

In the above example, problematic network nodes were identified using aggregated telemetric data for each network node. However, problematic network nodes might also be identified based on the observed performance problem (detected in act <NUM>). For instance, suppose that the observed performance problem was that calls were being dropped. Then, the telemetric data related to that specific performance problem may be evaluated for each network node.

For instance, the fraction of calls passing through a network node may be calculated as the number of calls passing through the network node divided by the total number of calls. In the example above, there are four total calls in the input communication set (corresponding to rows <NUM> through <NUM>). For network node <NUM>, this fraction would be <NUM> since there are two of those communications that pass through the network node (corresponding to rows <NUM> and <NUM>). Then, a performance specific fraction is calculated. As an example, a "call drop contribution per graph node" might be calculated as the number of dropped calls passing through the network node <NUM> (let's say two in our example) divided by the total number of dropped calls (let's say three in our example). Thus, in this example, the call drop contribution for network node <NUM> would be <NUM> (or <NUM>/<NUM>). A call drop hazard score for network node <NUM> may be calculated as the call drop contribution (<NUM>/<NUM>) divided by the fraction of calls passing through that node (<NUM>/<NUM>). This would be result in a call drop hazard score of <NUM>/<NUM>. Call drop hazard scores may similarly be determined for each of the other intermediaries <NUM> through <NUM>. Thus, network nodes might be evaluated based on aggregated telemetry data related to the specific detected performance problem
In one embodiment, network nodes are mapped to a physical location or approximate physical location. For instance, a particular set of internet protocol addresses may be mapped to a particular building, floor, or office in a business enterprise. Referring to <FIG>, that mapping <NUM> might be provided into the database <NUM>. That way, the reporting component <NUM> may receive the identification of the potentially problematic network node (from the problem node identification component <NUM>), and query the database for the physical location of each problematic network node. Then, the reporting component <NUM> may also report regarding the physical location of the problem network node.

In some embodiments, the problem network node might be within a network portion (e.g., a corporate intranet) managed by an entity (e.g., a corporation) that is affected by the performance problem. In that case, reporting the problem network node to the affected entity may allow that entity to employ resources to correct or troubleshoot the problem within the network portion that it manages. On the other hand, the problem network node might be outside of the network portion that is managed by the affected entity. In that case, the fact that the problem is occurring outside of the entity affected by the performance problem may be reported to the entity. That alleviates the entity having to take time to try to resolve a problem within the network portion that the entity manages, when such would be futile. In addition, the report to that entity might report the identity of the entity or service provider in which the problem appears to be occurring. Thus, the principles described herein allow for individuals to determine whether problems are occurring within their own networks, within a real-time service (e.g., voice and teleconferencing service), or perhaps just due to general Internet problems.

Accordingly, the principles described herein use telemetric and trace route data across multiple real-time communications in order to identify potentially problematic network nodes. There may be periods of time in which there are not sufficient real-time communications to be used to reliably identify potentially problematic network nodes. In that case, the principles described herein permit for the identification that there are presently insufficient telemetric and trace route data (act <NUM>) and as a result, formulate synthetic real-time communications for these periods of time (act <NUM>). Synthetic real-time communications might involve automatic connections or calls being made between bots on different computing systems. Such bots might be designed to issue audio or video to each other so as to simulate a connection or call between human participants.

For instance, outside of normal business hours for a given region, there might be insufficient real-time communications for applications that are business-oriented for networks that are limited to that region. Accordingly, during these times, there may be some number of synthetic real-time communications that may be performed, in order to measure telemetric and trace route data for those communications, and so as to be able to have some minimal level of real-time communications within the data set <NUM> to be able to detect poor network performance, and identify potential culprit network nodes.

Because the principles described herein may be performed in the context of a computing system, some introductory discussion of a computing system will be described with respect to <FIG>.

As illustrated in <FIG>, in its most basic configuration, a computing system <NUM> typically includes at least one hardware processing unit <NUM> and memory <NUM>. The processing unit <NUM> may include a general-purpose processor and may also include a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or any other specialized circuit. The memory <NUM> may be physical system memory, which may be volatile, non-volatile, or some combination of the two. The term "memory" may also be used herein to refer to non-volatile mass storage such as physical storage media. If the computing system is distributed, the processing, memory and/or storage capability may be distributed as well.

The computing system <NUM> also has thereon multiple structures often referred to as an "executable component". For instance, the memory <NUM> of the computing system <NUM> is illustrated as including executable component <NUM>. The term "executable component" is the name for a structure that is well understood to one of ordinary skill in the art in the field of computing as being a structure that can be software, hardware, or a combination thereof. For instance, when implemented in software, one of ordinary skill in the art would understand that the structure of an executable component may include software objects, routines, methods, and so forth, that may be executed on the computing system, whether such an executable component exists in the heap of a computing system, or whether the executable component exists on computer-readable storage media.

In such a case, one of ordinary skill in the art will recognize that the structure of the executable component exists on a computer-readable medium such that, when interpreted by one or more processors of a computing system (e.g., by a processor thread), the computing system is caused to perform a function. Such structure may be computer readable directly by the processors (as is the case if the executable component were binary). Alternatively, the structure may be structured to be interpretable and/or compiled (whether in a single stage or in multiple stages) so as to generate such binary that is directly interpretable by the processors. Such an understanding of example structures of an executable component is well within the understanding of one of ordinary skill in the art of computing when using the term "executable component".

While not all computing systems require a user interface, in some embodiments, the computing system <NUM> includes a user interface system <NUM> for use in interfacing with a user. The user interface system <NUM> may include output mechanisms 712A as well as input mechanisms 712B. The principles described herein are not limited to the precise output mechanisms 712A or input mechanisms 712B as such will depend on the nature of the device. However, output mechanisms 712A might include, for instance, speakers, displays, tactile output, virtual or augmented reality, holograms and so forth. Examples of input mechanisms 712B might include, for instance, microphones, touchscreens, virtual or augmented reality, holograms, cameras, keyboards, mouse or other pointer input, sensors of any type, and so forth.

Claim 1:
A method (<NUM>) for computer-aided identification of a potential problem node in a network (<NUM>) in response to detecting that a performance problem has occurred in the network, the method comprising:
gathering (<NUM>) telemetric and trace route data (<NUM>, <NUM>) for each of a plurality of real-time communications that occurred within a network (<NUM>);
detecting (<NUM>) that a performance problem has occurred in the network;
identifying (<NUM>) multiple real-time communications of the plurality of real-time communications that occurred within a window of time containing a time of the performance problem occurred in the network;
aggregating (<NUM>) the trace route data for the multiple real-time communications that were identified as occurring within the window of time containing the time of the performance problem; and
identifying (<NUM>) a potential problem network node in the network using the aggregation of the trace route data for at least some of the multiple real-time communications that were identified as occurring within the window of time containing the time of the performance problem.