Patent Description:
Performance indicators are used to provide quantifiable, objective indications of how complex systems, such as communication networks, are serving their users. At different stages of deployment a key set of performance indicators may be used to optimize operation of a system, and adherence to targets for various performance indicators may be used to determine whether the system is acceptable to be put into production. After deployment, performance indicators may continue to be used in system optimization, such as in monitoring the ongoing operation of the deployed system to identify trends and/or determining when a repair or other remediation may be necessary. However, there are problems associated with current approaches. For example, machine learning or artificial intelligence systems may be used to identify when an event impacting a performance indicator and requiring some type of remediation has occurred, but may not provide sufficient attribution information to explain their outputs to a user. Moreover, existing approaches may fail to address individual nodes in the context of the overall cluster in which they operate (or vice versa).

Accordingly, to address one or more of these or other issues, there is a need for improved technology for analysis of multi-node systems.

<CIT> relates to active measurements of performance indicators in a packet network, and more particularly to methods for determining a plurality of link performance indicators from a plurality of path measurements. An architecture is provided for active measuring and estimating of performance indicators (e.g., key performance indicators (KPIs)) in a packet network, such as packet delay (e.g., host-to-host latency), packet loss, or packet throughput, under network measurement resource constraints, such as limited use of network bandwidth. The method comprises.

The present disclosure is broadly directed to complex system analysis in which measurement values for individual nodes are used to evaluate those nodes' contributions to the performance of a cluster in which they operate. In a first aspect, as defined by the appended independent claim <NUM>, an embodiment of a method for multi-node system analysis comprises receiving a performance indicator equation, wherein the performance indicator equation comprises a corresponding set of measurement parameters, and the performance indicator equation defines a relationship between the corresponding set of measurement parameters and a performance indicator corresponding to the performance indicator equation. In the first aspect, the method further comprises, for each node from a set of nodes, receiving, from a database, a set of measurement values for that node, wherein the set of measurement values for that node comprises a current value for each measurement parameter comprised by the performance indicator equation, and determining that node's contribution to a cluster value for the performance indicator corresponding to the performance indicator equation based on the set of measurement values for that node. In the first aspect, the method further comprises reporting performance indicator information for the set of nodes wherein the reported performance indicator information is based on relative contributions of each node from the set of nodes to the cluster value for the performance indicator corresponding to the performance indicator equation.

In further aspect, as defined by the appended independent claim <NUM>, an embodiment of an apparatus for multi-node analysis comprising one or more processors configured with instructions operable to, when executed, perform methods set forth herein are provided.

In still further aspect, as defined by the appended independent claim <NUM>, an embodiment of a computer program product for multi-node analysis comprising a non-transitory machine readable storage medium having program instructions thereon, which are configured to, when executed by one or more processors, perform methods set forth herein are provided. Further aspects are defined by the appended dependent claims.

Additional benefits and advantages of the disclosed technology will be apparent in view of the following description and accompanying figures.

Embodiments of the present disclosure are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like references indicate similar elements. It should be noted that different references to "an" or "one" embodiment in this disclosure are not necessarily to the same embodiment, and such references may mean at least one.

The accompanying drawings are incorporated into and form a part of the specification to illustrate one or more exemplary embodiments of the present disclosure. Various advantages and features of the disclosure will be understood from the following detailed description taken in connection with the appended claims and with reference to the attached drawing figures in which:.

As set forth herein, aspects of the disclosed technology may be used in multi-node analysis of complex systems such as communication networks. This may include decomposing an equation defining a performance indicator used to measure the system's performance to create an equation for quantifying the contributions of individual nodes using the measurement parameters from the performance indicator equation. These contributions may then be used to increase the efficiency and effectiveness of monitoring, remediation, and other activities related to operation of the system, such as by providing insights into the performance of a cluster in which those nodes are located and/or supporting prioritization of repairs, replacements, upgrades or other actions which may impact nodes' operation. To illustrate how aspects of this technology may be made and used in practice, this disclosure provides various examples in the context of communication network analysis. In this context, references to "nodes" should be understood as referring to pieces of equipment (e.g., a router, switch, bridge, etc.) including hardware and software that communicatively interconnects with other equipment on a network. However, the disclosed technology may also be applied in other contexts (e.g., manufacturing), and so "nodes" can be more generally understood to refer to equipment which interacts with other nodes to perform a function (e.g., devices on an assembly line whose combined operation creates a finished product, or, in the context of a communication network, providing communication services). Accordingly, the examples provided of applications of the disclosed technology in the context of communication network analysis should be understood as being illustrative, rather than limiting.

Similarly, while the following description sets forth numerous specific details with respect to one or more potential embodiments, it should be understood that embodiments of the disclosed technology may be practiced without such specific details. In other instances, well-known circuits, subsystems, components, structures and techniques have not been shown in detail in order not to obscure the understanding of the example embodiments. Accordingly, it will be appreciated by one skilled in the art that one or more embodiments of the present disclosure may be practiced without such specific components-based details. It should be further recognized that those of ordinary skill in the art, with the aid of the detailed description set forth herein and taking reference to the accompanying drawings, will be able to make and use one or more embodiments without undue experimentation.

Turning now to the figures, <FIG> provides an example of a network architecture <NUM> which is in this disclosure to illustrate how aspects of the disclosed technology may be applied in practice. In the architecture of <FIG>, there is depicted a plurality of radio access networks (RANs) <NUM>-<NUM> to <NUM>-<NUM>, which may be understood as exemplary clusters of nodes for the purpose of illustrating how the operation of such systems may be optimized. In particular, each of the RANs <NUM>-<NUM> to <NUM>-<NUM> includes a plurality of towers <NUM>-<NUM> to <NUM>-<NUM> and nodes <NUM>-<NUM> to <NUM>-<NUM>, and the disclosed technology may be used to determine the contributions of each of those nodes to the operation of their respective RAN, and to prioritize actions (e.g., repairs, upgrades, etc.) which may impact the performance of those nodes so as to achieve maximum impact for optimizing a RAN. As described in more detail herein, this may include gathering measurement values for those nodes. This data may include, for example, measurement values for a set of measurement parameters for particular periods of time and particular nodes (e.g., the number of attempts by a management element to establish a radio resource control connection, which may be abbreviated pmRrcConnEstabAtt, every <NUM> minutes). This data may then be stored in databases <NUM>-<NUM> or <NUM>-<NUM> hosted in operator systems <NUM>-<NUM> or <NUM>-<NUM> of the network operators for the illustrated RANs <NUM>-<NUM> to <NUM>-<NUM>.

As shown in <FIG>, in addition to including databases <NUM>-<NUM> and <NUM>-<NUM> storing measurement values, operator systems <NUM>-<NUM> and <NUM>-<NUM> may also include sets of servers <NUM>-<NUM> and <NUM>-<NUM>. In some cases, these sets of servers <NUM>-<NUM> or <NUM>-<NUM> may be used to process the information in the operator systems' respective databases <NUM>-<NUM> or <NUM>-<NUM>. For example, in a system implemented using the architecture of <FIG>, the server(s) <NUM>-<NUM> of an operator system <NUM>-<NUM> associated with a first RAN <NUM>-<NUM> may use measurement values for the nodes <NUM>-<NUM> to <NUM>-<NUM> of that RAN <NUM>-<NUM> to calculate performance indicators such as the RAN's availability or accessibility. The operator systems' server(s) <NUM>-<NUM> and <NUM>-<NUM> may also interact with an external server <NUM> to provide additional information which may be used by the network operators to optimize their RANs. For example, an external server <NUM> may process the equation a network operator uses to calculate a performance indicator to determine how to calculate the contributions of each node from that operator's RAN(s). Those contributions may then be used to prioritize and implement remediation activities on the relevant nodes, either by the network operators themselves, by the operator of the external server <NUM>, or both. <FIG> and the associated text provide additional detail on, as well as concrete examples of, how these types of activities may be performed.

Turning now to <FIG>, that figure depicts a high level method <NUM> for analyzing the operation of a node cluster with respect to a performance indicator based on the contributions of individual nodes to the cluster as a whole. As shown in block <NUM>, this analysis may begin with receiving the equation used to define the performance indicator in question. This may be done in a variety of ways. To illustrate, consider an implementation following the architecture <NUM> of <FIG> in which a first operator system <NUM>-<NUM> was configured to interact with a server <NUM> to determine contributions of individual nodes in the operator system's RAN <NUM>-<NUM>. In this type of scenario, the server <NUM> may expose an application programming interface (API) that could allow the operator system <NUM>-<NUM> (e.g., via its set of servers <NUM>-<NUM>) to submit the equation used to define the performance indicator. In such a case, the server <NUM> would receive the equation as shown in block <NUM> when the operator system <NUM>-<NUM> passed the equation to the server <NUM> via the API. However, this is not the exclusive way to implement receiving performance indicator equation (block <NUM>), and other approaches may also be used. For example, in some cases, rather than providing a single API, a server <NUM> may provide a set of APIs, and an operator system <NUM>-<NUM> may pass different performance indicator equations to different APIs (e.g., based on a performance indicator classification such as described below in the context of <FIG>). As another alternative, there may be a setup process in which the performance equations used by the operator system <NUM>-<NUM> would be provided to a server <NUM>, in which case the server <NUM> would receive the performance indicator equation (block <NUM>) during the setup process, and the operator system <NUM>-<NUM> would not necessarily provide it as part of a subsequent request. Further variations (e.g., combinations in which a server <NUM> may receive an equation (block <NUM>) through a setup process, but also may expose an API which could receive an equation) are also possible, will be immediately apparent to, and could be implemented without undue experimentation by, one of ordinary skill in the art. Accordingly, the above examples of how receiving an equation (block <NUM>) may be achieved should be understood as being illustrative only, and should not be treated as limiting.

Continuing with the discussion of <FIG>, after receiving a performance indicator equation in block <NUM>, the method may continue with using that performance indicator equation to determine a contribution equation in block <NUM>. To illustrate how this type of determination may be performed, consider <FIG>, which depicts a method <NUM> which begins with classifying (block <NUM>) the performance indicator equation. This classification (block <NUM>) may allow an individual node's contribution to overall cluster performance to be calculated differently depending on how the performance indicator equation used its underlying measurement parameters in defining the performance indicator in question. For example, if the performance indicator was a rate defined by an equation of the form "PI = successes/attempts", then the contributions of the individual nodes may be reflected in their number of events leading to deviation from optimal performance (e.g., failures), which could be obtained using the contribution equation "contribution = attempts - successes". However, if the performance indicator was a weighted average defined by an equation of the form "PI = (weight<NUM> * parameter<NUM> + weight<NUM> * parameter<NUM> +. + weightn * parametern)/n" the determining a contribution equation for a number of failures might not be appropriate, and, instead, the contribution equation may be simply defined as the sum of the performance indicator equation's raw parameters - i.e., "F = parameter<NUM> + parameter<NUM> +. + parametern".

In practice, to perform a classification such as shown in block <NUM> of <FIG>, a module may be implemented to place an input performance indicator equation into one of a set of predefined classes. An example of a method which may be performed by such a module is provide in <FIG>, which illustrates how a performance indicator equation could be classified into one of a predefined set of five classes. In that method, the module would initially receive (block <NUM>) input in the form of a performance indicator equation. It would then determine if that performance indicator equation satisfied a condition defining one of its predefined classes (blocks <NUM>-<NUM>). Examples of these types of conditions, as well as exemplary performance indicator equations corresponding to those conditions, are provided below in table <NUM>.

Once it had been determined that an input performance indicator equation matched a condition corresponding to one of the predefined classes, a module performing the method of <FIG> could identify the class for the equation based on the condition that was satisfied (block <NUM>). For example, if the equation was determined to match the condition for class <NUM>, then it could be identified as being in class <NUM>. Alternatively, if the equation did not match any of the conditions for one of the module's predefined classes then, in block <NUM>, the module may throw an exception, such as may subsequently be used by the individual(s) responsible for the module to add additional classes or modify the conditions for the existing classes. In this way, a method such as shown in <FIG> may both classify equations which match one of a set of predefined classes, and may also provide a mechanism (i.e., throwing exceptions) which may be used to expand or modify the set of predefined classes if and as needed.

Continuing with the discussion of <FIG>, after a performance indicator equation had been classified in block <NUM> (e.g., using a module configured to perform a method such as described in the context of <FIG>), that classification could be used to determine a contribution equation in block <NUM>. An example of a method <NUM> which may be used to determine a contribution equation based on the classes set forth in table <NUM> is provided in <FIG>. In the method of <FIG>, initially, a determination <NUM> would be made of whether the input performance indicator equation included averaging or division - i.e., whether it was classified in class <NUM> from table <NUM>. If the performance indicator equation did not include averaging or division - i.e., it was classified in class <NUM> - then the contribution equation would be set in block <NUM> as the same as the performance indicator equation. Alternatively, if the performance indicator equation was not classified in class <NUM>, then it may be reduced to its raw parameters and their relationships in by removing any transformations or constant values in block <NUM>. For example, if the performance indicator equation was "PI = log10(C<NUM>-<NUM> * (W<NUM>-<NUM> * M<NUM>-<NUM> + W<NUM>-<NUM> * M<NUM>-<NUM> +. + W<NUM>-n * M<NUM>-n)/n)," block <NUM> may be implemented by removing the transformation (i.e., the logarithmic operation) and the constant values (i.e., n, C<NUM>-<NUM> and W<NUM>-<NUM> to W<NUM>-n). A further determination <NUM> may then be performed to determine if the performance indicator equation still had both a numerator and denominator - i.e., that it was included in class <NUM> or class <NUM> from table <NUM>. If the performance indicator equation did not include both a numerator and a denominator after being reduced to raw measurement parameters - i.e., if it was in either class <NUM> or class <NUM> - then the contribution equation may be set as the sum of the measurement parameters in block <NUM>. Alternatively, if the performance indicator equation included both a numerator and denominator after being reduced to its raw measurement parameters - i.e., if it was in either class <NUM> or class <NUM> - then the performance indicator equation may be decomposed in block <NUM> by parsing it into an abstract syntax tree (AST). An example of this type of transformation is provided in <FIG>, which illustrates how an input performance indicator equation <NUM> from class <NUM> of table <NUM> could be represented as an AST <NUM>. Such an AST may be traversed in block <NUM> to identify the numerators and denominators from the performance indicator equation. The contribution equation could then be defined by setting it as equal to the sum of the denominators minus the numerators in block <NUM>.

To further illustrate how contribution equations may be determined based on classifications as shown in table <NUM>, table <NUM>, below, illustrates how contribution equations could be determined on a class by class basis, and also provides an example of a performance indicator equation and a corresponding contribution equation for each class.

Other approaches will be immediately apparent to, and could be implemented without undue experimentation by, one of ordinary skill in the art in light of this disclosure, and so the exemplary approaches described above for determining a contribution equation should be understood as being illustrative only, and should not be treated as limiting on the protection provided by or any related document.

Returning to the overall method of <FIG>, after determining the contribution equation in block <NUM>, the method continues in block <NUM> with receiving measurement values for the measurement parameters used in the contribution equation. As with the other acts illustrated in <FIG>, this may be achieved in a variety of ways. As one example, consider the scenario in which an operator system <NUM>-<NUM> passes an equation for a performance indicator to an API exposed by a server <NUM>. In this type of scenario, in addition to passing in the equation, the operator system <NUM>-<NUM> may also use the API to submit measurement values for a time period during which a concerning trend was observed in the performance indicator. Thus, one way of receiving measurement values (block <NUM>) may be for a server <NUM> to receive them through an API. As another example, consider a scenario in which communications between an operator system <NUM>-<NUM> and a server <NUM> are facilitated through a setup process. In this type of scenario, the setup process may include software being installed on the operator system's set of servers <NUM>-<NUM> which would allow the server <NUM> to run queries against the operator system's database <NUM>-<NUM>. When this approach is used, the server <NUM> may receive measurement values (block <NUM>) by running a query against the database <NUM>-<NUM> requesting values for the relevant measurement parameters. Further variations (e.g., an implementation in which an operator system would submit an API request along with a link to a temporary location from which measurement values could be downloaded, passing measurement values to software running on an operator system, etc.) are also possible, will be immediately apparent to, and could be implemented without undue experimentation by, one of ordinary skill in the art. Accordingly, the above examples of how receiving measurement values (block <NUM>) may be achieved should be understood as being illustrative only, and should not be treated as limiting.

Once the measurement values had been received (block <NUM>), those measurement values may be used to determine contributions of individual nodes in block <NUM>. To illustrate this contribution determination, consider a case where a system such as a first radio access network <NUM>-<NUM> was found to have a performance indicator with a success rate of <NUM>%, while two of that system's nodes were each found to have a success rate for that performance indicator of <NUM>%. If there was a desire to improve the value of that performance indicator (e.g., because a value of <NUM>% was within a preconfigured distance of a threshold used to measure service failures), simply knowing that two nodes had values of <NUM>% may not be sufficient to appropriately allocate resources for improving the performance of the network overall. For example, if a first node <NUM>-<NUM> had <NUM> failures and <NUM> successes, while a second node <NUM>-<NUM> had <NUM> failures and <NUM> successes, both nodes would have a success rate of <NUM>%. However, the second node <NUM>-<NUM> would clearly make a greater contribution to the overall success rate for the network, and so, all else being equal, a remediation focusing on the second node <NUM>-<NUM> would be expected to have a greater impact than a remediation focusing on the first node <NUM>-<NUM>. Accordingly, to account for this, a method such as shown in <FIG> may assign each of the nodes a weight value corresponding to the impact of that node on the overall cluster's performance. For example, if the total number of failures for all of the nodes in the RAN from the above illustration was <NUM>,<NUM>, then the first node may be assigned a weight of <NUM>% (i.e., <NUM> failures for the first node divided by the total number of failures for all nodes in the RAN), while the second node may be assigned a weight of <NUM>% (i.e., <NUM>,<NUM> failures divided by the total number of failures for all nodes in the RAN).

In the method of <FIG>, once the contributions of individual nodes had been determined (block <NUM>), those contributions may be used in block <NUM> to report information on the performance indicator in question. To illustrate how this may take place, consider <FIG>, which illustrates a method <NUM> in which performance indicator information such as could be reported in block <NUM> is generated based on analysis of the variability of weights within a cluster's nodes. Initially, in block <NUM>, multiple types of contribution statistics are determined, including the maximum weight of any of the cluster's nodes (<NUM>-<NUM>), the minimum weight of any of the cluster's nodes (block <NUM>-<NUM>), the mean weight of any of the cluster's nodes (block <NUM>-<NUM>), and the median weight of any of the cluster's nodes (block <NUM>-<NUM>). Other types of statistics, such as the weights at the <NUM>th and <NUM>th percentile, and/or the standard deviation of the weights may also (or alternatively) be calculated as part of the contribution statistics determination (block <NUM>). These statistics may then be used to generate the performance indicator information in block <NUM>, such as by generating a report that could be presented to a user. This may include generating a report with both the statistics as well as the number of nodes whose performance those statistics represent. Such a report may also include, or statistics such as calculated in the method <NUM> of <FIG> may be used to generate, insights into the operation of the cluster. For example, if there is a large spread between minimum and maximum weight values, this may be treated as indicating that a drop in a performance indicator was the result of a localized failure, while if the minimum and maximum values are closer it may indicate that the cluster is experiencing a more widely distributed failure. Similarly, both the maximum and minimum values may be treated as indicating nodes that may require additional action. For example, the node with the maximum weight may be treated as a node which is currently experiencing a failure and in need of remediation, while the node with the minimum weight may be identified as a node which, while not needing remediation, may benefit from investigation (e.g., to determine if its low weight was the result of proper operation, or instead reflected a failure in data collection or node utilization).

Once the performance indicator information had been generated (block <NUM>), it may then be reported as shown in block <NUM>. This may be done by the server <NUM> transmitting the information in the form of a report to the operator system <NUM>-<NUM>, where it could be displayed to operator personnel who could use it for purposes such as prioritizing remediations (block <NUM>) among a cluster's nodes. In some cases, a report may also be provided with additional information, or with functionality allowing a user to access additional information, that would make the report more useful. For example, in addition to providing descriptive statistics such as described above, a report may also provide identifications of nodes which are most relevant to such statistics (e.g., the nodes with the maximum and minimum contributions to the performance indicator in question), or a link to an additional interface where such nodes could be identified. As another example, in some cases there may be location or other metadata available regarding the nodes in question which could be used to provide visualizations for a user, such as a map with locations of significant nodes (e.g., nodes whose contributions were greater than the <NUM>th percentile). Performance indicator information may also be reported (block <NUM>) in a form which provides data for more than a single performance indicator. For example, consider a scenario in which ten performance indicators were used to quantify a cluster's performance. In this type of scenario, performance indicator information may be reported (block <NUM>) by providing a table with descriptive statistics (e.g., mean, maximum and minimum contributions) for each of the performance indicators, thereby providing a tool for a user to obtain a more holistic understanding of the cluster. Other approaches to reporting performance indicator information (block <NUM>) will be immediately apparent to, and could be implemented without undue experimentation by, one of ordinary skill in the art in light of this disclosure. Accordingly, the above examples provided for how performance indicator information may be reported, like the discussion of the other acts illustrated in <FIG> and <FIG>, should be understood as being illustrative only, and should not be treated as limiting.

Additional benefits and/or optimizations may also be included in some implementations. For example, consider <FIG>, which illustrates a method <NUM> which may be integrated into performance of a method such as shown in <FIG> to focus the analysis of <FIG> on those nodes which are most likely to be significant to a cluster's operation. To illustrate how this type of focusing may be useful, consider a case in which the disclosed technology was deployed to support the operation of one or more radio access networks <NUM>-<NUM> to <NUM>-<NUM> as shown in <FIG>. In such a case, the nodes <NUM>-<NUM> to <NUM>-<NUM> may each be management elements, and the measurement values may be values captured for each management element during its real time operation. In this type of scenario, the measurement values for the management elements of even one radio access network could be a massive amount of data. Accordingly, a method <NUM> such as shown in <FIG> may help reduce the amount of data to be processed, which, in turn, may allow for increases in the sophistication and responsiveness of the processing which is actually performed.

In the data focusing method of <FIG>, initially a set of nodes which had undergone a change would be identified in block <NUM>. This may be done, for example, by considering all of the nodes in a cluster (e.g., all of the management elements in a radio access network), identifying the most recent software update that had been deployed to any of those nodes, and then identifying which of those nodes had been subject to the update. This information may then be used in block <NUM> to define the set of nodes for which data would be retrieved and whose individual contributions would be evaluated. In testing this was found to reduce the number of management elements for which data was retrieved when applying the disclosed technology to a radio access network by over <NUM>%, thereby significantly reducing the difficulty of both retrieving and subsequently processing that data. Variations on this type of approach may also be utilized in some situations. For example, while the above description mentioned identifying changed nodes (block <NUM>) based on software changes, it is possible that other types of changes may also, or alternatively, be considered, such as hardware changes (e.g., replacement of a management element or a component thereof with a new physical device or component) or network changes (e.g., adding a new node, or changing how nodes are connected to each other within a RAN).

To illustrate how a data focusing method <NUM> such as shown in <FIG> may be integrated into analysis such as shown in <FIG>, consider a case where a network operator used an API exposed by a server <NUM> to initiate performance of a method such as shown in <FIG> in response to an observed negative trend. In such a case, the network operator may, in addition to providing the relevant performance indicator equation, provide a date indicating the beginning of the trend, and the server <NUM> may identify changed nodes (block <NUM>) based on the most recent change prior to the beginning of the trend, even if there were subsequent changes that may have happened after the trend first started. As another alternative, in some cases a network operator may itself identify the nodes to consider prior to initiating performance of a process such as shown in <FIG> and the process of <FIG> may simply be performed with the data for all of those previously identified nodes. For example, a network operator may set a policy that a process such as shown in <FIG> should be performed any time there was a change (e.g., a hardware, software or network change) to the nodes in its network, and so, rather than requesting that an external server <NUM> identify nodes, may provide the external server with a list of nodes to be considered (e.g., the nodes impacted by the change, or all nodes in the network). Other variations are also possible, will be immediately apparent to, and could be implemented without undue experimentation by, one of ordinary skill in the art in light of this disclosure. Accordingly, the above discussion of node identification and potential integrations of the methods of <FIG> and <FIG> should be understood as being illustrative only, and should not be treated as limiting.

As another example of an additional type of feature which may be included in some implementations, consider the possibility of supplementing analysis and reporting such as described previously with inter-period or trend information. As an illustration of steps which may be performed to support this type of functionality, <FIG> depicts a method <NUM> for determining a historical evaluation window which may be used to gather data for intertemporal analysis. As shown in <FIG>, a method <NUM> for determining a historical evaluation window may begin with obtaining a current evaluation window, as shown in block <NUM>. This may be done, for example, by an operator system <NUM>-<NUM> making a call to an API exposed by a server <NUM> which allows the operator system to specify the current evaluation window. In such a case, a server <NUM> could obtain (block <NUM>) the current evaluation window simply by receiving it through the API, potentially in combination with other information relevant to the analysis to be performed, such as a performance indicator equation for a performance indicator where a downward trend had been observed. Alternatively, in some cases, rather than exposing an API which allowed for specification of a current evaluation window, a server <NUM> may only expose an API which allowed submission of some other information (e.g., a performance indicator equation, or a set of nodes to consider), and the current evaluation window may be obtained (block <NUM>) by the server <NUM> treating a call to the API as a request for real time analysis, and so defining the current evaluation window as a predefined period (e.g., <NUM> minutes) preceding the API call. Additional variations (e.g., combinations, in which an API is exposed which treats the current evaluation window as an optional parameter and the current evaluation window is defined differently based on whether it is provided via the API) are also possible and will be immediately apparent to those of ordinary skill in the art in light of this disclosure, and so the above description of examples for how a current evaluation window may be obtained (block <NUM>) should be understood as being illustrative only, and should not be treated as limiting.

In the method <NUM> of <FIG>, after the current evaluation window is obtained in block <NUM>, a determination is made in block <NUM> of whether the current evaluation window begins and ends on a single day. If it does, then the method <NUM> continues in block <NUM> by setting the historical evaluation as beginning and ending at the same time of one or more instances of the same day over a historical period. For example, if the current evaluation window is from <NUM>:<NUM> pm to <NUM>:<NUM> pm on a Wednesday, and the historical period is three weeks, then the historical evaluation window may be set in block <NUM> as <NUM>:<NUM> pm to <NUM>:<NUM> pm on each of the three Wednesdays preceding the current evaluation window. Alternatively, if the current evaluation window does not start and end on a single day, a second determination may be made in block <NUM> of whether the start and end times for the current evaluation window take place during a weekend (i.e., on a Saturday or Sunday). If so, then in block <NUM> the historical window is defined as weekend days with the same start and end times during the historical period. For example, if the current evaluation window starts at <NUM>:<NUM> pm on a Saturday and ends at <NUM>:<NUM> am on a Sunday, the historical window may be defined as <NUM>:<NUM> pm Saturday through <NUM>:<NUM> am Sunday for each of the three weekends preceding the current evaluation window. Finally, if the current evaluation window does not start and end on the same day and does not start and end on a weekend, then the method of <FIG> would define the historical evaluation window in block <NUM> as workdays with the same start and end as the current evaluation window. For example, if the current evaluation window starts at <NUM>:<NUM> pm on a Tuesday and ends on <NUM>:<NUM> am on a Wednesday, the historical evaluation window may be defined (block <NUM>) as <NUM>:<NUM> pm Monday to <NUM>:<NUM> am Tuesday, <NUM>:<NUM> pm Tuesday through <NUM>:<NUM> am Wednesday, <NUM>:<NUM> pm Wednesday through <NUM>:<NUM> am Thursday, and <NUM>:<NUM> pm Thursday through <NUM>:<NUM> am Friday for each of the three weeks preceding the current evaluation period.

Variations on the above-described exemplary approaches to defining a historical evaluation window are also possible, and will be immediately apparent to one or ordinary skill in light of this disclosure. For example, in some cases, when a current evaluation window does not start and end on the same day, the historical evaluation window may be defined as including partial portions of the period matching the current evaluation window, but truncated to remain on the weekend or work week as appropriate. To illustrate, consider the example above of a current evaluation window which begins on <NUM>:<NUM> pm on a Saturday and ends at <NUM>:<NUM> am on a Sunday. In some cases, in addition to including the period from <NUM>:<NUM> pm Saturday to <NUM>:<NUM> am Sunday on the preceding weekends during the historical period, the historical evaluation window may be defined (block <NUM>) as also including the period from <NUM>:<NUM> am to <NUM>:<NUM> am Saturday (i.e., the same four hour window, except ending instead starting on Saturday and having the period from <NUM>:<NUM> pm to <NUM>:<NUM> pm Friday removed) and the period from <NUM>:<NUM> pm to <NUM>:<NUM> pm Sunday (i.e., the same four hour window, except starting instead ending on Sunday and having the period from <NUM>:<NUM> am to <NUM>:<NUM> am Monday removed) for each of the preceding weekends during the historical period. Similarly, in some cases, when a current evaluation window is from <NUM>:<NUM> pm Tuesday to <NUM>:<NUM> am Wednesday, the historical evaluation window may be defined in block <NUM> as including the period from <NUM>:<NUM> am to <NUM>:<NUM> am Monday and from <NUM>:<NUM> pm to <NUM>:<NUM> pm Friday for each preceding Monday and Friday in the historical period. Other variations are also possible, such as having a different historical period than the three week period from the examples above, or identifying current evaluation windows which start on a weekend and end on a work day (or vice versa) and defining the historical evaluation window for those periods as beginning on the same day and time and ending on the same day and time for each preceding week during the historical period. Accordingly, the particular examples given above should be understood as being illustrative only, and should not be treated as implying limitations on the protection provided by this document or any other document claiming the benefit of this disclosure.

However, they are determined, the definition of, and retrieval of measurement values for, a historical evaluation window may allow for intertemporal analysis to be provided as part of reporting (block <NUM>) performance indicator information. To illustrate this type of intertemporal analysis, consider <FIG>, which depicts a method <NUM> which may be used to provide insights based on intertemporal correlations between node contributions. In such a method <NUM>, after current and historical contributions for each node to a plurality of performance indicators had been determined (e.g., using approaches described previously in the context of blocks <NUM> to <NUM>), correlations between the contributions of each node for each performance indicator in the historical and current evaluation windows could be calculated in block <NUM>. These correlations could then be compared in block <NUM> to provide a measure of system stability for each of the indicators across historical period to the present. To illustrate, consider a case where the contributions of individual nodes to a first performance indicator were very similar across time periods while the contributions of individual nodes to a second performance indicator changed significantly across time periods. In this type of scenario, the contributions of the nodes to the first performance indicator could be expected to have a relatively higher correlation value than those for the second performance indicator (e.g., <NUM>% versus <NUM>%). Accordingly, comparing those correlation values could reveal that the cluster was less stable with respect to the second performance indicator than the first performance indicator. This, in turn, could trigger further actions, such as root cause analysis for whether the changes with respect to the second performance indicator reflected an underlying issue requiring remediation. For example, if there had been a remedial action on the second performance indicator between the current and historical evaluation windows, and the second performance indicator had improved between the current and historical windows, then the instability may be treated as an innocuous result of the remedial action on previously lower performing nodes. Alternatively, if the second performance indicator had worsened between evaluation windows, then a further investigation could be initiated to identify a cause and potential remediations (e.g., were poorly performing nodes for the second performance indicator geographically localized such that they may have been damaged by a weather event and require physical repair or replacement). This type of information may then be reported, such as using approaches described previously in the context of block <NUM> from <FIG>.

It should be understood that, while the above discussion of <FIG> described how the disclosed technology may be utilized for intertemporal analysis, that description is intended to be illustrative only, as the disclosed technology may be applied to other types of intertemporal analysis as well. For example, in some cases, rather than correlating contributions across time periods, intertemporal analysis may comprise providing a list of nodes in a cluster and indicating whether their contributions to a performance indicator had increased, decreased or stayed the same from one period to the next. Similarly, in some cases intertemporal analysis such as described in the context of <FIG> may be combined with other types of statistical analysis such as described in the context of <FIG>. For example, in some cases performance indicator information generated (block <NUM>) in a method as shown in <FIG> may include performance indicator information (e.g., maximum and minimum contributions, etc.) for both a current and historical evaluation window rather than only including current information. As another example, while <FIG> illustrated a method <NUM> in which changes are used to identify nodes that would be analyzed, in some implementations the analysis of nodes may be used to identify the significance of changes. To illustrate, consider a case where a cluster had undergone a series of changes (e.g., a software update, deployment of new hardware, and a modification of connections between nodes) over a historical time period (e.g., <NUM> weeks). In such a case, an intertemporal analysis may be performed in which the contributions of the cluster's nodes were tracked over the historical period, and reported based on the periods following each change. In this way, changes in the performance of the cluster could be directly associated with each change, and potential remediations (e.g., reversing a software update for certain nodes if it appeared that that update actually damaged those nodes' performance) could be easily identified and understood by a user. Other variations, combinations and modifications are also possible, will be immediately apparent to, and could be implemented without undue experimentation by, one of ordinary skill in the art in light of this disclosure. Accordingly, the exemplary variations described above, like the description of intertemporal analysis provided in the context of <FIG>, should be understood as being illustrative only, and should not be treated as limiting.

Variations are also possible on the order or instrumentalities used for performing acts such as described herein. For instance, the description of <FIG> included various examples of how an external server could receive (block <NUM>) a performance indicator equation. However, it should be understood that in a method such as shown in <FIG> the performance indicator equation may not need to be received by (or exclusively by) a server external to an operator system. For example, in some cases a set of servers <NUM>-<NUM> at an operator system <NUM>-<NUM> may be configured with software which would process a performance indicator equation, such as by classifying it in a case where an external server provided different API's for different classes of performance indicator equation. In such a case, the software running on the servers <NUM>-<NUM> of the operator system <NUM>-<NUM> may be treated as receiving (block <NUM>) the performance indicator equation when it is provided for classification. As another example, consider the sequence in which receiving measurement values (block <NUM>) may be completed in a method performed based on this disclosure. In <FIG>, receiving measurement values (block <NUM>) is illustrated as following determining a contribution equation (block <NUM>). However, it is entirely possible that the measurement values may be received (block <NUM>) either prior to, or simultaneously with, determining the contribution equation (block <NUM>), such as in an implementation where server <NUM> queries a database for measurement values of the parameters in a performance indicator equation immediately when a request is received. In such a case, the determination of a contribution equation (block <NUM>) may be performed in parallel while the measurement values were being retrieved, thereby potentially increasing the overall throughput of the system. Accordingly, the above description of how various entities may perform various actions, or of a sequence in which various actions may be performed, should be understood as being illustrative only, and should not be treated as requiring any disclosed action to be performed in any particular sequence or by any particular entity.

Aspects of the disclosed technology may also be applied in a variety of use cases. For example, the disclosed technology may be used to target remediations in the event a performance anomaly is detected. However, it may also be used for performance monitoring even in the absence of an anomaly, such as by providing a dashboard (e.g., populated with performance indicator information such as may be reported in block <NUM> of <FIG>, intertemporal information such as described in the context of <FIG>, etc.) through which an operator could have continuous visibility into network operation. Similarly, the disclosed technology may be used to identify a remediation (e.g., identifying a node requiring an update or repair based on it having a disproportionate contribution to an observed performance indicator deviation). However, it may also be used to provide explanations for remediations suggested by another system. For example, in a case where the servers <NUM>-<NUM> on an operator system <NUM>-<NUM> host an artificial intelligence (AI) process which identifies and suggests an action to remediate an anomaly, those servers may interact with an external server <NUM> utilize the disclosed technology to provide an explanation for the recommendation of the AI process. In such a case, the set of servers <NUM>-<NUM> may provide the measurement data, a list of nodes in the relevant cluster, the formula(s) for the performance indicator(s) where the AI process identified an anomaly, and the output of the AI process itself, and the external server may analyze the contributions of the various nodes to identify factors which separate the nodes impacted by the AI process' suggested remediation from the others (e.g., those nodes have the highest weights of any nodes in the cluster, those nodes have weights which are trending upward, etc.). In this way, the disclosed technology may be used to essentially transform any artificial intelligence system into an explainable artificial intelligence system without requiring the underlying artificial intelligence system to be modified or replaced by its operator. Other applications of the disclosed technology (e.g., using system stability as measured by intertemporal correlations of the contributions of nodes to the performance of a cluster as a trigger for validating the behavior of newly installed infrastructure before putting it into productions) are also possible and will be immediately apparent to those of ordinary skill in the art. Accordingly, the exemplary uses and applications provided herein should be understood as being illustrative only, and should not be treated as limiting.

Based on the foregoing detailed description, it should be appreciated that embodiments of the present disclosure provide methods, systems and computer program products for multi-node analysis. Such multi-node analysis may be applied to detect performance drops in given clusters, narrowing down clusters based on changes such as software changes, hardware changes, or network changes. This may support analysis of current and historical events. Additionally, the identification of changes such as described may also be used for license and asset management, such as providing an inventory of nodes (e.g., management elements) and software installed thereon. Multi-node analysis as described herein may also support localizing and discovering area(s) of performance degradation in a given cluster, characterizing if an issue is local or spread across the enter cluster. In turn, this may decrease time and effort needed to reconstruct and quickly detect issues. Multi-node analysis as described herein can also allow targeted and preventive fixes when performance is degraded. For example, a node with high failure contributions in the cluster may be checked as a priority. Further, a priority fix approach is provided by the disclosed technology, as it provides evidence for and insight into network performance. This, in turn, provides a better experience for users of a network. The disclosed multi-node analysis technology may also provide a new level of automation and adaptiveness, and may provide improved efficiency in evaluating periodic or stochastic network issues with deep inspection of node level contributions to performance indicator metrics. The disclosed technology may also be fully scalable, being applied to the core network, transport network, interconnect network, and/or on multiple network layers rather than simply being applied to RANs. Similarly, the disclosed technology could be applied in a cloud-based deployment to any system in which performance indicator metrics are used for monitoring and anomaly detection. Multi-node analysis as described herein may also be used as a baseline for explainable artificial intelligence, providing attributive or causal explanations for outputs of other artificial intelligence systems. The disclosed technology may also be used to evaluate slicing on tenants or on deployed <NUM> networks. This may allow a service or application provider to monitor performance of slices over various wireless networks. Additionally, components of the disclosed multi-node analysis technology may be deployed as slaves in a fully automated implementation to provide preventive and priority fixes and avoid service disruption.

In the above-description of various embodiments of the present disclosure, it is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting on the scope of protection provided by this or any related document. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs as shown by a general purpose dictionary.

At least some example embodiments are described herein with reference to block diagrams and/or flowchart illustrations of computer-implemented methods, apparatus (systems and/or devices) and/or computer software. Such computer program instructions may be provided to a processor circuit of a general purpose computer circuit, special purpose computer circuit, and/or other programmable data processing circuit to produce a machine, so that the instructions, which execute via the processor of the computer and/or other programmable data processing apparatus, transform and control transistors, values stored in memory locations, and other hardware components within such circuitry to implement the functions/acts specified in the block diagrams and/or flowchart block or blocks, and thereby create means (functionality) and/or structure for implementing the functions/acts specified in the block diagrams and/or flowchart block(s). Additionally, the computer program instructions may also be stored in a tangible computer-readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instructions which implement the functions/acts specified in the block diagrams and/or flowchart block or blocks.

As alluded to previously, tangible, non-transitory computer-readable medium may include an electronic, magnetic, optical, electromagnetic, or semiconductor data storage system, apparatus, or device. More specific examples of the computer-readable medium would include the following: a portable computer diskette, a random access memory (RAM) circuit, a read-only memory (ROM) circuit, an erasable programmable read-only memory (EPROM or Flash memory) circuit, a portable compact disc read-only memory (CD-ROM), and a portable digital video disc read-only memory (DVD/Blu-ray). The computer program instructions may also be loaded onto or otherwise downloaded to a computer and/or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer and/or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions/acts specified in the block diagrams and/or flowchart block or blocks. Accordingly, embodiments of the present invention may be embodied in hardware and/or in software (including firmware, resident software micro-code, etc.) that runs on a processor such as a digital signal processor, which may collectively be referred to as "circuitry," "a module" or variants thereof.

Claim 1:
A method for multi-node analysis, the method comprising:
- receiving (<NUM>, <NUM>) a performance indicator equation, wherein:
• the performance indicator equation comprises a corresponding set of measurement parameters; and
• the performance indicator equation defines a relationship between the corresponding set of measurement parameters and a performance indicator corresponding to the performance indicator equation;
- for each node from a set of nodes (<NUM>-<NUM>, ..., <NUM>-<NUM>):
• receiving (<NUM>), from a database, a set of measurement values for that node, wherein the set of measurement values for that node comprises a current value for each measurement parameter comprised by the performance indicator equation; and
• determining (<NUM>, <NUM>) that node's contribution to a cluster value for the performance indicator corresponding to the performance indicator equation based on the set of measurement values for that node; wherein the cluster value for the performance indicator corresponding to the performance indicator equation is a value equal to a result of calculating the performance indicator equation with measurement values for all nodes from the set of nodes (<NUM>-<NUM>, ..., <NUM>-<NUM>) for each measurement parameter comprised by the performance indicator equation,
and
- reporting (<NUM>) performance indicator information for the set of nodes (<NUM>-<NUM>, ..., <NUM>-<NUM>) wherein the reported performance indicator information is based on relative contributions of each node from the set of nodes to the cluster value for the performance indicator corresponding to the performance indicator equation.