Patent Description:
In order to determine whether service quality and network quality fulfil certain requirements, network analytics systems are utilized. Such network analytics systems are part of the network management domain and monitor and analyze service and network quality at session level in mobile communications networks. These analytics systems have been exploited in Network Operation Centers (NOC), and in Service Operation Centers (SOC) so that service quality is monitored and detected service quality issues can be troubleshot for extended periods of time.

In network analytics systems, basic network key performance indicators (KPIs) are continuously monitored, where the KPIs are based on node and network events and counters. Often, KPIs are aggregated in time, for example for node or other dimensions (for example in relation to a device type, a service provider, or other network dimensions, such as a cell, a network node, etc.). Failures of a node or the network can be indicated via the KPIs. Furthermore, elementary network events and end-to-end service quality metrics may be collected and correlated and user level end-to-end KPIs may be computed based on the available data using advanced analytics systems.

Real-time analytics may be exploited in order to provide for fast reaction in network management. This may require real-time collection and correlation of specific node and protocol events from different radio and core nodes, probing signaling interfaces and the user-plane traffic. Besides such data collection and correlation functions, which may handle a large amount of information, an analytics system may require advanced databases, a rule engine, and a big data analytics platform. The per subscriber service quality and network KPIs may be aggregated for different network dimensions.

Current telecommunication network analytics products thus face the problem of processing an enormous amount of data coming from various sources. Correlation capabilities may allow to filter and split the hundreds of various KPIs by dozens of different dimensions, not to mention their combinations. The slicing and dicing of data leads to potentially endless possibilities for creating charts for visual inspection by a user. Even if the most relevant insights are captured by certain charts, the system might be unable to show the most important chart to the most relevant person.

To overcome this problem, the concept of dashboards was introduced: a thematic or responsibility area driven collection of charts. The motivation behind the creation of dashboards was to make sure that operators or supervising personnel (or more generally: users) keep an eye on charts showing some important piece of information. However, a complex network analytics system is typically targeting a wide variety of users, with different interests and/or responsibilities. Directing the dashboards to the most relevant audience and showing it at the moment when there is a recent change of high importance is a challenging task.

If there are just a few dashboards in the system, the dashboards could be easily assigned to a few clearly defined user roles. But once, with the inclusion of diverse data sources coming with complex <NUM> (but even <NUM>) networks, the number of dashboards and the variety of users explodes. Checking all possibly relevant dashboards one by one becomes an impossible task for a user.

A conventional dashboard recommendation system is described in <NPL>. This recommendation system provides visualization recommendations building upon a bookmarking system for organizing relevant resources into topic collections by deriving user interests from performed actions. The solution also builds upon how users define dynamic dashboards. Using text analytics tools, keywords are extracted from available meta-data and high-frequency keywords are added to topics of interest of specific user profiles, which can be edited manually.

<CIT> describes a method for displaying patient data. Predefined rules are used to define various conditions, and recommendations are made based on these conditions. Users may accept or reject these recommendations, and these actions are fed back to a recommendation engine for further analysis and for generating a new recommendation.

<CIT> describes a system for generating recommendations for dashboards, changes for dashboards or even new dashboards, based on usage tracking across consumers and a correlation of usage patterns with (portions of) dashboards. When a user makes a search, the system tries to give a prediction of the type(s) of data the user may be interested in.

<NPL>, describes a tool for exploratory visual data analytics. The tool allows users to share their dashboards and learns what kind of visualizations are appropriate for a specific dataset based on a voting mechanism. The system tries to learn the importance of datasets with the help of basic descriptive statistics, like the number of distinct elements, information entropy, coefficient of variation, kurtosis, and periodicity in the data.

<CIT> describes a system for making dashboard recommendations based on tracked user input patterns and predictive analytics. The system stores information on user behavior, including keywords, the way users interact with the dashboard, which other dashboard they open before and after the current one, which searches they do in search engines, which messages they send, etc. Using such usage patterns, a prediction is made for the kind of information that the user likely tries to access. The prediction is queried against dashboards to find recommendations. Characteristics of other users that accessed the same dashboard, and recommendations provided to them, can also be incorporated.

In view of the foregoing, improvements may be required for a network analytics tool, in particular in order to improve a user's awareness of the occurrence of significant changes in the values of monitored network parameters.

According to the present disclosure, there are provided a method, a computer program product and an apparatus according to the independent claims. Further developments are set forth in the dependent claims.

Therefore, according to an aspect of the present disclosure, there is provided a method for a network analytics tool capable of analyzing a time-varying set of parameter values for each of a plurality of measurable parameters and visually outputting a plurality of visualization objects, as defined by the appended independent claim <NUM>.

In certain implementations of the method, determining a value of the first object-related metric comprises determining, for each of two or more visualization items included in the visualization object, a value of an item-related metric based on the set of parameter values of the visualization item, and determining the value of the first object-related metric based on the determined value of the item-related metric of each of the two or more visualization items included in the visualization object.

According to certain implementations, determining a value of the first object-related metric includes performing an anomaly analysis on a group of parameter values comprising the set of parameter values of at least one visualization item included in the visualization object. Such anomaly analysis is performed, at least in certain implementation, separately on the set of parameter values of each of two or more visualization items included in the visualization object.

According to certain implementations, performing an anomaly analysis includes separating the parameter values of the group into at least one normally behaving subgroup and at least one anomalously behaving subgroup.

The method comprising in certain implementations obtaining, for each of a plurality of users, a value of a user-specific second object-related metric in relation to each of two or more of the plurality of visualization objects associated with the user. A user-specific object ranking is established, for each of the plurality of users, on the basis of the value of the first object-related metric and the value of the user-specific second object-related metric of each of the two or more visualization objects associated with the user. Generation of a sensory-perceptible output indicative of the user-specific object ranking is caused.

Establishing the user-specific object ranking includes in certain implementations determining, for each of the two or more visualization objects associated with the user, an object ranking value representative of a multiplicative product of the value of the first object-related metric and the value of the user-specific second object-related metric.

According to certain implementations, establishing the user-specific visualization object ranking includes obtaining an object usage matrix specifying for each of the plurality of users a usage value in relation to each of the two or more visualization objects associated with the user. The usage value is representative of a probability of usage of the visualization object by the user.

The object usage matrix is obtained, in certain implementations, by logging, for each of the plurality of users, visualization object interactions of the user and determining the usage value based on a logged number of visualization object interactions.

Establishing the user-specific visualization object ranking includes in certain implementations a matrix decomposition process which decomposes the object usage matrix into the product of a job role matrix and a toolkit matrix. The job role matrix specifies for each of the plurality of users a job role value in relation to each of two or more job roles, wherein the job role value is representative of a probability of the user being involved in a task related to the job role. The toolkit matrix specifies for each of two or more visualization objects a toolkit value in relation to each of the two or more job roles, with the toolkit value being representative of a probability of the visualization object requiring to be inspected in order to accomplish a task related to the job role.

According to certain implementations, establishing the user-specific visualization object ranking includes determining an object ranking matrix as a product of the job role matrix, the toolkit matrix and a vector including for each of the two or more visualization objects the value of the first object-related metric.

Obtaining a value of the user-specific second object-related metric may include in certain implementations, for each of one or more of the plurality of visualization objects, assigning a default value to the user-specific second object-related metric based on the non-availability of a sufficient amount of logged visualization object interactions associated with the user. Assigning a default value to the user-specific second object-related metric may be useful, e.g., in situations where a new user with an unknown behavior or with unknown preferences regarding the usage of visualization objects accesses the network analytics tool. The assigned default value of the user-specific second object-related metric may be subject to adjustment based on time. The adjustment may be dependent on the extent of information gathered for the new user on his or her usage of visualization objects.

In certain implementations, obtaining a value of the first object-related metric includes, for each of one or more of the plurality of visualization objects, assigning a default value to the first object-related metric based on the non-availability of a sufficient set of parameter values of the measurable parameter of at least one visualization item included in the visualization object. Assigning a default value to the first object-related metric may be useful, e.g., in situations where a new parameter is added to the parameter set being measured. The assigned default value of the first object-related metric may be subject to adjustment based on time, e.g., as more and more measured values of the parameter are obtained.

Certain implementations provide that the measurable parameters are derived at least in part from at least one of a radio access network part and a core network part of a mobile telecommunications network.

According to another aspect of the present disclosure, there is provided a computer program product comprising program code portions that, when executed on at least one processor, configure the processor to perform the method disclosed herein, as defined by the appended independent claim <NUM>.

Yet another aspect of the present disclosure provides an apparatus which is operable in a network analytics tool capable of analyzing a time-varying set of parameter values for each of a plurality of measurable parameters and visually outputting a plurality of visualization objects, as defined by the appended independent claim <NUM>.

These and other aspects of the disclosure will now be further described, by way of example only, with reference to the accompanying figures, wherein like reference numerals refer to like parts, and in which:.

While, for example, some embodiments of the following description consider an analytics tool for a mobile communications network, the present disclosure is not limited in this regard. In particular, the present disclosure may also be implemented in an analytics tool for a wired telecommunications system or in analytics tools for systems of an entirely different nature.

Those skilled in the art will further appreciate that the steps, services and functions explained herein may be implemented using individual hardware circuits, using software functioning in conjunction with a programmed microprocessor or general purpose computer, using one or more Application Specific Integrated Circuits (ASICs) and/or using one or more Digital Signal Processors (DSP). It will also be appreciated that when the present disclosure is described in terms of a method, it may also be embodied in one or more processors and one or more memories coupled to the one or more processors, wherein the one or more memories store one or more computer programs that perform the steps, services and functions disclosed herein when executed by one or more processors.

In the following description of exemplary embodiments, the same reference numerals denote the same or similar components.

Certain conventional network analytics tools provide a set of dashboards without any specific dashboard recommendation tools. A dashboard is an example implementation of a visualization object in the sense of the present disclosure and is a visual presentation (e.g., in the form of one or more of a chart, a diagram, a table, a graphic, text, etc.) of one or more measurable parameters each representing a visualization item in the sense of the present disclosure. A pure list of available dashboards may be sufficient if the number of dashboards is small (not more than a few dozen), and assigning them to the responsible users is straightforward, but network complexity nowadays will exceed such boundaries.

Analytics tools using rule-based recommendations may be suited for a user base with a static interest, i.e., interests which can be grasped by a fixed set of rules. Such tools are less effective in a dynamically changing environment where both the user base and the set of active dashboards changes over time. Certain conventional recommender systems only take into account a user's areas of interest, but do not consider the relevance of the current information visible on a given dashboard (note that dashboards generally represent content which varies over time, i.e., they do not represent static content).

Certain conventional analytics systems incorporate more sophisticated usage indicators than simply the number of clicks, such as the time spent by a user on specific dashboards, search keywords used by a user, the particular manner in which users interact with a dashboard, which other dashboards they open before and after a current one, which messages they send, etc., and apply machine learning techniques on top of them. Some conventional analytics systems include elements of basic descriptive statistics of the underlying data. However, these elements of statistics still fail to capture an important aspect of dashboard recommendation: the importance, or relevance, of the information represented by the underlying data. To give a purely illustrative example, if a dashboard is created with monitors a process that usually has no significant changes, e.g., a process measuring the seismic activity in a particular area or a nationwide outage, then most of the time the dashboard will not show anything of interest. However, in the unlikely case of an earthquake or an outage, it may become the absolute top priority dashboard. Naturally, in such cases an alarm system would be activated - but it is not trivial in less extreme scenarios when occasional failures are treated as acceptable, not necessarily calling for an immediate alarm, and there are no clear and easy rules to differentiate between favorable or problematic scenarios. Recommending the dashboard for inspection by an expert is exactly the action needed in such a scenario, as the expert's eye and understanding will be able to make a well-informed decision about whether or not to take action and which action, if any. Where the provision of an alarm system is no viable option (for example, because thresholds need to be updated frequently, which is infeasible), it would be desirable to incorporate a measure of importance of the underlying data into the recommendation process (apart from any other aspects already covered) based on, for example, an anomaly analysis.

It would also be desirable to allow the introduction of new dashboards without encountering cold-start problems, i.e., new dashboards should as much as possible not start with a disadvantage compared to existing, potentially highly used dashboards.

Implementations of the present disclosure are capable of providing a user-specific dashboard recommendation by learning each user's personal interest and behavior, instead of forcing users to play any of a number of pre-defined user roles (categories). Implementations of the present disclosure provide a content importance score, i.e., they establish a measure between a dashboard's content and its relevance. Such content importance score is an example realization of a first object-related metric in the sense of the present disclosure. This score can come, for example, from a detected anomaly's expansion, an inter-cluster distance between segmented data sets, etc., and may provide a more meaningful measure than simple descriptive statistics. Implementations of the present disclosure therefore recommend dashboards based on their current content, i.e., based on the content importance score.

Certain implementations of the present disclosure combine the above approaches of learning each user's personal interest and behavior and determining a content importance score. Such implementations create a dashboard-specific score based on a user-specific ranking and the content importance score. In this way, each user can be presented with a recommendation for a set of dashboards which represent the most adequate dashboards according to the particular user's interest, additionally taking into account the most relevant contents of the current data sets. The dashboard recommendations so created may be useful, e.g., for streaming database architectures having a retention for a given timeframe.

On analytics system in accordance with certain implementations of the present disclosure supports its users in defining new dashboards themselves. These newly defined dashboards become part of the set of dashboards which can be recommended; they may receive an initial popularity boost in order to become visible for other users (and not be dominated by longer-lived, already popular dashboards). This initial boost may fade over time through adjustment of a corresponding default boost value.

Implementations of the present disclosure provide for a re-calculation of the content importance score on a regular or irregular time basis. This allows the recommendation function of the analytics system to find new patterns in the underlying data set and rank them adaptively. The user-specific recommendation can be time-adaptive, as well. This may be particularly useful in circumstances where the analytics system stores session records for only a limited time duration or stores only a limited number of the last sessions; using the user interactions of the stored sessions may ensure that the recommendation function remains adaptive to user preferences.

Embodiments of the present disclosure support users of a telecommunication network analytics system to find and follow a number of most relevant dashboards with respect to their personal interest and responsibility. According to certain embodiments, the analytics system includes a dashboard recommendation function satisfying the following features:.

In the following, an example implementation of a user-specific dashboard recommendation solution for an advanced network analytics tool will be described in more detail. The network analytics tool may comprise, e.g., Ericsson's Expert Analytics (EEA). Complex, end-to-end correlated, multi-domain network analytics systems can monitor the network in various layers, e.g., the packet core control plane & user plane via node logs and/or probing, the Radio Access Network (RAN) by processing cell trace records (CTRs), the IMS control plane for VoNR/VoLTE services, etc. Typically, hundreds of different KPIs are derived for all of these domains, and the end-to-end correlation provides a large number of dimensions for filtering and comparison of network segments, user groups, nodes among others, etc. These altogether lead to virtually endless possibilities for the generation of charts, some of which may be recorded on thematic dashboards. The dashboard recommendation system described below is operable to work with a long list of such dashboards and with a registered user base; by learning user interests and monitoring the importance of information on the dashboards, the dashboard recommendation system is capable of making recommendations to the users regarding particular dashboards which the users should check at any given time.

Reference is now made to <FIG> which shows the overall architecture of an analytics system in accordance with an example embodiment of the present disclosure. The analytics system is generally designated <NUM> and comprises as its principal components a recommendation engine <NUM>, a user interface (UI) <NUM> and a number of databases <NUM>, <NUM>, <NUM>, <NUM>, <NUM>. The recommendation engine <NUM> and the user interface <NUM> are capable of accessing the databases <NUM> to <NUM>. The databases <NUM> to <NUM> represent the lowermost logical level of the system architecture. In the illustrated system architecture, the database <NUM> holds definitions for a plurality of dashboards, the database <NUM> holds the raw data (i.e., measured parameter values) to be visualized in the dashboards, the database <NUM> stores calculated content importance scores, the database <NUM> stores logged user interactions, and the database <NUM> holds user-specific dashboard recommendations. The actual number and content of the databases <NUM> to <NUM> can vary and is not limited to what is shown in <FIG>. It is to be understood that it is within the scope of the described embodiment for the "raw data" to be the result of preceding calculations. The UI <NUM> is configured to output dashboards <NUM> and other information in a visually perceptible manner on one or more display devices (not shown in the drawings) to users <NUM> and receive user input from the users <NUM> through, e.g., a keyboard, keypad, touchscreen or other type of input device (not shown, again). Each dashboard <NUM> comprises at least one chart <NUM> or other type of visualization item suitable for presenting a KPI, or more generally: a measurable parameter, in a manner allowing users <NUM> to grasp time-varying content conveyed by the parameter. Each measurable parameter represents a different data source, which is why the terms chart, data source and parameter can be used interchangeably herein.

A set of recommended dashboards <NUM> may be visually presented to a user <NUM>, for example, in a stacked, partially overlapping manner, with a dashboard ranked highest among the set appearing at the top of the stack and a dashboard ranked lowest among the set appearing at the bottom of the stack. In such a scenario, the order in which the dashboards are arranged in the stack follows their relative ranking. Whenever the dashboard ranking changes due to, e.g., a change in content importance of one or more dashboards of the set, the order of the dashboards in the stack can be re-adjusted to reflect the changed dashboard ranking. It is conceivable that whenever the dashboard ranking changes, a visually and/or audibly perceptible alert is outputted to the user <NUM>. The dashboard ranking can alternatively be presented to the user <NUM> using representative symbols, icons, characters, numbers, etc., instead of displaying the dashboards themselves in a particular order.

One of the functionalities of the recommendation engine <NUM> is the calculation of a content importance score in relation to each chart or data source or parameter representing a source or origin of raw data. Content importance is meant to capture "interesting" events in the associated data. An event may be considered interesting if it induces a large or unpredictable change in the data. Various algorithms coming from statistics and machine learning may be used to identify such anomalous behavior. These algorithms assign a score or probability for an event to be an anomaly, which in turn can be used to measure content importance.

The recommendation engine <NUM> includes, in the illustrated example implementation according to <FIG>, a content importance calculator node <NUM>, a dashboard ranking node <NUM>, a user profiler node <NUM>, a subjective weighting node <NUM>, a user interaction collector node <NUM> and a memory fading node <NUM>. In a <NUM> network, the functionalities ensured by these nodes <NUM> to <NUM> may wholly or partially be implemented as a network function (NF).

The content importance calculator node <NUM> is responsible for performing the requisite calculations to determine a content importance score for each chart/data source/parameter. The calculated content importance scores are stored in the content importance score database <NUM>. They can be viewed as an item-specific metric, i.e., specific to one visualization item.

The dashboard ranking node <NUM> is responsible for determining an importance score (first object-specific metric) in relation to each dashboard for which a definition is held in the dashboard database <NUM>. The dashboard ranking node <NUM> can access the dashboard database <NUM> and can query for each dashboard the associated dashboard definition (which describes what kind of data is available/included in the dashboard). The dashboard ranking node <NUM> then calculates a dashboard importance score based on the content importance scores belonging to the sets of raw data upon which the dashboard is defined. The dashboard importance scores can be viewed as object-specific scores as they are specific to distinct visualization objects/dashboards. Where a dashboard <NUM> includes a single data source/parameter/chart, the dashboard importance score can be simply the content importance score of the single data source/parameter/chart. Where a dashboard <NUM> includes a plurality of different data sources/parameters/charts, the content importance scores of the data sources/parameters/charts may be suitably aggregated. One exemplary implementation is simply selecting the largest content importance score. It is to be understood that different embodiments may choose other methods of aggregation. The dashboard importance score can thus adapt to changes in the underlying data.

The user profiler node <NUM> can run parallel to the dashboard ranking node <NUM>, obtaining user-specific importance scores (second object-specific metric) which define characteristics about the interests of specific users. This information is a useful supplement to the dashboard importance scores, because even though a dashboard <NUM> may show important content, some users might still deem it unimportant based on their own interests. User profiling has also a vast academic background - some exemplary embodiments may use clustering techniques or simple descriptive statistics. User-specific importance can be calculated for either specific dashboards, or for specific datasets.

The output of the dashboard ranking node <NUM> and the user profiler node <NUM> is finally processed by the subjective weighting node <NUM>, which is configured to combine the objective importance of the dashboards with the subjective perception of users. Again, there can be several solutions - the simplest example could be a simple linear combination of the dashboard importance scores and the corresponding user interest scores. The resulting subjective dashboard importance scores (per user) are stored in the database <NUM>, which can be queried by the UI <NUM> to show the best dashboards to each specific user. Note that it is also possible to build the process for user groups instead of specific users. As used herein, the term "user" thus does not necessarily refer to a single person, but may encompass any group of individuals or "personae" for which a specific dashboard recommendation is desired to be generated.

As users interact with the UI <NUM>, their actions are logged into database <NUM> to enhance the recommendation process with implicit feedback, supported by the user interaction collector node <NUM>. It may, however, be a desirable feature for the recommendation engine <NUM> to be adaptive to changes in user interests. For this reason, the memory fading node <NUM> is introduced, which is configured to remove entries from the user interactions database <NUM> based on suitably defined principles. Exemplary embodiments may foresee to remove entries from the user interactions database <NUM> after a predefined amount of time has passed since, or to always store no more than a predefined number of entries per user, and so on.

Users may additionally have the chance to create their own dashboards, e.g., through using the UI <NUM>. In such cases, newly generated dashboards are faced with a cold start problem, which may easily cause them to be dominated by already existing, highly used dashboards. In order to overcome this problem, each new dashboard may be given, by default and upon creation, a boost value representing a boosted importance, which fades away gradually. Information on the level of fading of the initial boost is part of the information which may be stored in the dashboard database <NUM>. Various solutions are conceivable to implement a fading of the initial boost, some including exponential smoothing over time, Laplace smoothing over usage,.

<FIG> shows a flowchart illustrating an exemplary flow of information in the analytics system <NUM> of <FIG>. From a high-level perspective there are two large components - one of them is responsible for calculating the objective dashboard importance scores, and the other one is responsible for calculating the subjective user-specific importance scores.

Calculating the dashboard importance scores can be done either separately or synchronized to the user profiling, possibly on a periodic basis. It should be noted, however, that while user profiles are expected to vary slowly, dashboard importance can change rather rapidly as an anomaly emerges in the underlying data. In order to capture such events in a timely manner, dashboards importance score calculation may be carried out at a higher frequency. This may amount to setting a higher trigger frequency for the content importance calculator node <NUM> or implementing it in a streaming environment. When the node <NUM> finishes, it automatically triggers the dashboard ranking node <NUM>, which can be implemented as part of a streaming pipeline, as well.

The calculation of the subjective user-specific importance scores can be implemented as a feedback loop. The user interaction database <NUM> may be queried occasionally in a periodic manner by the memory fading node <NUM> in order to remove unnecessary entries. This can be implemented in different ways. For example, a first approach is to store all interactions that accumulate over time (possibly considering some form of retention, though), and the node <NUM> simply filters the results, forwarding it to the user profiler node <NUM>. This can be implemented as part of a streaming pipeline, as well. Another approach is an offline approach, where the node <NUM> is triggered in a periodic way and updates the database <NUM> by actually removing unnecessary entries. The resulting stream or updated database is then accessed by the user profiler node <NUM>. The user profiler node <NUM> can be implemented as part of a streaming pipeline, although it may be more efficient from resource consumption point of view to run it in a batch mode due to the slowly varying nature of user profiles.

Finally, the results of the user profiler node <NUM> and the dashboard ranking node <NUM> are consumed by the subjective weighting node <NUM> to produce the subjective user specific dashboard recommendations. The subjective weighting node <NUM> can again be implemented either as part of a streaming pipeline or in a batch processing manner. However, due to the expected volatility of the dashboard importance scores it may be desirable if the subjective weighting node <NUM> is capable of providing reports with a high frequency. Note that the nodes <NUM>, <NUM>, <NUM> can be integrated into a single node if the nature of the recommendation algorithm requires so.

The interactions of the users on the UI <NUM> are then collected and fed back to the user interaction database <NUM>. In case a new dashboard is created by a user, it is stored in the dashboard database <NUM>. If a new dashboard is created, it is given an initial importance boost, which is gradually decreased in the dashboard ranking node <NUM>.

Here we present an exemplary implementation of a dashboard recommendation system that fits the concepts presented so far. First, we note that telecommunication analytics dashboards are by nature different than movies, books, and the like - therefore, their ranking by users is also different. The users of dashboards inside organizations usually belong to one or more, but definitely a finite number of job roles, which highly affects the selection of dashboards in which they are interested. In order for employees to perform their job successfully, they mostly have to view a given set of dashboards which help their everyday tasks - they do not rate the dashboard explicitly, but instead either use a dashboard or not. This kind of user rating is referred to as implicit feedback data in the academic literature and has the advantage of being available without the need for an employee to always update his or her rating on a dashboard. This is beneficial, since otherwise the user might either skip the ranking dialog, or even worse, he or she might be frustrated and give practically random or constant explicit ratings just to get over the ranking dialog - both cases rendering the recommendation process useless.

The implicit feedback of simply using a dashboard or not can be stored in usage matrices, where the rows represent the users and the columns represent the dashboards. Each row can be normalized to sum to one, which would result in the relative frequency of usage of each dashboard for the given user. In order to incorporate memory fading, one could continuously apply a sliding-window-like approach by retaining only the last N feedbacks from a specific user, which helps keeping the recommendation system adaptive. Another possible improvement is for the system to only count dashboard views if the user spends at least a prescribed amount of time on the dashboards (otherwise it is most probably a misclick, or the dashboard did not seem to be useful to the user after all).

We can also notice that in the environment of a network analytics system the users may have no means to specify dislike against a specific dashboard - they can only specify their liking. Such ratings are called unary in the literature. The benefit of this kind of rating is that missing data can be imputed with zeros without significantly biasing the recommender system.

Content importance builds on top of a process when an embodiment of the analytics system identifies anomalies for the available dashboards. Such embodiments generate a binary variable indicating whether a given object (parameter value) has anomalous behavior or not in a population (time-varying set of parameter values). The binary variable can be used to separate the population into a subset of anomalies and a subset of normally behaving objects. After this separation, a linkage-type distance can be calculated. First, data behind the dashboards can be standardized to obtain a common scale for all of them. After that, one may identify the point of the anomalously behaving subset which is closest to the normally behaving subset. A mean may be calculated from the normally behaving set and the standard deviation of the entire population may be calculated. A vector can then be generated which contains for all dashboards a number specifying how many times the standard deviation can be gauged on the mean-closest anomaly distance. This results in a vector y which can be applied as a natural definition of ranking between dashboards. The scores are interchangeable to <NUM>,. , m ranks, and weights can be added by converting them to an exponential scale: y<NUM> = exp(-<NUM>),. , ym = exp(-m). A normalized vector which sums to one can then be generated by dividing it with cons. = y<NUM> + y<NUM> + ··· + ym. We denote this vector with c ; it contains the following elements <MAT>.

When both the implicit user feedback and the content importance is available, there are various options for obtaining the final dashboard recommendations. There is a vast literature on recommender systems, and many of them can be implemented in the architecture and dataflow presented in this disclosure. As an example, we will describe an embodiment which uses matrix factorization. There are certain advantages of this approach, one of them being that the dashboard ranking node <NUM>, the user profiler node <NUM> and the subjective weighting node <NUM> can be combined into a single node.

Let there be n users who would like to see dashboard recommendations, and m dashboards. As has been discussed above, implicit user feedback can be represented in an n × m matrix F, where the element in the uth row and dth column, fud, is the probability of user u checking out dashboard d. This probability is calculated by counting (in the sliding window) how many times user u opened dashboard d, and dividing this by the total number of times user u opened any one of the dashboards.

Note that if user u belongs to more than one job role, then the probability that user u checks dashboard d due to fulfilling a task in job role j is simply the probability that user u performs a task related to job role j multiplied by the probability that a task belonging to job role j can be carried out by checking out dashboard d. Assume that the number of job roles is fixed to k. Note that k is not necessarily the number of jobs in an organization - it is more like a set of different user behaviors that can be expected to happen. It is not necessary to specify what these behaviors are, it suffices to state how many these behaviors the algorithm should try to learn (in the assumed case, k). Let P be an n × k matrix (job role matrix) where the element in the uth row and jth column gives the probability that user u is involved in a task related to job role j. Let us also define the m × k matrix Q (toolkit matrix) where the element in the dth row and jth column gives the probability that dashboard d has to be checked in order to finish a task related to job role j. Note that: <MAT>.

The probabilities in neither P nor Q are known. In order to find these, non-negative matrix factorization with certain additional constraints can be applied. The optimization task can be formulated as <MAT> with the following constraints: <MAT> <MAT> <MAT> <MAT> <MAT>.

Here <MAT> denotes the square of the Frobenius norm of a matrix. The first two constraints state that each element of P and Q must be a probability (i.e. in the range [<NUM>,<NUM>]). The second and third constraints state that each row of P and Q must sum to <NUM>, and that the rows of their product must also sum to one (i.e. satisfy the law of total probability). The optimization task can be solved, e.g., by using variants of the gradient descent method (like sequential quadratic programming), or by alternating least squares with the constraints included.

The matrix P actually gives the user profiles (meaning how different users typically behave). The relative importance of a dashboard for a job role is stored in Q - this can be interpreted as typical toolkits for different job roles.

A reason for the matrix factorization of F is that a <NUM> in a particular position of F allows no conclusion whether the user belonging to the row position of the <NUM> did not want to use the dashboard associated with the column position of the <NUM>, or whether the user simply did not know or forgot about the dashboard (since there can be so many dashboards). By deriving P and Q one can try to find common usage patterns across users having similar (but not exactly the same) roles who should be using similar dashboards.

Let us also introduce the n × m dashboard ranking matrix R. This matrix holds information about how relevant a given dashboard is to a given user, including the content importance. The content importance is, however, dependent on the job role. Let the m × k matrix C be the matrix of content importance of a dashboard relative to specific job roles. Each column of C is obtained by taking the Hadamard (or elementwise) product of the corresponding column of Q and the vector of dashbord content importance scores, c. Finally, we obtain the ranking as: <MAT>.

The element of R in the uth row and dth column specifies how important dashboard d is to user u given the current content importances.

The above described approach is based on the assumption that users may have to admit to different user behaviors or that sets of dashboards which have to be checked against different job roles are overlapping, so that both user behavior (i.e., P) and the typical toolkit for job roles (i.e., Q) must be learned by the algorithm. In certain cases, this assumption may not hold true; for example, different job roles may only require non-overlapping sets of dashboards. In such cases, the process can be simplified by manually fixing the elements of Q. Similarly, if users are guaranteed to only behave in a single specific manner, the elements of P can also be fixed. As long as at least one of them is not fixed, one can still find the other through non-negative matrix factorization, only allowing changes in the matrix under estimation. If both P and Q are fixed, non-negative matrix factorization is not necessary; in this case, C can be constructed right away each time a recommendation is necessary.

Note that the matrix factorization does not have to be computed each time a recommendation is necessary, because it is very unlikely that the user behavior and typical toolkit change too frequently. It is acceptable and computationally more feasible to update the matrices P and Q only occasionally, although "occasionally" depends very much on how rapidly things change inside the organization. On the other hand, updating c (and therefore C) is most likely done more frequently.

It is desirable to protect the recommender system against cold-start problems. There are different kinds of cold-start issues that can arise: a new user might enter, a new dashboard might appear, or the system itself has only been deployed recently and overall there is not much information in the database(s).

When new users enter the organization, there is no data about their behavior. As such, we can initialize their row in F to a uniform prior (or a different prior if we know better) and apply Laplacian smoothing as we gather more and more information about their dashboard usage.

When a new dashboard appears, users might not notice it, therefore it is important to give it a certain boost. This can be achieved by adding a (higher) number to the actual number of views for each user in the sliding-window (even if they have not seen the dashboard) and calculate F only after this. This extra number of views granted to the dashboard can be decremented gradually as time goes by. This could also be done with the relative frequencies right away, which may be preferable in certain instances, as the scale of the number of views will then not matter. This can be relevant when some users use the dashboards a lot more than others - in this case, adding a "higher" number might not be well-defined, but altering the relative frequencies works the same way for all users.

Finally, if the amount of data is low, it is possible to assign uniform priors over both rows and columns and apply Laplacian smoothing.

<FIG> shows an exemplary cloud architecture of the analytics system <NUM> of <FIG>. The cloud architecture comprises two separable logical units: at the bottom are the different databases <NUM> to <NUM> which are integrated into a persistent data storage <NUM> of the cloud (for example MinIO). The databases <NUM> to <NUM> can be accessed by the cloud's recommendation engine. The cloud is designated <NUM>. The databases <NUM> to <NUM> can be expanded and may be defined for additional sources, but the following is provided as a basis of the analytics system <NUM>: the dashboard definitions, the raw data that is visualized in the dashboards, the content importance scores, logged user interactions, and the user specific dashboard recommendations. The raw database <NUM> can be provided from an aggregation system which may likewise run on the cloud (possibly in a different namespace).

All interaction between software instances in the cloud and the persistent data storage <NUM> communicate through a cloud API <NUM>. Content importance scoring is run to detect anomalous behavior and to assign scores or probabilities to the events. It should receive input from the raw database <NUM> and writes its output into content importance scores database <NUM>. In a next step, user interaction collector instance should start and collect usage statistics of users. This node does not have to start at the system's first run, since at a new deployment there will be no interactions to collect. The pod will write the gathered data to the user interactions database <NUM>. As part of the user interaction collector instance the memory fading instance should run attached at every run; this instance creates changes in the user interactions database <NUM> by adapting and deleting database entries.

The user profiler instance can run parallel to the dashboard ranking instance; the first instance would provide the user specific importance scores from the dashboard database <NUM> and the user interactions database <NUM>. The dashboard database <NUM> may be refreshed at every start with new dashboard definitions of the system <NUM>. The dashboard ranking instance collects data from the dashboard database <NUM> and from the content importance scores database <NUM>. Both nodes <NUM>, <NUM> pass their output to the subjective weighting instance.

At last, the subjective weighting instance creates the user-specific subjective dashboard importance scores that are written into the user-specific dashboard recommendation database <NUM>, which is utilized by the UI <NUM> engine to provide the most important and valuable visualizations to the user.

<FIG> shows a flow diagram of steps of a method in accordance with an exemplary embodiment of the present disclosure. The method can be implemented in the analytics system <NUM> of <FIG>. The method comprises a step S100 of acquiring a time-varying set of parameter values for each of a plurality of measurable parameters of a network. The parameter values may be stored in the raw database <NUM>. At least some of the parameters each represent, e.g., a key performance indicator (KPI) of a telecommunications network. At least some of the parameters may be directly measurable, e.g., in a radio access network part and/or in a core network part of the telecommunications network; alternatively or additionally, at least some of the parameters may be derived from measurable quantities which can be measured directly in the telecommunications network. Whether directly measured or derived, the term "measurable parameters" as used in the present disclosure is meant to encompass all such possibilities. The acquiring step S100 acquires for each of the measurable parameters a set of values representing a time series of the parameter. For example, each of the parameters may be sampled or calculated at regular or irregular intervals, and whenever a new value is obtained for a parameter the new value may be added to the set of parameter values stored for the respective parameter in the raw database <NUM>. In turn, the oldest parameter value(s) may be removed from time to time from the raw database in accordance with predefined criteria. In this way, the set of parameter values stored for each parameter in the raw database <NUM> can be kept at a manageable size; at the same time, it can be ensured that occurrences giving rise to significant changes in the value of a parameter can be properly reflected in the sets stored in the raw database <NUM>. The sampling or update rate of the parameters may be the same or may be different for different parameters.

Each of the measurable parameters can be individually visualized on the user interface, UI, <NUM> by way of a visualization item <NUM>. Each visualization item <NUM> represents a single parameter and visualizes, at least in certain embodiments, a temporal behavior and/or a current state of the parameter in graphical and/or numerical form. The visualization item <NUM> may be also referred to as a chart in certain embodiments. Note that the particular form of visualization of a parameter is of no heightened relevance for the present disclosure and that any suitable manner of visualizing a temporal behavior and/or a current state of the parameter on an electrically/electronically controllable display device can be used.

The method of <FIG> proceeds with a step S110 of determining, for each of one or more visualization items included in a visualization object, a value of an item-related metric based on the acquired set of parameter values associated with the visualization item. As noted further above, a dashboard <NUM> as one implementation of a visualization object within the scope of the present disclosure may serve to visualize a single parameter and may consequentially include a single visualization item or may serve to visualize a plurality of parameters and may thus comprise a corresponding plurality of visualization items. Which parameter(s) is/are comprised in a dashboard <NUM> is defined in the dashboard definitions stored in the dashboard database <NUM>.

As also noted above, one aspect of implementations of the present disclosure is the determination of an objective score indicating an importance (or relevance) of the content of a dashboard <NUM> and the use of this score in establishing a dashboard ranking. The objective score is not determined based on user interest, but instead is determined based on a statistical analysis of the stored values of the parameter(s) comprised in each dashboard <NUM>. Where a dashboard <NUM> comprises a single parameter, a score indicating an objective importance (or relevance) of a parameter may directly express an objective importance (or relevance) of the dashboard <NUM>. Where a dashboard <NUM> comprises a plurality of parameters, however, an objective importance (or relevance) of the dashboard <NUM> may be derived from individual objective importances (or relevances) of the parameters included in the dashboard <NUM>. Before determining an objective score of a dashboard <NUM>, the exemplary method of <FIG> thus determines an objective score of each parameter included in a dashboard <NUM>. To be more specific, the method determines in step S110 an objective score for each parameter for which a set of parameter values is stored in the raw database <NUM> and/or which is referenced in any of the dashboard definitions held in the dashboard database <NUM>. The related scores are parameter-specific and each represent an item-related metric. The determination of the objective parameter-specific score may be performed by the content importance calculator node <NUM>, which may, for example, perform an anomaly analysis on each set of parameter values to separate the values of the set into a subset of normally behaving parameter values and a subset of anomalously behaving parameter values. With the parameter values of each set so subdivided, the content importance calculator node <NUM> may determine for each parameter (visualization item <NUM>), e.g., a parameter-specific distance metric representing a statistical distance measure between the normally behaving subset and the anomalously behaving subset. The content importance calculator node <NUM> may store the determined objective parameter-specific scores in the content importance scores database <NUM>.

The method of <FIG> proceeds with a step S120 of determining a value of a first object-related metric based on the determined value of the item-related metric of each visualization item included in the visualization object. As noted above, where a dashboard <NUM> comprises a single parameter, the objective parameter-specific score of the single parameter may be directly used as objective dashboard-specific score of the dashboard <NUM>. In a case where a dashboard <NUM> comprises a plurality of parameters, the method of <FIG> determines in step <NUM> an objective dashboard-specific score based on the objective parameter-specific scores of the various parameters included in the dashboard <NUM>. In a simple example, an objective dashboard-specific score may be determined by selecting the largest parameter-specific score among the various parameter-specific scores associated with the dashboard <NUM>. Other forms of aggregation of plural parameter-specific scores into an objective dashboard-specific score are likewise conceivable. The so-determined objective dashboard-specific score represents a value of a first object-related metric within the scope of the present disclosure. The determination of the objective dashboard-specific score may be performed by the dashboard ranking node <NUM>, which may determine an objective dashboard-specific score for each dashboard <NUM> for which a definition is held in the dashboard definitions database <NUM>.

In other embodiments, step S110 may be skipped and the determination of objective parameter-specific importance scores may not be needed. Instead, it is conceivable for the dashboard ranking node <NUM> to have direct access to the raw database <NUM> and determine an objective dashboard-specific score based on a statistical analysis of the entirety of the sets of parameter values associated with each respective dashboard <NUM>, provided that plural parameters are included in the respective dashboard <NUM>. For example, it is conceivable for the dashboard ranking node <NUM> to determine for a normally behaving subset and an anomalously behaving subset of each of several parameters included in a dashboard <NUM> a respective mean or median, to group the means (or medians) of the normally behaving subsets into a first group of means (or medians), to group the means (or medians) of the anomalously behaving subsets into a second group of means (or medians), and to determine a statistical distance measure such as, e.g., an interquartile distance between the first and second group of means (or medians) as an objective dashboard-specific score.

The determined objective dashboard-specific scores may be forwarded by the dashboard ranking node <NUM> to the subjective weighting node <NUM>.

The method of <FIG> proceeds with a step S130 of determining a value of a second object-related metric of the visualization object based on a user-specific ranking of two or more of the measurable parameters. This determination may be made by the user profiler node <NUM> based on logged user interactions; corresponding logging information can be retrieved by the user profiler node <NUM> from the user interactions database <NUM>. The second object-related metric represents a score assigned to a dashboard <NUM> based on an amount of usage of the dashboard <NUM> by a specific user; the second object-related metric thus reflects needs and/or preferences of a specific user <NUM> and is a subjective measure for the importance or relevance of a dashboard <NUM>. The second object-related metric can also be referred to as a subjective dashboard-specific score; it is independent of the values of the parameters included in the dashboard <NUM> and is instead determined based on the usage behavior of the user <NUM>. The user profiler node <NUM> determines a subjective dashboard-specific score for each dashboard <NUM> used by each respective user <NUM> of the analytics system <NUM>. The subjective dashboard-specific score is thus a user-specific score; it can be different for different users <NUM> of the same dashboard <NUM>.

The determined subjective dashboard-specific scores may be forwarded by the user profiler node <NUM> to the subjective weighting node <NUM>.

The method proceeds with a step S140 of determining a relative ranking position of the visualization object based on the determined values of the first object-related metric and the second object-related metric. This determination may be made by the subjective weighting node <NUM> using the received objective and subjective dashboard-specific scores from the dashboard ranking node <NUM> and the user profiler node <NUM>. For example, the subjective weighting node <NUM> may multiplicatively weigh each objective dashboard-specific score with each subjective dashboard-specific score available for each respective dashboard <NUM>, yielding for each dashboard <NUM> a plurality of user-specific final dashboard scores. The final dashboard scores are dashboard-specific and user-specific and each represent a user-specific dashboard recommendation score which determines a user-specific relative ranking position of the associated dashboard <NUM> with respect to other dashboards <NUM> used by the same user <NUM>. The so-determined final dashboard scores may be saved by the subjective weighting node <NUM> in the user-specific dashboard recommendation database <NUM>.

The method concludes with a step S150 of causing generation of a sensory-perceptible output indicative of the determined relative ranking position of the visualization object. This step may be implemented by the UI <NUM> by accessing the user-specific dashboard recommendation database <NUM> to retrieve therefrom the final dashboard scores associated with a specific user <NUM>. Based on the retrieved final dashboard scores, the UI <NUM> may determine a user-specific dashboard ranking in which the dashboards <NUM> of interest to a particular user are ranked in accordance with their final dashboard scores. As can be easily appreciated from the foregoing description, the relative position of a dashboard <NUM> in the ranking may change, e.g., as a result of significant changes in the parameter values of one or more parameters included in the dashboard <NUM>. The UI <NUM> may be configured to visualize on one or more display devices (not shown in the drawings) the dashboard ranking and any changes in the ranking in a manner which can be easily recognized by a user <NUM>.

<FIG> is a block diagram illustrating an embodiment of an analytics apparatus <NUM> in accordance with the present disclosure. The analytics apparatus <NUM> may implement some or all of the functionalities of the analytics system <NUM> and may implement some or all of the steps of the method of <FIG>. The analytics apparatus <NUM> comprises a processor <NUM> and a memory <NUM> coupled to the processor <NUM>. The memory <NUM> stores program code (e.g., in the form of a set of instructions) that controls operation of the processor <NUM> so that the analytics apparatus <NUM> is operative to perform any of the method aspects presented herein (see, e.g., <FIG>). As understood herein, a processor, such as processor <NUM>, may be implemented using any processing circuitry and is not limited to, for example, a single processing core, but may also have a distributed topology (e.g., using cloud computing resources).

The analytics apparatus <NUM> further comprises an input interface <NUM> and an output interface <NUM>. The two interfaces <NUM>, <NUM> may implement the UI <NUM> of the analytics system <NUM> of <FIG>. Referring to the flow diagram of <FIG>, operation of the analytics apparatus <NUM> may comprises execution of the steps S100 to S150.

Since, in certain implementations of the present disclosure, a first object-related metric is determined based on measured values of one or more parameters included in a visualization object, changes in these parameter values may result in a change in the value of the first object-related metric. The changed value of the first object-related metric may, in turn, result in a changed relative ranking position of the visualization object with respect to the relative ranking positions of other visualization objects composed of other parameters. Thus, an implementation of the present disclosure may adjust the visualization object ranking as the relative significance of the content of the visualization objects (as represented by the measured parameter values) changes. By suitably visualizing the changed visualization object ranking, it can be made sure that in spite of a potentially large number of visualization objects which a user needs to monitor, or check, in order to fulfil a particular job, the user can nevertheless take notice of important occurrences in the parameter(s) of a particular visualization object.

The visualization object ranking can be visualized in any conceivable manner. For example, an order in which the ranked visualization objects, or parts thereof or icons or other symbols representing the visualization objects, are displayed on a display device may reflect the relative ranking of the visualization objects. A change in the visualization object ranking may thus result in a change in the displayed order of the visualization objects (or representative icons/symbols). If a change in the visualization object ranking occurs, this may be notified to a user by outputting an audibly and/or visibly perceivable warning indication.

Disclosed herein are embodiments of a telecommunication network analytics tool which is extended to include a dashboard recommendation engine. The dashboard recommendation engine may:.

Claim 1:
A method for a network analytics tool capable of analyzing a time-varying set of parameter values for each of a plurality of measurable parameters and visually outputting a plurality of visualization objects (<NUM>), each of the visualization objects (<NUM>) comprising one or more visualization items (<NUM>), each visualization item (<NUM>) representative of one of the plurality of measurable parameters, the method comprising:
obtaining (S120), for each of the plurality of visualization objects (<NUM>), a value of a first object-related metric;
establishing (S140) a visualization object ranking on the basis of the value of the first object-related metric and a value of a second object-related metric of each of the plurality of visualization objects (<NUM>), the value of the second object-related metric determined based on a user-specific ranking of measurable parameters independently of the parameter values of the measurable parameter of each visualization item (<NUM>) included in the visualization object (<NUM>); and
causing (S150) generation of a sensory-perceptible output indicative of the visualization object ranking established on the basis of the obtained value of the first object-related metric and the obtained value of the second object-related metric of each of the plurality of visualization objects (<NUM>),
wherein obtaining a value of the first object-related metric includes:
for each of one or more of the plurality of visualization objects (<NUM>), determining a value of the first object-related metric based at least on the set of parameter values of the measurable parameter of each visualization item (<NUM>) included in the visualization object (<NUM>).