Patent Description:
Sleeping cells are a type of cell outage in a mobile communication system resulting in the mobile service being unavailable for subscribers, but in which the cell appears operable to a network. Although developments have been made, there remains a need for further developments in this field.

<CIT>discloses a method for predicting a state of a cell in a radio access network. The method comprising obtaining information of the cell, determining one or more sets of conditions based on the information and predicting that the cell will enter a sleeping state when at least one set of the one or more sets of conditions is fulfilled. The method further comprises outputting an action to prevent the cell from entering the sleeping state based on the probability and a number of wireless devices currently connected to the cell.

In a first aspect, this specification describes an apparatus comprising means for performing: obtaining initial training labels for first performance management data of a mobile communication network, wherein the initial training labels are generated by a first classification model to identify potentially sleeping cells of the mobile communication network; training a machine-learning model based on the initial training labels, wherein the model is configured to identify potentially sleeping cells within the mobile communication network; identifying potentially sleeping cells using the trained machine-learning model based on second performance management data of the mobile communication network; receiving feedback based on an impact of corrective action taken based on the identification, by the machine-learning model, of said potentially sleeping cells; updating said training labels based on said received feedback (e.g. updating said labels with ground truth labels); and retraining the machine-learning model based on the updated training labels.

The means for obtaining initial training labels may be further configured to perform: receiving said first performance management data from the mobile communication network; and generating said initial training labels from the received first performance management data using the first classification model.

In some example embodiments, the first classification model is configured to generate said initial training labels by identifying changes in statistical association between pairs of performance management data points that are suggestive of potentially sleeping cells. The statistical association may be a correlation. The statistical association/correlation may highlight windows containing the most relevant changes.

The first classification model may be configured to filter performance management data.

The initial training labels may include time window samples. Label of corresponding samples from different time windows are may be mapped to the same inputs of a neural network of said machine learning model.

In some example embodiments, the retraining of the machine-learning model is periodic.

In some example embodiments, the corrective action comprises restarting a cell of the mobile communication network identified as a potentially sleeping cell.

The means may be further configured to perform: determining whether a cell identified by said machine-learning model as a potentially sleeping cell was a sleeping cell based on the impact of said corrective action, wherein said feedback is based on said determination. The means for determining whether a cell identified by said model as a potentially sleeping cell was a sleeping cell may comprise a manual check. Alternatively, or in addition, the means for determining whether a cell identified by said model as a potentially sleeping cell was a sleeping cell may comprise an automated check using network management tools.

The machine-learning model may be organised according to a multiple interconnected hierarchical structure. For example, each layer of the hierarchical structure of the machine-learning model may be independently trainable.

The said means may comprise: at least one processor; and at least one memory including computer program code, the at least one memory and the computer program configured, with the at least one processor, to cause the performance of the apparatus.

In a second aspect, this specification describes a method comprising: obtaining initial training labels for first performance management data of a mobile communication network, wherein the initial training labels are generated by a first classification model to identify potentially sleeping cells of the mobile communication network; training a machine-learning model based on the initial training labels, wherein the model is configured to identify potentially sleeping cells within the mobile communication network; identifying potentially sleeping cells using the trained machine-learning model based on second performance management data of the mobile communication network; receiving feedback based on an impact of corrective action taken based on the identification, by the machine-learning model, of said potentially sleeping cells; updating said training labels based on said received feedback (e.g. updating said labels with ground truth labels); and retraining the machine-learning model based on the updated training labels.

Obtaining initial training labels may comprise: receiving said first performance management data from the mobile communication network; and generating said initial training labels from the received first performance management data using the first classification model.

The first classification model may be configured to generate said initial training labels by identifying changes in statistical association (e.g. a correlation) between pairs of performance management data points that are suggestive of potentially sleeping cells. The first classification model may be configured to filter performance management data. The initial training labels may include time window samples, wherein labels of corresponding samples from different time windows are mapped to the same inputs of a neural network of said machine learning model.

The retraining of the machine-learning model may be periodic.

The corrective action may comprise restarting a cell of the mobile communication network identified as a potentially sleeping cell.

The method may further comprise: determining whether a cell identified by said machine-learning model as a potentially sleeping cell was a sleeping cell based on the impact of said corrective action. The said feedback may be based on said determination.

Determining whether a cell identified by said model as a potentially sleeping cell was a sleeping cell may comprise a manual check or an automated check using network management tools.

The machine-learning model may be organised according to a multiple interconnected hierarchical structure (e.g. wherein each layer of the hierarchical structure of the machine-learning model is independently trainable).

In a third aspect, this specification describes an apparatus configured to perform (at least) any method as described with reference to the second aspect.

In a fourth aspect, this specification describes computer-readable instructions which, when executed by computing apparatus, cause the computing apparatus to perform (at least) any method as described with reference to the second aspect.

In a fifth aspect, this specification describes a computer program comprising instructions for causing an apparatus to perform at least the following: obtaining initial training labels for first performance management data of a mobile communication network, wherein the initial training labels are generated by a first classification model to identify potentially sleeping cells of the mobile communication network; training a machine-learning model based on the initial training labels, wherein the model is configured to identify potentially sleeping cells within the mobile communication network; identifying potentially sleeping cells using the trained machine-learning model based on second performance management data of the mobile communication network; receiving feedback based on an impact of corrective action taken based on the identification, by the machine-learning model, of said potentially sleeping cells; updating said training labels based on said received feedback (e.g. updating said labels with ground truth labels); and retraining the machine-learning model based on the updated training labels.

In a sixth aspect, this specification describes a computer-readable medium (such as a non-transitory computer-readable medium) comprising program instructions stored thereon for performing (at least) the method of the second aspect.

In a seventh aspect, this specification describes an apparatus comprising: at least one processor; and at least one memory including computer program code which, when executed by the at least one processor, causes the apparatus to perform (at least) the method of the second aspect.

In an eighth aspect, this specification describes an apparatus comprising: means (such as a first classification model, e.g. forming parting of an automatic label generation module) for obtaining initial training labels for first performance management data of a mobile communication network, wherein the initial training labels are generated by a first classification model to identify potentially sleeping cells of the mobile communication network; means (such as a training or retraining module) for training a machine-learning model based on the initial training labels, wherein the model is configured to identify potentially sleeping cells within the mobile communication network; means (such as a (trained) classification module) for identifying potentially sleeping cells using the trained machine-learning model based on second performance management data of the mobile communication network; means (such as a feedback module) for receiving feedback based on an impact of corrective action taken based on the identification, by the machine-learning model, of said potentially sleeping cells; means (such as a ground truth label updating module) for updating said training labels based on said received feedback; and means (such as the training or retraining module) retraining the machine-learning model based on the updated training labels.

Example embodiments will now be described, by way of example only, with reference to the following schematic drawings, in which:.

The embodiments and features, if any, described in the specification that do not fall under the scope of the independent claims are to be interpreted as examples useful for understanding various embodiments of the invention.

<FIG> is a block diagram of a system, indicated generally by the reference numeral <NUM>, in accordance with an example embodiment. The system <NUM> comprises a network management system (NMS) <NUM> in two-way communication with a mobile communications network <NUM>, such as an LTE (long term evolution) RAN (radio access network). The NMS <NUM> includes a performance management (PM) data collector <NUM> (along with many other modules, which are omitted from <FIG> for clarity). The PM data collector <NUM> provides performance management (PM) data to a sleeping cell detector <NUM>. The NMS <NUM> and sleeping cell detector <NUM> may form part of an operator Network Operation Center (NOC) <NUM>.

The sleeping cell detector <NUM> seeks to identify sleeping cells based on the PM data received from the NMS <NUM>. The PM data may, for example, be available on an hourly basis, but such time granularity for PM collection can be also reduced (e.g. to <NUM> minutes) or increased (e.g. to daily collection). As noted above, sleeping cells are a kind of cell outage which makes mobile service either unavailable for subscribers or with reduced performance, but in which the cell typically appears to be operable from the network point of view. The cell results provided to the operator (e.g. to the NMS <NUM>) typically appears as if it has no or reduced traffic, as would be the case if most users within the cell were inactive.

The sleeping cell detector <NUM> provides indications of identified sleeping cells to the NMS <NUM>, such that suitable action can be taken.

<FIG> is a flow chart showing an algorithm, indicated generally by the reference numeral <NUM>, in accordance with an example embodiment.

The algorithm <NUM> starts at operation <NUM> where initial training labels are obtained. The initial training labels may be generated based on first performance management (PM) data received from a mobile communication network (such as the mobile network <NUM>). As discussed in detail below, the initial training labels are generated by a first classification model (e.g. forming part of the sleeping cell detector <NUM>) that is used to identify potentially sleeping cells of the mobile communication network, for example based on a statistical analysis of the PM data.

At operation <NUM>, a machine-learning model is trained based on the initial training labels generated in operation <NUM>. The model (e.g. implemented by the sleeping cell detector <NUM>) is configured to identify potentially sleeping cells within the mobile communication network <NUM>.

At operation <NUM>, the model trained in operation <NUM> is used to identify potentially sleeping cells based on further performance management data of the mobile communication network <NUM>.

As discussed in detail below, a determination of the validity of the operation <NUM> in identifying potentially sleeping cells can be used, in operation <NUM>, in the updating of the training labels and the retraining of the machine-learning model based on the updated training labels. The updating of the machine-learning model may be periodic. Alternatively, or in addition, the updating of the machine-learning model may be triggered by other factors, such as the quality (perceived or measured) of the model.

The algorithm <NUM> provides a feedback loop that can be used to continuously evolve a set of training labels for using in training the machine-learning model. The initial training labels are generated (in the operation <NUM>), with no ground truth needed, and the training labels are then evolved, as discussed further below, by mixing the training set with new training examples that constitute the ground truth, as derived from a feedback from the network and the operators.

<FIG> is a block diagram of a system, indicated generally by the reference numeral <NUM>, in accordance with an example embodiment. The system <NUM> may be used to implement the algorithm <NUM> described above.

The system <NUM> comprises network cells <NUM>, an automatic label generation module <NUM>, training set <NUM>, model training module <NUM>, online classification module <NUM>, reset module <NUM>, no action module <NUM>, feedback module <NUM>, ground truth label update module <NUM> and manual check module <NUM>.

The automatic label generation module <NUM> generates initial training labels based on performance management data from the network cells <NUM> (such as the mobile network <NUM>). The automatic label generation module generates those initial training labels (which identify potentially sleeping cells of the network cells <NUM>) using a first classification model. The automatic label generation module <NUM> can therefore be used to implement the operation <NUM> described above.

On the basis of the initial training labels, a training set of data <NUM> is generated for use by the model training module <NUM> to generate the online classification module <NUM> (which model takes the form of a machine-learning model, as discussed in detail below). Thus, the training set of data <NUM> can be used in the implementation of the operation <NUM> described above.

The online classification module <NUM> can then be used to identify potentially sleeping cells based on further performance management (PM) data received from the network cells <NUM>, thereby implementing operation <NUM> of the algorithm <NUM>.

The output of the online classification module <NUM> can then be used by the rest of the system <NUM> to implement the operation <NUM> of the algorithm <NUM>, as discussed in detail below.

<FIG> is a flow chart showing an algorithm, indicated generally by the reference numeral <NUM>, in accordance with an example embodiment. The algorithm <NUM>, which may be implemented by the system <NUM>, can be used to evolve the initial training set generated by the automatic label generation module <NUM> (and therefore improve the training of the online classification module <NUM>).

The algorithm <NUM> starts with the operation <NUM> of the algorithm <NUM> described above in which a potentially sleeping cell is identified (e.g. based on an ML model trained in operation <NUM>) based on initial training labels (obtained or generated in operation <NUM>).

When a potentially sleeping cell is identified in the operation <NUM>, the algorithm <NUM> moves to operation <NUM>, where corrective action is taken. For example, a cell identified as a potentially sleeping cell may be restarted (e.g. under the control of the restart module <NUM> of the system <NUM>). Note that no action is taken regarding a cell that is identified by the model as operating normally (i.e. not sleeping), as indicated by the no action module <NUM>.

The algorithm <NUM> moves to operation <NUM>, where the impact of the corrective action (or no correction action) taken in operation <NUM> is determined. For example, feedback from the network cells <NUM> can be analysed by the feedback module <NUM> of the system <NUM> to determine whether the prediction was correct, and this allows the ground truth label update module <NUM> to be updated with a ground truth label.

In an alternative embodiment, the impact of the corrective action may be carried out in operation <NUM> by performing a manual check (see the manual check module <NUM> of the system <NUM>).

With the impact of the corrective action determined, the algorithm <NUM> moves to operation <NUM>, where the labels of the training set <NUM> can be updated, enabling the machine learning module to be retrained, if desired.

The system <NUM> therefore provides an arrangement for sleeping cell detection based on a feedback loop that allows the continuous improvement and updating of the machine learning algorithm. In the algorithms <NUM> and <NUM>, the training set <NUM> is continuously evolved, starting from a set that is generated (in operation <NUM>), with no ground truth needed, and then by mixing the training set with new training examples that constitute the ground truth, as derived from a feedback from the network and the operators. The mixing strategy may rely on both an aging mechanism, by which the oldest labels of a cell are replaced by the newest labels, and a 'learning by errors' strategy, by which the label of a cell is replaced or confirmed thanks to the feedback from the network.

In practice, when a potentially sleeping cell is identified (e.g. in the operation <NUM>), corrective action can be integrated in the network operation center <NUM>, such that an appropriate reset command (using the reset module <NUM>) can be sent to the suspected sleeping cell. In many circumstances, this is sufficient to recover the normal cell behaviour. Then after restart, the cell behaviour can be monitored and if normal behaviour is detected, the corresponding sleeping cell pattern can be labelled as truth set.

The reset in most cases results in repair of the malfunctioning in the networks. However, monitoring and verifying the correctness of the provided classification can be carried out by analyzing and checking the status of the cells after the reset. This feedback regarding the classification from the network contributes to enriching the ground truth set with trusted labels and it allows mixing of the automatic training set labels with the validated data.

As noted above, resetting is not the only way to obtain ground truth data, it is also possible obtain such data via manual checks (remotely or from the field, see the manual check module <NUM>); this can be in addition to, or instead of, the reset method described above. By way of example, when a domain expert provides the ground truth derived from manual detection process, either remotely or in the field, the expert may provide one or more of: specific explicit label (sleeping), detection interval of relevant KPIs, reset time and in service time.

At the same time, the ground truth module <NUM> may also be updated with randomly chosen normal patterns in order to have a balanced training set.

Thus, it is possible to start the training and the classification described above even in the absence of any ground truth, as it is often the case in real world applications, and then to enhance the quality of the classification by keeping track of the monitoring output and of the feedback from the network.

In some example embodiments, prior to generating or obtaining training labels in the operation <NUM>, the available performance management (PM) data coming from the network <NUM> are subjected to a data preparation step, in order, for example, to remove missing values and any possible outliers, remove seasonality and/or to normalize the data. The presence of missing values inside the data can cause the failure of the entire method. Moreover, raw data may show a periodic behavior due to the day/night cycle that is not relevant in the analysis, so the seasonality can be removed; and seasonality removal can be applied or not, depending on the type of KPI. The period of the seasonal component may be chosen as one day (e.g. where an hourly PM collection period is used). Finally, the data may be normalized in order to be able to compare different KPIs in different windows.

<FIG> is a flow chart showing an algorithm, indicated generally by the reference numeral <NUM>, in accordance with an example embodiment. The algorithm <NUM> comprises a vector labelling phase <NUM>, a pattern recognition phase <NUM> and a sleep cell detector generation phase <NUM>.

The vector labelling phase <NUM> takes raw data (e.g. raw performance management data) as input and provides labelling for following sequential blocks of neural networks. Vector labels generated in the vector labelling phase <NUM> may be used as the training labels of the operation <NUM> described above.

The pattern recognition phase <NUM> uses pattern recognition to train a plurality of neutral networks, as described in detail below. Then, the sleeping cell detector generation phase <NUM> is used to generate a single machine-learning module that can be used as the online classification module <NUM> described above.

The performance management (PM) data comprises key performance indicators (KPIs). To better combine these KPIs, we create the so-called KPI couples (i.e. pairs of KPIs), created with the purpose of avoiding problems due to external causes that usually affects only a single KPI.

Sleeping cells may be identified by identification of a synchronous variation in more than one key performance indicators (KPIs) that form part of the performance management data described above (e.g. KPI couples). In order to generate the learning vectors for the training of successive blocks of neural networks (e.g. in the operation <NUM>), it is possible to extract the training labels for each cell at a couple's level and the training labels for the cell classification. One of the advantages of using a structured algorithm for the extraction of the training labels is the possibility of building a layered architecture for the pattern recognition and the sleeping cell detection method. Indeed, without a structure able to identify the KPI couples mostly related to the sleeping behavior of a cell, it may not be possible to obtain clear labels for the KPI couples and therefore a clear distinction between two blocks of neural networks; otherwise, a global cell classification may have been the only possibility.

The vector labelling phase <NUM> comprises: a Principal Component Analysis (PCA) that selects pairs of KPIs (i.e. KPI couples) most responsible for the change of correlation structure among KPI couples; a first step selecting KPIs with highest PCA loadings over a given threshold; a correlation analysis that highlights time windows containing the most relevant changes; and a selection of the most relevant KPI couples in the relevant time windows provides a <NUM>/<NUM> classification of the couples. Couples classified with "<NUM>" may be kept and along with the detection window and the temporal position in the detection window constitute the labelled training vector; thus, the data labels discussed above are generated by combining the most relevant couples in the relevant windows.

The data provided as input of the vector labelling phase <NUM> are a subset of non-labelled cells coming from the network cells <NUM> that enter this step after data preparation. Even if this step is based on a time window analysis, the data are considered all together in the Principal Components Analysis, in order to extract the weights of the couples from a cumulative point of view. Then the cell information is taken into account in the correlation step, to obtain the detection of the most relevant windows, with a cell specific analysis. The output of this phase, i.e. the labeled couples in every window, is the input of the next phase.

From this point forward, the analysis proceeds window by window in such a way that all the windows, including the windows belonging to the same cell, are independent from each other and then the behavior of an adjacent window will not affect the classification of another.

In some example embodiments, the initial training labels may include time window samples, wherein labels of corresponding samples from different time windows are mapped to the same inputs of said neural network. A time window may include multiple samples that are all provided to a particular neural network, such that the neural network can examine temporal relationships.

The structure of multiple neural networks is built in order to recognize the KPI couples responsible for the sleeping behavior. The idea is to use a parallel structure of multiple Neural Networks (one for each KPI couple), with the purpose of analyzing each couple independently from the others and training a model that is able to recognize the sleeping patterns of each couple. The neural networks have the same structure, in terms of number of hidden layers, nodes and hyperparameters, in order to be able to combine and compare the results, to obtain a final classification.

The vector labelling phase <NUM> described above may include an activity of filtering of KPI couples to be included in the training set based on rules defined by domain Knowledge.

In fact, thanks to technical domain knowledge, it is possible to a priori inspect the meaning of the relation among the KPIs. In particular, before performing the training phase on the set of neural networks used in the operation <NUM>, a check on a previously provided set of rules may be recommended, in order to avoid the use of particular training vectors responsible of describing a relation between two KPIs that shows a high correlation not related to sleeping behavior. For example, considering Block Error Rate and throughput KPIs, it is expected they have a negative correlation. If it is the case for a particular training vector, it may be necessary to remove it from the training set.

In some example embodiments the general method <NUM> only is used. In some example embodiments the application of the general method <NUM> is followed the training vector filtering procedure based on domain knowledge described above. Thus, patterns that are not related to sleeping cells may be removed from consideration.

The creation of a method based on neural networks implies the use of classified inputs to perform the training phase. On the other hand, it is the previous phase that automatically provides the label vectors for the neural networks. This implies that the method described so far is working in an unsupervised mode, since no trusted information is required for the achieving of the labeled output.

As already described, the input data for this second phase are the KPIs coming from the data preparation step and the KPI couples' labels coming from the vector labelling operation <NUM>, combined together to gather the information of all the couples in the different windows. The output of those Neural Networks will be a continuous value between [<NUM>, <NUM>] for every couple in every window.

<FIG> is a block diagram of one of a plurality neural networks, indicated generally by the reference numeral <NUM>, used in example embodiments. The neural network <NUM> may be used in the pattern recognition operation <NUM>.

Each of a plurality of parallel neural networks used in the operation <NUM> is structurally the same, while each is trained on a specific KPI couple, in order to be specialized on those two particular KPIs and on their specific interactions. Indeed, here the number of neural networks is equal to the number of KPI couples, i.e. <MAT> couples, if n is the number of available KPIs. It should be noticed that the <MAT> neural networks are independent; they only share the KPIs. Each of these neural networks is trained with the same hyperparameters, in such a way that each couple maintains its own weight with respect to all the other couples, in terms of the relative importance that has been associated to a KPI couple on the others. The second phase of the architecture based on parallel neural networks is devoted to the detection and identification of the observed variables in terms of patterns, by looking at the couples' behavior highlighted by the vector generation phase and by generalizing them. The sleeping cell detector generation phase <NUM> uses a set of cells, whose labels come from the automatic label generator or from the trusted feedback from the network, to build a single neural network. This network is the effective sleeping cell detector, able to recognize the sleeping cells using the provided patterns and to provide the final classification. The neural network of the sleeping cell detector phase <NUM> consists of a single neural network which aggregates the outputs of the pattern recognition phase for each KPI couple and it defines when a certain set of KPI couples in a specific combination of patterns is responsible for the cell to be sleeping, by giving the <NUM>/<NUM> classification (normal or sleeping) for every window. A cell is then considered to be sleeping if it holds at least one sleeping window.

<FIG> is a block diagram of a neural network, indicated generally by the reference numeral <NUM>, that may be used in the sleeping cell detector phase <NUM>.

The neural network <NUM> used in the operation <NUM> may be trained with either automatically generated labels (e.g. the outputs of the automatic label generation module <NUM>) or trusted labels (e.g. as updated by the ground truth label update module <NUM>). The neural network <NUM> can be retrained when new (trusted) data are available, leaving the previous steps untouched. At the same time, the neural network <NUM> is able to perform the classification with automatic labelled data, allowing to obtain a final result without the need of trusted cells. Moreover, the neural network <NUM> can be trained with a mixed set of automatic labelled cells and trusted cells, a characteristic that allows to faster increase the accuracy during time, bringing trusted knowledge when available.

Thus, in the operation <NUM>, the training operating is performed with the use of the evolving training set of classified windows and cell, i.e. only automatic labels, ground truth labels, or a mix of them.

The operation <NUM> may be based on a single neural network (e.g. the neural network <NUM>) created in order to provide a classification of a single window. For this reason, the outputs of the <MAT> parallel neural networks used in the operation <NUM> are combined together to obtain the information window by window. Then, the <MAT> continuous values describing a single window are used as input for the neural network constructed in this phase.

The automatic labelled training set generated at step <NUM> is split into two subsets, the first used to train the neural networks at step <NUM> and the second used to train the sleeping cells detector at step <NUM>, to avoid the overfitting of this second step. In order to perform the operation <NUM> on a second subset of cells, those new data do not enter the first phase for the vector labelling, but their value is obtained at the end of the test phase of the parallel neural networks and then is used, in combination with the window classification, to train this new single neural network.

The expected output of this layer is a final classification of the window and then, all the windows of a cell can be combined to obtain the cell classification based on the fact that a cell can be considered as sleeping if it has at least a window classified as sleeping.

Finally, the intrinsic non-linearity of a neural network is reflected in the capability of the system to catch some non-linear failure mechanisms. On the contrary, the linear Euclidean metric compared with a threshold applied on the correlation matrices in the vector generation phase would not have been able to detect those non-linear failures. In particular, the non-linear detector also recognizes the failures caused by the activation or the variation of particular subsets of KPIs, while a linear detector could only find a small variation of all the KPIs or a big variation of just one KPI.

The multi-layer structure of the algorithm allows to separately train each of the blocks and fully control them, in order to completely understand their output. In this perspective, it is suggested to train the parallel neural network used in the operation <NUM> and the neural network used in the operation <NUM> with different data, to be sure that the two learning phases are independent. This is particularly useful in the operation <NUM>, when the automatic label mechanism may be responsible for both the training vectors of the multiple neural networks and of the last neural network.

The embodiments described above can be used with all phases in the network operation center <NUM> in order to provide a real-time observation of the network and a classification of the cells. Moreover, the described variants composed by a subset of phases can be used with the same purpose, in combination with other cell monitoring techniques. In both cases, the procedure for the detection of sleeping cells may require the user to provide a list of input values and data that can be based on operator domain knowledge or can be network specific.

In the following, some parameters that may be used as input of the method in some example embodiments are described. These values may be used for the construction of the method, together with the different training steps related to the neural networks. During the algorithm preparation, the size of the neural networks may be defined and stored in the model construction, and the parameters fixed. The idea is that the algorithm is prepared and trained at the beginning and then the user can proceed with the testing of the data or the user can create a flow that automatically analyzes the data available and classifies them using the presented method.

In order to obtain a method that is as accurate as possible, there are some criteria to evaluate how much the model is correctly performed, that may be taken into account during the building phase of the method, but are not needed for its use. In particular, the values of the accuracy, the sensitivity and the other values mainly obtained from the neural networks, can be used for the setup of the parameters, such as the number of iterations required both in the operation <NUM> and the operation <NUM>. Even though they play a role in the model preparation, they are not available in the table below, since they depend on the choice of other parameters.

The parameters may include the following.

For completeness, <FIG> is a schematic diagram of components of one or more of the example embodiments described previously, which hereafter are referred to generically as a processing system <NUM>. The processing system <NUM> may, for example, be the apparatus referred to in the claims below.

The processing system <NUM> may have a processor <NUM>, a memory <NUM> closely coupled to the processor and comprised of a RAM <NUM> and a ROM <NUM>, and, optionally, a user input <NUM> and a display <NUM>. The processing system <NUM> may comprise one or more network/apparatus interfaces <NUM> for connection to a network/ apparatus, e.g. a modem which may be wired or wireless. The network/ apparatus interface <NUM> may also operate as a connection to other apparatus such as device/apparatus which is not network side apparatus. Thus, direct connection between devices/apparatus without network participation is possible.

The memory <NUM> may comprise a non-volatile memory, such as a hard disk drive (HDD) or a solid state drive (SSD). The ROM <NUM> of the memory <NUM> stores, amongst other things, an operating system <NUM> and may store software applications <NUM>. The RAM <NUM> of the memory <NUM> is used by the processor <NUM> for the temporary storage of data. The operating system <NUM> may contain code which, when executed by the processor implements aspects of the algorithms <NUM>, <NUM> and <NUM> described above. Note that in the case of small device/apparatus the memory can be most suitable for small size usage i.e. not always a hard disk drive (HDD) or a solid state drive (SSD) is used.

The processor <NUM> may take any suitable form. For instance, it may be a microcontroller, a plurality of microcontrollers, a processor, or a plurality of processors.

The processing system <NUM> may be a standalone computer, a server, a console, or a network thereof. The processing system <NUM> and needed structural parts may be all inside device/apparatus such as IoT device/apparatus i.e. embedded to very small size.

In some example embodiments, the processing system <NUM> may also be associated with external software applications. These may be applications stored on a remote server device/apparatus and may run partly or exclusively on the remote server device/apparatus. These applications may be termed cloud-hosted applications. The processing system <NUM> may be in communication with the remote server device/apparatus in order to utilize the software application stored there.

<FIG> show tangible media, respectively a removable memory unit <NUM> and a compact disc (CD) <NUM>, storing computer-readable code which when run by a computer may perform methods according to example embodiments described above. The removable memory unit <NUM> may be a memory stick, e.g. a USB memory stick, having internal memory <NUM> storing the computer-readable code. The internal memory <NUM> may be accessed by a computer system via a connector <NUM>. The CD <NUM> may be a CD-ROM or a DVD or similar. Other forms of tangible storage media may be used. Tangible media can be any device/apparatus capable of storing data/information which data/information can be exchanged between devices/apparatus/network.

Reference to, where relevant, "computer-readable medium", "computer program product", "tangibly embodied computer program" etc., or a "processor" or "processing circuitry" etc. should be understood to encompass not only computers having differing architectures such as single/multi-processor architectures and sequencers/parallel architectures, but also specialised circuits such as field programmable gate arrays FPGA, application specify circuits ASIC, signal processing devices/apparatus and other devices/ apparatus. References to computer program, instructions, code etc. should be understood to express software for a programmable processor firmware such as the programmable content of a hardware device/apparatus as instructions for a processor or configured or configuration settings for a fixed function device/apparatus, gate array, programmable logic device/apparatus, etc..

Similarly, it will also be appreciated that the flow diagrams of <FIG>, <FIG> and <FIG> are examples only and that various operations depicted therein may be omitted, reordered and/or combined.

It will be appreciated that the above described example embodiments are purely illustrative and are not limiting on the scope of the invention. Other variations and modifications will be apparent to persons skilled in the art upon reading the present specification.

Moreover, the disclosure of the present application should be understood to include any novel features or any novel combination of features either explicitly or implicitly disclosed herein or any generalization thereof and during the prosecution of the present application or of any application derived therefrom, new claims may be formulated to cover any such features and/or combination of such features.

Claim 1:
An apparatus comprising means for performing:
obtaining (<NUM>) initial training labels for first performance management data of a mobile communication network (<NUM>), wherein the initial training labels are generated by a first classification model (<NUM>) to identify potentially sleeping cells of the mobile communication network (<NUM>);
training (<NUM>) a machine-learning model (<NUM>) based on the initial training labels, wherein the model is configured to identify potentially sleeping cells within the mobile communication network;
identifying (<NUM>) potentially sleeping cells using the trained machine-learning model based on second performance management data of the mobile communication network;
receiving feedback (<NUM>) based on an impact of corrective action (<NUM>) taken based on the identification, by the machine-learning model, of said potentially sleeping cells;
updating (<NUM>) said training labels based on said received feedback; and
retraining (<NUM>) the machine-learning model based on the updated training labels
wherein the means for obtaining initial training labels is further configured to perform:
receiving said first performance management data from the mobile communication network (<NUM>); and
generating said initial training labels from the received first performance management data using the first classification model.