Patent Description:
For a telecommunication network operator there is a need to monitor operation and performance of the telecommunication network and to take corrective actions when necessary in order to ensure quality of service for the users of the telecommunication network. A network operation center (NOC) is generally a location from which NOC personnel exercises monitoring and control over the network. NOC personnel are responsible for monitoring alarms and other incidents appearing in the network and for deciding upon required actions based on the information from the network.

In order to improve efficiency of network monitoring and control and to reduce the amount of manual work and human errors, various methods have been developed to automatize the NOC and to automatically analyze alarms and other incidents from telecommunication networks and to perform automatic actions based on the information from the network. Such automation is not a straightforward task to implement.

<CIT> discloses testing macrodiversity and handover functionality of a radio network controller. <CIT> discloses automated network monitoring and control. <CIT> discloses quality of experience aware transport self organizing network framework.

Now new tools are provided for use in the context of automation of network alarm monitoring and network control.

According to a first example aspect there is provided a computer implemented method according to enclosed independent method claim. The method is for providing a test system for testing a network control system configured to perform automated alarm monitoring and control of a telecommunication network. The method is performed by.

In some example embodiments, the method further comprises updating and maintaining states of network devices of the telecommunication network in the test system; and taking the states of the network devices into account in the one or more responses.

In some example embodiments, the method further comprises interacting with the control interface of the telecommunication network over a second interface of the test system;
collecting, over the second interface, states of one or more network devices of the telecommunication network; and taking the collected states of the one or more network devices into account in the one or more responses.

In some example embodiments, the method further comprises sending, to the network control system, over the first interface, one or more alerts and/or network performance data in order to trigger the network control system to send the command.

In some example embodiments, the method further comprises monitoring, over the second interface, alerts and/or network performance data appearing in the telecommunication network; and monitoring the first interface in expectation of receiving a command sent responsive to the alerts and/or network performance data appearing in the telecommunication network.

In some example embodiments, the control interface of the telecommunication network is an OSS interface.

In some example embodiments, the received command specifies one or more actions in the network device of the telecommunication network.

In some example embodiments, the command and/or the next action relates to one or more of the following: registering to a network device, resetting a network device, shutting down a network device, login to a network device, blocking a cell.

In some example embodiments, the one or more responses comprise indication of one or more of the following: successful operation, failed operation, successful registration to the network device, failed registration to the network device, successful reset operation, failed reset operation, successful shut down operation, failed shut down operation, successful login operation, failed login operation, successful cell block operation, failed cell block operation, failed connection to the network device, unexpectedly dropped connection to a network device, network device not found, unexpected error.

According to a second example aspect of the present invention, there is provided an apparatus comprising a processor and a memory including computer program code; the memory and the computer program code configured to, with the processor, cause the apparatus to perform the method of the first aspect or any related embodiment.

According to a third example aspect of the present invention, there is provided a computer program comprising computer executable program code which when executed by a processor causes an apparatus to perform the method of the first aspect or any related embodiment.

According to a fourth example aspect there is provided a computer program product comprising a non-transitory computer readable medium having the computer program of the third example aspect stored thereon.

According to a fifth example aspect there is provided an apparatus comprising means for performing the method of the first aspect or any related embodiment. Any foregoing memory medium may comprise a digital data storage such as a data disc or diskette, optical storage, magnetic storage, holographic storage, optomagnetic storage, phase-change memory, resistive random access memory, magnetic random access memory, solid-electrolyte memory, ferroelectric random access memory, organic memory or polymer memory. The memory medium may be formed into a device without other substantial functions than storing memory or it may be formed as part of a device with other functions, including but not limited to a memory of a computer, a chip set, and a sub assembly of an electronic device.

In the following description, the invention is described with particular reference to <FIG>, while the description of the remaining figures is provided for illustrative purposes for a better understanding of the invention.

Various embodiments of present disclosure provide a test system for testing whether network automation operates as intended. The network automation comprises a network control system that monitors network and performs automatic corrective actions and failure correction in attempt to improve operation of the network. The test mechanisms provided are operable to test whether the network control system results in performing sensible actions.

<FIG> shows an example scenario. The scenario shows a telecommunication network <NUM> comprising a plurality of physical network sites comprising base stations and other network devices, and a network control system <NUM> configured to automatically monitor and control the telecommunication network <NUM>. The network control system <NUM> is configured to interact with the telecommunication network (e.g. through an operations support system, OSS, interface), to automatically analyze alarms and other incidents from the telecommunication network <NUM> and to perform automatic actions and failure correction in the telecommunication network <NUM> based on this information. In this scenario, the network control system is in normal operation.

<FIG> shows an example scenario according to an embodiment. The scenario shows a test system <NUM> in addition to the telecommunication network <NUM> and the network control system <NUM> of <FIG>. In this scenario, operation of the network control system <NUM> is being tested. The test system <NUM> is configured to interact with the network control system <NUM> and to emulate operation of the telecommunication network <NUM> in view of the network control system <NUM>. Additionally, the test system <NUM> may be configured to interact with the telecommunication network <NUM> for example to obtain real time state information from production environment. The test system <NUM> is configured to implement at least some example embodiments of present disclosure. The messages that would travel between the network control system and the control interface of the telecommunication network, when the network control system is in production use, are now configured to travel to or through the test system. However, also in this scenario, the network control system <NUM> and the telecommunication network <NUM> may interact with each other at least to some extent.

<FIG> shows a block diagram of an apparatus <NUM> according to an embodiment. The apparatus <NUM> is for example a general-purpose computer or server or some other electronic data processing apparatus. The apparatus <NUM> can be used for implementing at least some embodiments of the invention. That is, with suitable configuration the apparatus <NUM> is suited for operating for example as the test system <NUM> of foregoing disclosure.

The apparatus <NUM> comprises a communication interface <NUM>; a processor <NUM>; a user interface <NUM>; and a memory <NUM>. The apparatus <NUM> further comprises software <NUM> stored in the memory <NUM> and operable to be loaded into and executed in the processor <NUM>. The software <NUM> may comprise one or more software modules and can be in the form of a computer program product.

The processor <NUM> may comprise a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a graphics processing unit, or the like. <FIG> shows one processor <NUM>, but the apparatus <NUM> may comprise a plurality of processors.

The user interface <NUM> is configured for providing interaction with a user of the apparatus. Additionally or alternatively, the user interaction may be implemented through the communication interface <NUM>. The user interface <NUM> may comprise a circuitry for receiving input from a user of the apparatus <NUM>, e.g., via a keyboard, graphical user interface shown on the display of the apparatus <NUM>, speech recognition circuitry, or an accessory device, such as a headset, and for providing output to the user via, e.g., a graphical user interface or a loudspeaker.

The memory <NUM> may comprise for example a non-volatile or a volatile memory, such as a read-only memory (ROM), a programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), a random-access memory (RAM), a flash memory, a data disk, an optical storage, a magnetic storage, a smart card, or the like. The apparatus <NUM> may comprise a plurality of memories. The memory <NUM> may serve the sole purpose of storing data, or be constructed as a part of an apparatus <NUM> serving other purposes, such as processing data.

The communication interface <NUM> may comprise communication modules that implement data transmission to and from the apparatus <NUM>. The communication modules may comprise a wireless or a wired interface module(s) or both. The wireless interface may comprise such as a WLAN, Bluetooth, infrared (IR), radio frequency identification (RF ID), GSM/GPRS, CDMA, WCDMA, LTE (Long Term Evolution) or <NUM> radio module. The wired interface may comprise such as Ethernet or universal serial bus (USB), for example. The communication interface <NUM> may support one or more different communication technologies. The apparatus <NUM> may additionally or alternatively comprise more than one of the communication interfaces <NUM>.

A skilled person appreciates that in addition to the elements shown in <FIG>, the apparatus <NUM> may comprise other elements, such as displays, as well as additional circuitry such as memory chips, application-specific integrated circuits (ASIC), other processing circuitry for specific purposes and the like. Further, it is noted that only one apparatus is shown in <FIG>, but the embodiments of the invention may equally be implemented in a cluster of shown apparatuses.

<FIG> shows a flow diagram illustrating example methods according to certain embodiments. The methods may be implemented in the test system <NUM> of <FIG> and/or in the apparatus <NUM> of <FIG>. The methods are implemented in a computer and do not require human interaction unless otherwise expressly stated. It is to be noted that the methods may however provide output that may be further processed by humans and/or the methods may require user input to start. Different phases shown in the flow diagrams may be combined with each other and the order of phases may be changed except where otherwise explicitly defined. Furthermore, it is to be noted that performing all phases of the flow diagram is not mandatory.

The method of <FIG> comprises the following phases:
<NUM>: The test system interacts with the network control system over a first interface. The network control system is configured to perform automated alarm monitoring and control of a telecommunication network. The network control system may be further configured to perform automated failure correction in the telecommunication network. The purpose of the test system is to test operation of the network control system. The first interface is configured to emulate operation of a control interface of the telecommunication network. Such control interface is used by the network control system for interacting with the telecommunication network when the network control system is in use and monitors production environment (i.e. telecommunication network that is in actual use). The control interface may be for example OSS interface and the first interface may emulate operation of the OSS interface.

The use of the test system may be initiated by setting the network control system to interact with the first interface of the test system instead of interacting with the control interface of the telecommunication network. Alternatively, any interaction intended to the control interface of the telecommunication network may be intercepted by the test system so that changes are not needed in the network control system.

<NUM>: The test system optionally interacts with the control interface of the telecommunication network (i.e. the production environment) over a second interface. the test system may interact with the OSS interface of the telecommunication network.

<NUM>: The test system executes a test case or a chain of interrelated test cases by monitoring operation of the network control system through the first interface. Additionally or alternatively, plurality of independent test cases may be consecutively executed. Each test case comprises performing at least some of the phases <NUM>-<NUM>. Each test case may comprise certain predefined parameters or rules according to which respective test case is executed.

<NUM>: Alerts and/or network performance data are provided to the network control system to trigger a test case. These may be provided through the first interface or obtained from other source as will be explained below. An alert is in general a notification of a problem in a network device. There may be different types of alerts. Network performance data refers for example to Key Performance Indicator KPI data that is available in telecommunication networks. KPI data may relate to signal quality, signal levels, signal to noise ratio, handover parameters, number of users, number of dropped calls and the like.

In an embodiment, the network control system receives alerts and/or network performance data from production environment and the test system monitors operation of the network control system triggered by the alerts and/or network performance data from production environment. In such embodiment, the test system may monitor, over the second interface, alerts and/or network performance data appearing in the telecommunication network in order to know when and which actions should be triggered in the network control system. The triggered actions may then be monitored through the first interface.

In another embodiment, the test system provides alerts and/or network performance data to the network control system to trigger one or more actions in the network control system. This may be done by the test system through the first interface or by writing the alerts and/or network performance data to an input source that is normally monitored by the network control system. In such case, interaction with the production environment is not necessarily needed.

<NUM>: The test system monitors whether a command associated with a network device of the telecommunication network is received over the first interface. The command is expected to be triggered responsive to the alerts and/or network performance data provided in phase <NUM>. The command should be received through the first interface if the network control system is operating normally. The receipt of the command may be monitored for a predefined period of time. Depending on the test case and the alerts/network performance data that is involved, the command should be seen within certain time window. The predefined period of time is set to cover at least this time window. The test case that is being executed may define the time period that is to be used. It is to be noted that setting the predefined period of time is not mandatory. Alternatively, the test system may be set to wait until a command is received from the network control system.

<NUM>: In some embodiments, the test system obtains states of network devices of the telecommunication network.

In an embodiment, states of the network devices are updated and maintained internally in the test system. Initial state of the network devices may be defined in the respective test case. Additionally or alternatively, the states of the network devices may be reset to initial setting between test cases (i.e. before starting the next test case). In an embodiment, where a chain of interrelated test cases are executed, states of network devices of the telecommunication network are in general updated and maintained over the chain of test cases without resetting the states between the cases. The states of the network devices may be updated according to commands from the network control system. Additionally or alternatively, predefined parameters of the respective test case may be taken into account in updating the states of the network devices. Updating and maintaining states of the network devices provides that the test system emulates operation of the telecommunication network.

In another embodiment, the test system collects the states of the network devices over the second interface form the production environment. In this way, an actually existing combination of states of network devices can be obtained. Thereby, improved emulation of operation of the telecommunication network may be achieved. This option suits well with the test cases triggered by alerts and/or network performance data received from the production environment. By triggering the test cases by alerts and/or network performance data received from the production environment and using states collected from the production environment, the test system tends to test operation of the network control system in real situations that actually exist in the production environment.

In some embodiments, the command specifies one or more actions in a network device of the telecommunication network e.g. for the purpose of failure correction. The command may relate to one or more of the following: registering to a network device, resetting a network device, shutting down a network device, login to a network device, blocking a cell.

<NUM>: Based on the received command, one or more responses are provided, over the first interface, to the network control system. Optionally the states of the network devices may be taken into account in the responses.

The responses may be selected based on states of network devices of the telecommunication network as maintained in the test system or based on states collected from the production environment. Additionally or alternatively, the responses may be selected at least partially based on predefined parameters of the respective test case.

In a yet further embodiment, the test system interacts with the production environment over the second interface to check how the production environment would respond to the command. In an embodiment, this is applied only to certain type of commands, such as registering to a network device, but not to some that may cause unintended interruptions in the production environment, such as resetting a network device or blocking a cell.

In some embodiments, the responses may comprise indication of one or more of the following: successful operation, failed operation, successful registration to the network device, failed registration to the network device, successful reset operation, failed reset operation, successful shut down operation, failed shut down operation, successful login operation, failed login operation, successful cell block operation, failed cell block operation, failed connection to the network device, unexpectedly dropped connection to a network device, network device not found, unexpected error.

<NUM>: Next actions of the network control system are then monitored. The next actions may comprise further commands that specify action(s) in the telecommunication network. Again, the monitoring may be performed for a predefined period of time.

<NUM>: Operation of the network control system is evaluated based on the one or more next actions or lack of next actions of phase <NUM>. Further, the evaluation may be based on the command and/or lack of commands of phase <NUM>.

The evaluation of the operation of the network control system may be based on checking type, syntax and/or timing of the command(s) and/or next action(s). Checking the type may comprise checking that an expected command or action is submitted and/or that a command or action is directed to an expected network device. Lack of expected command or action may be an indication of malfunctioning network control system. Likewise, wrong type, syntax and/or timing of the command(s) and/or next action(s) may be an indication of malfunctioning network control system. The test system may comprise rules that define expected command(s) and actions(s) in different cases and the evaluation may be performed based on such rules.

Results of the evaluation may be output from the test system as such without further processing. That is, the test system may output a listing of test cases and respective information on whether the type, syntax and/or timing of the responsive commands was expected or not. Alternatively, the test system may output an indication of the network control system operating as expected or not operating as expected.

Phase <NUM> provides conditions that the network control system would normally see in production environment. In this way, the test system emulates operation of the telecommunication network.

The following lists some example combinations of commands from the network control system and responses that the test system may provide depending on the case:.

Some example embodiments provide possibility to freely define test cases to be executed. There may be for example a testing interface though which tests can be defined and execution of the tests can be monitored. Some example embodiments that involve test system interacting with the production environment provide possibility to monitor operation of the network control system in situations that substantially correspond to real situations in the production environment.

As an illustrative non-limiting example, a test case is defined for confirming whether the network control system performs resetting of a base station as expected. First, alert(s), which should result in base station reset, are provided to the network control system (either from the production environment or from the test system). Then the test system waits until reset counter in the target base station is incremented by <NUM>. After this, execution of the test case may continue to next phase. In this way, minor delay in the reset operation (or other expected operation) does not fail the test.

However, if the expected reset is not detected, it is concluded that the test failed and the network control system does not operate as intended.

<FIG> shows a flow diagram of an example case. <FIG> shows a network control system, a test system and a telecommunication network. A dashed line <NUM> illustrates a first interface of the test system and a dashed line <NUM> illustrates a second interface of the test system. The first interface is configured for interacting with the network control system and the second interface is configured for interacting with the telecommunication network.

The shown example case operates as follows:.

Additionally, the snapshot of base station states may be provided to data researchers and domain experts for further analysis. In this way, the collected states may be used in further analysis and network planning.

<NUM>: Based on the base station states, a registering successful response is provided to the network control system over the first interface.

<NUM>: The network control system determines an action for the base station x. The action in this example is to reset the base station x.

<NUM>: The network control system sends a command to reset the base station x. The command is received at the first interface of the test system. This is a next action monitored by the test system.

<NUM>: The test system resets the base station x in the current configuration of the test system. That is, the test system changes state of the base station x in the test system.

<NUM>: The test system returns a reset successful response to the network control system. In an alternative case, the test system could return a reset failed response to the network control system in order to monitor, how the network control system reacts to such situation.

<NUM>: The network control system continues with determining a next action and the test system monitors whether expected next action is taken.

The test system evaluates operation of the network control system based on the commands <NUM> and <NUM> and possible further commands seen after phase <NUM>. For example, if the command <NUM> to reset the base station x would not be seen or some other command would be seen instead of the command <NUM> to reset the base station x, the test system may conclude that the network control system failed the test.

It is to be noted that <FIG> shows only one example flow of actions. It is clear that many other alternative options are possible.

Without in any way limiting the scope, interpretation, or application of the claims appearing below, a technical effect of one or more of the example embodiments disclosed herein is an effective method to test automatic monitoring and control of telecommunication networks. By the test system of various embodiments, testing in real world production environment is not needed. From the network control system perspective, the first interface of the test system is equal to control interface of the telecommunication network and therefore substantial changes are not needed in the network control system in order to execute the tests. Merely a pointer to the test system may be needed instead of a pointer to the control interface of the telecommunication network. Once the network control system has been tested using the test system of present disclosure, the network control system may be taken into actual use in the production environment.

Various embodiments have been presented. It should be appreciated that in this document, words comprise, include and contain are each used as open-ended expressions with no intended exclusivity.

Claim 1:
A computer implemented method for providing a test system (<NUM>) for testing a network control system (<NUM>), characterized in that the network control system is configured to automatically analyze alerts and other incidents from a telecommunication network (<NUM>) and to perform automatic actions and failure correction in the telecommunication network (<NUM>) based on this information, the method comprising performing in the test system:
interacting (<NUM>) with the network control system (<NUM>) over a first interface of the test system, wherein the first interface is configured to emulate operation of a control interface of the telecommunication network;
monitoring (<NUM>) over the first interface, whether a command associated with a network device of the telecommunication network is received from the network control system;
based on the received command, providing (<NUM>), over the first interface, to the network control system, one or more responses;
monitoring (<NUM>) one or more next actions of the network control system; and
evaluating (<NUM>) operation of the network control system based on the received command or lack of received command and/or based on the one or more next actions or lack of next actions.