Patent ID: 12229677

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

Cellular networks may suffer from an array of network issues (e.g., degrading hardware, misconfigurations between network elements, unreliable updates or upgrades to network equipment, etc.). The network issues may impact network performance and cause users of a cellular network (i.e., subscribers of a cellular network) to have a poor user experience with the cellular network. The poor user experience may result in user frustration and perhaps even a user switching network operators (i.e., network providers) as a means to resolve the network performance issues.

Network providers (or operators) have an incentive to address these issues because network issues may affect their customer loyalty and may have a detrimental impact on their cellular services. Without resolving network issues, these issues could cost network operators business and potentially damage a network operator's goodwill and/or brand. Yet often network operators do not experience the network performance issues firsthand. In other words, users of a cellular network are the ones generally impacted by network performance issues. This means that network operators often may have to rely on the network users to report network issues when they occur. However, there are a few problems with user-reporting to address network issues. First off, network users not only need to recognize that the issues they are experiencing are likely due to their cellular network, but also to take their time to report the issue to the network operator in some manner. Clearly, this approach is not likely to work well for users who fail to recognize that they are experiencing less-than-ideal performance. For instance, a user becomes accustom to below-average network performance or does not realize that the network performance should be better. Here, this type of user may never inform the network operator that a network performance issue is present and simply changes cellular network providers thinking that another provider might result in better performance. In other words, the original cellular provider may never have the opportunity to address the problem. Furthermore, when a user does report a network performance issue to a network operator, the network operator performs an investigation of the reported issue. These investigations may be a labor intensive process that may leave some user issues unsolved due to a lack of available resources to investigate/address all reported problems. Particularly, network operators may often have to prioritize labor resources to operating the cellular network rather than investigating reported user issues.

Another approach is that the network operator monitors the cellular network to detect anomalies that may indicate a network performance issue. An anomaly refers to a unique occurrence (or different behavior) during signaling for a cellular network. Here, an anomaly itself is agnostic as to whether the unique occurrence is an occurrence that indicates detrimental behavior (e.g., a network performance issue) or an occurrence that indicates non-detrimental behavior (e.g., not a network performance issue). Yet by identifying anomalies, a network operator may analyze an anomaly to determine whether the anomaly corresponds to a network performance issue.

Detecting anomalies within a cellular network has traditionally had its drawbacks. For instance, depending on the cellular usage and traffic, cellular networks could have an immense amount of log data (e.g., network logs, inter-process logs, usage statistics, etc.). Sifting through the immense amounts of data to identify an anomaly may be resource intensive. Therefore, when an anomaly was detected that impacted network performance, an entity detecting the anomaly (e.g., the network operator) may develop a rule to more easily detect the same or similar anomaly in other instances. This traditional form of anomaly detection therefore generates one or more rules to identify a deviation from normal behavior. For instance, a rule defines that a certain message type typically occurs at a rate of five times a second. When that certain message type occurs more or less times per second, this rule would allow a system to detect this deviation as an anomaly. Unfortunately, the issue with this form of anomaly detection is that an entity must first specify what is considered normal behavior to identify anomalies with behavior outside of the specified normality. Here, this method only works for known anomalies dictated by known rules. In other words, a new anomaly that impacts network performance will be undetected until a rule specifically addresses the new anomaly (or the normal behavior that should be occurring instead of the new anomaly). This approach lacks any ability to be predictive for new anomalies that may cause performance issues. Thus, a predictive anomaly detector may more accurately use anomalies to detect network performance issues.

FIG.1illustrates a communication network100(also referred to as a cellular network), which may be a Long-Term Evolution (LTE) network, a 5G network, and/or a multiple access network supporting numerous access technologies specified by the 3rd Generation Partnership Project (3GPP), such as the General Packet Radio Service (GPRS), the Global System for Mobile Communications/Enhanced Data Rates for GSM Evolution (GSM/EDGE), the Universal Mobile Telecommunication System/High Speed Packet Access (UMTS/HSPA), LTE and LTE advanced network technologies. The cellular network100(e.g., LTE network) enables wireless communication of high-speed data packets between subscriber devices102,102a-b, such as mobile phones and data terminals, and a base station104. The subscriber devices102may be interchangeably referred to as user equipment (UE) devices and/or mobile devices102. For instance, LTE is a wireless communication standard that is based on the GSM/EDGE and UMTS/HSPA network technologies and configured to increase the capacity and speed of the telecommunication by using different radio interfaces in addition to core network improvements. Different types of cellular networks100may support different bands/frequencies at various bandwidths to allow UE devices102to communicate data (e.g., data packets). To illustrate, LTE supports scalable carrier bandwidths, from 1.4 MHz to 20 MHz and supports both frequency division duplexing (FDD) and time-division duplexing (TDD) while 5G supports bandwidths ranging from 5 MHz to 100 MHz where some bandwidths overlap with LTE.

UE devices102may be any telecommunication device that is capable of transmitting and/or receiving voice/data over the network100. UE devices102may include, but are not limited to, mobile computing devices, such as laptops, tablets, smart phones, and wearable computing devices (e.g., headsets and/or watches). UE devices102may also include other computing devices having other form factors, such as computing devices included in desktop computers, smart speakers/displays, vehicles, gaming devices, televisions, or other appliances (e.g., networked home automation devices and home appliances). UE devices102subscribe to network services provided by a network operator of the communication network100. The network operator may also be referred to as a mobile network operator (MNO), a wireless service provider, wireless carrier, cellular company, or mobile network carrier.

The UE devices102may communicate with an external network30, such as a packet data network (PDN), through the communication network100(or 5G/3G/2G network). Referring toFIG.1, the communication network100is an LTE network that includes a first portion, an Evolved Universal Terrestrial Radio Access Network (e-UTRAN) portion106, and a second portion, an Evolved Packet Core (EPC) portion108. The first portion106includes an air interface110(i.e., Evolved Universal Terrestrial Radio Access (e-UTRA)) of 3GPP's LTE upgrade path for mobile networks, UE devices102, and multiple base stations104. The LTE air interface110uses orthogonal frequency-division multiple access (OFDMA) radio-access for the downlink and Single-carrier FDMA (SC-FDMA) for the uplink. Accordingly, the first portion106provides a radio access network (RAN) that supports radio communication of data packets and/or other surfaces from the external network30to the UE devices102over the air interface110via one or more base station104.

Each base station104may include an evolved Node B (also referred as eNode B or eNB). An eNB104includes hardware that connects to the air interface110(e.g., a mobile phone network) for communicating directly with the UE devices102. For instance, the eNB104may transmit downlink LTE/3G/5G signals (e.g., communications) to the UE devices102and receive uplink LTE/3G/5G signals from the UE devices102over the air interface110. A base station104may have an associated coverage area104area that corresponds to an area where one or more UE devices102communicate with the network100by way of the base station104. The eNBs104use a S1 interface for communicating with the EPC108. The S1 interface may include an S1-MME interface for communicating with a Mobility Management Entity (MME)112and an S1-U interface for interfacing with a Serving Gateway (SGW)116. Accordingly, the S1 interface is associated with a backhaul link for communicating with the EPC108.

The EPC108provides a framework configured to converge voice and data on the LTE network100. The EPC108unifies voice and data on an Internet Protocol (IP) service architecture and voice is treated as just another IP application. The EPC108includes, without limitation, several network elements, such as the MME112, a Serving GPRS Support Node (SGSN)114, the SGW116, a Policy and Charging Rules Function (PCRF)118, a Home Subscriber Server (HSS)120, and a Packet Data Node Gateway (PGW)122, The PGW122may also be referred to as a network gateway device122, and when the network corresponds to a 3G network, the network gateway device122includes a Gateway GPRS Support Node (GGSN) instead of the PGW122. Optionally, when the network corresponds to a 5G or 5G+ network, the network gateway device122may include a gateway node with a naming convention as defined by the 5G and/or 5G+ network. The MIME112, the SGSN114, the SGW116, the PCRF118, the HSS120, and the PGW122may be standalone components, or at least two of the components may be integrated together. The EPC108communicates with the UE devices102and the external network30to route data packets therebetween.

The network100includes interfaces that allow the UE devices102, the base stations104, and various network elements (e.g., the MME112, the SGSN114, the SGW116, the PCRF118, the HSS120, and the PGW122) to cooperate with each other during use of the network100. Information flows along these interfaces throughout the network100and generally these interfaces may be divided into a user plane and a control plane. The user plane routes user plane traffic and includes a user plane protocol stack between the UE devices102and the base station104with sublayers, such as packet data convergence protocol (PDCP), radio link control (RLC), and medium access control (MAC). Some interfaces specific to the user plane, shown in solid lines between the network elements, are as follows: a S1-U interface between the base station104and the SGW116for per bearer user plane tunneling and inter base station path switching during handover; a S4 interface between a UE device102with 2G access or 3G access and the PGW122for control and mobility support and, in some cases, user plane tunneling; and a S12 interface (not shown) between the E-UTRAN portion106(e.g., UE device102) and the SGW116for user plane tunneling as an operator configuration option. Other types of communication networks (e.g., 3G, 5G, etc.) may include other user plane interfaces besides the ones depicted inFIG.1for the network100.

The control plane is responsible for controlling and supporting user plane functions with control plane protocols. Particularly, the control plane controls E-UTRAN access connections (e.g., attaching and detaching from the E-UTRAN portion106of the network100), controls attributes of an established network access connection (e.g., an activation of an IP address), controls routing paths of an established network connection (e.g., to support user mobility), and/or controls an assignment of network resources based on demands to the network100(e.g., by a user of a UE device102). Some interfaces specific to the control plane, shown in dotted lines between network elements, are as follows: a S1-MME interface between the base station104and the MME112that guarantees delivery of signaling messages; a S3 interface between the SGSN114and the MME112that enables user/bearer information exchange for inter 3GPP access network mobility in idle and/or active states; a S5/S8 interface between the SGW116and the PGW122where the S5 interface is used in a non-roaming scenario to serve relocation based on UE device102mobility and to connect to a non-collocated gateway of a PDN while the S8 interface connects to public land mobile networks (PLMN); an S10 interface that coordinates handovers between MMES112; a S11 interface between the MME112and the SGW116for transferring signal messages; a S6a interface between the MME112and the HSS120that enables transfer of subscription and authentication data related to user access; a S6d interface between the HSS120and the SGSN114that also enables transfer of subscription and authentication data related to user access; and a S13 interface (not shown) that supports a UE device102identity check. Other types of communication networks (e.g., 3G, 5G, etc.) may include other control plane interfaces besides the ones depicted inFIG.1for the network100.

When a particular UE device102connects to the network100, one or more control messages128are sent among the various network elements (e.g., between the network elements of the evolved packet core108and the E-UTRAN portion106). For instance, as illustrated byFIG.1, the base station104sends a control message128to the MME112indicating that a new UE device102is attempting to connect to the network100. As another example, the SGW116sends a control message128to the MME112indicating that data from the external network30has arrived for a particular UE device102and that the UE device102needs to be awoken (or paged) to establish tunnels in order to accept the waiting data. The control plane interfaces may transmit such control messages128using control plane protocols, such as a general packet radio service tunneling control (GTP-C) protocol or a Diameter protocol. The type of protocol used to transmit a control message128may depend on the interface. For instance, the S3, S5/S8, and S10 interfaces use GTP-C protocol while the S11, S6a, S6d, and S13 interfaces use Diameter protocol.

The MME112is a key control-node for the LTE network100. The MME112manages sessions and states and authenticates and tracks a UE device102across the network100. For instance, the MME112may perform various functions such as, but not limited to, control of signaling and security for a Non Access Stratum (NAS), authentication and mobility management of UE devices102, selection of gateways for UE devices102, and bearer management functions. The SGSN114may act in some ways similar to the MME112. For instance, the SGSN114tracks the location of a UE device102and performs security and access control functions. In some examples, the SGSN114is responsible for mobility management (e.g., of a standby mode UE device102), logical link management, authentication, charging functions, and/or handling overload situations. The SGW116performs various functions related to IP data transfer for user devices102, such as data routing and forwarding, as well as mobility anchoring. The SGW116may perform functions such as buffering, routing, and forwarding of data packets for mobile devices102.

The PCRF118is a node responsible for real-time policy rules and charging in the EPC108. In some examples, the PCRF118is configured to access subscriber databases (i.e., UE device users) to make policy decisions. Quality of service management may be controlled by dynamic policy interactions between the PCRF118and the network gateway device122. Signaling by the PCRF118may establish or modify attributes of an EPS bearer (i.e., a virtual connection between the UE device102and the PGW122). In some configurations, such as voice over LTE (VoLTE), the PCRF118allocates network resources for establishing calls and distributing requested bandwidth to a call bearer with configured attributes.

The HSS120refers to a database of all UE devices102that includes all UE device user data. Generally, the HSS120is responsible for authentication for call and session setup. In other words, the HSS120is configured to transfer subscription and authentication data for user access and UE context authentication. The HSS120interacts with the MME112to authenticate the UE device102and/or UE device user. The MME communicates with the HSS120on the PLMN using Diameter protocol (e.g., via the S6a interface).

The PGW122(i.e., network gateway device) performs various functions such as, but not limited to, internet protocol (IP) address allocation, maintenance of data connectivity for UE devices102, packet filtering for UE devices102, service level gating control and rate enforcement, dynamic host configuration protocol (DHCP) functions for clients and servers, and gateway general packet radio service (GGSN) functionality.

In some implementations, data processing hardware124of the network gateway device122(e.g., PGW or GGSN or a gateway node with another naming convention as defined by 5G and/or 5G+ networks) receives control messages128associated with at least one UE device102. The data processing hardware124may receive the control messages128based on interaction(s) that at least one UE device102has with the network100within the coverage area104area of the base station104.

Referring further toFIG.1, the communication network100also includes an anomaly detector200. In some examples, the anomaly detector200is part of the network gateway device122(e.g., PGW or GGSN or a gateway node with another naming convention as defined by 5G and/or 5G+ networks). For instance, data processing hardware124and/or memory hardware126of the network gateway device122host the anomaly detector200and execute the functionality of the anomaly detector200. In some implementations, the anomaly detector200communicates with the E-UTRAN portion106and the EPC108, but resides on the external network30(e.g., data processing hardware corresponding to the external network30). In other words, the external network30may be a distributed system (e.g., a cloud environment) with its own data processing hardware or shared data processing hardware (e.g., shared with the network gateway device122). In other configurations, a network element other than the network gateway device122implements the anomaly detector200. Additionally or alternatively, the anomaly detector200resides across more than one network element of the network100.

Generally, the anomaly detector200is configured to detect anomalies that occur within the network100based on one or more control messages128. With a detected anomaly, the anomaly detector200analyzes whether the anomaly corresponds to a network performance issue202that impacts a performance of the network100. In other words, the anomaly detector200identifies a unique occurrence (i.e., the anomaly) within the network100and determines whether the unique occurrence is detrimental to the performance of the network100(or negatively impacts a user experience). When the anomaly detector200identifies that the detected anomaly impacts network performance, the anomaly detector200is configured to inform a network entity40responsible for the network performance issue202or relay the network performance issue202to an entity that knows or communicates with the responsible entity. For instance, the anomaly detector200may signal or inform the network operator of the network performance issue202corresponding to the detected anomaly. In some implementations, the anomaly detector200communicates the one or more control messages128that indicated the network anomaly to the network entity40. Here, the network entity40may further analyze one or more control messages128to help resolve the network issue202.

Referring toFIGS.2A-2D, the anomaly detector200generally includes a collector210, an extractor220, a predictor230, and an analyzer240. The collector210is configured to receive at least one control message128from the network100. In some implementations, the collector210includes a datastore212to collect control messages128from the network100in order to function as a central database for logging data corresponding to the control messages128. With the collector210, the anomaly detector200may process the control messages128in a variety of ways to create training data (e.g., training control messages) that may be used to detect anomalies. For instance, the collector210groups together (e.g., within the datastore212) control messages128from a single session of a UE device102. In some examples, a session refers to a time period from when a user (via the UE device102) initiates a CreateSessionRequest or CreatePdpRequest message to when the user terminates the session with a Delete SessionResponse or DeletePdpContextRequest message. As another example, the collector210groups control messages128together to indicate an amount of data129that was transferred (e.g., either in an uplink direction, a downlink direction, or both) within a certain time period (e.g., during a session). With these control messages128grouped together, the collector210forms a representation of a total amount of data129for a certain time period.

In other configurations, the collector210collects the log data as a sequence such that the control messages128are strung together as a time series (e.g., t0-t3. Here, the string of control messages128may be aggregated by an entity (e.g., a particular user or UE device102) or by sessions of the entity. If these sequences become too long, the collector210may be configured to dissect these sequences into sub-sequences of a fixed length and associate any identifiers of the original sequence to each sub-sequence. Otherwise, a sequence may have a label (e.g., a particular entity or UE device102) that when the collector210dissects the sequence would fail to transfer to one or more sub-sequences.

The extractor220is configured to extract information from one or more control messages128and/or log data corresponding to control messages128. The extractor220may extract one or more features222and/or one or more labels224from the one or more control messages128(or parts thereof). Each feature222and/or label224refers to a characteristic derived from a control message128. In some examples, a label224is a characteristic of a network element, a UE device102, a user of a UE device, or a base station104that is generally obfuscated due to 3GPP standardization of the network100. In other words, although the extractor220may generate an actual label224directly from a control message128(or log data relating to a control message128), it should not be possible to contextually determine the actual label224simply from one or more control message128when the network100is 3GPP compliant. One such example of a label224is a type allocation code (TAC) that identifies a wireless device (e.g., a mobile phone type of a UE device102). Other examples of labels224may include, without limitation, identifiers corresponding to network elements of the network100(e.g., a MME identifier, a base station identity code (B SIC), an international mobile equipment identity (IMEI), E-UTRAN cell identity (ECI)/E-UTRAN cell global identifier (ECGI), etc.)

On the other hand, a feature222corresponds to another characteristic derived from a control message128that is different than the characteristic forming the label224. Here, unlike for a label224, a feature222of a control message128may be discernable even when the network100is 3GPP compliant. Some examples of features222include a control message type (e.g., represented as an integer), a cause type for GTP-C messages, an amount of time elapsed between adjacent messages (e.g., when the collector210sequences the control messages128), etc. In some examples, the extractor220extracts different features222from different control message protocols. For instance, features222extracted from GTP-C messages would be different than features222extracted from Diameter messages. In some examples, features222extracted by the extractor220are crossed to create new features222. A cross of features222refers to a combination of a portion of two or more features222. For example, the extractor220crosses the message type feature222and the cause value feature222to generate a message type-cause value feature222. By crossing features222, the extractor220may provide additional training data sets potentially increasing the ability of the anomaly detector200to detect anomalies.

Whether the extractor220extracts a feature222and/or a label224may depend on a stage of the anomaly detector200. In a first stage (e.g., training stage), the anomaly detector200trains to be able to predict network anomalies. In order to train the anomaly detector200, the extractor220extracts information from one or more control messages128at the collector210. The extracted information forms a training control message226that includes one or more features222and an actual label224. By including the actual label224as a ground truth with the training control message226, the anomaly detector200learns which features222may correspond to which label224. In a second stage (e.g., inference), after the anomaly detector200is trained, the extractor220no longer provides training control messages226with both features222and a label224. Instead, the extractor220extracts one or more features222from a control message128and relies on the trained anomaly detector200to predict the label224. In other words, as processing each control message128to extract an actual label224therefrom is time-sensitive, and therefore not practical in real-time, the trained anomaly detector200may predict potential labels234using only the features22extracted from the control message128as feature inputs.

The predictor230is configured to use a predictive model232to predict a potential label234for a control message128associated with the one or more features222extracted from the control message128by the extractor220. Ideally, because of the standardization of 3GPP, it should not be possible for the predictor230to generate a prediction P where the potential label234matches (i.e., correctly predicts) the actual label224for a given control message128. Thus, when the predictor230predicts a potential label234that matches the actual label224from at least one control message128(e.g., features222of a control message128), this match indicates a unique correlation (i.e., a detected anomaly) between the control message(s)128and the labels224,234.

When the predictor230generates a correct prediction P, the analyzer240analyzes the related control message128and/or the log data corresponding to the control message128. Here, the analyzer240analyzes the control message128to determine whether the control message128corresponds to a network performance issue202impacting network performance of the network100. In other words, the analyzer240determines whether the detected anomaly is a unique correlation due to detrimental behavior or whether the detected anomaly is simply unique behavior with little to no impact on network performance or user experience. When the analyzer240determines that the detected anomaly of the control message128impacts network performance, the analyzer240flags this detrimental behavior to be fixed. To fix the behavior, the analyzer240may communicate the network performance issue202to the network entity40(e.g., a network operator or a UE device provider) responsible for the network performance issue202.

In some configurations, the analyzer240performs clustering. Clustering may be beneficial where there are too many anomalies occurring with a network100to investigate. Instead of investigating each and every detected anomaly, the analyzer240clusters the detected anomalies into similar groups. By clustering into groups, the analyzer240may prioritize larger clusters that potentially may have more detrimental impact on the network100(e.g., ranking clusters by network impact or likelihood/probability of network impact). Furthermore, when the analyzer240relies on human analysis to determine whether or not the detected anomaly corresponds to a network performance issue202, the analyzer240may use an autoencoder to perform dimensionality reduction. Dimensionality reduction by an autoencoder is configured to reduce large data sets (i.e., a large number of anomalies) by correlating redundant features in the large data sets. Here, as a neural network trained according to gradient descent, an autoencoder performs dimensionality reduction by trying to identify new structures or uniqueness in a data set. In other words, the autoencoder may isolate more unique anomalies for the network100that may more likely correlate to network performance issues202that should be analyzed. By combining clustering and autoencoding, a large number of anomalies may be formed into smaller groups (clusters) and then further reduced to make efficient use of human and/or computations resources.

The predictor230predicts the potential label234using the predictive model232. In some examples, the predictive model232is a neural network (e.g., a deep neural network (DNN), a recurrent neural network (RNN), or a convolution neural network (CNN)). To generate predictions P, the predictive model232undergoes model training. Here, training for the predictive model232occurs using examples (also referred to as training data or a training data set) that correspond to control messages128and/or their related log data. In some implementations, the extractor220generates a set228of training control messages226as examples to train the predictive model232(e.g., shown inFIG.2B). In some configurations, each training control message226corresponds to a control message128processed at the collector210. The extractor220may form each training control message226by extracting one or more features222from a control message128along with the actual label224for the control message128. In some examples, when more than one control message128has the same label224, the features222of these control messages128are combined into one example or set228of training control messages226. For example, the extractor220creates a message type vector summary to account for each type of control message128included in a combination. The message type vector summary may include one entry for each possible message type to represent a number of times that a particular control message128was encountered (e.g., within a single session).

In order to train the predictive model232, the predictor230divides the set228of training control messages226into a training set226Tand validation set226V. In some examples, in addition to the training set226Tand validation set226V, the training control messages226are also split into a test set. The predictive model232trains on the training set226Twhile using the validation set226Vto determine when to stop training (e.g., to prevent over-fit). The training may stop when a performance of the predictive model232reaches a particular threshold or when the performance of the predictive model232on the validation set226Vstops decreasing. In some examples, the training set226Tevaluates the final performance for the predictive model232. In some implementations, the predictive model232is trained as a multiclass classification model. As a multiclass classification model, the predictive model232outputs a probability distribution PBdisrepresenting an opinion regarding the probability PBfor each class. For instance, when the predictive model232predicts TAC, each TAC will be a different class such that the predictive model232will output a probability distribution for each class of TAC.

In some examples, the process of training and evaluating the predictive model232occurs continuously to provide early detection of new network issues202that may arise. Once the training is complete, predictions P from the training may be fed back into the predictive model232. These predictions P may correspond to the training sets226T, the validations sets226V, the test sets, or any combination thereof. In other words, the predictive model232is configured to evaluate its predictions P from training on the training data (e.g., the set228of training control messages226). This approach may ensure the predictive model232has completed training and is ready to predict potential labels234

With reference toFIGS.2B and2D, in some examples, the predictive model232of the predictor230generates a probability PBfor a prediction P of a potential label234. To evaluate the probability PBof the potential label234, the predictor230may apply a confidence threshold236. The confidence threshold236indicates a level of confidence that the probability PBof the potential label234corresponds to an anomaly that requires evaluation by the analyzer240for detrimental behavior. In other words, when the prediction probability PBof the potential label234satisfies the confidence threshold236, the predictor230communicates the control message128corresponding to the potential label234to the analyzer240. For instance, when the confidence threshold236is 90%, a probability PBfor a prediction P of a potential label234indicative of a TAC that is greater than 90% indicates a confident prediction P that should pass to the analyzer240to be further analyzed.

In some configurations, the predictive model232outputs/predicts a probability distribution PBdisover potential labels234a-n. In these configurations, each potential label234a-nin the probability distribution PBdisincludes a corresponding probability PB. In some examples, the predictor230predicts the potential label234by selecting the potential label234a-nhaving the highest probability PBin the probability distribution PBdisover potential labels234a-n. In the example shown inFIGS.2B and2D, the potential label234ahas the highest probability PBof ninety-one percent (91%) in the probability distribution PBdisover potential labels234a-n, and therefore the predictor230selects the potential label234aand compares the probability PB(91%) to the confidence threshold (90%). Thus, in the example, the predictor230determines that the probability PBof the selected potential label234asatisfies the confidence threshold236and passes the corresponding control message128to the analyzer240to determine whether the control message128corresponds to a respective network performance issue202impacting network performance. In some scenarios, the predictor230communicates to the analyzer240each potential label234a-nin the in the probability distribution PBdisthat has a corresponding probability PBsatisfying the confidence threshold236.

In some configurations, the predictive model232is an RNN model that is better suited (than a DNN model) for sequential data. For an RNN model, the extractor220generates sequences for the features222. In other words, the extractor220may form the training control messages226from sequential control messages128(or sequential features222from sequential control messages128). With sequential features222, each sequence may be a training example such that sequential features222would be split into a training data set, a validation data set, and a test data set. Besides preferring sequential data, the RNN model operates relatively similar to the previously described predictive model232.

In some examples, the predictive model232has difficulty distinguishing different potential labels234that perform similarly. For instance, when predicting TAC, there may be several TACs (e.g., three TACs) that perform identically. This identical behavior results in the predictive model232confidently knowing that the TAC is one of the three TACs, but not being able to predict exactly which TAC. To overcome this issue, the predictor230may use principal component analysis (PCA) to identify groupings of labels234that perform similarly (e.g., like the three TACs). Using PCA, the prediction P of the potential label234may be a vector where PCA identifies which groupings of labels224are commonly predicted together. For example, the PCA will identify that the three TACs should be considered together because the principal component vectors of these three TACs will have strong peaks indicating that they should be grouped (or considered) together.

Referring toFIGS.2C and2D, the anomaly detector200may also include a filter250. The filter250is configured to prevent redundant analysis of similar detected anomalies. In other words, the anomaly detector200generates a filter250when an anomaly has been detected. The filter250may be for an anomaly of detrimental behavior or for an anomaly of non-detrimental behavior (i.e., acceptable behavior). Once the analyzer240has determined whether or not a control message128corresponding to an anomaly affects network performance, performing this same analysis for a similar control message128or sequence of similar control messages128may defer anomaly detection resources from detecting new anomalies or anomalies that need to be analyzed. Accordingly, the filter250attempts to prevent repeat analysis. For instance, when the analyzer240determines a control message128corresponds to a respective network issue202that affects network performance, the respective network issue202and/or control message128is reported to the responsible network entity40. Here, it would be redundant to re-analyze and report similar control messages128to the network entity40because the respective network issue202has been reported and will be addressed by the responsible network entity40in due course. On the other hand, when the analyzer240determines a control message does not affect network performance, the anomaly associated with the control message128is non-detrimental, and therefore acceptable. Accordingly, it would be pointless to re-analyze subsequent similar control messages128.

The anomaly detector200may generally apply the filter250in two scenarios: (1) on features222extracted from control messages128prior to input to the predictive model232; or (2) on the set228of training control messages226used to train the predictive model232. In some examples (i.e., the first scenario), the anomaly detector200applies the filter250after the predictive model232has been trained, but before one or more features222extracted from a subsequent control message128are input to the trained predictive model232for prediction P of a subsequent potential label234. Here, the anomaly detector200identifies that at least one of the one or more of the corresponding features222extracted from the subsequent control message128match the one or more features222extracted from a previous control message128having a predicted potential label234indicative of a network anomaly, (i.e., the predicted potential label234satisfies a confidence threshold236). Thereafter, prior to using the predictive model232to predict a corresponding potential label234for the subsequent control message128, the anomaly detector200applies the filter250to remove the identified at least one of the one or more corresponding features222extracted from the subsequent control message128from use as feature inputs to the predictive model232. Accordingly, any prediction P output by the predictive model232at the predictor230for a potential label234will not be based on features222extracted from previous control messages128having predicted potential labels234indicative of a network anomaly, regardless of whether the analyzer240determined the network anomaly was non-detrimental or impacted network performance. For example,FIG.2Cillustrates the filter250in grey blocking and/or removing one of the three features222being communicated to the predictor230to predict a potential label234for a subsequent control message128.

In other examples (i.e., the second scenario), such as inFIG.2D, the anomaly detector200re-trains the predictive model232so that any features222extracted from control messages128previously identified as having a prediction P of a potential label234indicative of a network anomaly are removed from the set228of training control messages226. This approach may also be applicable whether or not the control message128corresponds to a network performance issue202. To re-train the predictive model232, the anomaly detector200first identifies the one or more features222extracted from a prior control message128having a potential label234indicative of the network anomaly. Then, prior to using the predictive model232to predict P a corresponding potential label234for a subsequent control message128, the anomaly detector200modifies the set228of training control messages226by removing each training control message226that includes one or more corresponding features222that match any of the identified one or more features222extracted from the prior control message128. Thereafter, anomaly detector200re-trains the predictive model232on the modified set228of training control messages226. For instance,FIG.2Ddepicts the filter250modifying the set228of training control messages226by removing one of the three training control messages226from a retraining set (i.e., modified set228) of training control messages226. Once the one or more training control messages226have been removed, the filter250retrains the predictive model232one the modified set228of training control messages226. In other words, if the predictive model232is not trained to detect which features222are indicative of an anomaly, the anomaly will subsequently be undetected, and thus ignored.

Additionally or alternatively, when a detected anomaly indicates a respective network performance issue202and the network performance issue202has subsequently been resolved, the anomaly detector200may be configured to remove any filter250relating to the resolved network performance issue202. In configurations where the predictive model232is an RNN model, the anomaly detector200may selectively apply a filter250. In other words, rather than removing an entire sequence as a feature222, the filter250may remove part of a sequence of the feature222that correspond to a particular control message(s)128of a detected anomaly. Advantageously, the filter250may remove this part of the sequence before the sequence splits into smaller sequences. For instance, when the filter250identifies when there are too many CreateSessionRequest messages with a small time period, these individual messages can be completely or partially removed.

FIG.3illustrates a flow diagram of an example method300for detecting network anomalies. At operation302, the method300receives a control message128from a cellular network100. At operations304, the method300extracts one or more features222from the control message128. At operation306, the method300predicts a potential label234for the control message using a predictive model232configured to receive the one or more extracted features222from the control message128as feature inputs. The predictive model232is trained on a set of training control message226where each training control message226includes one or more corresponding features222and an actual label224. At operation308, the method300determines that a probability PBof the potential label234satisfies a confidence threshold236. At operation310, the method300analyzes the control message128to determine whether the control message128corresponds to a respective network performance issue202impacting network performance of the cellular network100. At operation312, when the control message128corresponds to the respective network performance issue impacting network performance, the method300communicates the network performance issue202to a network entity40responsible for the network performance issue202.

In some examples, when the control message128fails to correspond to the respective network performance issue202, the method300receives a subsequent control message128from the cellular network100and extracts one or more corresponding features222from the subsequent control message128. In these examples, the method300also identifies that at least one of the one or more corresponding features222extracted from the subsequent control message128match the one or more features222extracted from the control message128. Here, prior to using the predictive model232to predict a corresponding potential label234for the a subsequent control message, the method300removes the identified at least one of the one or more features222extracted from the subsequent control message128as feature inputs to the predictive model232. In some implementations, when the control message128fails to correspond to the respective network performance issue202, the method300identifies the one or more features222extracted from the control message128. Here, in addition to identifying the one or more features222, the method300, prior to using the predictive model232to predict a corresponding potential label234for a subsequent control message128, modifies the set228of training control messages226by removing each training control message226that includes one or more of corresponding features that match any of the identified one or more features222extracted from the control message128and re-training the predictive model232with the modified set228of training control messages226.

FIG.4is schematic view of an example computing device400that may be used to implement the systems (e.g., the anomaly detector200) and methods (e.g., the method300) described in this document. The computing device400is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed in this document.

The computing device400includes a processor410(i.e., data processing hardware), memory420(i.e., memory hardware), a storage device430, a high-speed interface/controller440connecting to the memory420and high-speed expansion ports450, and a low speed interface/controller460connecting to a low speed bus470and a storage device430. Each of the components410,420,430,440,450, and460, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor410can process instructions for execution within the computing device400, including instructions stored in the memory420or on the storage device430to display graphical information for a graphical user interface (GUI) on an external input/output device, such as display480coupled to high speed interface440. In other implementations, multiple processors and/or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices400may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

The memory420stores information non-transitorily within the computing device400. The memory420may be a computer-readable medium, a volatile memory unit(s), or non-volatile memory unit(s). The non-transitory memory420may be physical devices used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by the computing device400. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM)/programmable read-only memory (PROM)/erasable programmable read-only memory (EPROM)/electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM) as well as disks or tapes.

The storage device430is capable of providing mass storage for the computing device400. In some implementations, the storage device430is a computer-readable medium. In various different implementations, the storage device430may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory420, the storage device430, or memory on processor410.

The high speed controller440manages bandwidth-intensive operations for the computing device400, while the low speed controller460manages lower bandwidth-intensive operations. Such allocation of duties is exemplary only. In some implementations, the high-speed controller440is coupled to the memory420, the display480(e.g., through a graphics processor or accelerator), and to the high-speed expansion ports450, which may accept various expansion cards (not shown). In some implementations, the low-speed controller460is coupled to the storage device430and a low-speed expansion port490. The low-speed expansion port490, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

The computing device400may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server400aor multiple times in a group of such servers400a, as a laptop computer400b, or as part of a rack server system400c.

Various implementations of the systems and techniques described herein can be realized in digital electronic and/or optical circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transitory computer readable medium, apparatus and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and/or data to a programmable processor.

The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, one or more aspects of the disclosure can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen for displaying information to the user and optionally a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the disclosure. Accordingly, other implementations are within the scope of the following claims.