Patent Description:
Wireless communication networks (e.g., cellular networks) provide communication content such as voice, video, packet data, messaging, and broadcast for user equipment (UE), such as mobile devices and data terminals. The communication network may include a number of base stations that can support communication for a number of user equipment across dispersed geographic regions.

In some configurations, wireless networks are rather large and may employ a large number of base stations. These larger networks may have extensive site plans where telecommunication operators deploy many base stations (e.g., thousands of base stations). With respect to these base stations, site plans often dictate base station details, such as antenna location, antenna feeder cables, antenna tilt angle, antenna azimuth, etc. Typically, these details and specifications for site plans have been preconfigured for network performance. Yet during network operation, it is not uncommon for user equipment to experience network issues as a result of site plan deviation. Document <CIT> refers to an apparatus and method for pairing measurements and position estimate as an information pair from multiple mobile devices and reporting these information pairs to a server without over burdening the mobile device and without requiring the mobile devices to establish a new link. Also, disclosed is an apparatus and method for collecting these information pairs and compiling network maps based on the information pairs.

The present invetion is set out in the appended claims.

One aspect of the disclosure provides a method for detecting radio or signal coverage problems. The method includes receiving, at data processing hardware, from at least one user equipment (UE) (e.g., any device capable of receiving signal emissions), observations. Each observation includes a radio signal measurement of a signal (e.g., a radio signal, WiFi signal, etc.) emitted from a base station and a corresponding location of the signal measurement. The method also includes generating, by the data processing hardware, a coverage map for the base station based on the received observations. The coverage map indicates a signal characteristic of the emitted signal about the base station. The method may further include generating, by the data processing hardware, an observation map based on the coverage map and the observations. The method includes determining, by the data processing hardware, an estimated characteristic of the base station by feeding the coverage map into a neural network configured to output the estimated characteristic of the base station. In some examples, the method includes determining, by the data processing hardware, an estimated characteristic of the base station by feeding the coverage map and the observation map into a neural network configured to output the estimated characteristic of the base station. In some implementations, the signal measurement includes a location uncertainty measurement.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, generating the coverage map for the base station includes dividing a coverage area about the base station into pixels, each pixel corresponding to a geographical portion of the coverage area. Here, for each observation, the method may include identifying the pixel having the corresponding geographical portion of the coverage area that contains the location of the signal measurement of the respective observation and associating the observation with the identified pixel. For each pixel, the method may include averaging the signal measurements of any observations associated with the respective pixel. For example, associating the observation with the identified pixel may include placing the observation in a pixel bin associated with the identified pixel.

In some examples, the coverage map includes a grid having cells, and each pixel corresponds to one of the cells. Generating the observation map may also include, for each pixel of the coverage map, generating a metric that monotonically expresses a number of any observations associated with the respective pixel. When generating the coverage map for the base station, the method may include generating a terrain map of a geographical area about the base station and feeding the terrain map into the neural network. The terrain map may describe at least one of a terrain altitude of the geographical area or a presence and/or height of objects extending above a ground surface of the geographical area.

In some configurations, when generating a metric that monotonically expresses a number of any observations associated with the respective pixel, the metric includes at least one of: a number of any observations associated with the respective pixel; a log of the number of any observations associated with the respective pixel; or a monotonic function of the number of any observations associated with the respective pixel. Additionally or alternatively, the metric may include determining a number of any observations associated with the respective pixel, when the number is greater than zero, assigning a value of the respective pixel to one and when the number equals zero, assigning the value of the respective pixel to zero.

In some implementations, the method includes feeding side information into the neural network, the side information including at least one of a frequency of operation of the base station, a height of an antenna of the base station, an antenna beam width, an antenna tilt angle, or a predetermined location of the base station. The characteristic of the base station may include an estimated location of the base station, an estimated pointing direction of the base station, or an antenna azimuth of the base station. Optionally, the neural network may be configured to output a confidence indicator of the estimated characteristic of the base station.

Optionally, the method includes generating, by the data processing hardware, a location uncertainty map based on location uncertainty measurements. The method may also include determining, by the data processing hardware, the estimated characteristic of the base station by feeding the coverage map and the location uncertainty map into the neural network configured to output the estimated characteristic of the base station.

In some examples, generating the location uncertainty map for the base station includes dividing a coverage area about the base station into pixels, each pixel corresponding to a geographical portion of the coverage area. Here, for each observation, the method may include identifying the pixel having the corresponding geographical portion of the coverage area that contains the location of the location uncertainty measurement of the respective observation and associating the observation with the identified pixel. For each pixel, the method may include averaging the location uncertainty measurements of any observations associated with the respective pixel.

Another aspect of the disclosure provides a system for detecting signal coverage problems. The system includes data processing hardware and memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations. The operations includes receiving from at least one user equipment (UE), observations, each observation comprising a signal measurement of a signal emitted from a base station and a corresponding location of the signal measurement. The operations also include generating a coverage map for the base station based on the received observations, the coverage map indicating a signal characteristic of the emitted signal about the base station and generating an observation map based on the coverage map and the observations. The operations may further include generating an observation map based on the coverage map and the observations. The operations also include determining an estimated characteristic of the base station by feeding the coverage map into a neural network configured to output the estimated characteristic of the base station. In some examples, the operations include determining an estimated characteristic of the base station by feeding the coverage map and the observation map into a neural network configured to output the estimated characteristic of the base station. In some implementations, the signal measurement includes a location uncertainty measurement.

This aspect may include one or more of the following optional features. In some examples, the operation of generating the coverage map for the base station includes dividing a coverage area about the base station into pixels, each pixel corresponding to a geographical portion of the coverage area. For each observation, the operation may include identifying the pixel having the corresponding geographical portion of the coverage area that contains the location of the signal measurement of the respective observation and associating the observation with the identified pixel. For each pixel, the operation may include averaging the signal measurements of any observations associated with the respective pixel. For example, associating the observation with the identified pixel may include placing the observation in a pixel bin associated with the identified pixel.

In some implementations, the coverage map includes a grid having cells, and each pixel corresponds to one of the cells. The operation of generating the observation map may include, for each pixel of the coverage map, generating a metric that monotonically expresses a number of any observations associated with the respective pixel. The metric may include at least one of a number of any observations associated with the respective pixel, a log of the number of any observations associated with the respective pixel, or a monotonic function of the number of any observations associated with the respective pixel. Optionally, generating the metric may include determining a number of any observations associated with the respective pixel, when the number is greater than zero, assigning a value of the respective pixel to one and when the number equals zero, assigning the value of the respective pixel to zero.

In some configurations, the operation of generating the coverage map for the base station includes generating a terrain map of a geographical area about the base station and feeding the terrain map into the neural network. The terrain map may describe at least one of a terrain altitude of the geographical area or a presence and/or height of objects extending above a ground surface of the geographical area. Optionally, the operations may include feeding side information into the neural network, the side information including at least one of a frequency of operation of the base station, a height of an antenna of the base station, an antenna beam width, an antenna tilt angle, or a predetermined location of the base station. The estimated characteristic of the base station may include an estimated location of the base station, an estimated pointing direction of the base station, or an antenna azimuth of the base station. The neural network may be configured to output a confidence indicator of the estimated characteristic of the base station.

Optionally, the operation include generating a location uncertainty map based on location uncertainty measurements. The operations may also include determining the estimated characteristic of the base station by feeding the coverage map and the location uncertainty map into the neural network configured to output the estimated characteristic of the base station.

In some examples, generating the location uncertainty map for the base station includes dividing a coverage area about the base station into pixels, each pixel corresponding to a geographical portion of the coverage area. Here, for each observation, the operations may include identifying the pixel having the corresponding geographical portion of the coverage area that contains the location of the location uncertainty measurement of the respective observation and associating the observation with the identified pixel. For each pixel, the system may include averaging the location uncertainty measurements of any observations associated with the respective pixel.

Like any project, telecommunication operators may have to modify site plans during the deployment of base stations. For example, it generally proves difficult to deploy a large number of base stations without some degree of deviation from original site plans. Telecommunication operators may have to move antenna locations (e.g., due to site deployment terrain) or swap feeder cables (e.g., cable supply issues). In other examples, telecommunication operators simply make inadvertent errors when deploying the infrastructure for networks. In the case of large networks, small errors may be amplified over a large site and/or system. Whether these deviations are minor or major, these deviations may affect later network performance during network operation. Issues caused by these deviations may even go undetected for long periods of time. Unfortunately, to verify the operation of each and every transmitter (e.g., base station) is usually prohibitively expensive. As a result, there is a need to detect signal coverage problems during network operation. With user equipment (UE) devices operating within a geographic coverage area of the base station, server devices receiving information from the UE devices may use predictive modeling to detect characteristics for a given base station. An advantage of this UE feedback detection based system is that the network provider or network manager may virtually verify characteristics of network infrastructure (e.g., base stations) without a need to physically verify the network infrastructure. The concepts disclosed may also be applied to signal coverage detection for WiFi access point and other signal emitting devices.

<FIG> depicts an example communication network <NUM>, which may be a Long-Term Evolution (LTE) network, a <NUM> network, and/or a multiple access network supporting numerous access technologies specified by the <NUM>rd Generation Partnership Project (3GPP), such as the General Packet Radio Service (GPRS), the Global System for Mobile Communications/Enhanced Data Rates for GSM Evolution (GSM/EDGE), the Universal Mobile Telecommunication System/High Speed Packet Access (UMTS/HSPA), LTE and LTE advanced network technologies. LTE is a standard for wireless communication of high-speed data packets between user equipment <NUM>, 102a-c, such as mobile phones and data terminals, and base stations <NUM>. LTE is based on the GSM/EDGE and UMTS/HSPA network technologies. LTE is configured to increase the capacity and speed of the telecommunication by using different radio interfaces in addition to core network improvements. LTE supports scalable carrier bandwidths, from <NUM> to <NUM> and supports both frequency division duplexing (FDD) and time-division duplexing (TDD). In other examples, the communication network <NUM> is a WiFi network or other wireless signal network. The user equipment <NUM> may be interchangeably referred to as user equipment (UE) devices and mobile devices <NUM>.

The UE devices <NUM>, 102a-c may communicate with an external network <NUM>, such as a packet data network (PDN), through the communication network <NUM> (or <NUM>/<NUM>/<NUM> network). In the example shown, the network <NUM> includes a first portion, an Evolved Universal Terrestrial Radio Access Network (e-UTRAN) portion <NUM>, and a second portion, an Evolved Packet Core (EPC) portion <NUM>. The first portion <NUM> includes an air interface <NUM> (i.e., Evolved Universal Terrestrial Radio Access (e-UTRA)) of 3GPP's LTE upgrade path for mobile networks, UE devices <NUM>, and base station <NUM>. The LTE air interface <NUM> uses orthogonal frequency-division multiple access (OFDMA) radio-access for the downlink and Single-carrier FDMA (SC-FDMA) for the uplink. Accordingly, the first portion <NUM> provides a radio access network (RAN) that supports radio communication of data packets and/or other surfaces from the external network to the UE devices <NUM> over the air interface <NUM> via one or more base stations <NUM>.

The EPC <NUM> provides a framework configured to converge voice and data on the communication network <NUM>. The EPC <NUM> unifies voice and data on an Internet Protocol (IP) service architecture and voice is treated as just another IP application. The EPC <NUM> includes several key components that include, without limitations, a Mobility Management Entity (MME) <NUM>, a Serving Gateway (SGW) <NUM>, and a Packet Data Node Gateway (PGW) <NUM>. The PGW <NUM> may be referred to as a network gateway device <NUM>, and when the network corresponds to a <NUM> network, the network gateway device <NUM> includes a Gateway GPRS Support Node (GGSN) instead of the PGW <NUM>. Optionally, when the network corresponds to a <NUM> or <NUM>+ network, the network gateway device <NUM> may include a gateway node with a naming convention as defined by the <NUM> and/or <NUM>+ network. The MME <NUM>, the SGW <NUM>, and the PGW <NUM> may be standalone components, or at least two of the components may be integrated together. The EPC <NUM> communicates with the UE devices <NUM> and the external network <NUM> to route data packets therebetween.

The MME <NUM> is a key control-node for the communication network <NUM>. The MME <NUM> manages sessions and states and authenticates and tracks a UE device <NUM> across the network <NUM>. For instance, the MME <NUM> may perform various functions such as, but not limited to, control of signaling and security for a Non Access Stratum (NAS), authentication and mobility management of UE devices <NUM>, selection of gateways for UE devices <NUM>, and bearer management functions. The SGW <NUM> performs various functions related to IP data transfer for UE devices <NUM>, such as data routing and forwarding, as well as mobility anchoring. The SGW <NUM> may perform functions such as buffering, routing, and forwarding of data packets for UE devices <NUM>. The SGW <NUM> and the MME <NUM> also communicate with one another over an S11 interface <NUM>.

The PGW <NUM> (i.e., network gateway device) performs various functions such as, but not limited to, internet protocol (IP) address allocation, maintenance of data connectivity for UE devices <NUM>, packet filtering for UE devices <NUM>, service level gating control and rate enforcement, dynamic host configuration protocol (DHCP) functions for clients and servers, and gateway general packet radio service (GGSN) functionality. The PGW <NUM> communicates with the SGW <NUM> over an S5 interface <NUM>.

Each base station <NUM> has a coverage area <NUM>Area that generally corresponds to a geographic area where a UE device <NUM> may receive a radio signal S emitted from the base station <NUM>. In some implementations, the base station <NUM> includes an evolved Node B (also referred as eNode B or eNB). An eNB <NUM> includes hardware that connects to the air interface <NUM> (e.g., a mobile phone network) for communicating directly with the UE devices <NUM>. For instance, the eNB <NUM> may transmit downlink LTE/<NUM>/<NUM> signals (e.g., communications) to the UE devices <NUM> and receive uplink LTE/<NUM>/<NUM> signals from the UE devices <NUM> over the air interface <NUM>. The eNBs <NUM> use an S1 interface <NUM> for communicating with the EPC <NUM>. The S1 interface <NUM> may include an S1-MME interface for communicating with the MME <NUM> and an S1-U interface for interfacing with the SGW <NUM>. Accordingly, the S1 interface <NUM> is associated with a backhaul link for communicating with the EPC <NUM>. In additional implementations, the base station <NUM> is a wireless access point or other wireless signal emitter.

UE devices <NUM> may be any telecommunication device that is capable of transmitting and/or receiving voice/data over the network <NUM>. UE devices <NUM> may include, but are not limited to, mobile computing devices, such as laptops, tablets, smart phones, and wearable computing devices (e.g., headsets and/or watches). UE devices <NUM> may also include other computing devices having other form factors, such as computing devices included in desktop computers, vehicles, gaming devices, televisions, or other appliances (e.g., networked home automation devices and home appliances).

In some implementations, data processing hardware <NUM> of the network gateway device <NUM> (e.g., PGW or GGSN or a gateway node with another naming convention as defined by <NUM> and/or <NUM>+ networks) receives from at least one UE device <NUM> (e.g., shown as UE 102a-c) observations <NUM> (i.e. observation data). The data processing hardware <NUM> may receive the observations <NUM> based on interaction(s) the at least one UE device <NUM> has with the network <NUM> within the coverage area <NUM>area of the base station <NUM>. Each observation <NUM> includes a radio signal measurement <NUM> of the radio signal S emitted from the base station <NUM> and a corresponding location <NUM> of the radio signal measurement <NUM>. The location <NUM> refers to coordinates of the UE device <NUM> at a time the observation <NUM> is transmitted. For example, the location <NUM> is a global longitude and/or latitude of the UE device <NUM>. In other examples, the location <NUM> refers to a position relative to an object, such as the base station <NUM> emitting the radio signal S measure by an observation <NUM>. In some examples, the radio signal measurement <NUM> is a measurement of signal strength. Some examples of measurements of signal strength include received signal strength indicators (RSSI), reference signal received power (RSRP), reference signal received quality (RSRQ), and/or timing advance. In some implementations, each observation <NUM> corresponds to a particular base station <NUM>. In other implementations, each observation <NUM> corresponds to more than one base station <NUM> (e.g., adjacent base stations near the UE <NUM>). When each observation <NUM> corresponds to more than one base station <NUM>, the observation <NUM> may be parsed according to each observed base station <NUM>. For example, the network gateway device <NUM> determines a particular base station <NUM> corresponding to an observation <NUM> received from an UE device <NUM> and may estimate characteristics of the particular base station <NUM> based on the observation <NUM> or a collection of observations <NUM> from multiple UE devices <NUM> within the coverage area <NUM>Area corresponding to the base station <NUM>.

Referring to <FIG>, the communication network <NUM> further includes a mapper <NUM> and a modeler <NUM>. The mapper <NUM> and/or the modeler <NUM> may be implemented by the data processing hardware <NUM> of the network gateway device <NUM>. In some examples, the mapper <NUM> and/or the modeler <NUM> are executed by data processing hardware corresponding to the external network <NUM>. For example, the external network <NUM> may be a distributed system (e.g., a cloud environment) with its own data processing hardware or shared data processing hardware (e.g., shared with the network gateway device <NUM>). In yet other examples, the mapper <NUM> and the modeler <NUM> are implemented on different data processing hardware in communication within the communication network <NUM>.

Generally, the mapper <NUM> is configured to generate maps <NUM>, 210a-n for a given base station <NUM> based on the received observations <NUM> corresponding to that base station <NUM>. Some examples of maps <NUM> generated by the mapper <NUM> for a given base station <NUM> include a coverage map 210a, an observation map 210b, and a terrain map 210c. Each of these maps <NUM> may include different types of map information <NUM> relating to the received observations <NUM>. In some examples, each map <NUM> represents the coverage area <NUM>area for a base station <NUM>, but varies with regard to map information <NUM> depicted within that coverage area <NUM>area. For instance, a map <NUM> is divided into geographic portions <NUM> (i.e. geographic subsections) sometimes referred to as pixels, bins, or cells. In other words, these geographic portions <NUM> may be units of the coverage area <NUM>area. For example, when the coverage area <NUM>area for a base station <NUM> is one hundred square kilometers, the coverage area <NUM>area is divided into a 10x10 grid with units of one square kilometer geographic portions <NUM>. Here, more granular geographic portions <NUM> (i.e. smaller geographic areas) may correspond to greater accuracy at the modeler <NUM> (e.g., more accurate estimated characteristics <NUM> for a base station <NUM>). Once the mapper <NUM> generates at least one map <NUM>, the at least one map <NUM> is then fed as an input into a model <NUM> of the modeler <NUM> to output estimated characteristics <NUM> for the base station <NUM>.

The modeler <NUM> is generally configured to receive inputs (e.g., maps <NUM> from the mapper <NUM>) and output estimated characteristics <NUM> of a base station <NUM> associated with the map(s) <NUM>. The modeler <NUM> may be designed such that it may receive any number of maps <NUM> generated by the mapper <NUM>. In some examples, the modeler <NUM> has models <NUM> that correspond to different combination of maps <NUM>. For example, an input of a particular combination of maps <NUM> outputs a particular estimated characteristic <NUM> for the base station <NUM>. Some examples of these estimated characteristics <NUM> include an estimated location (e.g., coordinate location) of the base station <NUM>, an estimated pointing direction of the base station <NUM>, an antenna azimuth of the base station <NUM>, or an uncertainty estimate related to a characteristic of the base station <NUM> or UE device <NUM>.

In some examples, the models <NUM> of the modeler <NUM> correspond to algorithms configured to determine the output of a particular estimated characteristic <NUM>. In other examples, a model <NUM> is a machine learning model <NUM> where the model <NUM> is taught (or trained) based on data sets and result sets to predict its own output based on input data similar to the data sets. For example, the model <NUM> receives radio signal measurements <NUM> associated with a base station <NUM> and based on an aggregate of the radio signal measurements <NUM>, predicts the location of the base station <NUM> as an estimated characteristic <NUM>. In some examples, operators of a base station <NUM> train a model <NUM> with training data corresponding to the base station <NUM> and, based on iterative learning, enable the model <NUM> to determine estimated characteristics <NUM> for a different base station <NUM> and/or different operator based on observations <NUM> associated with that different base station <NUM>. In other examples, the operators of a base station <NUM> who train a model <NUM> may then use the model <NUM> to determine estimated characteristics <NUM>. For instance, the estimated characteristics <NUM> can identify anomalies when compared to expected characteristics. Detection of these anomalies has the benefit that it may inform an operator of potential issues related to the base station <NUM> (e.g., discrepancies with antenna azimuth, pointing direction, or location) or confirm issues being experienced by UE devices <NUM>.

Additionally or alternatively, the model <NUM> is a neural network <NUM> that is fed the map(s) <NUM> as inputs and configured to output the estimated characteristics <NUM> of the base station <NUM>. The neural network <NUM> may be a convolution neural network (CNN) or a deep neural network (DNN). In some examples, the model <NUM> is a combination of a convolution neutral network and a deep neutral network such that the convolution neural network filters, pools, then flattens information to send to a deep neural network. Much like a machine learning model <NUM>, a neural network <NUM> is trained to generate meaningful outputs that may be used as accurate estimated characteristics <NUM>. For example, when training a neural network <NUM> to output the estimated characteristic <NUM> of the location of a base station <NUM>, a mean squared error loss function trains the neural network <NUM>. Typically for training purposes, data is segregated into training and evaluation sets (e.g., <NUM>% training and <NUM>% evaluation) and the neural network <NUM> is trained until a performance of the neural network <NUM> on the evaluation set stops decreasing. Once the performance stops decreasing on the evaluation set, the neural network <NUM> may be ready to determine estimated characteristics <NUM> based on map(s) <NUM> with map information <NUM> related to observations <NUM>.

As an illustration, the neural network <NUM> may function like an image recognition neural network. In an image recognition neural network, the neural network receives layers of an image. For example, these layers may correspond to colors, such as red, green, and blue (i.e. RGB). With layers of the image and a trained familiarity with the colors red, green, and blue, the neural network is configured to identify the image. Here, much like the color layers (i.e. red, green, and blue), the modeler <NUM> receives map(s) <NUM>. With the map(s) <NUM> and a trained familiarity with the types of maps <NUM> provided to the modeler <NUM>, the model <NUM> (e.g., the neural network <NUM>) is configured to identify estimated characteristics <NUM> of the base station <NUM>.

<FIG> is an example of a mapper <NUM> generating maps <NUM> for a base station <NUM>. <FIG> depicts the generated map <NUM> as a coverage map <NUM>, 210a. The coverage map 210a corresponds to a geographic area referred to as the coverage area <NUM>area of a base station <NUM>. Typically, the mapper <NUM> determines the coverage area <NUM>area of the coverage map 210a from each observation <NUM> received by the data processing hardware <NUM>. An observation <NUM> commonly contains data (e.g., metadata) identifying the base station <NUM> emitting the radio signal S measured as the radio signal measurement <NUM> of the observation <NUM>. Therefore, a collection of all observations <NUM>, 130a-n associated with a base station <NUM> define geographic boundaries of the coverage map 210a.

In some examples, the coverage map 210a indicates, as map information <NUM>, at least one radio signal characteristic 212a of the emitted radio signal S about the base station <NUM>. The radio signal characteristic 212a may relate to a single received observation <NUM> or a collection of received observations <NUM>. In some implementations, the mapper <NUM> determines an average radio signal measurement 212a for a cluster of radio signal measurements 132a in a similar location 134a. In other implementations, the mapper <NUM> determines a variance 212a corresponding to a cluster radio signal measurements 132a in a similar location 134a. Additionally or alternatively, the mapper <NUM> may determine both an average and a variance for a cluster of radio signal measurements 132a in a similar location 134a as radio signal characteristics <NUM>.

In some configurations, the mapper <NUM> generates the coverage map 210a for the base station <NUM> by dividing a coverage area <NUM>area into pixels (e.g., geographic portions <NUM>). For example, <FIG> illustrates the coverage map 210a divided into a matrix (or grid). Each cell of this matrix (or grid) may represent a pixel referring to a geographic portion <NUM>. As seen in the example <FIG>, for each observation <NUM>, 130a-n, the mapper <NUM> identifies the pixel having the corresponding geographic portion <NUM> of the coverage area <NUM>area that contains the location <NUM> of the radio signal measurement <NUM> of the respective observation <NUM>. The mapper <NUM> may associate the observation <NUM> with the identified pixel. For example, observation 130a has a signal measurement 132a at location 134a that corresponds to geographic portion <NUM><NUM>, where "<NUM>" is the third row and the second column pixel associated with location 134a. Similarly, location 134b of observation 130b corresponds to geographic portion <NUM><NUM>. Location 134c of observation 130c also maps to geographic portion <NUM><NUM>. Location 134d of observation 130d corresponds to geographic portion <NUM><NUM> and location 134n of observation 130n maps to geographic portion <NUM><NUM> as well. With the locations <NUM> of each observation <NUM> for the base station <NUM> mapped along with the radio signal measurement <NUM> at that location <NUM>, the mapper <NUM> may generate radio signal characteristics <NUM> as map information <NUM> for each pixel. As previously stated, for each pixel, the mapper <NUM> may determine the average (e.g., a RSRP value) and/or the variance of the radio signal measurements <NUM> of any observations <NUM> associated with the respective pixel. Referring further to <FIG>, the mapper <NUM> generates the average (e.g., a RSRP value) and/or the variance as radio signal characteristics 212a<NUM>, 212a<NUM>, and 212a<NUM> for the respective pixels <NUM><NUM>, <NUM><NUM>, and <NUM><NUM>. Additionally or alternatively, the coverage map 210a may include data structures, referred to as bins, for each pixel, cell, and/or geographic portion <NUM> where an observation <NUM> associated with the pixel, cell, and/or geographic portion <NUM> may be stored. An advantage of bins is that the data structure of the coverage map 210a may enable to mapper <NUM> to quickly generate additional maps <NUM> (e.g., observation maps 210b) from the observations <NUM> within the bins. This may prevent the mapper <NUM> from once again mapping the locations <NUM> of the observations <NUM>.

<FIG> is an example of the mapper <NUM> generating an observation map <NUM>, 210b based on the coverage map 210a and the observations <NUM>. With respect to the observation map 210b, the map information <NUM> refers to a metric 212b, generated by the mapper <NUM>, that expresses the presence of an observation <NUM> in a geographic portion <NUM> (e.g., pixel or cell) of the coverage map 210a. The metric 212b may be a number N of observations <NUM> associated with a geographic portion <NUM> (e.g., pixel or cell), a log of the number N of any observations <NUM> associated with a respective geographic portion <NUM> (e.g., pixel or cell), or a monotonic function of the number N of any observations <NUM> associated with the respective geographic portion <NUM> (e.g., pixel or cell).

In some configurations, the mapper <NUM> generates the metric 212b by determining a number N of any observations <NUM> associated with a respective geographic portion <NUM> (e.g., pixel or cell). When the number N is greater than zero, the mapper <NUM> assigns a value to the respective geographic portion <NUM> (e.g., pixel or cell). The value may be a "<NUM>" or a number that corresponds to the number N of observations <NUM>. When the number N equals zero, the mapper <NUM> assigns the value of "<NUM>" to the respective geographic portion <NUM> (e.g., pixel or cell). For example <FIG> illustrates the metric 212b is a binary metric where a "<NUM>" is assigned to a geographic portion <NUM> when the geographic portion <NUM> does not contain any observation <NUM> and where a "<NUM>" is assigned to a geographic portion <NUM> when the geographic portion <NUM> contains one or more observation <NUM>. In this example, only geographic portions <NUM><NUM>, <NUM><NUM>, and <NUM><NUM> of the observation map 210b<NUM> are assigned a value of "<NUM>"; while all other geographic portions <NUM> are assigned a value of "<NUM>. " In some examples, the modeler <NUM> may skip geographic portion <NUM> assigned a value of "<NUM>. " In models <NUM> with machine learning and/or neutral networks, the model <NUM> may, over time, also learn to skip geographic portion <NUM> assigned a value of "<NUM>" during the modeling process. In other implementations, such as <FIG>, the value assigned as a metric 212b to a geographic portion <NUM> of the observation map 210b<NUM> directly corresponds to the number N of observations <NUM> within that geographic portion <NUM>. Here, because geographic portions <NUM><NUM>, <NUM><NUM>, and <NUM><NUM> correspond to <NUM>, <NUM>, and <NUM> observations <NUM>, the observation map 210b<NUM> assigns geographic portions <NUM><NUM>, <NUM><NUM>, and <NUM><NUM> a value of <NUM>, <NUM>, and <NUM>, respectively.

Although the observation map 210b may seem like a redundant map to provide to the modeler <NUM> in light of the coverage map 210a, each map 210a, 210b provided to the modeler <NUM> may provide different degrees of accuracy for outputs (e.g., the estimated characteristic <NUM>) determined by the modeler <NUM>. In some examples, the modeler <NUM> outputs a confidence indicator <NUM> for the estimated characteristic <NUM>. Here, the observation map 210b enables the modeler <NUM> to output a confidence indicator <NUM> for the estimated characteristic <NUM> based on the number of observations <NUM> indicated in provided observation maps 210b. For example, the modeler <NUM> has a higher confidence indicator <NUM> when the observation map 210b corresponds to a significant number of observations <NUM>. In other words, the modeler <NUM> gains confidence in the estimated characteristics <NUM> of the base station <NUM> when the estimated characteristics <NUM> are based on a larger volume of observations <NUM> (i.e., data measurements to generate the estimated characteristics <NUM>). In yet other examples, the modeler <NUM> outputs a confidence indictor <NUM> without an observation map 210b such that a model <NUM> of the modeler <NUM> may be configured to determine the confidence indicator <NUM> based on other map information <NUM> provided with the map(s) <NUM> input into the modeler <NUM>.

<FIG> and <FIG> illustrate that other maps <NUM> may be provided to the modeler <NUM> to determine an estimated characteristic <NUM> for a base station <NUM>. In <FIG> and <FIG>, the mapper <NUM> generates a terrain map <NUM>, 210c<NUM>-<NUM>. The terrain map 210c may include at least one of a terrain altitude of a geographical area associated with the coverage area <NUM>area or a presence and/or height of objects extending above a ground surface of the geographical area as the map information 212c. For example, the mapper <NUM> generates the terrain map 210c from terrain images, such as topography (<FIG>) or images of the coverage area <NUM>area (<FIG>) from which the mapper <NUM> can determine the presence and/or height of objects. With these images, the mapper <NUM> may generate various types of terrain maps 210c depending on the granularity desired by an entity such as an operator or a network provider seeking to determine an accurate estimated characteristic <NUM> of the base station <NUM>. In other words, the complexity of the terrain map 210c that the mapper <NUM> may generate may vary depending on application and/or design of the modeler <NUM> and model(s) <NUM>. For example, <FIG> illustrates a simple terrain map 210c where, much like the observation map 210b<NUM>, the mapper <NUM> generates metrics 212c corresponding the terrain (e.g., based on a terrain image). In <FIG>, the mapper <NUM> uses a binary metric 212c to assign geographic portions <NUM> with a "<NUM>" value when the terrain contains an object or feature (e.g., a river or a forest) and to assign geographic portions <NUM> with a "<NUM>" when the geographic portion <NUM> does not contain a terrain object or feature. For example, the terrain map 210c<NUM> of <FIG> includes a "<NUM>" value in each geographic portion <NUM> (e.g., pixel or cell) of columns one and two because of the river and the forest in the terrain image.

<FIG> is a more complicated terrain map 210c<NUM>. The terrain image corresponding to terrain map 210c<NUM> is a topographical map identifying elevation within the coverage area <NUM>area. The mapper <NUM> may generate several different terrain maps 210c with varying map information <NUM> relating to the elevation information. Some examples include, terrain maps 210c designing, as the map information <NUM>, the extrema within a geographic portion <NUM>, the average elevation for the respective geographic portion <NUM>, and/or a metric to designate elevations that may be predetermined to pose an issue with determining estimated characteristics <NUM> for a base station <NUM>. In the example of <FIG>, each geographic portion <NUM> has map information 212c corresponding to the maxima elevation height within the geographic portion <NUM>. For example, geographic portion <NUM><NUM> has a maxima elevation height of <NUM>,<NUM> feet based on the topography image.

In other configurations, the mapper <NUM> providers the modeler <NUM> with another map <NUM>, a timing advance map 210d, whose map information <NUM> includes a measured timing advance 212d. The measured timing advance 212d corresponds to an estimate of how far the UE device <NUM> is from the base station <NUM>. Generally, the timing advance is a measurement performed by a UE device <NUM> such that, for example, when the UE device <NUM> generates radio signal measurements <NUM> (e.g., RSSI, RSRP, or RSRQ), the UE device <NUM> also measures the timing advance (i.e. generates a measured timing advance 212d).

In other examples, the mapper <NUM> provides the modeler <NUM> with a location uncertainty map 210e. For example, the location uncertainty map 210e corresponds to a radio signal measurement <NUM> (e.g., signal strength measurement such as RSSI, RSRP, RSRQ) generated by the UE device <NUM> at a given location (e.g., location X) where the radio signal measurement <NUM> also indicates a location uncertainty 212e of some distance Y. The location uncertainty map 210e may be a map <NUM> of the location uncertainty 212e or a map <NUM> of a statistic relating to the location uncertainty 212e. For example, the location uncertainty map 210e is divided into geographic portions <NUM> (e.g., pixel or cell) where each geographic portion <NUM> represents an average of location uncertainties 212e from radio signal measurements <NUM> within a geographic portion <NUM>. Although coverage maps 210a, observations maps 210b, and terrain maps 210c are discussed in detail, any map <NUM> may be provided to the modeler <NUM> to compile more data for the model <NUM> to effectively provide an estimated characteristic <NUM> for a base station <NUM>.

<FIG> are examples of the modeler <NUM>. The modeler <NUM> is configured to receive maps <NUM> and/or side information <NUM>. With the map(s) <NUM> and/or side information <NUM>, the modeler <NUM> uses at least one model <NUM> to determine an estimated characteristic <NUM> and/or confidence indicator <NUM> for the estimated characteristic <NUM>. The side information <NUM> fed into the modeler <NUM> may include at least one of frequency of operation of a base station <NUM>, a height of an antenna of the base station <NUM>, an antenna beam width, an antenna tilt angle, a predetermined (i.e. expected) location <NUM>Lexp of the base station <NUM>, or any other parameter associated with the base station <NUM>. In configurations where the modeler <NUM> receives side information <NUM> in addition to the map(s) <NUM>, the modeler <NUM> does not receive, as side information <NUM>, information related to the estimated characteristic <NUM>. The information related to the estimated characteristic <NUM> may be later compared to the estimated characteristic <NUM> for identification of anomalies or discrepancies with the network <NUM>. In other words, the modeler <NUM> may be configured such that estimated characteristic <NUM> (i.e. output of the modeler <NUM>) is never an input (e.g., side information <NUM> or map information <NUM>).

<FIG> is an example where the modeler <NUM> does not receive side information <NUM>. Here, the inputs to the modeler <NUM> are the coverage map 210a indicating radio signal characteristics 212a and the observation map 210b based on metrics 212b relating to observations <NUM>. In some examples, the modeler <NUM> receives more than one observation map 210b as an input (e.g., a first observation map 210b<NUM> with a binary metric 212b and a second observation map 210b<NUM> with a metric 212b relating to the number N of observations <NUM>). Using at least one model <NUM> (e.g., a neural network <NUM>), the modeler <NUM> outputs an estimated characteristic <NUM> (e.g., the estimated location 312a of the base station <NUM>). As shown in <FIG> and <FIG>, the estimated location 312a of the base station <NUM> may be mapped and/or compared to a map <NUM> identifying an expected location <NUM>Lexp of the base station <NUM>. <FIG> and <FIG> indicate the estimated location 312a with a circle containing an "X" and the expected location <NUM>Lexp of the base station <NUM> with a bullseye shape. In <FIG>, the expected location <NUM>Lexp of the base station <NUM> and the estimated location 312a are nearby and not a significant discrepancy. <FIG>, on the other hand, indicates that the estimated location 312a of the base station <NUM> significantly deviate from the expected location <NUM>Lexp of the base station <NUM>. With an identified discrepancy or anomaly, an entity (e.g., network administrator, operator or network provider) looks to the confidence indicator <NUM> to further understand the identified discrepancy or anomaly. According to the claimed invention, the entity identifies a ratio of the prediction error to the uncertainty (i.e. confidence indicator <NUM>) where the prediction error corresponds to the apparent deviation between the estimated characteristic <NUM> and the expected characteristic. For example, a large prediction error between the expected location <NUM>Lexp of the base station <NUM> and the estimated location 312a (e.g., <FIG>) appears justified by a large uncertainty for the confidence indicator <NUM>; whereas, a large prediction error between the expected location <NUM>Lexp of the base station <NUM> and the estimated location 312a appears troublesome with a high confidence indicator <NUM>.

<FIG> is another example of a communication network <NUM>. In this example, the mapper <NUM> may receive the observations <NUM> from data processing hardware <NUM> of the network gateway device <NUM> or from a topology data store <NUM> (e.g., a central server). The topology data store <NUM> may continuously scan the communication network <NUM> to receive measurement data from UE devices <NUM>. The measurements may then be interpreted as observations <NUM> that the mapper <NUM> may use to generate maps <NUM> for the model <NUM> to determine an estimated characteristic <NUM> for a base station <NUM>. With continuous scanning of the topology data store <NUM>, the communication network <NUM> may seek to resolve issues that UE devices <NUM> encounter with the communication network <NUM> using the combination of the mapper <NUM> and the modeler <NUM> to detect issues with base station parameters. In some examples, an estimated characteristic store <NUM> stores the estimated characteristics <NUM> determined by the modeler <NUM>. By storing the estimated characteristics <NUM>, the communication network <NUM> may be able to queue encountered issues to later repair and resolve. In some examples, these identified issues are resolved at the base station <NUM> and/or UE device <NUM>. Additionally or alternatively, the estimated characteristic store <NUM> may provide a data sets for models <NUM> of the modeler <NUM> to further learn via machine learning models and/or neural network models. Learning from these data sets may further hone a predictive accuracy of the models <NUM> of the modeler <NUM> to determine estimated characteristics <NUM> of the base station <NUM>. In some configurations, the estimated characteristic store <NUM> and/or the topology data store <NUM> can create numerous data sets and/or maps to utilize the data contained therein to train a model <NUM> of the modeler <NUM>. For example, the observations <NUM> sent from the UE devices <NUM> and collected by the topology data store <NUM> may create more than one learning map <NUM> from the same set of data by translating, rotating, or inverting about an axis an initial generated map <NUM> of a set of data. With potentially volumes of observations <NUM> collected at the topology data store <NUM> combined with the ability to generate multiple learning maps <NUM> for a model <NUM> with one data set, the topology data store <NUM> may function as a powerful tool to train models <NUM> for the modeler <NUM>.

<FIG> illustrates a method <NUM> for detecting radio coverage problems based on UE devices <NUM> and observations <NUM> within a coverage area <NUM>area of a base station <NUM>. While the method is described with respect to radio base stations, the method may also be applied to television stations, WiFi access points and coverage maps, and other communication devices. At block <NUM>, the method <NUM> includes receiving from at least one UE <NUM> observations <NUM>. Each observation <NUM> including a radio signal measurement <NUM> from a base station <NUM> and a corresponding location <NUM> of the radio signal S. At block <NUM>, the method <NUM> further includes generating, by data processing hardware, a coverage map 210a for the base station <NUM> based on the received observations <NUM>. The coverage map 210a indicates a radio signal characteristic 212a of the emitted signal S of about the base station <NUM>. At block <NUM>, the method <NUM> also includes generating an observation map 210b based on the coverage map 210a and the observations <NUM>. At block <NUM>, the method <NUM> further includes determining, by data processing hardware, an estimated characteristic <NUM> of the base station <NUM> by feeding the coverage map 210a and the observation map 210b into a neural network <NUM> configured to output the estimated characteristic <NUM> of the base station <NUM>.

Claim 1:
A method comprising:
feeding, by data processing hardware (<NUM>), a map (<NUM>) into a neural network (<NUM>), wherein the map identifies an expected location (<NUM>exp) of a base station (<NUM>), the neural network configured to output an estimated characteristic (<NUM>) of the base station and to output a confidence indicator (<NUM>) for the estimated characteristic of the base station, the confidence indicator representing an uncertainty of the estimated characteristic;
determining, by the data processing hardware, the estimated characteristic of the base station by feeding a coverage map (210a) into the neural network, the coverage map indicating a signal characteristic of the emitted signal about the base station, the estimated characteristic of the base station comprising an estimated location of the base station; and
determining, by the data processing hardware, a prediction error, wherein the prediction error (i) corresponds to a deviation between the estimated location of the base station and the expected location of the base station and (ii) is compared with the confidence indicator to determine a ratio of the prediction error to the confidence indicator for identifying an anomaly.