Patent Description:
The present disclosure relates to identification of radio frequency (RF) signals.

Radio frequency (RF) waveforms are prevalent in many systems for communication, storage, sensing, measurements, and monitoring. RF waveforms are transmitted and received through various types of communication media, such as over the air, under water, or through outer space. In some scenarios, RF waveforms transmit information that is modulated onto one or more carrier waveforms operating at RF frequencies. In other scenarios, RF waveforms are themselves information, such as outputs of sensors or probes. Information that is carried in RF waveforms is typically processed, stored, and/or transported through other forms of communication, such as through an internal system bus in a computer or through local or wide-area networks.

<NPL> describes modulation classification using two subsystems. A first subsystem extracts key features from the instantaneous amplitude and instantaneous phase of the signal. These features are used by the second subsystem to train a multilayer perceptron.

<NPL>) compares the efficacy of radio modulation classification using naively learned features against using expert features based methods. A set of expert features are extracted and used to train several classifiers. A convolutional neural network provided with a windowed raw radio time series is also trained for modulation classification. The classification results of the two methods are compared.

In general, the subject matter described in this disclosure is embodied in methods and systems for training and deploying machine-learning networks to identify or process, or both, radio frequency (RF) signals. The invention is defined in the claims.

In one aspect, a method is disclosed for training at least one machine-learning network to classify radio frequency (RF) signals, as defined in claim <NUM>.

Another aspect includes a system as defined in claim <NUM>.

The details of one or more implementations of the subject matter of this disclosure are set forth in the accompanying drawings and the description below.

Systems and techniques are disclosed herein that enable machine learning and deployment of identification of radio frequency (RF) signals. In some implementations, such systems and techniques are able to identify an RF signal and label the RF signal, which may be a series of received radio samples, with classification information regarding a characteristic of the RF signal or a characteristic of an environment in which the RF signal is communicated. Such classification information may include for example, a high-level human understandable name, a category, a radio protocol, information about the transmission method or contents that are transmitted, or a radio device name.

The machine-learning RF signal classification system may learn features to discriminate radio signal types from one another, based on raw sampled time-series radio data as well as labels for that data. The labels may be provided manually, programmatically (e.g., in scenarios where RF data and labels are generated by simulations), or through other techniques of label estimation. In some implementations, the system may combine learned features with additional expert features to form estimates of the RF signal type that is present.

A machine-learning system may be trained to learn a system for RF signal identification, for example by training on a classification model based on a large set of RF signal and labeling data. The machine-learning system may then be deployed with a fixed training model or with online model adaptation that are updated based on observations, experience, performance metrics, and/or feedback from users or higher level applications. As such, implementations disclosed herein enable quick and accurate sensing of the radio spectrum, which is a critically important task for a number of important applications.

Techniques disclosed herein may be applied to various scenarios in the field of digital radio signal processing. For example, implementations disclosed herein may enable federal regulators or operators to identify radio-signal emitter devices in licensed and unlicensed radio bands, thereby helping ensure that the signal is authorized to be emanating from an intended location or is not malfunctioning and producing unintended interference. As another example, implementations disclosed herein may enable organizations to perform quick and accurate sensing of the radio spectrum to identify hostile signals or wireless cyber-attacks more efficiently, and/or to identify and organize optimal allocation among friendly radio emitter devices.

Radio sensing techniques typically have significant power budgets and are developed using highly specialized algorithms that rely on specific analytic knowledge of the composition of certain types of target radio signals. The significant power budgets or the highly specialized nature of algorithms, or both, can severely limit the extent to which radio sensing and classification techniques are deployed and scaled to different signal types, signal environments or deployment devices in actual use. Furthermore, such algorithms may degrade under certain hardware effects or channel propagation conditions, such as when fading or intermodulation distortion is present and changes the radio signal from its ideal form.

By contrast, signal identification systems and techniques disclosed herein provide novel approaches that utilize machine learning to learn appropriate techniques for identifying RF signals, such as distinguishing between different signal types, directly from time-series RF waveforms, such as sampled radio recording of a radio transmission. In some implementations, machine learning may be performed without, or with minimal, reliance on expert feature design, such as analytic derivation of specific hand crafted features designed for specific modulations. By learning features directly from RF signals, implementations disclosed herein enable signal identification systems to adapt rapidly to different types of new signals while reducing labor-intensive and time-consuming engineering and design work to change a large number of internal algorithms in order to support a new modulation type, channel effect or other signal label.

In some scenarios, implementations disclosed herein may provide energy efficiency advantages by performing signal identification in a concurrent way, which facilitates many-core or distributed FPGA implementations. For example, systems and techniques disclosed herein may be implemented using a highly concurrent (e.g. distributed many-core or distributed logic synthesis) architecture and low clock rates (e.g., approximately <NUM> or <NUM> of MHz instead of GHz). In another example, the disclosed approach also allows for the usage of lower precision data types (e.g., <NUM> or <NUM> bit integer/float or less, instead of <NUM> or <NUM> bit floats) requiring less bits per operation to preserve the dynamic range and performance of the signal, allowing as well for lower power implementations which utilize fewer logic gates than more traditional feature-based approaches in many cases. As such, implementations disclosed herein may enable improved power efficiency of mili-Watts or single-digit Watts as compared to higher power systems which may consume tens or hundreds of Watts.

<FIG> illustrates an example of a radio frequency (RF) system <NUM> that implements at least one machine-learning network to perform learned identification of RF signals. In some implementations, the system <NUM> is implemented in a training scenario to train a machine-learning network <NUM>. In other implementations, the system <NUM> is used in a deployment scenario to utilize a pre-trained machine-learning network <NUM>.

As shown in <FIG>, the system <NUM> includes a machine-learning network <NUM> that may be trained and deployed to receive an RF signal <NUM> and generate predicted classification information <NUM>. In general, the machine-learning network <NUM> may utilize any suitable mapping of RF signals <NUM> to predicted RF signal classification information <NUM>, as discussed in more detail with reference to <FIG>, below.

The predicted classification information <NUM> may represent various features related to the RF signal <NUM>. For example, classification information <NUM> may represent a characteristic of the RF signal <NUM> and/or a characteristic of an environment in which the RF signal <NUM> is communicated. In some implementations, the classification information may include one or more labels, and may indicate deterministic or statistical classification information (e.g., likelihoods and/or probabilities). The classification information <NUM> may, in some implementations, include human-readable information, such as human-readable labels.

As examples, the classification information may indicate characteristics of the RF signal and/or the environment of the RF signals, such as a modulation type of RF signals (e.g. QPSK, BPSK, OOK, FSK, GMSK, OFDM, etc.); a type of protocol or standard that is carried on or that is utilized to transmit RF signals (e.g., GSM, LTE, Wi-Fi <NUM>, LTE-<NUM>-TDD, HSPA, P25, etc.); a specific radio device utilized to transmit or receive RF signals, a tower or user associated with RF signals. As another example, the classification information may indicate a type of traffic, behavior or other contents that are being carried by the RF signal, such as normal usage on a wireless channel, types of application data, or nefarious attacks on the wireless channel (e.g., flooding, probing or jamming attacks). In these cases, detection of such events may result in triggering an alarm, triggering a configuration change in an electronic device such as a router or firewall, reconfiguration of a wireless system, or otherwise.

As further examples, the classification information may indicate a type of RF emission or a type of RF emission pattern (e.g., a particular sequence of data bits or other pattern in the RF emission that can be distinguished and identified) occurring, which may indicate electromagnetic interference (EMI), malicious interference (e.g., snooping), or a noise source (e.g., communication channel noise or hardware noise). The type of RF emission or RF emission pattern may be utilized to determine specific events of interest, such as equipment malfunction (e.g., a broken cable or connector on a base station generating impedance mismatch/inter-modulations/distortions, or a heater or electronic device emitting harmful EMI). As such, implementations disclosed herein may be utilized to identify EMI emitter types and/or identify wireless changes, failures, interferers, threats or anomalies (e.g., cyber-threats, attacks).

In yet another example, the classification information may indicate a presence of particular types of communication signals. For example, the classification may involve a binary decision indicating whether a particular type of RF signal is present, therefore enabling an RF signal detection technique. In this instance one or more of the signal labels may indicate the presence of background noise, or the absence of a known signal type or label. In some examples information derived from the RF signal may be used to perform analytics on emitters (e.g. passing people, vehicles, devices, etc.) which could include detection, on-times, usage level, location, or other statistical information about their use.

In a training scenario, the machine-learning network <NUM> takes an input RF signal <NUM> and learns how to generate predicted RF signal classification information <NUM> that corresponds to the signal <NUM>. During training, various network features <NUM> of the machine-learning network <NUM> may be updated to achieve a desired performance objective, such as weights of one or more network layers, parameters, or architecture choices for the machine-learning network <NUM>.

In some implementations, training may involve comparing the predicted RF signal classification information <NUM> with known or pre-labeled classification information, to determine a classification error for input signal <NUM>. The machine-learning network <NUM> may be trained to achieve various types of objective functions, which may involve a measure of classification error, a measure of computational complexity (e.g., as measured by the number of parameters, number of multiplies/adds, execution time, Kolmogorov complexity, etc.), or combinations thereof. Further details of training are described below, for example with reference to <FIG> and <FIG>.

Once trained, the machine-learning network <NUM> may be deployed in various application scenarios to perform learned identification of RF signals, using the identification techniques that were learned during training. In some implementations, the machine-learning network <NUM> may be further updated during deployment based on real-time performance results of identification. Further details of deployment are described below, for example with reference to <FIG> and <FIG>.

In some implementations, the input RF signal <NUM> may represent the result of an analog RF waveform that was received by one or more antennas over a medium, such as over the air. The RF signal <NUM> may be processed by the machine-learning network <NUM> in analog or digital form. For example, the RF signal <NUM> may represent a digitized representation, such as a raw sampled time series, of an RF waveform that was received and processed by a receiver before being input into the machine-learning network <NUM>. In some implementations, the RF signal <NUM> may be an analog waveform, and the machine-learning network <NUM> may implement various filters, samplers, analog-to-digital (A/D) converters, or other circuitry and modules for processing the RF signal <NUM>. In some scenarios, the input RF signal <NUM> may be a composite signal that includes multiple RF signals, one or more of which may be distinguished and identified by the machine-learning network <NUM>.

In some implementations, the machine-learning network <NUM> may include at least one artificial neural network that consists of one or more connected layers of parametric multiplications, divisions, additions, and non-linearities, or other operations (e.g., normalization). In such scenarios, updating the machine-learning network <NUM>, which may occur during training or deployment, includes updating weights of the neural network layers, or updating connectivity in the neural network layers, or other modifications of the neural network architecture, so as to modify a mapping of inputs to outputs.

The machine-learning network <NUM> may be configured to identify RF signals using any suitable machine-learning technique. For example, the machine-learning network <NUM> may implement a mapping from RF signal inputs <NUM> into a lower-dimension space, from which the predicted RF signal classification information <NUM> is generated. As an example, the lower-dimensional mappings may be utilize a set of basis functions and transforming the input RF signal <NUM> into a lower-dimensional projection of the signal <NUM> onto those basis functions. The predicted RF signals classification information <NUM> may then be generated based on analyzing those projections onto the basis functions.

RF signals that are processed by system <NUM> may include any suitable radio-frequency signal, such as acoustic signals, optical signals, or other analog waveforms, typically of human-designed communications system or radar/sonar system. In some instances, the signals may have been designed by automated processes (e.g., learned communications or radar systems). The spectrum of RF signals that are processed by system <NUM> may be in a range of <NUM> to <NUM>. For example, such RF signals include very low frequency (VLF) RF signals between <NUM> to <NUM>, low frequency (LF) RF signals between <NUM> to <NUM>, medium frequency (MF) RF signals between <NUM> to <NUM>, high frequency (HF) RF signals between <NUM> to <NUM>, and higher-frequency RF signals up to <NUM>.

The RF signals may themselves be information, such as measurements that are output from sensors or probes, including medical devices and equipment monitoring devices. Alternatively, the RF signals may be fixed waveforms that have been modulated to carry other information, such as carrier waveforms modulated by a communication system to carry data. In both scenarios, implementations disclosed herein enable learning identification of RF signals, and provide efficient classification, analysis, and high-level understanding of those signals.

<FIG> illustrates an example of a network structure <NUM> of an RF system implementing at least one machine-learning network to perform learned identification of RF signals. The network structure <NUM> includes machine-learning network <NUM> that is implemented using an artificial neural network that consists of one or more layers. The output of each network layer is used as input to the next network layer.

The machine-learning network <NUM> may include one or more such layers that are shown. For example, in some implementations, the machine-learning network <NUM> may include a plurality of networks that may be collectively or iteratively trained. As such, the network input <NUM> in <FIG> may be the original received RF signal (e.g., RF signal <NUM> in <FIG>, above), or may be an output of one or more layers in the machine-learning network <NUM>. Analogously, the network output <NUM> may represent the final predicted classification information (e.g., classification information <NUM> in <FIG>, above), or may be an input into one or more subsequent layers in the machine-learning network <NUM>.

In general, the machine-learning network <NUM> may include one or more collections of multiplications, divisions, and summations of inputs and intermediate values, optionally followed by non-linearities (such as rectified linear units, sigmoid function, or otherwise) or other operations (e.g. normalization), which may be arranged in a feed-forward manner, including optional bypass or residual connections or in a manner with feedback and in-layer connections (e.g., a recurrent or quasi-recurrent network). Parameters and weight values in the network may be used for a single multiplication, as in a fully connected neural network (DNN), or they may be "tied" or replicated across multiple locations within the network to form one or more receptive fields, such as in a convolutional neural network, a dilated convolutional neural network, a residual network unit, or similar. A collection of one or more of these layers may constitute the machine-learning network <NUM>. The specific structure for the networks may be explicitly specified at design time, or it may be selected from a plurality of possible architecture candidates to ascertain the best performing candidate.

The machine-learning network <NUM> may implement dense network layers and/or convolutional network layers, and non-linear activations. In the example of <FIG>, the machine-learning network <NUM> implements convolutional neural network (CNN) layers and dense neural network (DNN) layers with rectified linear-unit (ReLU) and SoftMAX activations In general, however, implementations are not limited to these specific types of layers, and other configurations of layers and non-linearities may be used, such as sigmoid, tanh, and others. One common configuration may include a sequence of convolutional layers, utilizing pooling or striding, resulting in a set of feature maps which are combined using a fully connected layer to produce classification information.

One or more layers of the machine-learning network <NUM> (e.g., dense, convolutional, or otherwise), may include a set of parameters, such as weights <NUM> for the network layers. The machine-learning network <NUM> adapts these layers, including connectivity between layers and parameter such as network layer weights, to learn techniques for generating a network output <NUM> in response to a network input <NUM>.

The network weights <NUM> may be real or complex valued numbers such as floating point values, fixed point numbers, or integer numbers which are used by multiplication or addition operations within the model architecture of machine-learning network <NUM> to be combined with the input RF signal (e.g., network input <NUM>), or intermediate values within the model.

The convolutional layers in the machine-learning network <NUM> may facilitate time-shift invariance learning, and may reduce the number of parameters used to classify the input RF signal (e.g., network input <NUM>). For example, by using convolutional layers with only one or two filters, implementations disclosed herein may achieve a maximally matched small set of time-basis filters. As such, convolutional layers may be well suited for reducing parameter space and forming a compact front-end for radio data. Learned weights can be extremely useful in the analysis or reception of unknown or partially known signal types, for instance filters can often learn the modulation basis functions or pulse shapes used within a transmission system very quickly which can be used to aid the reception or analysis of an unknown system type.

In addition to the convolutional layers, dense layers with non-linear activations may be implemented in between these convolutional layers to provide an estimation of the logic for what the classification should be for those basis filters occurring at different times. For example, non-linearities or activation functions may be applied to an intermediate layer summation (t) and produce an intermediate layer or final layer output (s). Example include a sigmoid function given by S(t) = <NUM>/(<NUM> + e-t); a rectifier (or rectified linear units, ReLU) given by S(t) = max(<NUM>, t); a hyperbolic tangent ("tanh") given by S(t) = (et - e-t)/(et + e-t); a Softmax function given by S(t) = ejt/ Σj ejt; or another function providing a similar non-linear effect.

In some implementations, the machine-learning network <NUM> may include at least one regularization layer having at least one of weight regularization on convolutional network layer weights, activity regularization on dense network layer activations, or other stochastic impairments on activations or weights, such as dropout or dropconnect. For example, regularization may be used to prevent over-fitting. Dropout, such as a penalty on the convolutional layer weights, may be utilized to encourage minimum energy bases, and a penalty on the first dense layer activation may be utilized to encourage sparsity of solutions.

In training scenarios, various features of the machine-learning network <NUM> may be updated to learn a classification technique. For example, if an artificial neural network is implemented in the machine-learning network <NUM>, then the specific arrangement and composition of artificial neural network layers, types of layers, and parameterizations such as weights in the layers may be updated based on the specific dataset or training process. As such, the machine-learning network <NUM> may be trained based on different configurations of input values, network layer weights, multiplications of weights with inputs, summations of these scaled weights, and the generation of output activations from these summations. In the case of dense network layers, every input to output combination may have a unique weight scaling, while in the case of convolutional or other types of network layers, weights may be re-used at different shifts or patterns across many input activations.

Various different configurations of CNN, DNN, activation, or some combination of these layers which may be sequential, parallel, or connected with various bypass or cross connections and other network layer types may be used in the machine-learning network <NUM> to achieve a desired objective function. Any suitable search process, such as greedy search, guided search, or evolution, may be used to assist in classification model selection during training scenarios.

A general design objective for the machine-learning network <NUM> may be to obtain a minimum complexity network, while still achieving desired classification performance.

<FIG> illustrates an example of a network structure <NUM> of an RF system implementing at least one machine-learning network to learn and merge raw time-series RF signal information with features that are extracted from the RF signal to perform learned identification RF signals. The network structure <NUM> includes a machine-learning network <NUM> implementing an artificial neural network including one or more layers to learn RF signal classification information (e.g., network output <NUM>) from an input RF signal (e.g., network input <NUM>), utilizing one or more network layer weights <NUM>, as in the example described in <FIG>, above.

Compared to the example of <FIG>, however, the example of <FIG> additionally utilizes a feature extractor <NUM> that leverages prior knowledge about the RF signal (e.g., network input <NUM>). The feature extractor <NUM> may utilize, for example, features and/or transforms that are obtained from expert hand crafted techniques or automated feature-extracting techniques. Therefore, the structure <NUM> in <FIG> enables using both machine-learned information and extracted features from RF signals to generate the network output <NUM> (which may be the RF signal classification information or an input to another one or more layers, as discussed in regards to <FIG>, above).

The features extracted by feature extractor <NUM> may, in some implementations, thus be included as additional inputs to the machine-learning network <NUM>. In some scenarios, this may provide a more accurate classification as compared to using machine learning alone. For example, if an artificial neural network is implemented in the machine-learning network <NUM>, then the specific arrangement and composition of artificial neural network layers, types of layers, and parameterizations such as weights in the layers may be updated based on the specific dataset, extracted features or labels, or training process. One example where this may improve performance is when a smaller amount of training data is available, and a small machine learning network is used (e.g., to avoid over-fitting or poor generalization of the network).

In some implementations, the feature extractor <NUM> may also be trained, as with the machine-learning network <NUM>, based on results of classification performance.

<FIG> is a flowchart illustrating an example method <NUM> of training at least one machine-learning network to perform learned identification of RF signals. This type of training may be utilized to train a machine-learning network (e.g., machine learning network <NUM> in <FIG>, above) to classify RF signals. The method <NUM> may be performed by one or more processors, such as one or more CPUs, GPUs, DSPs, FPGAs, ASICs, TPUs, neuromorphic chips, or vector accelerators that execute instructions encoded on a computer storage medium.

The example training method <NUM> includes determining an RF signal that is configured to be transmitted through an RF band of a communication medium (<NUM>). The RF signal (e.g., RF signal <NUM> in <FIG>, above) may be generated from a training dataset of sample RF signals, and may be in digital or analog form. In some implementations, the RF signals in the training dataset may have been obtained from an RF waveform that was received by one or more antennas, or may have been obtained by other techniques, such as simulation, as further discussed in regards to <FIG>, below.

The method <NUM> further includes determining first classification information that is associated with the RF signal (<NUM>). The first classification information may be, for example, one or more labels that are known to be associated with the RF signal, and that are used as reference points for evaluating the performance of the machine-learning network during training. The first classification information may include a representation of at least one of a characteristic of the RF signal or a characteristic of an environment in which the RF signal is communicated. As examples, the first classification may be pre-stored label information (e.g., stored label information <NUM> in <FIG>, below), and/or may include human labeling and/or machine labeling (e.g., <NUM> and/or <NUM> in <FIG>, below).

The method <NUM> further includes using at least one machine-learning network to process the RF signal and generate second classification information, which is a prediction of the first classification information (<NUM>). In some implementations, the machine-learning network (e.g., machine learning network <NUM> in <FIG>, above) is an artificial neural network that may utilize any suitable mapping from input RF signals (e.g., RF signal <NUM> in <FIG>, above) to predicted second classification information (e.g., predicted classification information <NUM> in <FIG>, above). The second classification information may include a representation of at least one of a characteristic of the RF signal or a characteristic of an environment in which the RF signal is communicated. Details of this classification process are discussed further in regards to <FIG>, below.

The method <NUM> further includes computing a measure of distance between (i) the second classification information that was generated by the machine-learning network, and (ii) the first classification information that was associated with the RF signal (<NUM>). This measure of distance may be implemented as a loss function (e.g., loss function <NUM> in <FIG>, below) and represents a difference or error between the second classification information and the first classification information. As examples, the measure of distance may include cross-entropy, mean squared error, or other geometric distance metric (e.g., MSE, MAE, KL divergence), or may combine several geometric, entropy-based and/or other classes of distance metrics into an aggregate expression for distance or loss, as discussed further in regards to <FIG>, below. In some instances, for example where cross-entropy and a SoftMax or hierarchical SoftMax output activation are used during training, the network may output pseudo-probabilities of the presence of each class, in cases such as these the pseudo-probabilities, or ratios between pseudo-probabilities may be used to form a likelihood or other confidence metric.

The method <NUM> further includes updating the at least one machine-learning network based on the measure of distance between the second classification information and the first classification information (<NUM>). This updating may generally include updating any suitable machine-learning network feature, such as network weights, architecture choice, machine-learning model, or other parameter or connectivity design, as further discussed in regards to <FIG>, below.

<FIG> illustrates an example system <NUM> of a training an RF system that implements at least one machine-learning network to learn how to identify RF signals. The system <NUM> includes a machine-learning network <NUM> that is trained to learn how to classify input RF signals <NUM> into predicted RF signal classification information <NUM>.

During training, the machine-learning network <NUM> takes an RF signal <NUM> as input and generates predicted RF signal classification information <NUM>, subject to constraints or other objectives. In general, the machine-learning network <NUM> may be configured to utilize any suitable mapping from the input RF signal <NUM> to generate the output predicted RF signal classification information <NUM>, examples of which are described further below. For example, in some implementations, the machine-learning network <NUM> may convert the RF signal <NUM> as raw time-domain samples or features into information (such as one or more likelihoods or probabilities) enabling the prediction of RF signal classification (such as class labels).

The system <NUM> then compares the resulting predicted RF signal classification information <NUM> with known classification information <NUM>, to compute a loss function <NUM> between the predicted and known classifications. The known classification information <NUM> may be obtained by various techniques, such as human labeling <NUM> and/or machine labeling <NUM> of the RF signal <NUM>. Alternatively or additionally, the known classification information <NUM> may include stored label information <NUM> that is associated with the RF signal <NUM>, which is discussed further below.

In general, classification information, such as the predicted classification information <NUM> and the known classification information <NUM>, may represent various features related to an RF signal. For example, classification information may represent a characteristic of the RF signal <NUM> and/or a characteristic of an environment in which the RF signal <NUM> is communicated. The classification information may, in some implementations, include human-readable information, such as human-readable labels.

As examples, the classification information may indicate a modulation type of RF signals (e.g. QPSK, BPSK, OOK, FSK, GMSK, OFDM, etc.); a type of protocol or standard that is carried on or that is utilized to transmit RF signals (e.g., GSM, LTE, Wi-Fi <NUM>, LTE-<NUM>-TDD, HSPA, P25, etc.); a specific radio device utilized to transmit or receive RF signals, a tower or user associated with RF signals. As another example, the classification information may indicate a type of traffic, behaviors, or contents being carried by the RF signal, such as normal usage on a wireless channel, or nefarious attacks on the wireless channel (e.g., flooding attacks).

As further examples, the classification information may indicate a type of RF emission or a type of RF emission pattern (e.g., a particular sequence of data bits occurring, which may indicate electromagnetic interference (EMI), malicious interference (e.g., snooping), or a noise source (e.g., communication channel noise or hardware noise). The type of RF emission or RF emission pattern may be utilized to determine specific events of interest, such as equipment malfunction (e.g., a broken cable on a base station generating impedance mismatch/inter-modulations, or a heater or electronic device emitting harmful EMI). As such, implementations disclosed herein may be utilized to identify EMI emitter types and/or identify wireless threats or anomalies (e.g. cyber-threats, attacks).

In yet another example, the classification information may indicate a presence of particular types of communication signals. For example, the classification may involve a binary decision indicating whether a particular type of RF signal is present, therefore enabling an RF signal detection technique.

The loss function <NUM> may be any suitable measure of distance between the two classifications. For example, the loss function <NUM> may indicate a cross-entropy, mean squared error, or other geometric distance metric (e.g., MSE, MAE, KL divergence) between the predicted classification information <NUM> and the known classification information <NUM>. In some implementations, the loss function <NUM> may combine several geometric and/or entropy-based distance metrics into an aggregate expression for distance.

The update process <NUM> may utilize the computed loss function <NUM> to update the machine-learning network <NUM>. In general, the update process <NUM> may perform updates based on the loss function <NUM>, as well as other performance metrics, such as network complexity of the machine-learning network <NUM>, or computational power or throughput. Other performance metrics may include a computational throughput or delay (e.g., time duration or operation count) achieved by the at least one machine-learning network in classifying RF signals, a computational dwell time associated with the at least one machine-learning network in classifying RF signals, or a computational duty cycle associated with the at least one machine-learning network in classifying RF signals. In some implementations, the update process <NUM> may additionally or alternatively include one or more hard constraints on such metrics.

The update process <NUM> may utilize various techniques to determine a suitable update of the machine-learning network <NUM>, such as an optimization method including evolution, gradient descent, stochastic gradient descent, or other solution techniques. In some implementations, the update process <NUM> may include user preferences or application specifications.

As an example, the update process <NUM> may seek to optimize (or nearly optimize) an objective function that combines one or more performance metrics, such as classification error or complexity, discussed above. In some implementations, the objective function may be a weighted combination of such metrics.

The update process <NUM> may calculate a rate of change of the objective function relative to variations in the machine-learning network <NUM>, for example by calculating or approximating a gradient of the objective function. Such variations may include, for example, variations in network layer weights or other network parameters, as discussed further below. The variation may be determined based on a desired change in the objective function, for example, using Stochastic Gradient Descent (SGD) style optimizers, such as Adam, AdaGrad, Nesterov SGD, or others. In some implementations, these variations may be determined using other scalable methods for direct search, such as evolutionary algorithms or particle swarm optimizations.

Once the variations have been determined, the update process <NUM> then applies those variations to update the machine-learning network <NUM>. For example, the update process <NUM> may update at least one network weight in one or more layers of the machine-learning network <NUM>. In general, updating the machine-learning network <NUM> is not limited to updating network weights, and other types of updates may be implemented, such as layer parameters, additions, multiplications, applications of activation functions, and other tunable algorithm specific features of the machine-learning network <NUM>. In some implementations, updating the machine-learning network <NUM> may include selecting a machine-learning model from among a plurality of models. In such implementations, selecting machine-learning models may include selecting a specific network architecture, such as choice of layers, layer-hyperparameters, or other network features.

The input RF signal <NUM> may be generated from a set of example RF signals in a training dataset <NUM>. The training dataset <NUM> may be generated by various techniques. For example, the training dataset <NUM> may be generated from a set of analog RF waveforms <NUM> that have been processed by an RF receiver <NUM>. The RF receiver <NUM> may perform sampling, filtering, and/or analog to digital (A/D) conversion of the analog RF waveforms <NUM>, to form a set of sampled time-series radio signal examples in the dataset <NUM>. Alternatively, or additionally, the training dataset <NUM> may be generated through simulation, such as software simulation of RF signals <NUM>. In general, the training dataset <NUM> consists of a large number of recordings or simulations of RF signal examples, in time or in another representation.

In some implementations, one or more RF signals from the training dataset <NUM> may be augmented to include additional effects, for example to model real-world phenomena or variations in signaling (e.g., hardware imperfections, wireless propagation, or other variations). For example, the particular RF signal <NUM> may be stored and labeled (e.g., with stored label information <NUM>) in the training dataset <NUM>. During training, when the RF signal <NUM> is processed by the machine-learning network <NUM>, one or more additional effects may be introduced in the RF signal <NUM> to broaden the types or number of examples of signals that are modelled by the RF signal <NUM>. For example, the system may introduce random phase/frequency offsets, time offsets, time dilations, delay spreads, fading effects, distortion effects, interference effects, spatial propagation effects, dispersion effects, differing contents, non-linear effects, noise, and/or other signal effects in the RF signal. Such effects may be implemented, for example, as regularization layers in the machine-learning network <NUM> or as other augmentation effects while training or while pre-processing the input data prior to training. As such, the training may be made more robust (e.g., to generalize well) to identify not only the particular RF signal <NUM>, but also to identify a range of RF signals that correspond to the RF signal <NUM> having been affected or perturbed by real-world variability (e.g., through a plurality of different propagation modes or differing channel state information).

The training dataset <NUM> may include a same type or different types of example RF signals. Depending on the composition of RF signals in the training dataset <NUM>, the machine-learning network <NUM> may learn to identify a certain type of radio signal or a wide range of different types of radio signals. For example, the training dataset <NUM> may be limited to a particular class of RF signals. In such scenarios, the machine-learning network <NUM> will be trained to learn predicted RF signal classification information <NUM> that are tuned to that particular class of RF signals. By training on different classes of RF signals, the machine-learning network <NUM> may learn to classify different classes of RF signals.

In some implementations, the machine-learning network <NUM> may determine an appropriate time-scale over which to process RF signals in the training dataset <NUM>. For example, the machine-learning network <NUM> may initially process the RF signal over a first time scale, and may then subsequently update the processing to a second time scale based on classification performance. By updating the duration over which identification is performed, the machine-learning system <NUM> is able to learn an appropriate time scale to identify the RF signal more accurately, thus taking into account time-variant characteristics of RF signals.

As an example, if classification error (e.g., the measure of distance between predicted and known classification information) does not satisfy threshold criteria, then the machine-learning network <NUM> may lengthen the time scale over which identification is performed, or combine the results of multiple classifications to obtain better accuracy over more samples of the input RF signal. As another example, if computational complexity or power does not satisfy threshold criteria, then the machine-learning network <NUM> may shorten the time scale over which identification is performed, to reduce computational burden.

In some implementations, the example RF signals in the training dataset <NUM> may be associated with stored label information <NUM>. The stored label information <NUM> may include labels that are stored for one or more RF signals in the training dataset <NUM>. In general, the stored label information <NUM> may include labels that represent various types of information regarding the RF signals in the training dataset <NUM>, as discussed regarding classification information, above.

In some implementations, the system <NUM> may implement a feature extraction process <NUM> that extracts additional statistics or features from the input RF signal <NUM>. In some implementations, the feature extraction process <NUM> may leverage expert information about the structure of a modulation of the RF signal <NUM>. For example, the feature extraction process <NUM> may leverage parameters such as higher-order-moments, or other integrated statistics about the input RF signal <NUM>, or in other instances it may perform steps such as synchronization or multicarrier extraction prior to classification. Such features may then be utilized with the machine-learning network <NUM> to form an expert-augmented machine-learning model which can be trained, as previously described. In some implementations, the feature extractor <NUM> may also be trained based on classification performance, such as based on the measure of distance between the predicted classification <NUM> and known classification <NUM>.

In some implementations, the machine-learning network <NUM> may process the input RF signal <NUM> using a set of basis functions. For example, the machine-learning network <NUM> may process the input RF signal <NUM> and compute projections onto the basis functions to determine basis coefficients that represent the input RF signal <NUM>. Using these basis coefficients, or using further transformations thereof, the machine-learning network <NUM> may determine appropriate RF signal classification information <NUM>, for example using clustering algorithms or other suitable classification algorithms.

The basis functions may be any suitable set of orthogonal or non-orthogonal set of basis functions that can represent the RF signal <NUM>. For example, the basis functions may be In-Phase and Quadrature-Phase (I/Q) signals, Fourier basis functions, polynomial basis functions, Gaussian basis functions, exponential basis functions, wavelet basis functions, or combinations of these and/or other suitable set of basis functions that can be utilized represent RF signals. The basis functions may have different phase, amplitude, and/or frequency components.

The machine-learning network may then project the input RF signal <NUM> onto the set of basis signals to determine basis coefficients. As a specific example, if the basis functions are Fourier basis functions, then the machine-learning network <NUM> may implement a bank of filters each tuned to a particular frequency, and may process the RF signal <NUM> by correlating the RF signal <NUM> with the filters to generate a set of basis coefficients. In some implementations, the basis functions may be parameterized and the training may involve optimizing over parameters of the basis functions.

The training of the machine-learning network <NUM> may begin with any suitable set of initial conditions. For example, the training may begin with a random set of basis functions subject to certain conditions. Alternatively, the training may begin with a fixed set of basis functions, such as commonly used RF communication basis functions including Quadrature Phase-Shift Keying (QPSK) or Gaussian Binary Frequency Shift Keying (GFSK), orthogonal frequency division multiple access (OFDM), or other fixed set of basis functions. In other instances, the training may begin with a set of initial conditions which has been learned in the training process for another set of data (e.g., similar signals, or different set of examples of the same signals).

During training, the machine-learning network <NUM> may attempt to learn improved basis functions, according to the results of the predicted RF classification information <NUM>. In such scenarios, training the machine-learning network <NUM> may involve optimizing over a set of basis functions or over different sets of basis functions, for example using greedy search or other optimization-type algorithm. However, implementations are not limited to using basis functions, and the machine-learning network <NUM> may perform any suitable processing of the input RF signal <NUM> to predict the RF signal classification information <NUM>.

Upon achieving a desired objective during training, a particular choice of model architecture, network weights, and training parameters are selected and stored to represent the model that will be implemented in a deployed system.

<FIG> is a flowchart illustrating an example method <NUM> of deploying an RF system implementing at least one machine-learning network that has been trained to perform learned identification of RF signals. Such deployment may utilize at least one machine-learning network (e.g., machine-learning network <NUM> in <FIG>, above) that has been previously trained, for example by using a training technique as shown in <FIG>, above, or similar training techniques. In some implementations, the machine-learning network may be an artificial neural network, as discussed above, that has been trained to learn a set of network features (e.g., network layer weights, parameters, and/or network architectures) to achieve a performance-related objective.

The method <NUM> includes determining at least one machine-learning network has been trained to classify RF signals configured to be transmitted through an RF band of a communication medium (<NUM>). For example, at least one trained machine-learning network may have been previously trained using a training technique as shown in <FIG>, above, or similar training techniques.

The method <NUM> further includes setting at least one parameter of an RF receiver based on the at least one trained machine-learning network or based on updates to the machine-learning network during deployment (<NUM>). The RF receiver and its parameter settings may be implemented, for example, similar to RF receiver <NUM> in <FIG>, below.

The method <NUM> further includes using the RF receiver to receive an analog RF waveform from an RF spectrum of the communication medium, and to process the analog RF waveform to generate a discrete-time representation of the analog RF waveform as a received RF signal (<NUM>). For example, the RF receiver may include one or more antennas to receive an analog RF waveform over the air (or other medium), and may process the analog RF waveform to generate an RF signal (e.g., RF signal <NUM> in <FIG>, above) that is to be classified.

The method <NUM> further includes using the at least one trained machine-learning network to process the received RF signal and generate predicted RF signal classification information (<NUM>). The predicted RF signal classification information (e.g., predicted classification information <NUM> in <FIG> or <NUM> in <FIG>, above) may include a representation of at least one of a characteristic of the received RF signal or a characteristic of an environment in which the received RF signal was communicated.

The RF signal classification information may be stored in memory, or transmitted to another automated system over some communications bus. In some implementations, the RF signal classification information may be displayed to a user directly or as an annotation on top of a displayed visualization of the received RF signal (e.g., as a spectrogram or power spectral density).

<FIG> illustrates an example system <NUM> of deploying an RF system including at least one machine-learning network that has been trained to perform learned identification of RF signals. The system <NUM> deploys learned radio signal identification based on pre-trained network features, allowing a model trained as in <FIG>, above, to be leveraged for identification in an efficient manner.

System <NUM> includes a machine-learning network <NUM> that receives input RF signals <NUM> and generates predicted RF signal classification information <NUM>. The system <NUM> further predicts the most likely type(s) of radio signals <NUM> that correspond to the input RF signal <NUM>, based on the generated signal classification information <NUM>. However, implementations are not limited to predicting the type(s) of radio signals, and in general may utilize the generated signal classification information <NUM> for any suitable classification purpose, as discussed in regards to <FIG>.

In this example system <NUM>, a radio waveform <NUM> is received and processed by a radio receiver <NUM>, which may perform tuning and filtering to an appropriate radio band. The RF processed waveform may be converted to digital representation, for example by an analog to digital converter in the receiver <NUM>, to generate an RF signal <NUM> as a discrete-time series of sampled values.

The machine-learning network <NUM> then utilizes a set of network parameters <NUM>, such as network weights, biases, architecture choices, etc., which have been learned through training or architecture search, as in <FIG>, above, to process the RF signal <NUM>. For example, the machine-learning network <NUM> may implement a neural network classification model that has previously been trained, e.g., as described with respect to <FIG>, by loading previously learned and stored model architecture choices, network weights and parameters <NUM> from training. In some implementations, the machine-learning network <NUM> may be further trained or refined during deployment, for example by utilizing feedback and/or metrics to determine performance of the classification.

In some implementations, the system <NUM> further includes system controller <NUM> for control of radio tuning and attention of the RF receiver <NUM>. The system controller <NUM> may be is used to isolate and extract signals from a large collection of various types of signals present within the radio spectrum, for example, by controlling tuning, filtering, or channelization parameters in the RF receiver <NUM>. The controller <NUM> may set at least one of a tuning parameter, a filtering parameter, a scheduling of operations, or a channelization parameter of the RF receiver <NUM> according to the trained at least one machine-learning network. In some implementations, the controller <NUM> may control the RF receiver <NUM> to effect scheduling and execution of changes in the model architecture, channelization of the RF receiver and input signals, target class labels, or specific application configuration or task based on the outcome of the classification information. In general, the controller <NUM> may be configured to control other aspects of the system <NUM> as the machine-learning network is trained. For example, the controller <NUM> may control the scheduling of one or more software operations that are executed by the system <NUM>.

In some implementations, the system <NUM> may also include a feature extractor <NUM> to provide additional inputs into the machine-learning network <NUM>, e.g., as discussed in regards to <FIG>, above. The features that are extracted by feature extractor <NUM> may be used in conjunction with the RF signal <NUM> itself when deploying a production classification system for a given training scenario and dataset.

As such, the system <NUM> described above may be used to provide best estimates of over-the-air RF signals <NUM> to determine the type of signal that is present at the network output values <NUM>. The network output values <NUM> can be used to compute which are the most likely signals <NUM> that are probabilistically likely to correspond to the input signal <NUM>, and the resulting information may be transmitted to a user or a control system.

In general, the classification information <NUM> may be leveraged by various types of systems or processes to perform analysis of RF signals. For example, a set of automated signal reasoning systems may process the classification information <NUM> to analyze the RF input signals <NUM> to determine a similarity or difference between different RF signals, or attempt to label the RF signals with additional information. In some implementations, the automated reasoning system may determine a change of one or more RF signals <NUM> over time, predict future values of the RF signal <NUM>, predict the underlying state of a communications system through which the RF signal <NUM> was received, or determine a difference between several known types or modes of underlying RF signals. In some implementations, the automated reasoning system may implement training mappings from the classification information <NUM> to alternate sequences of representations, such as words, bits, other sparse representations.

In some implementations, a user interface may be implemented to enable user interaction with the classification information <NUM>. The user interface may generate output results that help explain to a user what types of RF signals are present, what actions may be occurring in the radio or physical environments, what spectrum resources are used or free, or other types of high-level human-interpretable information about the environment determined from a radio antenna that resulted in the original RF signal <NUM>.

<FIG> is a flowchart illustrating an example method <NUM> of using at least one machine-learning network to process an RF signal and generate predicted signal RF signal classification information by determining a plurality of basis signals. The method <NUM> may be implemented during training and/or deployment, for example, in step <NUM> of <FIG> and/or in step <NUM> of <FIG>. Further details of using basis signals were discussed in regards to <FIG>, above.

In this example, the method <NUM> includes determining a plurality of basis signals that can be used to represent the received RF signal (<NUM>). As discussed in regards to <FIG>, above, the basis functions may be any suitable set of orthogonal or non-orthogonal set of basis functions, such as In-Phase and Quadrature-Phase (I/Q) signals, Fourier basis functions, polynomial basis functions, Gaussian basis functions, exponential basis functions, wavelet basis functions, or combinations of these and/or other suitable set of basis functions.

The method <NUM> further includes processing the received RF signal using the plurality of basis signals to generate a plurality of basis coefficients for the received RF signal (<NUM>). For example, as discussed in regards to <FIG>, above, the input RF signal may be projected onto the plurality of basis signals to determine the basis coefficients. As a specific example, if the basis signals are Fourier basis functions, then a bank of filters may be implemented, each filter tuned to a particular frequency to process the input RF signal, to generate a corresponding plurality of basis coefficients.

The method <NUM> further includes performing classification of the RF signal based on the plurality of basis coefficients (<NUM>). The classification may directly utilize the basis coefficients, or may utilize further transformations of the basis coefficients. For example, the classification may involve using clustering algorithms or other suitable classification algorithms that determine an appropriate classification of the RF signal based on the basis coefficients.

The method <NUM> further includes outputting a result of the classification as the predicted RF signal classification information (<NUM>). For example, as discussed in regards to <FIG> and <FIG>, above, the classification information may include one or more labels, and may indicate deterministic or statistical classification information (e.g., likelihood probabilities).

<FIG> is a diagram illustrating an example of a computing system that may be used to implement one or more components of an RF system that implements at least one machine-learning network to perform learned identification of RF signals.

The example system of <FIG> shows a computing device <NUM> and a mobile computing device <NUM> that can be used to implement the techniques described herein. For example, one or more parts of an encoder machine-learning network system or a decoder machine-learning network system could be an example of the system <NUM> described here, such as a computer system implemented in any of the machine-learning networks, devices that access information from the machine-learning networks, or a server that accesses or stores information regarding the RF signal identification techniques performed by the machine-learning networks.

The mobile computing device <NUM> is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart-phones, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to be limiting.

The computing device <NUM> includes a processor <NUM>, a memory <NUM>, a storage device <NUM>, a high-speed interface <NUM> connecting to the memory <NUM> and multiple high-speed expansion ports <NUM>, and a low-speed interface <NUM> connecting to a low-speed expansion port <NUM> and the storage device <NUM>. Each of the processor <NUM>, the memory <NUM>, the storage device <NUM>, the high-speed interface <NUM>, the high-speed expansion ports <NUM>, and the low-speed interface <NUM>, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor <NUM> can process instructions for execution within the computing device <NUM>, including instructions stored in the memory <NUM> or on the storage device <NUM> to display graphical information for a GUI on an external input/output device, such as a display <NUM> coupled to the high-speed interface <NUM>. In addition, multiple computing devices may be connected, with each device providing portions of the operations (e.g., as a server bank, a group of blade servers, or a multi-processor system). In some implementations, the processor <NUM> is a single-threaded processor. In some implementations, the processor <NUM> is a multi-threaded processor. In some implementations, the processor <NUM> is a quantum computer.

In some implementations, the storage device <NUM> may be or may include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configurations.

The high-speed interface <NUM> manages bandwidth-intensive operations for the computing device <NUM>, while the low-speed interface <NUM> manages lower bandwidth-intensive operations. Such allocation of functions is an example only. In some implementations, the high-speed interface <NUM> is coupled to the memory <NUM>, the display <NUM> (e.g., through a graphics processor or accelerator), and to the high-speed expansion ports <NUM>, which may accept various expansion cards (not shown). In the implementation, the low-speed interface <NUM> is coupled to the storage device <NUM> and the low-speed expansion port <NUM>. The low-speed expansion port <NUM>, which may include various communication ports (e.g., LiSB, Bluetooth, Ethernet, wireless Ethernet) may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

Each of such devices may include one or more of the computing device <NUM> and the mobile computing device <NUM>, and an entire system may be made up of multiple computing devices communicating with each other.

The memory <NUM> stores information within the mobile computing device <NUM>. An expansion memory <NUM> may also be provided and connected to the mobile computing device <NUM> through an expansion interface <NUM>, which may include, for example, a SIMM (Single In Line Memory Module) card interface. The expansion memory <NUM> may provide extra storage space for the mobile computing device <NUM>, or may also store applications or other information for the mobile computing device <NUM>. Specifically, the expansion memory <NUM> may include instructions to carry out or supplement the processes described above, and may include secure information also. Thus, for example, the expansion memory <NUM> may be provide as a security module for the mobile computing device <NUM>, and may be programmed with instructions that permit secure use of the mobile computing device <NUM>.

The memory may include, for example, flash memory and/or NVRAM memory (non-volatile random access memory), as discussed below. In some implementations, instructions are stored in an information carrier such that the instructions, when executed by one or more processing devices (for example, processor <NUM>), perform one or more methods, such as those described above. The instructions can also be stored by one or more storage devices, such as one or more computer- or machine-readable mediums (for example, the memory <NUM>, the expansion memory <NUM>, or memory on the processor <NUM>). In some implementations, the instructions can be received in a propagated signal, for example, over the transceiver <NUM> or the external interface <NUM>.

The mobile computing device <NUM> may communicate wirelessly through the communication interface <NUM>, which may include digital signal processing circuitry. The communication interface <NUM> may provide for communications under various modes or protocols, such as GSM voice calls (Global System for Mobile communications), SMS (Short Message Service), EMS (Enhanced Messaging Service), or MMS messaging (Multimedia Messaging Service), CDMA (code division multiple access), TDMA (time division multiple access), PDC (Personal Digital Cellular), WCDMA (Wideband Code Division Multiple Access), CDMA2000, or GPRS (General Packet Radio Service), among others. Such communication may occur, for example, through the transceiver <NUM> using a radio frequency. In addition, short-range communication may occur, such as using a Bluetooth, Wi-Fi, or other such transceiver (not shown). In addition, a GPS (Global Positioning System) receiver module <NUM> may provide additional navigation- and location-related wireless data to the mobile computing device <NUM>, which may be used as appropriate by applications running on the mobile computing device <NUM>.

The term "system" as used in this disclosure may encompass all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. A processing system can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

A computer program (also known as a program, software, software application, script, executable logic, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile or volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks or magnetic tapes; magneto optical disks; and CD-ROM and DVD-ROM disks. Sometimes a server is a general-purpose computer, and sometimes it is a custom-tailored special purpose electronic device, and sometimes it is a combination of these things.

Implementations can include a back end component, e.g., a data server, or a middleware component, e.g., an application server, or a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described is this specification, or any combination of one or more such back end, middleware, or front end components.

The features described can be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. The apparatus can be implemented in a computer program product tangibly embodied in an information carrier, e.g., in a machine-readable storage device, for execution by a programmable processor; and method steps can be performed by a programmable processor executing a program of instructions to perform functions of the described implementations by operating on input data and generating output. The described features can be implemented advantageously in one or more computer programs that are executable on a programmable system including at least one programmable processor coupled to receive data and instructions from, and to transmit data and instructions to, a data storage system, at least one input device, and at least one output device. A computer program is a set of instructions that can be used, directly or indirectly, in a computer to perform a certain activity or bring about a certain result.

While this disclosure contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features that may be specific to particular implementations of particular inventions. Certain features that are described in this disclosure in the context of separate implementations can also be implemented in combination in a single implementation.

Claim 1:
A method (<NUM>) of training at least one machine-learning network to classify radio frequency "RF" signals, the method performed by at least one processor executing instructions stored on at least one computer memory coupled to the at least one processor, the method comprising:
determining (<NUM>) an RF signal that is configured to be transmitted through an RF band of a communication medium;
extracting one or more features of the RF signal using prior knowledge about the RF signal;
determining (<NUM>) first classification information associated with the RF signal, the first classification information comprising a representation of at least one of a characteristic of the RF signal or a characteristic of an environment in which the RF signal is communicated;
providing the RF signal and the extracted one or more features of the RF signal as inputs to a machine-learning network to generate (<NUM>) second classification information as a prediction of the first classification information;
calculating (<NUM>) a measure of distance between (i) the second classification information that was generated by the at least one machine-learning network, and (ii) the first classification information that was associated with the RF signal; and
updating (<NUM>) the at least one machine-learning network based on the measure of distance between the second classification information and the first classification information.