Patent Description:
<CIT> refers to a solution of detecting faults that affect communication links.

<CIT> refers to impairment diagnosis systems in communication systems.

<CIT> B <NUM> discloses an approach for impairment diagnosis in communication systems.

Subscriber networks sometimes experience physical layer impairments that can result in service degradation, service interruption, poor quality of customer experience, among others. For example, a bad splice, e.g., a poor connection between one or more electrical wires in a copper network, can impair or sometimes prevent transmission of electrical signals along a line of a subscriber network. Other examples of physical impairments, such as conductors not having proper electrical contacts can appear and behave like a bad splice. The presence of physical impairments can make a telecommunications system unreliable. Additionally, depending on its location, the presence of a physical impairment on a line in vectored systems can have network impacts on other lines, such as lines that are dependent on signals transmitted along the line with physical impairments, which then impacts customer experience on these other lines.

Physical impairments in a subscriber network can be challenging to locate and monitor. For example, a bad splice on a line within a copper network can be challenging to locate because they can vary over time, are impacted by variable environmental conditions, and may not be apparent from a visual inspection. Furthermore, in a vectored system, since ports are mathematically coupled, physical impairments on a given line may affect communications on other lines of the system, and a line experiencing service problems may not necessarily be the line that has the physical impairment.

As described herein, "per-tone data" refers to data corresponding to each of multiple different tones in the upstream or downstream ends of a DSL line. For instance, in systems that are capable of multi-channel transmission techniques (e.g., Discrete Multi-tone (DMT)), a "tone" can represent one sub-carrier, and DMT can use <NUM> tones to carry bits/data for ADSL and each tone can carry up to <NUM> bits. As one example of per-tone data, a per-tone signal-to-noise ratio (SNR) refers to the SNR associated with modulating tones at certain kilohertz ranges.

In DSL networks, per-tone data is highly dimensional, e.g., XLIN data is a 384x384x256 matrix, which makes it challenging to identify the presence of physical impairments. Additionally, techniques commonly used for analytical models, such as dual-ended line testing (DELT), to detect the presence and/or location of physical impairments causes service disruptions. Given that physical impairments are time-varying and can affect other lines, obtaining network data using a DELT that characterizes the physical impairments without creating service disruptions is often challenging.

To address these and other limitations, systems and techniques disclosed herein use machine learning techniques to dynamically detect physical impairments in lines of a subscriber network. Physical impairments can be detected by developing predictions based on primary per-tone data, such as XLIN, HLOG, SNR, QLN, as well as secondary information, such as modem retrain reasons, changes in loop lengths over time, among others. A machine learning model can be trained to make predictions using feature subset ensembles representing different network measurements. The model can use a form of stacked generalization that increases performance, allows inspectability, and does not require all of the primary and secondary data points to be present. Depending on how much of the considered data is fed to the model, the output probability can be scaled based on the learned importance of each piece of data. The use of the techniques described throughout this document enable the detection of physical impairments in lines of a subscriber network without having to take the subscriber network offline. The techniques can also be used to allow a computing system to perform self-healing processes to improve the repair and/or maintenance of access network resource devices.

This document describes methods, systems, and apparatus for predicting a physical impairment on a line of a subscriber network with minimal or no service disruptions. For example, a system can evaluate subsets of feature ensembles representing different network measurements to determine the likelihood that a physical impairment exists on the line. Physical impairments can be detected by developing predictions based on primary per-tone data, such as XLIN, HLOG, SNR, QLN, as well as secondary information, such as retrain reasons, changes in loop lengths over time, among others. The predictions can be performed by training an ensemble machine learning model (referred throughout as a "prediction model").

The prediction model disclosed herein can use a form of stacked generalization that increases performance, allows inspectability, and does not require all of the primary and secondary data points to be present. Depending on how much of the considered data is fed to the model, the output probability can be scaled based on the learned importance of each piece of data.

As described herein, a "subscriber network" refers to a telecommunications network that can be used to transmit digital data to customers. Examples of "subscriber networks" include a Digital Subscriber Line (DSL) network used to digital data over telephones or Passive Optical Network (PON) implementing point-to-multipoint architectures to provide fiber to end consumers. In some instances, the DSL network can be configured to implement a "G. fast" protocol standard for local loops shorter than <NUM> meters, with performance targets between <NUM> and <NUM> Gbit/s, depending on loop length. In such instances, high speed can be achieved over very short loops.

As described herein, "XLIN" refers to data representing availability of actual crosstalk coupling among pairs of a vectored system, in upstream and downstream directions. For example, in vectored DSL systems, XLIN can be reported through a management interface for diagnostics, troubleshooting, management, and planning.

As described herein, "SNR" refers to signal-to-noise ratio associated with signals transmitted over a subscriber network. The SNR of a network can be expressed as a ratio between signal strength and signal noise such that a higher SNR represents stronger signal quality. For example, SNR data can be used to indicate how modems have analyzed a line and determined SNR per tone. This data can change over time as line conditions change due to temperature variations or moisture in a cable.

As described herein, "QLN" refers to quiet line noise data gathered by modems when no signal is active on a line during initialization. QLN indicates noise levels in dBm/Hz over frequency) across applicable DSL spectrum in use. For example, QLN can be used to show noise spikes, which may be indications of high crosstalk.

As described herein, "HLOG" refers to data reported during a modem initialization phase and used to show attenuation over frequency. For example, a clean line HLOG should exhibit a plot where the slope of the attenuation curve typically declines slowly and evenly from lower frequencies to higher frequencies. Thus, increases in HLOG is a signature of a network problem. In some instances, HLOG data can be represented as a plot representing the relationship between transfer function magnitude (represented in decibels, dB) and tone frequency.

As described herein, "BAT" refers to a bit allocation table that specifies how many bits are used and/or can be used within a sub-carrier channel. The bit allocation table is maintained by modem routers of a subscriber network, and can be used for network diagnostics.

As described herein, a "loop length" refers to the length of a local loop of a subscriber network from a DSLAM located in a central office or remote terminal. The loop length is inversely correlated with maximum connection speed such that maximum DSL connection speed decreases as the loop length increases.

<FIG> is a diagram illustrating an example of machine learning architecture <NUM> for predicting a physical impairment on a line of a subscriber network. The architecture <NUM> includes multiple estimator models, such as a XLIN model 110A, a SNR model 110B, a QLN model 110C, a HLOG model 110D, a retrain model 110E, a loop length model 110F, and a BAT model <NUM>. The architecture <NUM> also includes a prediction model <NUM>.

As shown in <FIG>, estimator models 110A-D receive primary per-tone data <NUM> as input and estimator models 110E-G receive secondary per-tone data <NUM> as input. The estimator models 110A-G evaluate the respective input data to compute a set of scores <NUM> that are provided as input to the prediction model <NUM>. The prediction model <NUM> evaluates the input data and outputs a confidence score <NUM> representing an overall likelihood that a physical impairment is detected on the line of the subscriber network.

The estimator models 110A-G can be level-<NUM> estimators that are trained to evaluate a given feature subset ensemble of network data generated for the subscriber network. For example, the XLIN model 110A is trained to evaluate network data with respect to an XLIN feature subset ensemble and output a score representing a conditional probability that a physical impairment is detected on a line of a subscriber network given the evaluation of the network data with respect to the XLIN feature subset ensemble. In this respect, each of the estimator models 110A-G evaluate network data with respect to different feature subset ensembles and thereby compute conditional probabilities representing different types of likelihoods that a physical impairment exists based on unique evaluation criteria. Since network performance impacts resulting from physical impairments can be challenging to detect, using conditional probabilities can improve the predictive accuracy of physical impairment detection by considering multiple factors that are impacted by physical impairments (e.g., multiple feature subset ensembles).

The prediction model <NUM> can be a machine learning model that is trained to output, for different subsets of feature ensembles, a confidence score <NUM> representing an overall likelihood that a physical impairment is detected on a line of a subscriber network. In some instances, the prediction model <NUM> is any suitable neural network that employs multiple layers of operations to predict one or more outputs from one or more inputs.

The prediction model <NUM> can include an input layer that receives per-tone data as input. The per-tone data includes scores computed by the estimator models 110A-G representing conditional probabilities of a detected physical impairment given a specific feature subset ensemble, as discussed above. The prediction model <NUM> can also include one or more hidden layers situated between an input layer and an output layer. The output of each layer can be used as input to another layer in the network, e.g., the next hidden layer or the output layer. Each layer of a prediction model <NUM> can specify one or more transformation operations to be performed on input to the layer. Some layers have operations that are referred to as neurons. Each neuron receives one or more inputs and generates an output that is received by another neural network layer. Often, each neuron receives inputs from other neurons, and each neuron provides an output to one or more other neurons.

The architecture of the prediction model <NUM> can specify what layers are included in the network and their properties, as well as how the neurons of each layer of the network are connected. In other words, the architecture specifies which layers provide their output as input to which other layers and how the output is provided. The transformation operations of each layer can be performed by computers having installed software modules that implement the transformation operations. Thus, a layer being described as performing operations means that the computers implementing the transformation operations of the layer perform the operations. Each layer generates one or more outputs using the current values of a set of parameters for the layer. Training the prediction model <NUM> can therefore involve continually performing a forward pass on the input, computing gradient values, and updating the current values for the set of parameters for each layer. Once the prediction model <NUM> is trained, the final set of parameters can be used to make predictions in a production system.

The prediction model <NUM> can be a simple neural network, such as a perceptron, that assigns weights to each of the inputs received from the estimator models 110A-G. This provides a simple way for the prediction model <NUM> to interpret the importance of each data point. In other implementations, the prediction model <NUM> is a multilayer perceptron that employs more challenging evaluation techniques but better captures tradeoffs. For example, if the XLIN model 110A predicts a physical impairment based on a XLIN feature subset ensemble, but the SNR model 110B does not predict a physical impairment based on a SNR feature subset ensemble, the prediction model <NUM> can independently evaluate the per-tone data to resolve the competing conditional probabilities. In doing so, the prediction model <NUM> can evaluate, for example, the reliability of each conditional probability, a precision level associated with the conditional probability, historical assessments of the estimator models, among other factors.

In some implementations, the prediction model <NUM> is a convolutional network. Convolutional neural networks include convolutional neural network layers. Convolutional neural network layers have a neuron connectivity that takes advantage of spatially local correlation in the input data. To do so, convolutional neural network layers have sparse connectivity, with neurons in one convolutional layer receiving input from only a small subset of neurons in the previous neural network layer. The other neurons from which a neuron receives its input defines a receptive field for that neuron. Convolutional neural network layers have one or more filters, which are defined by parameters of the layer. A convolutional neural network layer generates an output by performing a convolution of each neuron's filter with the layer's input. In addition, each convolutional network layer can have neurons in a three-dimensional arrangement, with depth, width, and height dimensions. The width and height dimensions correspond to the two-dimensional features of the layer's input. The depth-dimension includes one or more depth sublayers of neurons. Generally, convolutional neural networks employ weight sharing so that all neurons in a depth sublayer have the same weights. This provides for translation invariance when detecting features in the input. Convolutional neural networks can also include fully-connected layers and other kinds of layers. Neurons in fully-connected layers receive input from each neuron in the previous neural network layer.

In some implementations, the prediction model <NUM> receives only a subset of the data output by the estimator models 110A-G. For example, in some instances, only XLIN and SNR feature ensembles are evaluated by the XLIN model 110A and the SNR model 110B, but QLN features ensembles are not evaluated by the QLN model 110C. In this example, where not all of the per-tone data is present, the QLN model 110C is bypassed and the conditional probability typically outputted by the QLN model 110C is set to a predetermined static value (e.g., <NUM>). In this example, the prediction model <NUM> is allowed to assess tradeoffs among the remaining data points even with missing data.

The techniques of gathering data and making predictions discussed above can be performed with minimal or no impact on service associated with the subscriber network. In other words, the subscriber network can remain in showtime while the data is gathered, and the predictions made, such that the subscriber network can continue to provide service to subscribers while these techniques are carried out. As described herein, "showtime" refers to the state that is reached after an initialization procedure has been completed in which channel data is transmitted. Lines that are in showtime therefore represent lines that are in a state of post-initialization data transmission after the initialization procedure has been completed.

The prediction model <NUM> therefore provide improvements over other prediction techniques that often employ double ended line testing (DELT) or single ended line testing (SELT), which require the subscriber network to be taken offline (i.e., taken out of showtime). Additionally, in the case of very-high-bit-rate digital subscriber line (VDSL) networks, the prediction model <NUM> can be used to consider an entire vectoring system (i.e., that includes multiple lines), not just an individual line. Predictions and data from prediction model <NUM> can be aggregated over time, which allow a number of unique advantages. For example, the time-varying nature of physical impairments can be addressed by taking a window over a set of predictions rather than relying on a single prediction at a single time. As another example, cause and effect behavior can be addressed through successive operations. For instance, if no physical impairment was initially observed, a later observation that the noise floor on other lines subsequently rose can be used to modify the initial prediction that a physical impairment was actually observed.

<FIG> are diagrams illustrating an example of an impairment detection system <NUM> that uses predicted physical impairments to repair components of a subscriber network. Referring initially to <FIG>, an example of a self-healing repair process is depicted. The self-healing repair process can be used by the system <NUM> to repair components of a subscriber network based on predicting the detection of physical impairments on lines of the subscriber network.

The system <NUM> can be deployed by a service provider that is associated with (e.g., that provides, operates, and/or maintains) a subscriber network. For example, the system <NUM> can be implemented on one or more computing devices of a facilities-based service provider that manages service provided to customers over the subscriber network, e.g., a member organization listed on the Cellular Telecommuting & Internet Association (CTIA). In other examples, the system <NUM> can be implemented on one or more computing devices of a third-party provider that is contracted by a facilities-based service provider to manage service interruptions and/or other performance-related impacts associated with the subscriber network.

As shown in <FIG>, the system <NUM> evaluates network data <NUM> collected for the subscriber network using a set of monitoring catalogs <NUM>. The network data <NUM> can include primary per-tone data, such as XLIN, HLOG, SNR, QLN, as well as secondary information such as retrain reasons, changes in loop length over time, or a bit allocation table. The monitoring catalogs <NUM> specify triggers and conditions associated with the subscriber network, such as service assurance or fault scenarios, and repair actions to be performed in response to satisfaction of certain triggers and conditions. Examples of monitoring catalogs <NUM> are depicted in <FIG> and discussed in detail below.

In a typical self-healing process, the system <NUM> performs operations that include, but are not limited to, gathering data, scanning data, diagnosing data, and performing a repair action (or providing a recommendation). Specifically, the system <NUM> gathers the network data <NUM>. The system <NUM> scans for conditions and/or triggers specified by the monitoring catalogs <NUM> by, for instance, applying the estimator models 110A-G to evaluate the gathered network data with respect to different feature subset ensembles.

The system <NUM> then diagnoses the gathered network data based on the evaluation. The diagnosis can include determining the likelihood that one or more physical impairments are on lines of the subscriber network, locations of physical impairments along the lines, identified network conditions that might be resulting in performance degradation, among others.

The system <NUM> prepares a report <NUM> that includes the diagnoses for the subscriber network and/or recommended repair actions that can be performed to address the diagnoses. The report <NUM> can be provided to technician <NUM> of a service provider that manages the subscriber network. For example, the report <NUM> can identify a predicted location of detected physical impairment along a line of the subscriber network, e.g., a distance from a Digital Subscriber Line Access Multiplexer (DSLAM) in a copper network. As another example, the report <NUM> can identify a type of physical impairment that is detected on the line, e.g., a bad splice. In some instances, the report <NUM> includes relevant network information that can be used by the technician <NUM> for additional diagnosis or identify the cause of the physical impairment. For example, the report <NUM> can include a monitored temperature, a monitored humidity, or prior detected physical impairments in nearby locations. This information can be used in conjunction with the location of the impairment (e.g., feet from the access device) by the technician <NUM> to identify a root cause of the physical impairment or determine whether the physical impairment may be caused by a larger problem with the line or other associated lines in a vectored system.

Once the report <NUM> has been provided to the technician <NUM>, the system <NUM> can monitor repair or maintenance actions performed on the line to map actions to diagnoses. This information can be used to retrain the estimator models 110A-G and/or the prediction model <NUM> or adjust subsequent prediction techniques. For example, the system <NUM> can use the performed actions to determine whether an automated diagnosis was correct, and in response, use this determination to augment predictions performed at a later time point at the same location. As another example, if the actions performed indicate that the prediction was incorrect (i.e., the technician provides confirmation that no physical impairment was found at the predicted location), then the system <NUM> can use this information as a feedback mechanism to re-train and/or re-calibrate the estimator models 110A-G or the prediction model <NUM> to reduce the likelihood of subsequent false-positive predictions. In this way, a feedback loop is created that enables the prediction model to continue to learn which actions to recommend when certain predictions are made, and to more accurately make predictions.

Referring now to <FIG>, examples of catalogs used by the system <NUM> during the self-healing repair process are depicted. In this example, the system <NUM> performs a self-healing process <NUM> similar to the self-healing process discussed above in reference to <FIG>. Specifically, the system <NUM> uses a set of catalogs <NUM> to predict physical impairments that are detected in the network infrastructure <NUM>.

In the example depicted in <FIG>, the system <NUM> monitors and evaluates network data collected by the access network infrastructure <NUM> for performance degradation due to physical impairments. The access network infrastructure <NUM> includes virtual network functions (VNFs) <NUM> that are used for managing performance issues associated with physical resources <NUM>. The VNFs can include traffic management modules, a virtual subscriber session manager (vSSM), and virtual customer premises equipment (vCPE). As described herein, a "module" can refer to software and/or a combination of software and hardware (e.g., one or more processors). The VNFs <NUM> can include software that is used to deliver managed services, such as managing network deployments, or configuring on-premises hardware. The physical resources <NUM> include access network devices associated with the subscriber network, such as routers, modems, power supplies, network access devices, among others.

As shown in <FIG>, the system <NUM> initially gathers data network collected by physical resources <NUM>. The system <NUM> obtains a service assurance catalog <NUM>, which provides a set of tests to be performed to determine whether service over the subscriber network meets certain specifications. In the example depicted in <FIG>, the service assurance catalog <NUM> includes service assurance specifications for a PON and a DSL. The system <NUM> selects the appropriate service assurance specification depending on the type of subscriber network operated by the physical resources <NUM>. Each of the specifications can identify, for example, network performance, customer complaints, fault tolerances, workflow management, among other types of information related to network Quality-of-Service (QoS).

The system <NUM> scans the gathered data using a fault scenario catalog <NUM> to diagnose any network performance issues detected for the physical resources <NUM> using, for instance, root cause analysis. The fault scenario catalog <NUM> includes examples of physical impairments that are likely to cause performance degradation, such as bridged taps, bad splices, fiber faults, capacitive coupling, splitter shelf manufacturing defects, or microbends/macrobends. As discussed throughout, the system <NUM> predicts whether one or more of the physical impairments specified by the fault scenario catalog have been detected on the lines of the subscriber network based on evaluating different feature subset ensembles in the gathered data.

If the system <NUM> predicts that one or more physical impairments have been detected on the lines of the subscriber network, the system <NUM> then accesses a repair action catalog <NUM> to identify the appropriate action to be performed. The repair action catalog <NUM> specifies maintenance or repair actions that can be performed to address the detected physical impairments. In some examples, a repair action includes automated actions that can be executed without intervention by a technician <NUM>, such as rebooting network components or adjusting network traffic parameters. In other examples, a repair action includes recommendations provided to a technician <NUM> to perform additional diagnosis and/or maintenance, displaying a repair profile, providing a list of replacement options for certain network components that are likely to be malfunctioning, or increasing the INP.

The system <NUM> generates a report that includes repair actions that are identified as being relevant to the diagnoses of the subscriber network as discussed above in reference to <FIG>. In some instances, where automated repair actions are selected from the repair action catalog <NUM>, the system <NUM> may automatically execute a repair action. Alternatively, where the system <NUM> determines that additional intervention may be needed, the system <NUM> generates a report that includes pertinent information and provides the report for output to a service provider system.

<FIG> is a diagram illustrating an example of self-healing process for generating repair reports based on monitoring network data generated by a subscriber network. At <NUM>, data for a line of a subscriber network is obtained. The data can indicate a set of one or more scores. The scores indicate a conditional likelihood that the line has a type of impairment with respect to a different feature ensemble. For instance, as depicted in <FIG>, the set of one or more scores can be computed by the estimator models 110A-G based on evaluating a different type of network measurement associated with the subscriber network. As shown in <FIG>, the estimator models can include a XLIN model 110A, a SNR model 110B, a QLN model 110C, a HLOG model 110D, which are trained to evaluate primary per-tone data <NUM> of the subscriber network. The estimator models also include other models, such as a retrain model 110E, a loop length model 110F, and a BAT model <NUM>, which are trained to evaluate secondary data <NUM>.

As an example, the XLIN model 110A receives network data of the subscriber network as input and evaluates the data using an XLIN feature subset ensemble. The XLIN model 110A generates a score representing a conditional probability Pi reflecting the likelihood of a physical impairment given only the information included in the XLIN feature subset ensemble. As another example, the SNR model 110B receives network data of the subscriber network as input and evaluates the data using an SNR feature subset ensemble. The SNR model 110B generates a score representing a conditional probability Pi reflecting the likelihood of a physical impairment given only the information included in the SNR feature subset ensemble. As discussed above, each score included in the set of scores <NUM> represents conditional probabilities in that it reflects a prediction of physical impairment for a specific type of network measurement (e.g., XLIN, SNR, QLN, HLOG).

In some implementations, the prediction model <NUM> is a perceptron. The prediction model <NUM>, in such implementations, can be trained to identify a respective weight assigned to each of the network measurements specified in the feature subset ensembles. For example, the prediction model <NUM> can identify different weights to each of the outputs of the estimator models 110A-G based on their relative importance to the detection of a physical impairment along the line of the subscriber network. The prediction model <NUM> computes the confidence score <NUM> as a single score by combining the scores according to the identified weights. For example, the prediction model <NUM> can combine scores by multiplying each score by its assigned weight and summing the scores together to compute the confidence <NUM>. As another example, the prediction model <NUM> can normalize the confidence score between a minimum value and a maximum value (e.g., between <NUM> and <NUM>) to represent the probability of a physical impediment being detected on a line.

At <NUM>, the obtained data indicating the set of one or more scores is provided as input to the prediction model <NUM>. The prediction model <NUM> is trained to output, for each of different sets of feature ensembles, a confidence score representing an overall likelihood that a particular line has a physical impairment. As discussed above, in some implementations, the prediction model <NUM> is a perceptron that assigns weights to each of the scores included in the obtained data and computes the confidence score based on combining the scores according to the assigned weights. In such implementations, the assigned weight can represent the importance of each data point that can be based on, for instance, the accuracy of the information represented by the data point, the predictive strength of data point, the frequency of data collection, among others. For example, if the prediction model <NUM> determines that the XLIN data has greater importance to physical impairment prediction relative to the QLN data, then the score representing conditional probability Pi is assigned a higher weight compared to the weight assigned to the score representing conditional probability P<NUM>.

At <NUM>, data indicating a particular confidence score representing an overall likelihood that the line of the subscriber network has the physical impairment is received. For example, data indicating the confidence score <NUM> is received from the prediction model <NUM> responsive to the input provided in step <NUM>. As discussed above, the value of the confidence score <NUM> can represent an overall likelihood that the line has a physical impairment.

At <NUM>, the particular confidence score is provided for output. For example, the confidence score <NUM> is provided for output in a repair report to a service provider associated with the subscriber network. As depicted in <FIG>, the confidence score <NUM> can be included in a report <NUM> that is provided to a technician <NUM>. The report <NUM> can identify a predicted location of the physical impairment along the line of subscriber network, the probability associated with the existence of the physical impairment, and any relevant network data <NUM> that may be useful to the technician <NUM> for repairing the predicted physical impairment and/or performing a maintenance operation to identify the cause of the physical impairment.

In some implementations, the process <NUM> includes additional operations. For example, where a physical impairment has been detected on the line of the subscriber network, the process <NUM> can also include determining a location of the physical impairment along the line of subscriber network. For example, the distance can be represented as a distance from a Digital Subscriber Line Access Multiplexer (DSLAM) in a copper network. In such implementations, a report indicating the location of the physical impairment can be provided to a service provider system associated with the subscriber network. For example, as illustrated in <FIG>, the system <NUM> can provide a report <NUM> to the technician <NUM> of a service provider of the subscriber network.

In some implementations, the process <NUM> includes comparing a value of the confidence score <NUM> to a predetermined threshold. The predetermined threshold can be specified by a service provider and used to adjust the sensitivity by which the system determines whether the impairment detection system determines that a physical impairment is likely to be on a line of the subscriber network. For example, the system can detect a physical impairment on the line of the subscriber network if the value of the confidence score <NUM> exceeds the predetermined threshold. Alternatively, the system can determine that no physical impairment has been detected on the line if the value of the confidence score <NUM> does not exceed the predetermined threshold.

Claim 1:
A method comprising:
- providing input to multiple estimator models that are each trained to output a conditional likelihood with respect to a particular feature subset ensemble from among different feature subset ensembles;
- obtaining, by a computing device, data for a line of a subscriber network that indicates a set of one or more scores, wherein each score included in the set of scores indicates the conditional likelihood that the line has a type of impairment with respect to a feature subset ensemble of different feature subset ensembles;
- providing, by the computing device, the obtained data as input to a prediction model that is trained to output, for each of different sets of feature subset ensembles, a confidence score representing an overall likelihood that a particular line has a physical impairment;
- receiving, by the computing device and from the prediction model, data indicating a particular confidence score representing an overall likelihood that the line has the physical impairment; and
- providing, by the computing device, the particular confidence score for output.