# Detection and Localization of Failures in Hybrid Fiber-Coaxial Network Using Big Data Platform
## 5 Failure Detection and Localization Based on Big Data
### Failure Localization  
Once the failure is detected for a particular MAC domain, the failure localization is triggered. The main goal of this part is to detect the network device (AP, AMP, or ON) that is the root cause of the problem. Prerequisites for failure localization are topology mapping information (for example, comma-separated relation between parent and child network element) as well as the network element to which the CPE device is connected (for example, modern FF:3B:2A:8C:AE:DC is connected to AP_cc43.) In addition, mapping between MAC domains and CPE is mandatory (this can be populated querying CMTS metrics for docsIf3 CmtsCmRegStatusMdlIndex, docsIfCmtsCmStatusIpAddress, and docsIfCmtsCmStatusMacAddress [34]) and IP (Internet Protocol) address for each CPE device. The localization algorithm can be split into several steps:  
1. Ping all CPE devices that are connected to the problematic MAC domain. 2. Join ping results by the topology and calculate the percentage of offline CPE devices. 3. If there are 100% of offline modems, go one level up in the topology hierarchy and repeat the second step. 4. Repeat the third step until the percentage of offline CPE devices decreases. This means that the algorithm reached out of the root-cause element. Because most of the CPE devices are constantly turned on, this algorithm gives almost no false positives in practice (theoretically, it would be possible that devices are simultaneously turned off on purpose) and precisely detects the failed network element. We developed the failure localization algorithm by exploring the possible data that can be obtained from a CMTS and the ones that the network operator has in inventory system. The algorithm was tested and verified in the real HFC network. Real data sets used for algorithm development and verification are confidential and we are not able to disclose them. However, the same results can be achieved by generating appropriate test data sets, creating different case scenarios (no offline devices, some devices are offline, all devices are offline) and running the algorithm on top of it. The following example shows the execution of the failure localization algorithm for the topology given in Figure 3. The given example is artificial and simplified but \begin{table}
\begin{tabular}{c c c c} \hline \hline \(T_{curr}\) & \(N_{out}[T_{curr}]\) & \(N_{out\_avg}\) & \(\begin{array}{c}N_{out}[T_{curr}]V\\ N_{out\_avg}\end{array}\) \\ \hline
1584659988 & 417 & 455.55 & 0.915377017 \\ \hline
1584660048 & 413 & 451.15 & 0.915438324 \\ \hline
1584660108 & 415 & 446.9 & 0.928619378 \\ \hline
1584660168 & 412 & 442.6 & 0.930863082 \\ \hline
1584660288 & 120 & 424.6 & 0.1826447 \\ \hline
1584660288 & 121 & 405.85 & 0.298139707 \\ \hline
1584660348 & 121 & 388.15 & 0.311735154 \\ \hline
1584660408 & 118 & 370.2 & 0.318746623 \\ \hline
1584660468 & 429 & 368.2 & 1.165127648 \\ \hline
1584660528 & 427 & 366.6 & 1.164757229 \\ \hline
1584660588 & 442 & 365.95 & 1.207815275 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Failure detection and resolution example. reflects the behavior of the real network. Failure detection discovers a sudden drop in number of active users and triggers failure localization. First, mapping between the MAC domain and CPE (MAC address and IP address) is correlated with metrics (docslf3 CmtsCmRegStatusMdlfIndex, docslfCmtsCmStatusDAddress, and docslfCmtsCmStatusMacAddress) gathered from CMTS. The example of the mapping:  
1. MAC_domain1, 10.0.1.17, FF:3C:2A:1C:FE:AA
2. MAC_domain1, 10.0.1.22, FF:1B:A7:C9:D9:34
3. MAC_domain1, 10.0.1.63, FF:AE:FE:2B:98:99
4. MAC_domain1, 10.0.1.64, FF:B4:93:64:E7:AE
5....
6. MAC_domain1, 10.0.1.211, FF:9E:FC:98:86:F1
7. MAC_domain2, 10.0.1.214, FF:DA:E3:5A:ED:79  
Given the fact that such information slowly changes over time, it is collected periodically and used directly to reduce the localization time. The topology mapping shows a relation between the CPE MAC address and the network element to which CPE is connected first in the tree topology:  
1.