# Practical Anomaly Detection in Internet Services: An ISP centric approach

Alex Huang Feng, Pierre Francois, Kensuke Fukuda, Wanting Du, Thomas

Graf, Paolo Lucente, Stéphane Frénot

► To cite this version:

Alex Huang Feng, Pierre Francois, Kensuke Fukuda, Wanting Du, Thomas Graf, et al. Practical Anomaly Detection in Internet Services: An ISP centric approach. NOMS 2024-2024 IEEE Network Operations and Management Symposium, May 2024, Seoul, South Korea. pp.1-4, 10.1109/NOMS59830.2024.10575071. hal-04655324

HAL Id: hal-04655324

https://hal.science/hal-04655324v1

Submitted on 22 Jul 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Practical Anomaly Detection in Internet Services:
An ISP centric approach

Alex Huang Feng*, Pierre Francois*, Kensuke Fukuda†, Wanting Du†,
Thomas Graf†, Paolo Lucente§, Stéphane Frénot*

*INSA Lyon, Inria, CITI, UR3720, Villeurbanne, France

alex.huang-feng@insa-lyon.fr, pierre.francois@insa-lyon.fr, stephane.frenot@insa-lyon.fr

†National Institute of Informatics, Tokyo, Japan
kensuke@nii.ac.jp

†Swisscom, Zurich, Switzerland

wanting.du@swisscom.com, thomas.graf@swisscom.com

§pmacct.net, Barcelona, Spain

paolo@pmacct.net

**Abstract**—Identifying anomalies in a network is a crucial endeavor for Internet Service Providers (ISPs). Anomalies that impact the traffic of the ISP customers can lead to a degradation in the reputation of the company. Moreover, silent anomalies that do not break connectivity can impact the revenue and business of ISPs. Therefore, monitoring and anomaly detection has become essential for ISPs. In this paper, we present an ongoing research project aimed at identifying anomalies in Internet services provided by an ISP. We aim at detecting anomalies within the domain managed by the ISP that impact the customers and the business of the ISP. We propose a rule-based approach designed to promptly detect and provide reporting for such anomalies in near real time, giving information that allows the operator to identify whether a solution can be brought. In this paper, we describe the collected network telemetry metrics and illustrate how they are processed using open-source solutions. We introduce a set of use cases showing that an ISP can monitor Internet services using IETF standard metrics.

# I. INTRODUCTION

Internet services include providing global Internet reachabil-
ity for customer Autonomous Systems (ASes) connected to an
Internet Service Provider (ISP) and serving private customers
within the ISP (e.g. FTTH). Disruptions in the network that
affect the connectivity of an ISP not only significantly degrade
the organization's reputation but also have implications on the
company's revenue. Customers subscribed to Internet services
depend on the ISP peerings to reach the Internet and an
incident between them and the Internet can have detrimental
implications for their business.

Today, routing between different ASes is established using BGP [1]. ISPs managing an AS configure policies in their routers based on the business relationship they have with their neighboring ASes. Generally, ISPs classify their BGP neighbors into Customers, Settlement-free Peers and Transit Providers. Customer ASes compensate the ISP to reach the Internet, Settlement-free peers are mutual arrangements between two ISPs to exchange Internet traffic without any financial compensation and Transit Providers provide access to the global Internet.

ISPs rely on collected BGP messages and traffic counters to monitor peerings and detect anomalies that could impact their customers. They closely supervise network traffic to identify unexpected patterns or potential abuses by peers. This underscores the importance for ISPs to receive prompt alerts when anomalous or unwanted traffic behaviors occur, enabling network operators to rapidly implement solutions and address the detected issues.

Anomaly detection (AD) has been a hot topic in the last decade where researchers have proposed new ways to detect irregularities in the data. Most research projects aiming at detecting anomalies in BGP networks use public repositories such as Routeviews and RIPE NCC archives [2, 3], allowing researchers to identify problems in Internet from a global point of view. In conjunction with publicly known incidents, researchers have been able to develop methods to detect anomalies in data from the public domain, with a focus on detecting anomalies in the global Internet topology [4, 5].

Simulated environments mimicking the deployed network and manually generated anomalies have also been used to test anomaly detection [6]. Very few projects use production data coming from an ISP to detect anomalies and root cause analysis within a single domain. AD within an AS have only been investigated by very few researchers having access to production data [7]–[9].

In this paper, we focus on detecting anomalies within a
single AS to potentially help them fixing their configuration
and find unwanted traffic flows impacting their business. We
describe the target use cases in Section II. Instead of solely
using BGP activity as a source of data, as done in [7], we
use a larger set of monitoring information, allowing us to
cover a broader set of service anomalies (Sec. III-A). The
authors in [8] focus on detecting performance issues from
end-to-end users, while the work presented in this paper also
covers anomalies impacting the traffic from peerings. In [9],
anomaly detection is based on traffic information with a focus
on network intrusion detection, while the project presented
in this paper focuses on detecting operational anomalies, taking forwarding plane, control plane and management plane information into account (Sec. III-B). This work is an extension to the architecture presented in [10], that is targeting VPN services (Fig. 1). We investigate whether using expert knowledge based checks also allows for the detection of issues in the Internet services provided by an ISP (Sec. III-C).

<figure><img src="image_2.png" /><figcaption>Fig. 1. Anomaly Detection System architecture.</figcaption></figure>

## II. MONITORING ISP NETWORKS

Detecting anomalies in network services can make use of metrics categorized into: forwarding plane, e.g. counters characterising traffic behaviors, control plane, e.g. BGP events impacting the routing table of the network and management plane, information related to the interface state of the routers.

Netflow [11] and IPFIX [12] are the preferred solutions for traffic counters since they allow aggregation directly from the node [13]. To monitor the control plane, BGP [1] and BMP [14] are currently used within the industry. Finally, in management plane, SNMP [15] has been the de facto standard to get information from the node but the industry has acknowledged its limitations due to its polling-based approach and have been developing NETCONF [16] and YANG-Push [17].

The first two things an operator wants to know when there is a service anomaly are (1) if it is impacting their customer traffic and (2) whether the root cause is within their own network or due to a change within the Internet, outside of their scope of operation.

In this project, examples of service disruption we aim to detect are: (i) losing the traffic from an AS sending a high volume of traffic to the customers of the ISP; (ii) undergoing saturation on a link with a neighboring AS or (iii) having traffic traversing all the ISP core network instead of being forwarded over a shorter path. Additionally, we are detecting anomalies which have an impact on the ISP business even though they are not breaking connectivity. For example, (iv) Internet traffic that has moved from a settlement-free peering to a transit provider is of concern to an ISP; (v) trending analysis can also be performed on inbound/outbound traffic ratios on settlement-free peering; (vi) traffic from a top traffic receiver (typically a customer) is not traversing the ISP anymore, possibly because it moved to another ISP. Finally, we also plan to use the same platform to detect (vii) security-related anomalies, such as prefix hijacks or DDoS attacks.

## III. DETECTING SERVICE ANOMALIES

In this section, we explain how we plan to detect the anomalies applying expert knowledge to the collected metrics.

### A. Data collection and correlation

We consider the deployed network as the source of truth instead of a model mimicking the network behavior. We collect network telemetry metrics using IETF standards to allow ISPs to not rely on proprietary solutions. We are currently collecting forwarding plane metrics using IPFIX and control plane events using BGP collectors. The metrics are collected from the Swisscom production network allowing us to test our use cases based on real data.

All metrics are collected by leveraging the open-source solution pmacct [18]. The pmacct collector decodes the collected BGP messages and aggregates them into four event types: peer ups, peer downs, updates and withdraws. After ingestion, the BGP events result in the following tuple: Timestamp, IP prefix, AS path, BGP communities, Event type, Node id and Platform Id.

For forwarding plane metrics, pmacct aggregates IPFIX flows based on the 5-tuples (IP source, IP destination, ports and protocol). pmacct aggregates IPFIX flows to timebins of 1 minute and associates the IPs to some of the BGP attributes it learned the prefix from. This results in having, for each flow, the aggregated metrics from IPFIX correlated to attributes from BGP resulting in the following tuple: Timestamp, IP src, IP dst, IP src BGP Communities, IP dst BGP Communities, IP src AS path, IP dst AS path, Bytes, Packets, Node Id and Platform Id. Once the data is ingested, our Anomaly Detection System aggregates again these metrics during anomaly detection on an AS basis to reduce cardinality. We elaborate our methodology in Section III-C.

Due to BGP asymmetry, the path from the source AS to the ISP cannot be known from local data. We however only use the AS Path to the src IP to extract the source AS of the flows.

Some of the fields, such as the Platform Id and the Node Id, are internal references to the ISP to facilitate the understanding of the collected data. In our case, the Node Id identifies the node within the network and the Platform Id identifies a set of nodes belonging to the same network. The mapping between the source IP and the Node Id and Platform Id is set in the configuration file by the operator.

These dimensions and metrics are pushed to Apache Kafka [19] and are ingested to Apache Druid [20] using a Kafka connector. At this point, any system can either consume the data in streaming mode using the Kafka topic or poll the database.

### B. Anomaly Detection System (ADS)

The Anomaly Detection System presented in [10] is implemented in polling mode and is currently deployed in production for a subset of Swisscom VPN customers. In [10], the profiles were aimed at grouping BGP/MPLS VPN customers having similar traffic behaviors.
ADS operates by triggering minute-by-minute jobs for each customer profile. These jobs specify the algorithm, customer identifiers, and the timeframe in which we want to execute the algorithm. Different threads consume jobs, executing algorithms and generating a concern score, a numerical value from 0 to 1 representing the operator's concern. In cases where multiple algorithms collaborate to detect a specific anomaly, an aggregation function (e.g., weighted sum) computes an alert level. If the alert level surpasses an operator-defined threshold, an alarm is raised to the Network Operations Center.

### C. Use cases and knowledge based rules

In this paper we detail three anomalies: (1) losing a top talker, (2) the traffic from a Settlement-free Peer has moved to a Transit Provider and (3) trending analysis on traffic ratios from peerings. The implemented algorithms in ADS are based on interviews with the network operator. We mimick what operators do when looking at the collected metrics during an ongoing incident or anomaly within their network.

1. Losing a top talker: Top talkers are the ASes that transmit the highest volume of data across the ISP. ISPs are invested in monitoring these ASes, as any disruptions or incidents affecting their traffic could potentially have a cascading impact on their customers.

To compute the top talkers, we classify the flows traversing our network by their origin AS, the AS that advertised the prefix covering the source IP of the flows. The origin AS number (ASN) is determined by extracting the last value of the AS path in the BGP path towards the source. Consider the example illustrated in Figure 2, where inbound customer traffic originates from two distinct transit peer ASes. In this scenario, the collector generates two flows in IPFIX, one characterized by the IP source AS_path of <AS3303;ASYYYY;ASZZZZ> and the second one with an IP source AS_path of <AS3303;ASTTTT;ASZZZZ>. We aggregate the traffic coming from these two ASes by the origin ASN ASZZZZ. Then, we sort the different ASes transmitting traffic to our customers by their total amount of traffic.

<figure><img src="image_3.png" /><figcaption>Fig. 2. Example of Top talker traffic.</figcaption></figure>

After identifying the top talkers within our network, we can closely monitor both traffic patterns and control plane events on a per AS basis. Based on past experience, operators tend to become concerned when observing a spike of BGP withdraws and BGP updates, especially if they are followed by a decline in overall traffic. This concern was experienced lively during the DNS outage from AS32934 in 2021 October 4th [21]. Swisscom experienced a spike of BGP withdraws at 17:39 where prefixes from AS32934 were withdrawn from

the Internet. Suddenly, as shown in Figure 3, inbound traffic decreased 85% and remained almost linear until 00:41 when the issue was fixed and traffic was restored.

<figure><img src="image_4.png" /><figcaption>Fig. 3. Swisscom inbound traffic during the DNS outage from AS32934 (blue) compared to traffic of one week before (gray).</figcaption></figure>

Therefore, for this use case, we can implement two jobs. One job generating a concern score when there is a spike in BGP withdraw events. And a second job for traffic information which produces a high concern score when the traffic compared to the week before has a big negative difference.

2. Settlement-free peer traffic going to provider: Our strategy involves monitoring the flows on an AS basis and track both directions independently: outbound traffic originating from our customer and inbound traffic destined to our customer. In this scenario, our focus is on tracking traffic typically received or forwarded over a settlement-free peer which has shifted over a transit provider peering.

<figure><img src="image_5.png" /><figcaption>Fig. 4. Example of Customer traffic received from a Settlement-free peering and a Transit provider.</figcaption></figure>

For this use case, we leverage BGP communities identifying where the prefixes are learned from: a Transit peering, a Settlement-free peer or a Customer. By using the correlated data from pmacct, we can directly aggregate the flows based on the customer ASN and the BGP communities we want to track. Take the example illustrated in Figure 4 where ASVVVV decides to forward some of the traffic to ASZZZZ. In this case, the local AS (AS3303) receives traffic to its customer peer through two different ASes. The first flow uses a transit provider to reach the customer AS and the second flow traverses a settlement-free peer. In the collected data, the flows have the customer ASN in the IP dst AS_path field and the community tag telling us if the prefix was learned from a transit or a peering in the src BGP communities field. Therefore, we can aggregate, on the one hand, the amount of traffic coming from the transit and, on the other hand, the amount of traffic arriving from the settlement-free peer. Once having both values computed, we can track the ratio on a sliding window and alert the network operator when the
percentage of traffic going through the transit provider has augmented considerably. We can apply the same logic to the outbound traffic of the customer AS using the equivalent fields IP src AS_path and dst BGP communities.

In our ADS, we can implement two jobs to track these ratios, one for the inbound customer traffic and one for the outbound customer traffic. The job computes both traffic amounts and tracks the ratios every minute. Then, it raises the concern score when the ratios show that more traffic is shifting to the transit ASN.

Detecting a large shift of traffic does not explain the root cause, but a third job tracking the traffic on a prefix level, checking if some of the same traffic from that prefix has moved to the provider peering gives more insights to the network operator to establish actions to solve it.

3) Exceeding a threshold ratio in settlement-free peer traffic: Settlement-free peers allow ISPs to bypass transit providers when exchanging traffic among their customer bases. Typically, through a mutual agreement between the peers, a window of inbound vs. outbound ratio of traffic is expected. As a result, some ASes are vigilant in monitoring the evolution of such ratios to reconsider future peering agreements. To address this aspect, we perform trending analysis on the traffic ratios, focusing on the balance between inbound and outbound traffic across the link connecting the ISP and the peer.

For each IPFIX flow, pmacct exports the traffic counters with the ingress and egress interface along with the Node Id. Assuming that an inventory mapping the ASN to the node and egress interface exists, we can aggregate the traffic counters per node and interface and obtain the inbound and outbound traffic for that peering. We then track the ratio of inbound traffic and outbound traffic for that peering giving more insights to the operator to reconsider their peering strategies.

In our ADS, this is implemented within a single job, where it computes both aggregated traffic amounts and tracks the ratio evolution. When the ratio passes a threshold defined by the operator, the concern score is raised and an alert to the Network Operations Center can be generated. Abrupt changes within the traffic exchange can also be tracked by implementing a second job. This would allow the operator to spot a policy change of their peer or an unexpected behavior after a configuration change within their network.

## IV. CONCLUSION

In this paper, we present an ongoing project to detect service anomalies in Internet services provided by an ISP. We focus on use cases impacting the business of an ISP and argue that we can use IETF standards and open-source solutions to obtain the data necessary to detect these anomalies. We show the different collected metrics and present the first rule-based methods to detect some of the mentioned anomalies. We are in the early phase of this project and are still studying the rest of the anomalies to see if an operator can use its own network telemetry data to detect the anomalies.

Currently, our ADS relies on local BGP events and IPFIX counters. Future plans include deploying the BGP Monitoring

Protocol [14] for monitoring Adj-RIB-in and Loc-RIB, and implementing YANG-Push [17] to directly export interface counters and other device-related statistics. Exploring external data sources like Routeviews may be considered for investigating anomalies, integrating them with locally collected network telemetry.

## ACKNOWLEDGEMENTS

This work is partially supported by the NII internship program and Swisscom.

## REFERENCES

[1] Y. Rekhter, S. Hares, and T. Li, "A Border Gateway Protocol 4 (BGP-4)," RFC 4271, Jan. 2006.

[2] S. Peng, J. Nie, X. Shu, Z. Ruan, L. Wang, Y. Sheng, and Q. Xuan, "A multi-view framework for bgp anomaly detection via graph attention network," Computer Networks, vol. 214, p. 109129, 2022.

[3] B. Al-Musawi, P. Branch, and G. Armitage, "BGP Anomaly Detection Techniques: A Survey," IEEE Communications Surveys & Tutorials, vol. 19, no. 1, pp. 377-396, 2017.

[4] J. Qiu, L. Gao, S. Ranjan, and A. Nucci, "Detecting bogus BGP route information: Going beyond prefix hijacking," in 2007 Third International Conference on Security and Privacy in Communications Networks and the Workshops - SecureComm 2007. Nice, France: IEEE, 2007, pp. 381-390.

[5] P. Fonseca, E. S. Mota, R. Bennesby, and A. Passito, "BGP Dataset Generation and Feature Extraction for Anomaly Detection," in 2019 IEEE Symposium on Computers and Communications (ISCC). Barcelona, Spain: IEEE, Jun. 2019, pp. 1-6.

[6] A. Putina and D. Rossi, "Online anomaly detection leveraging stream-based clustering and real-time telemetry," IEEE Transactions on Network and Service Management, vol. 18, no. 1, pp. 839-854, 2021.

[7] Y. Huang, N. Feamster, A. Lakhina, and J. J. Xu, "Diagnosing network disruptions with network-wide analysis," in Proceedings of the 2007 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems, ser. SIGMETRICS '07. New York, NY, USA: Association for Computing Machinery, 2007, pp. 61-72.

[8] H. Yan, A. Flavel, Z. Ge, A. Gerber, D. Massey, C. Papadopoulos, H. Shah, and J. Yates, "Argus: End-to-end service anomaly detection and localization from an isp's point of view," in 2012 Proceedings IEEE INFOCOM, 2012, pp. 2756-2760.

[9] J. Dromard, G. Roudière, and P. Owezarski, "Online and scalable unsupervised network anomaly detection method," IEEE Transactions on Network and Service Management, vol. 14, no. 1, pp. 34-47, March 2017.

[10] A. Huang Feng, P. Francois, S. Frenot, T. Graf, W. Du, and P. Lucente, "Daisy: Practical Anomaly Detection in Large BGP/MPLS and BGP/SRv6 VPN Networks," in Proceedings of the Applied Networking Research Workshop, ser. ANRW '23. New York, NY, USA: Association for Computing Machinery, 2023, pp. 8-14.

[11] B. Claise, "Cisco Systems NetFlow Services Export Version 9," RFC 3954, Oct. 2004.

[12] P. Aitken, B. Claise, and B. Trammell, "Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of Flow Information," RFC 7011, Sep. 2013.

[13] B. Trammell, A. Wagner, and B. Claise, "Flow Aggregation for the IP Flow Information Export (IPFIX) Protocol," RFC 7015, Sep. 2013.

[14] J. Scudder, R. Fernando, and S. Stuart, "BGP Monitoring Protocol (BMP)," RFC 7854, Jun. 2016.

[15] M. Fedor, M. L. Schoffstall, J. R. Davin, and D. J. D. Case, "Simple Network Management Protocol (SNMP)," RFC 1157, May 1990.

[16] R. Enns, M. Björklund, A. Bierman, and J. Schönwälder, "Network Configuration Protocol (NETCONF)," RFC 6241, Jun. 2011.

[17] A. Clemm and E. Voit, "Subscription to YANG Notifications for Datastore Updates," RFC 8641, Sep. 2019.

[18] P. Lucente, "pmacct," http://pmacct.net, [Accessed: 2024-01-04].

[19] "Apache Kafka," https://kafka.apache.org, [Accessed: 2024-01-04].

[20] "Apache Druid," https://druid.apache.org, [Accessed: 2024-01-04].

[21] C. Martinho and T. Strickx. (2021) Understanding how facebook disappeared from the internet. [Accessed 2024-01-10]. [Online]. Available: https://blog.cloudflare.com/october-2021-facebook-outage