# Utilizing Duplicate Announcements for BGP Anomaly Detection
## 3 Proposed Methodology
### Data Pre-Processing and Feature Extraction  
The BGP data consists of the routing information, which carries network prefixes and network layer reachability information. On the internet, this routing information is exchanged among various autonomous systems. The pre-processing of the BGP datasets starts with cleaning of the data. After the data are cleaned, they are normalized and prepared for further analysis. The raw data are collected from the available repositories From RIPE and ROUTE Views collection sites [2]. The BGP data are stored in a multi-threaded routing tool kit. The ASCII file is parsed in C#, and a list of 37 features is extracted. In large BGP datasets, the occurrence of these duplicate announcements is a common and challenging task. The incomplete or corrupted entries, such as invalid AS paths, are discarded during the process of data cleaning. The complete data cleaning process includes the following operation:  
1. Duplicated Announcements Removal: In the case of duplicate announcements, we detected them by comparing two consecutive update messages from the same AS and checking if their attributes (prefix, AS-path, next-hop, etc.) are identical within a time window. If yes, one instance was kept, and others were selected for anomaly detection. 2. Removing Incomplete or Corrupted Entries: To prevent errors during feature extraction, the entries with missing fields (e.g., AS-paths with NULL values, incorrect timestamps, or incorrectly extended network prefixes) were excluded. 3. Outlier Removal: Updates were filtered using Interquartile Range (IQR) filtering to remove extreme outlier values in attributes such as AS-path length or number of announcements per time window. 4. The IP Address Standardization: to make IPv4 and IPv6 addresses compatible with machine models, we parsed and formatted them into a common representation. \begin{table}
\begin{tabular}{l l c c c c} \hline \hline \multirow{2}{*}{**Collection Site**} & \multirow{2}{*}{**Dataset**} & **Regular** & **Anomaly** & \multicolumn{2}{c}{**Collection Date**} \\ \cline{3-6}  & & **(min)** & **(min)** & **Start** & **End** \\ \hline \multirow{4}{*}{ RIPE} & WannaCrypt & 5760 & 5760 & 10 May 2017 & 17 May 2017 \\  & & & & 00:00:00 & 23:59:59 \\  & Moscow b/o & 6960 & 240 & 23 January 2005 & 27 January 2005 \\  & & & & 00:00:00 & 23:59:59 \\  & & & & 23 May 2003 & 27 May 2003 \\  & & & & 00:00:00 & 23:59:59 \\ \hline \multirow{4}{*}{ Route Views} & WannaCrypt & 5760 & 5760 & 10 May 2017 & 17 May 2017 \\  & & & & 00:00:00 & 23:59:59 \\ \cline{1-1}  & & & & 23 January 2002 & 27 January 2005 \\ \cline{1-1}  & & & & 00:00:00 & 23:59:59 \\ \cline{1-1}  & & & & 23 May 2003 & 27 May 2003 \\ \cline{1-1}  & & & & 00:00:00 & 23:59:59 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Data points in the BGP dataset from the RIPE and ROUTE Views. Timestamp Synchronization: BGP updates from separate collection points had slightly different timestamps owing to logging latency. Timestamps were standardized to make sure there was no mixture of UTC and local time, and we resampled the ticks to one-minute intervals. Once data cleaning was performed, the following step was normalization, which was very important to avoid numerical instability during deep learning model training. Normalization was then applied as follows:  
1. Min-Max Normalization: Continuous numerical features such as number of announcements, withdrawals, and AS-path lengths were normalized using the following Min-Max scaling: \[X_{norm}=\frac{X-X_{min}}{X_{max}-X_{min}}\] (1) This ensured all values were scaled between 0 and 1, preventing large numerical disparities between features from dominating the learning process. 2. One-Hot Encoding for Categorical Variables: Features like BGP update types (announcement, withdrawal, implicit withdrawal) were categorical and encoded using one-hot encoding to allow numerical representation while maintaining feature independence. 3. Time-Stamp Corrections: It is assured that all the lettering and inscriptions are in place and hold actual timestamps.