This is the post-print version of the following article: Loc Gia Nguyen and Kohei Watabe, ``Flow-based Network Intrusion Detection Based on BERT Masked Language Model,'' in Proceedings of the 18th International Conference on emerging
Networking EXperiments and Technologies (CoNEXT 2022) Student Workshop, 2022. (c) ACM, 2022. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive
version was published in https://doi.org/10.1145/3565477.3569152 (DOI: 10.1145/3565477.3569152).

Flow-based Network Intrusion Detection Based on BERT Masked
Language Model
Loc Gia Nguyen

Kohei Watabe

Nagaoka University of Technology
Nagaoka, Niigata, Japans203145@stn.nagaokaut.ac.jp

Nagaoka University of Technology
Nagaoka, Niigata, Japank_watabe@vos.nagaokaut.ac.jp

ABSTRACT
A Network Intrusion Detection System (NIDS) is an important
tool that identifies potential threats to a network. Recently, different flow-based NIDS designs utilizing Machine Learning (ML)algorithms have been proposed as potential solutions to detect intrusions efficiently. However, conventional ML-based classifiers
have not seen widespread adoption in the real-world due to theirpoor domain adaptation capability. In this research, our goal is toexplore the possibility of improve the domain adaptation capability of NIDS. Our proposal employs Natural Language Processing(NLP) techniques and Bidirectional Encoder Representations fromTransformers (BERT) framework. The proposed method achieved
positive results when tested on data from different domains.

CCS CONCEPTS• Security and privacy → Intrusion detection systems.

KEYWORDS
Intrusion Detection System, Domain adaptation, BERTACM Reference Format:
Loc Gia Nguyen and Kohei Watabe. 2022. Flow-based Network Intrusion
Detection Based on BERT Masked Language Model. In CoNEXT StudentWorkshop 2022 (CoNEXT-SW ’22), December 9, 2022, Roma, Italy. ACM, NewYork, NY, USA, 2 pages. https://doi.org/10.1145/3565477.3569152

1

INTRODUCTION

It is common in practical application of NIDS for there to be achange in the data distribution between its training data and thedata it encounters when deployed. Conventional ML algorithms
often adapt poorly to such change, which limit their usefulness
in real-world scenarios [1]. To address this, Energy-based FlowClassifier (EFC) [3] was proposed as a solution. Despite having good
adaptability, EFC produces high false positives rate for domains
where the distribution of features of malicious flows overlap withthat of benign flows. We theorize that the reason for the limitationsof conventional ML algorithms and EFC is the use of singular flows
as input data, as the classifier can only model the distribution offeatures within a flow. This limitation can be overcome with theuse of sequences of flows, allowing the classifier to further models
Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
CoNEXT-SW ’22, December 9, 2022, Roma, Italy© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9937-1/22/12.
https://doi.org/10.1145/3565477.3569152

6-dim vector

6 features

Src Pt: 445
Dst Pt: 48888
Proto+Flags: TCP.AP...
# Packets: 1# Bytes: 108
Duration: 0.000[12,43,23,54,85,23]

768-dim vector(128 x 6)

768-dim vector(128 x 6)[0.1, -0.2, ...][-0.0, -0.2, ...]
[0.3, 0.9, ...]
[0.7, -0.2, ...]
[0.8, -0.3, ...]
[0.2, 0.1, ...]

[0.3, 0.4, ...]
[0.7, -0.3, ...][-0.9, 0.7, ...]
[0.5, -0.8, ...][-0.3, -0.3, ...]
[0.4, 0.6, ...]

sequenceof flows

...

...

...
BERT

2-dim vector

Linearlayer
768 to 2[0.99,0.01]

benignbenign 0.99
malicious 0.01

benign...

...

predictions
probabilities

outputpredictions

Figure 1: Proposed system architecture

the distribution of a flow in relation to other flows. To utilize thecontext information from a sequence of flow, we use the BERT
framework, which is able to process inputs in relation to all theother inputs in a sequence.
BERT[2] is a transformer-based machine learning technique for
NLP developed by Google. The BERT framework is comprised oftwo steps: pre-training and fine-tuning. In pre-training, the BERT
model is trained on unlabeled data. For fine-tuning, the model isfirst initialized using the pre-trained parameters, and then trainedusing labeled data from the downstream tasks. BERT is pre-trainedwith two unsupervised tasks, which are Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). In MLM, some of
the words in a sentence are replaced with a different token. Theobjective is to predict the original value of the masked words basedon other unmasked words in the sentence. In NSP, BERT takes
sentence pairs as input. The objective is to predict whether thesecond sentence in the pair is the next sentence in the document.
For fine-tuning, task-specific inputs and outputs are added to a
pre-trained BERT model.
Our research employs CIDDS-001[4] and CIDDS-002[5] data
sets that contains flow samples from a small business environment
emulated using OpenStack. CIDDS-001 also contains real traffic
flow samples captured from an external server directly deployedon the internet.

2PROPOSAL

We first organize network traffic flows into structures similar to alanguage, treating a flow as a word and a sequence of flows as asentence. In this study, the BERT model is pre-trained with only
the MLM task. For fine-tuning, a linear layer with softmax output
is used. It is important to preserve the distribution of flows within asequence; therefore, during training, the data set is not shuffled. Atraining sample is generated by selecting a segment of flows fromthe data set at random.

CoNEXT-SW ’22, December 9, 2022, Roma, Italy

Loc Gia Nguyen and Kohei Watabe

Table 1: Average composition of data sets
CIDDS-001
OpenStack
1 set
CIDDS-001 large
label#
𝑛𝑜𝑟𝑚𝑎𝑙25178585
𝑑𝑜𝑠2775655
𝑝𝑜𝑟𝑡𝑆𝑐𝑎𝑛194642
𝑝𝑖𝑛𝑔𝑆𝑐𝑎𝑛5266
𝑏𝑟𝑢𝑡𝑒𝐹𝑜𝑟𝑐𝑒
4992
Total
28159140

10 sets
CIDDS-001 internal
label#
𝑛𝑜𝑟𝑚𝑎𝑙
10000
𝑑𝑜𝑠
9000
𝑝𝑜𝑟𝑡𝑆𝑐𝑎𝑛935
𝑝𝑖𝑛𝑔𝑆𝑐𝑎𝑛45
𝑏𝑟𝑢𝑡𝑒𝐹𝑜𝑟𝑐𝑒
20Total
20000

Table 2: Average classification performance and standard
error

CIDDS-001
External Server
10 sets
CIDDS-001 external
label#
𝑢𝑛𝑘𝑛𝑜𝑤𝑛
10000
𝑠𝑢𝑠𝑝𝑖𝑐𝑖𝑜𝑢𝑠
10000

10 sets
CIDDS-002
label#
𝑛𝑜𝑟𝑚𝑎𝑙 10000𝑠𝑐𝑎𝑛
10000

Total

Total

20000

CIDDS-002

20000

The overall architecture of the system is illustrated in Figure 1.
Six features from each flow (Src Pt, Dst Pt, Proto+Flags, Packets,Bytes, Duration) are used as input data, these features are discretized as described in EFC’s original paper [3]. The discrete valueof each features are encoded as numbers (flow𝑖 ). BERT decodes
each number into a 128-dimension vector, concatenates them toform a 768-dimension vector (𝑒 flow𝑖 ), and processes it to produce a
different 768-dimension vector (ℎ flow𝑖 ). The output of BERT is thenpassed through a Multilayer Perceptron classifier (a linear layerwith softmax output), which reduces the dimension from 768 to2. This 2-dimension vector represents the predicted probability of
the flow being benign and malicious (e.g. benign 0.99, malicious0.01). The predicted class of the flow is the class with the higher
probability (e.g. benign).
We used data from three different domains: CIDDS-001 OpenStack, CIDDS-001 External Server, and CIDDS-002 to evaluate thedomain adaptation capability of the proposed method. Averagecomposition of the data sets used in the experiment are shown in
Table 1. Training was performed on one set of CIDDS-001 large.
While testing was performed on CIDDS-001 internal, CIDDS-001 external, and CIDDS-002, each containing ten sets randomly selectedfrom the full data sets. This testing scheme mimics the one used byCamila et al. [3] to make the results of the proposed method morecomparable to those of EFC. Flows labeled 𝑛𝑜𝑟𝑚𝑎𝑙 are consideredbenign, while those labeled otherwise are considered malicious.
For CIDDS-001 external, flows labeled 𝑢𝑛𝑘𝑛𝑜𝑤𝑛 and 𝑠𝑢𝑠𝑝𝑖𝑐𝑖𝑜𝑢𝑠 are
considered benign and malicious respectively.
We assess the performance of our method in comparison toEFC and ML classifiers including Decision Tree (DT), K-Nearest
Neighbors (KNN), Multilayer Perceptron (MLP), Naive Bayes (NB),and Support Vector Machine (SVM). Each classifier’s performanceis measured using Accuracy and F1 score [1].

3RESULTS AND DISCUSSION

Table 2 shows the average performance and standard error for eachclassifier. All classifiers achieved higher Accuracy and F1 Score on
CIDDS-001 internal test sets (same domain as training data) compared to the other test sets (different domains from training data).
Both the proposed method and EFC maintained performance acrossthe two different domains, with the proposed method outperforming EFC.
We also experimented with training the classifiers on smaller but
balanced data sets (containing 80000 flows with the same proportionof labels as in CIDDS-001 internal). However, performance wasClassifier
Proposal
EFCDT
KNN
MLP
NB
SVM

Test CIDDS-001 internal
Accuracy
F1 score
0.994(0.002)
0.994(0.002)
0.941(0.005)
0.941(0.004)
0.996(0.001) 0.996(0.001)
0.989(0.002)
0.989(0.003)
0.992(0.001)
0.992(0.001)
0.903(0.002)
0.892(0.002)
0.570(0.015)
0.245(0.047)

Train CIDDS-001 large
Test CIDDS-001 external
Accuracy
F1 score
0.895(0.027) 0.904(0.022)
0.747(0.039)
0.796(0.025)
0.870(0.028)
0.874(0.024)
0.839(0.008)
0.811(0.011)
0.573(0.009)
0.285(0.023)
0.500(0.000)
0.000(0.000)
0.738(0.016)
0.718(0.021)

Test CIDDS-002
Accuracy
F1 score
0.916(0.045) 0.877(0.072)
0.846(0.046)0.800(0.070)
0.818(0.061)
0.707(0.106)
0.818(0.061)
0.707(0.106)
0.832(0.059)
0.729(0.108)
0.499(0.000)
0.001(0.000)
0.513(0.013)
0.090(0.037)

worse for all classifiers when compared to those trained on CIDDS001 large. Notably, the performance of the proposed method wassignificantly affected for all domains. By creating balanced data
sets, the distribution of flows within a sequence was also altered.
This suggests that the distribution of flows within a sequence islearned by the model of the proposed method.

4CONCLUSION AND FUTURE WORKIn this study, we suggest the use of singular flows input to be apossible explanation for the poor domain adaptation capability of
conventional ML-based classifiers. Then we proposed the used ofsequences of flows to address this limitation. We utilized BERT
model for the representation of flow sequences and an MLP classifier to discriminate between benign and malicious flows. Earlyexperimental results showed that the proposed method is capableof achieving good and consistent results across different domains.
However, more extensive testing on recent data sets is needed tofurther evaluate its domain adaptation capability.
In future work, we plan to investigate the impact of flow sequence sampling method on result, such as using only benign flows
or grouping flows into sequences that originate from the same hosts.
Making the system less reliant on labeled data is another researchgoals of ours. We aim to achieve this by modeling sequences ofbenign flows then look for anomalous representations produce byBERT, indicating malicious flows.

ACKNOWLEDGMENTS
This work was partly supported by JSPS KAKENHI Grant Number
JP20H04172.

REFERENCES[1] Zeeshan Ahmad, Adnan Shahid Khan, Cheah Wai Shiang, Johari Abdullah, andFarhan Ahmad. 2021. Network Intrusion Detection System: A Systematic Studyof Machine Learning and Deep Learning Approaches. Trans. Emerg. Telecommun.
Technol. 32, 1 (jan 2021), 30 pages. https://doi.org/10.1002/ett.4150[2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT:
Pre-training of Deep Bidirectional Transformers for Language Understanding. In
Proceedings of the 2019 Conference of the North American Chapter of the Associationfor Computational Linguistics: Human Language Technologies, Volume 1 (Long andShort Papers). Association for Computational Linguistics, Minneapolis, Minnesota,
4171–4186. https://doi.org/10.18653/v1/N19-1423[3] Camila F. T. Pontes, Manuela M. C. de Souza, João J. C. Gondim, Matt Bishop, andMarcelo Antonio Marotta. 2021. A New Method for Flow-Based Network Intrusion
Detection Using the Inverse Potts Model. IEEE Transactions on Network and Service
Management 18, 2 (2021), 1125–1136. https://doi.org/10.1109/TNSM.2021.3075503[4] Markus Ring, Sarah Wunderlich, Dominik Grüdl, Dieter Landes, and Andreas
Hotho. 2017. Flow-based benchmark data sets for intrusion detection. In Proceedings of the 16th European Conference on Cyber Warfare and Security. 361–369.
[5] Markus Ring, Sarah Wunderlich, Dominik Grüdl, Dieter Landes, and Andreas
Hotho. 2017. Creation of Flow-Based Data Sets for Intrusion Detection. Journal ofInformation Warfare 16 (12 2017), 41–54.