A Method for Network Intrusion Detection
Using Flow Sequence and BERT Framework
Loc Gia Nguyen

Kohei Watabe

Nagaoka University of Technology
May 30, 2023

L. G. Nguyen, K. Watabe (NUT)

IEEE ICC 2023

May 30, 2023

1 / 19Motivation▶ Concept drift in flow features from different environments(domains) made it diﬀicult for machine learning model toidentify flows.
▶ Domain adaptation capability describe the ability of an NIDS
to identify flows from different environments.

GoalImprove the domain adaptation capability of Network Intrusion
Detection Systems.
Related Work: Energy-based Flow Classifier (EFC)1
1
Pontes et al., “A New Method for Flow-Based Network Intrusion Detection
Using the Inverse Potts Model”.
L. G. Nguyen, K. Watabe (NUT)

IEEE ICC 2023

May 30, 2023

2 / 19

Flow-based NIDS

A network traﬀic flow is▶ a sequence of packets carrying information between two hoststhat shares common properties, such as the 5-tuple: Src IP, Src
Pt, Dst IP, Dst Pt, Proto.
Flow-based NIDS is▶ a system that utilized the properties of network traﬀic flows todetermined if they represent anomalous activity.

L. G. Nguyen, K. Watabe (NUT)

IEEE ICC 2023

May 30, 2023

3 / 19Transformers (1/2)

Bidirectional Encoder Representations from Transformers (BERT) is alanguage model that utilizes Transformer Encoders and Masked
Language Modeling (MLM) task pre-training.
In MLM task, 15% of tokens were selected for prediction, and thetraining objective was to predict the selected token given its context.
The selected token is▶ replaced with a [MASK] token with probability 80%,
▶ replaced with a random word token with probability 10%,▶ not replaced with probability 10%.

L. G. Nguyen, K. Watabe (NUT)

IEEE ICC 2023

May 30, 2023

4 / 19Transformers (2/2)

To identify anomalous flows, BERT is then fine-tuned with Named
Entity Recognition (NER) task.
NER is structured as:▶ taking an unannotated block of text (a sequence of flows).
▶ producing an annotated block of text that highlights the namesof entities (marks flows as anomalous).

L. G. Nguyen, K. Watabe (NUT)

IEEE ICC 2023

May 30, 2023

5 / 19Hypotheses (1/2)▶ Features within a flow underwent major shift in theirdistributions between different environments.
▶ Context are flows that appear at roughly the same time at thetarget flow.
▶ A sequence of flows is the collection of the target flow and its
context.
▶ Relation information between each flows can be infer from thesequence.

L. G. Nguyen, K. Watabe (NUT)

IEEE ICC 2023

May 30, 2023

6 / 19Hypotheses (2/2)

Figure: Two flow sequences. Though the flows in bold share the same properties,
in sequence one they are benign, in sequence two they are anomalous.
L. G. Nguyen, K. Watabe (NUT)

IEEE ICC 2023

May 30, 2023

7 / 19

Why use Transformer (BERT)?
and not RNN (LSTM)?

▶ Flows are not captured in perfect chronological order.
▶ RNN relies on the state of the previous input.
▶ Transformer process the sequence as a whole, small changes in
the ordering of flow will probably not impact its performance.

L. G. Nguyen, K. Watabe (NUT)

IEEE ICC 2023

May 30, 2023

8 / 19Proposal (1/4)
Overview

Pre-processing▶ discretize and tokenize features.
Model▶ BERTTraining▶ Pre-training with Masked Language Modeling task
▶ Fine-tuning with Named-entity Recognition task

L. G. Nguyen, K. Watabe (NUT)

IEEE ICC 2023

May 30, 2023

9 / 19Proposal (2/4)
Pre-processing▶ Perform data binning, where each bin contains similar number ofvalues▶ Each bin is represented as a token

Figure: Feature binning upper thresholds

L. G. Nguyen, K. Watabe (NUT)

IEEE ICC 2023

May 30, 2023

10 / 19

Proposal (3/4)
Training

Pre-training▶ Train with Masked Language Modeling task
▶ Train with only benign flows▶ Weights of the output layer is shared with the embedding layer▶ The output is the probability of tokens
Fine-tuning▶ Train with Named Entity Recognition task
▶ Train with benign and anomalous flows▶ Output layer is an MLP with two output neuron▶ The output is the probability of being benign and anomalous
L. G. Nguyen, K. Watabe (NUT)

IEEE ICC 2023

May 30, 2023

11 / 19

Proposal (4/4)
Inference

The fine-tuned model is used as an NIDS
6-dim vector

6 features

Src Pt: 445
Dst Pt: 48888
Proto+Flags: TCP.AP...
# Packets: 1# Bytes: 108
Duration: 0.000[12,43,23,54,85,23]

768-dim vector(128 x 6)

768-dim vector(128 x 6)[0.1, -0.2, ...][-0.0, -0.2, ...]
[0.3, 0.9, ...]
[0.7, -0.2, ...]
[0.8, -0.3, ...]
[0.2, 0.1, ...]

[0.3, 0.4, ...]
[0.7, -0.3, ...][-0.9, 0.7, ...]
[0.5, -0.8, ...][-0.3, -0.3, ...]
[0.4, 0.6, ...]

sequenceof flows

...

...

...
BERT

2-dim vector

Linearlayer
768 to 2[0.99,0.01]

benignbenign 0.99malicious 0.01

benign...

...

predictions
probabilities

outputpredictions

Figure: Proposed method model and data flow

L. G. Nguyen, K. Watabe (NUT)

IEEE ICC 2023

May 30, 2023

12 / 19

Data sets (1/2)
CIDDS-001 and CIDDS-002 intrusion detection benchmark data setsare used for training and evaluation

Figure: CIDDS features, Date first seen, Src IP, Dst IP discarded

L. G. Nguyen, K. Watabe (NUT)

IEEE ICC 2023

May 30, 2023

13 / 19

Data sets (2/2)

Figure: Training data: 4 data sets from 3 network environments▶ CIDDS-001 balanced is derived from CIDDS-001 internal bysubsampling the benign flows▶ Due to subsampling, the distribution of flows in contexts aredifferent from the original

L. G. Nguyen, K. Watabe (NUT)

IEEE ICC 2023

May 30, 2023

14 / 19

Experiment Setup
BERT configuration: 1 encoder, 1 attention head, hidden size 768
Training parameters:▶ batch size 512, sequence length 128▶ Optimizer: Adam, learning rate 1e−5
▶ Criterion: CrossEntropyLoss▶ Mask Language Modeling 400 iterations, Named Entity
Recognition 1500 iterations
Comparison targets:▶ Energy-based Classifier (EFC), Decision Tree (DT), K-Nearest
Neighbors (KNN), Multilayer Perceptron (MLP), Naive Bayes(NB), Support Vector Machine (LinSVM), AdaBoost (AB),Random Forest (RF)L. G. Nguyen, K. Watabe (NUT)

IEEE ICC 2023

May 30, 2023

15 / 19Performance Evaluation (1/2)▶ Our proposal exhibit better Accuracy and F1-score compare toclassical machine learning techniques.
▶ Training on CIDDS-001 result in better performance incomparison to CIDDS-001 balanced

Figure: Test results on CIDDS-001 external

L. G. Nguyen, K. Watabe (NUT)

IEEE ICC 2023

May 30, 2023

16 / 19Performance Evaluation (2/2)▶ Similar results are achieved on CIDDS-002▶ Changing the distribution of flows in a context greatly affectanomaly detection performance

Figure: Test results on CIDDS-002

L. G. Nguyen, K. Watabe (NUT)

IEEE ICC 2023

May 30, 2023

17 / 19

Conclusion and Future Work

▶ Better domain adaptation capability is observed for theproposed model over traditional ML approaches.
▶ There is contextual information in flow data. Classification
performance of the proposed model degrade when flows are
shuffle during training.
▶ Using the full transformer architecture is very resource
intensive for an NIDS.

L. G. Nguyen, K. Watabe (NUT)

IEEE ICC 2023

May 30, 2023

18 / 19

Future Work▶ Statistical approach, using unlabeled, score-based classification.
▶ Lightweight mechanism to summarize information from flow
sequence.

L. G. Nguyen, K. Watabe (NUT)

IEEE ICC 2023

May 30, 2023

19 / 19