Session 12C: Traffic Analysis and Side Channels

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

New Directions in Automated Traffic Analysis
Jordan Holland

Paul Schmitt

Princeton University
Princeton, New Jersey, USA
jordanah@princeton.edu

Princeton University
Princeton, New Jersey, USA
pschmitt@cs.princeton.edu

Nick Feamster

Prateek Mittal

University of Chicago
Chicago, Illinois, USA
feamster@uchicago.edu

Princeton University
Princeton, New Jersey, USA
pmittal@princeton.edu

ABSTRACT

1

Machine learning is leveraged for many network traffic analysis
tasks in security, from application identification to intrusion detection. Yet, the aspects of the machine learning pipeline that ultimately determine the performance of the model—feature selection
and representation, model selection, and parameter tuning—remain
manual and painstaking. This paper presents a method to automate
many aspects of traffic analysis, making it easier to apply machinelearning techniques to a wider variety of traffic analysis tasks.
We introduce nPrint, a tool that generates a unified packet representation that is amenable for representation learning and modeltraining. We integrate nPrint with automated machine learning(AutoML), resulting in nPrintML, a public system that largely eliminates feature extraction and model tuning for a wide variety of
traffic analysis tasks. We have evaluated nPrintML on eight separate
traffic analysis tasks and released nPrint and nPrintML to enable
future work to extend these methods.

Many traffic analysis tasks in network security rely on machinelearning (e.g., application identification [8, 25, 56], device identification [17, 26]). Although research has paid much attention to
the machine learning models applied to these tasks and the performance of those models, in practice these tasks rely heavily ona pipeline that involves manually engineering features and selecting and tuning models. Arriving at an appropriate combinationof features, model, and model parameters is typically an iterative
process. Indeed, the effectiveness of applying machine learning to
network traffic analysis tasks often depends on the selection andappropriate representation of features as much as the model itself,
yet this part of the process has remained exacting and manual.
Feature engineering and model selection are painstaking processes, typically requiring substantial specialized domain knowledge to engineer features that are both practical to measure orderive and result in an accurate model. Even with expert domain
knowledge, feature exploration and engineering remains largelya brittle and imperfect process, since the choice of features andhow to represent them can greatly affect model accuracy. Suchmanual extraction may omit features that either were not immediately apparent or involve complex relationships (e.g., non-linear
relationships between features). Furthermore, traffic patterns andconditions invariably shift, rendering models and hand-craftedfeatures obsolete [4, 19]. Finally, every new network detection orclassification task requires re-inventing the wheel: engineering newfeatures, selecting appropriate models, and tuning new parametersby hand.
This paper reconsiders long-held norms in applying machinelearning to network traffic analysis; namely, we seek to reducereliance upon human-driven feature engineering. To do so, we
explore whether and how a single, standard representation of a
network packet can serve as a building block for the automation ofmany common traffic analysis tasks. Our goal is not to retread any
specific network classification problem, but rather to argue thatmany of these problems can be made easier—and in some cases,
completely automated—with a unified representation of traffic thatis amenable for input to existing automated machine learning (AutoML) pipelines [14]. To demonstrate this capability, we designed astandard packet representation, nPrint, that encodes each packet in
an inherently normalized, binary representation while preserving
the underlying semantics of each packet. nPrint enables machinelearning models to automatically discover important features fromCCS CONCEPTS• Security and privacy → Network security; • Networks →Network algorithms; Packet classification; • Computing methodologies → Supervised learning by classification;

KEYWORDS
Network Traffic Analysis; Automated Traffic Analysis; Machine
Learning on Network Traffic
ACM Reference Format:
Jordan Holland, Paul Schmitt, Nick Feamster, and Prateek Mittal. 2021. New
Directions in Automated Traffic Analysis. In Proceedings of the 2021 ACM
SIGSAC Conference on Computer and Communications Security (CCS ’21),
November 15–19, 2021, Virtual Event, Republic of Korea. ACM, New York, NY,
USA, 18 pages. https://doi.org/10.1145/3460120.3484758

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.
Thisallwork
licensed
underthea Creative
Commons Attribution International 4.0 License.
For
otheris uses,
contact
owner/author(s).
CCS
15–19, 2021,
2021,Virtual
VirtualEvent,
Event,Republic
RepublicofofKorea.
Korea
CCS’21,'21, November 15–19,
2021Copyright
Copyright heldis held
owner/author(s).
©©2021byby
thetheowner/author(s).
ACMISBNISBN 978-1-4503-8454-4/21/11.
978-1-4503-8454-4/21/11.
ACM
https://doi.org/10.1145/3460120.3484758
https://doi.org/10.1145/3460120.3484758

3366

INTRODUCTION

Session 12C: Traffic Analysis and Side Channels

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

Problem Overview
Description
Active Device Fingerprinting (§5.1)

Passive OS Detection (§5.2)

Network Device Dataset [22]CICIDS 2017 [48]# Classes

Sample Size(# Packets)

Balanced
Accuracy

ROC
AUC

Macro
F1

Score

Source-4 -t -i

21

95.4

99.7

95.5

92.9 (Macro-F1)

ML-Enhanced
Nmap [31]

-4 -t

1
10
100
100

99.5
99.9
99.9
77.1

99.9
100
100
97.5

99.5
99.9
99.9
76.9

43

99.8
99.9
95.0
99.9
99.9
99.8

96.9
99.7
78.8
99.7
99.7
99.9

99.7
99.5
77.4
99.7
99.7
99.8

99.8 (Average Accuracy)

10

92.4
86.1

99.5
96.9

93.2
84.1

99.9 (True Positive Rate)
39.7 (Balanced F1)

10

81.9
76.1
66.2
60.9

98.0
94.2
91.3
92.2

79.5
75.8
63.7
57.6

ConfigurationeAppendix A.4)

15
3
13

Application Identification via DTLS Handshakes (§5.3)

Malware Detection for IoT Traces (§5.4.1)

Comparison

nPrintML
DatasetDTLS Handshakes [32]

7

-4
-u
-p 10
-p 25
-p 100-4 -u -p 10

netML IoT [6, 28]

219-4 -t -u

7

-4 -t -u -p 10

81.3 (Macro-F1)

p0f [40]

No Previous Work67.3 (Balanced F1)

Hand-Curated Features [32]NetML Challenge
Leaderboard [37]Type of Traffic in Capture (§5.4.1)

netML Non-VPN [6, 12]

Intrusion Detection (§5.4.1)

netML CICIDS 2017 [6, 48]

2
8-4 -t -u

5

99.9
99.9

99.9
99.9

99.9
99.9

Cross Platform [44]3

-4 -t -u -p 50

25

96.8

90.2

90.4

No Previous Work

-4 -t -u -R

10
25
50

77.9
90.2
98.4

96.0
98.6
99.9

78.9
90.4
98.6

No Previous Work

Determine Country of Origin for(§5.4.2)
Android & iOS Application Traces
Identify streaming video (DASH)(§5.4.3)
service via device SYN packets

Streaming Video Providers [10]-4 -t -u18
31

442.1 (Balanced F1)
34.9 (Balanced F1)
98.9 (True Positive Rate)
99.2 (Balanced F1)Table 1: Case studies we have performed with nPrintML. nPrintML enables automated high performance traffic analysis across a wide range and combination of
protocols. In many cases, models trained on nPrint are able to outperform state-of-the-art tools with no hand-derived features.

sets of packets for each distinct classification task without the needfor manual extraction of features from stateful, semantic network
protocols. This automation paves the way for faster iteration and deployment of machine learning algorithms for networking, lowering
the barriers to practical deployment.
The integration of nPrint with AutoML, a system we namenPrintML, enables automated model selection and hyperparametertuning, enabling the creation of complete traffic analysis pipelines
with nPrint—often without writing a single line of code. Our evaluation shows that models trained on nPrint can perform fine-grained
OS detection, achieve higher accuracy than Nmap [31] in device fingerprinting, and automatically identify applications in traffic withretransmissions. Further, we compare the performance of models
trained on nPrint to the best results from netML, a public traffic
analysis challenge for malware detection, intrusion detection, andapplication identification tasks [6, 37]. nPrintML outperforms thebest performing hand-tuned models trained on the extracted features in all cases but one. Finally, to explore generality of nPrintML
for a broader range of traffic analysis tasks, we use nPrintML totrain models on public datasets to identify the country of origin formobile application traces and to identify streaming video service
providers [10, 44].
Table 1 highlights the performance of nPrintML. To enable others to replicate and extend these results—and to apply nPrint andnPrintML to a broader class of problems—we have publicly released
nPrint, nPrintML, and all datasets used in this work, to serve as abenchmark for the research community and others.
Many problems, such as capturing temporal relationships acrossmultiple traffic flows, and running nPrintML on longer traffic sequences remain unsolved and unexplored. In this regard, we view
nPrint and nPrintML as the first chapter in a new direction forresearch at the intersection of machine learning and networking

that enables researchers to focus less time on fine-tuning models
and features and more time interpreting and deploying the bestmodels in practice.
Figure 1 summarizes the following sections of this paper. Sections 2 and 3 discuss the design and implementation of nPrint.
Section 4 introduces AutoML and nPrintML, which combines AutoML and nPrint. Section 5 applies nPrintML for the case studies
in Table 1. Finally, Sections 6 and 7 examine related works andsummarize our contributions.

2

DATA REPRESENTATION

For many classification problems, the data representation is at least
as important as the choice of model. Many machine learning models are well-tuned for standard benchmarks (e.g., images, video,
audio), but unfortunately network traffic does not naturally lend
itself to these types of representations. Nonetheless, the nature of
the models dictate certain design requirements, which we outline
in Section 2.1. Section 2.2 explores three possible standard representations of network traffic, including a semantic encoding, an
unaligned binary representation, and a hybrid approach, whichfuses a binary representation of the packet with an encoding thatsimultaneously preserves the semantic structure of the bits.

2.1

Design Requirements

Complete. Rather than select for certain features (or representations), we aim to devise a representation that includes every bit of apacket header. Doing so avoids the problems of relying on domain
knowledge that one packet header field (or combination of fields)
is more important than others. Our intuition is that the models canoften determine which features are important for a given problem
without human guidance, given a complete representation.

3367

Session 12C: Traffic Analysis and Side Channels

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

Figure 1: nPrint produces a standard network traffic representation that can be combined with AutoML tools to create nPrintML, a standard, largely automatedtraffic analysis system.

Constant size per problem. Many machine learning models assume that inputs are always the same size. For example, a trainedneural network expects a certain number of inputs; other models
such as random forests and even regression models expect thatthe input conforms to a particular size (i.e., number of elements).
Thus, each representation must be constant-size—even if the sizesof individual packets or packet headers vary. Common approaches
used in image recognition, such as scaling, do not directly apply
to network traffic. An ideal representation has a size per-problem
that is determined a priori; this knowledge avoids the need for multiple passes over a stored packet trace, and it is essential in data
streaming contexts.

IP, TCP, and UDP. Each header has semantic fields such as the IP
TTL, the TCP port numbers, and the UDP length fields. A standard semantic representation of network traffic collects all of thesesemantic fields in a single representation. This semantic representation is complete and constant size but has drawbacks. Figure 2
shows an example of this semantic representation and some of its
drawbacks as a standard representation. First, the representation
does not preserve ordering of options fields, which have been long
used to separate classes of devices in fingerprinting [31, 52].
We examine two of the datasets (presented in Sections 5.2 and 5.1)
and find 10 and 59 unique TCP option orderings, respectively. We
further explore the effect that option ordering can have on classification performance in Appendix A.1, finding that a binary representation of the TCP options, which preserves ordering, outperforms
a semantic representation by 10% across a wide range of models interms of F1-score. Second, domain expertise is required to parse thesemantic structure of every protocol, and even with this knowledge,
determining the correct representation of each feature is often asignificant exercise. For example, domain knowledge might indicate that the TCP source port is an important field, but further(often manual) evaluation may be needed to determine whether
it should be represented as a continuous value, or with a one-hot
encoding, as well as if the feature needs to be normalized before
training. These decisions must be made for every field extracted ina semantic manner, from IP addresses to each unique TCP option
to ICMP address masks.

Inherently normalized. Machine learning models typically perform better when features are normalized: even simple linear models (e.g., linear and logistic regression) operate on normalized features; in more complex models that use gradient-based optimization,
normalization decreases training time and increases model stability [36, 50]. Of course, it is possible to normalize any feature set,
but it is more convenient if the initial representation is inherently
normalized.
Aligned. Every location in the representation should correspondto the same part of the packet header across all packets. Alignment
allows for models to learn feature representations based on the factthat specific features (i.e., packet headers) are always located at
the same offset in a packet. While human-driven feature engineering leads to aligned features by extracting information from eachpacket into a well-formatted structure, this requirement is needed
when considering packets in binary form, as both protocols andpackets differ in length. Any misaligned features inject noise into
the learning process, reducing the accuracy of the trained model.

2.2

2.2.2 Naive Binary Representation. We can preserve ordering andmitigate reliance on manual feature engineering with a raw bitmap
representation. This choice leads to a consistent, pre-normalized
representation akin to an “image” of each packet. We see an example of this representation in Figure 2. However, transforming
each packet into its bitmap representation ignores many intricate
details, including varying sizes and protocols. These issues cancause feature vectors for two packets to have different meanings
for the same feature. For example, a TCP packet and a UDP packet
with the same IP header would have entirely different information
represented as the same feature. Figure 2 illustrates this problem.

Building a Standard Data Representation

Network traffic can be represented in multiple ways. We discuss
three options: semantic, unaligned binary, and a hybrid representation that we term nPrint.
2.2.1 Semantic Representation. A classic view of network traffic
examines packets as a collection of higher-level headers, such as

3368

Session 12C: Traffic Analysis and Side Channels

IP
Verison

IP
IHL

IP
...

IP
Options

TCP
Source Port

TCP
...

TCP
Options

Payload

4

5

...

?

80

...

?

?

TCP Options
Bit 1

Payload
Bit 1

Semantic: (TCP / IP)IP
Bit 1
Naive Binary: (TCP / IP)

TCP
Bit 1

TCP
Bit 1

Payload
Bit 1

0 1 0 0 0 1 0 1 ... ... ... ... 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 ... ... ... ... 1 0 1 1
IP
Bit 1

Naive Binary: (UDP / IP)

IP Options
Bit 1

0 1 0 0 0 1 0 1 ... ... ... ... 0 1 1 1 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 ... ... ... ... 1 1 0 0 1 0 1 1
IP
Bit 1

Naive Binary: (TCP / IP)(No Options)

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

UDP
Bit 1

Payload
Bit 1

0 1 0 0 0 1 0 1 ... ... ... ... 1 0 0 1 0 0 1 0 ... ... ... ... 1 0 1 1

Figure 2: Semantic and naive binary nPrint representations introduce various problems that make modeling difficult: Semantic loses ordering of specific fields suchas the TCP options. Binary nPrints lack alignment, which can degrade performance due to misaligned features.

Such misalignment can occur between two packets of the same
protocol: a TCP/IP packet with IP options and a TCP/IP packet
without IP options will cause the bits to be misaligned in the two
representations. Misalignment can manifest in two ways: 1) decreasing model performance as the misaligned bits introduce noisein the model where important features may exist; and 2) the resulting representation is not interpretable, as it is impossible tomap each bit in the representation to a semantic meaning. The
ability to understand the features that determine the performanceof a given model is especially important in network traffic where
the underlying data has semantic structure, in contrast to image
classification, where the underlying data is pixels.

non-existing headers with -1, each feature takes on one of three values: -1, 0, or 1, removing the need for parsing and representing thevalues for each field in every packet. Further, filling non-existingheader values with -1 allows us to differentiate between a bit being
set to 0 and a header that does not exist in the packet. Finally, nPrint
is constant size per problem: each packet is represented in the same
number of features. We make the payload an optional number ofbytes for a given problem: with the increasing majority of network
traffic being encrypted, the payload is unusable for many trafficclassification problems.
nPrint is modular and extensible. First, other protocols (e.g.,
ICMP) can be added to the representation. Second, nPrint, which
is a single-packet representation, can be used as a building blockfor classification problems that require sets of packets, as we haveshown in several cases. If we consider each nPrint fingerprint as a
1xM matrix, where M is the number of features in the fingerprint,
we can build multi-packet nPrint fingerprints by concatenatingthem.

2.2.3 Hybrid: nPrint. Figure 3 shows nPrint, a single-packet representation that can be provided, unmodified, as input to machinelearning models. nPrint is a hybrid of semantic and binary packet
representations, representing packets to models as raw binary data,
but aligning the binary data in such a way that recognizes thatthe packets themselves have specific semantic structure. By using
internal padding, nPrint mitigates misalignment that can occur
with an unaligned binary representation while still preserving ordering of options. Further, nPrint encodes the semantic structure of
protocols while only requiring knowledge of the maximum lengthof the protocol compared to parsing and representing each field ina protocol.
nPrint is complete, aligned, constant size per problem, and inherently normalized. nPrint is complete: any packet can be represented
without information loss. It is aligned: using internal padding andincluding space for each header type regardless of whether thatheader is actually present in a given packet ensures that each packet
is represented in the same number of features, and that each feature
has the same meaning. Alignment gives nPrint a distinct advantage
over many network representations in that it is interpretable atthe bit level. This allows for researchers and practitioners to map
nPrint back to the semantic realm to better understand the featuresthat are driving the performance of a given model. Not all models
are interpretable, but by having an interpretable representation, we
can better understand models that are. nPrint is also inherently
normalized: by directly using the bits of the packets and filling

3

NPRINT IMPLEMENTATION

We implemented nPrint in C++ and evaluated it in different contexts,
including its memory footprint and a proof-of-concept evaluation
on an operational 10 Gbps link. nPrint currently supports Ethernet,
IPv4, fixed IPv6 headers, UDP, TCP, ICMP, and any corresponding
packet payloads. Appendix A.4 shows the full configuration options
available in nPrint. nPrint can either process offline packet packet
capture formats such as PCAP and Zmap [13] output, or capture
packets directly from a live interface. nPrint can also reverse theencoding, creating a PCAP from the nPrint format. The nPrint
source code is publicly available.
nPrint transforms over 1 million packets per minute. We evaluate
the performance of nPrint on a system with a 4-core, 2.7 GHz
CPU (Intel Core i7-8559U) and 32GB of RAM. On this machine,nPrint transformed the three datasets used in Section 5, containing
approximately 2.35 million, 274K, and 49K packets, in 49, 13, and
12 seconds respectively∗ On average, nPrint currently transforms∗ Each dataset is in a different file format resulting in varying packet processing
rates. The file formats are a single PCAP, a CSV of hex-encoded raw packets, and over
7,000 PCAPs, respectively.

3369

Session 12C: Traffic Analysis and Side Channels

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

nPrintIPv4
480 Features

TCP
480 Features

UDP
64 Features

ICMP
64 Features

Payloadn Features

Maximum Sizeof IPv4 Header(60 Bytes)

Maximum Sizeof TCP Header(60 Bytes)

Size ofUDP Header(8 Bytes)Size of
ICMP Header(8 Bytes)

User Deﬁned
Number of Bytes

nPrint (TCP / IP) Packet
0 1 0 0 0 1 0 1 1 1 ... ... ... ... 0 1 0 1 0 1 0 1 0 1 ... ... ... ... -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 1 1 0 ... ... ... ...
nPrint (UDP / IP) Packet
0 1 0 0 0 1 0 1 1 1 ... ... ... ... -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 0 1 1 ... ... ... ... -1 -1 -1 -1 -1 -1 -1 1 1 0 ... ... ... ...

Figure 3: nPrint, the complete, normalized, aligned packet representation. Headers that do not exist in the packet being transformed are noted with sentinels
accordingly, while headers that exist but are not of maximum size are zero padded for alignment across nPrints. nPrint removes reliance on manual featureengineering while avoiding misaligned features.

packets at about 1.5 million packets per minute using a single
thread. We note here a few performance bottlenecks of the nPrint
implementation. nPrint writes output in a structured CSV format,
chosen for interoperability with popular machine learning libraries.
Replacing nPrint’s output to a binary format could yield significant
performance benefits. Further, nPrint leverages libpcap for packet
processing. Leveraging optimized packet libraries, such as zerocopy pf_ring could also increase the performance of nPrint on live
traffic [38, 54]. We leave such optimizations for future work.

by experts over the course of days, weeks, or years, and ultimately
trained on one (or a small set of) models that experts believe will
work best on the developed features. Finally, the models are tuned
either by hand or through a search process such as a grid search [47].
We highlight an opportunity to simplify the standard pipeline in
Figure 1 through nPrint and new AutoML systems.
First, we designed nPrint to be directly used in a machine learning pipeline, standardizing the tedious feature development process
for a large set of traffic analysis problems. Next, our design choices
for nPrint enable it to be combined with new AutoML tools tostandardize the second half of the pipeline.
AutoML tools are designed to automate feature selection, model
selection, and hyperparameter tuning to find an optimized model
for a given set of features and labels. Rather than running hyperparameter optimization on one model, AutoML tools use optimization
techniques to perform combined model selection and hyperparameter optimization, searching for the highest performing model in amore principled manner than by hand.
We build upon our guiding principle that models can extract thebest features for each task and allow AutoML to determine the best
type of model and hyperparameters for that problem. This decision
provides multiple benefits: 1) we can train and test more modeltypes, 2) we can optimize the hyperparameters for every model wetrain, and 3) we are certain that the best model is chosen for a given
representation. Finally, we note that although the case studies inthis work and the public implementations of nPrint and nPrintML
focus on supervised learning, there is opportunity for future workin combining the nPrint representation with unsupervised learningtechniques for other applications.

nPrint has a constant memory footprint. nPrint has a constantmemory footprint that depends only on the output configuration.
As an example, we profile memory usage with valgrind while configuring nPrint to include IPv4 and TCP output (used for p0f evaluationin Section 5.2). This configuration results in a constant memory
footprint of about 295 KB. A configuration consisting of IPv4, UDP,
ICMP, TCP, and 10 payload bytes yields a constant memory footprint of about 310 KB.
Proof of Concept: nPrint on an operational link. nPrint processes
each packet independently, making it amenable to parallelization.
We run a proof-of-concept evaluation to understand of the baseline
performance of our public nPrint implementation on a 10 Gbps
link comprised of traffic from a consortium of universities using acommodity server. The server has two Intel Xeon 6154 CPUs, eachwith 18 cores, running at 3 GHz and 376 GB of RAM. We observethat by load balancing the traffic across multiple receive queue/CPU
pairs using Receive-Side Scaling (RSS† ) we can run multiple, parallel nPrint processes without incurring penalties for moving data
between cores. We verified that nPrint is able to run on a live trafficload of roughly 8 Gbps with near zero loss using 16 queues and 16
nPrint processes. Given this performance, nPrint should be capableof processing higher rates by leveraging further parallelization andoptimized packet libraries, such as zero-copy pf_ring [38]. We hopethat our publicly available implementation serves as groundworkfor those who wish to further optimize performance.

4

4.1

AutoGluon-Tabular AutoML

We use AutoGluon-Tabular to perform feature selection, model
search, and hyperparameter optimization for all eight problems weevaluate [14]. We choose AutoGluon as it has been shown to outperform many other public AutoML tools given the same data [14],and it is open source, though nPrint’s well-structure format makeit amenable for any AutoML library. While many AutoML tools
search a set of models and corresponding hyperparameters, AutoGluon achieves higher performance by ensembling multiple single
models that perform well. AutoGluon-Tabular allows us to train,
optimize, and test over 50 models for each problem stemming fromNPRINTML

Much previous work has used a pipeline similar to the full pipeline
seen in Figure 1. Packets are transformed into features developed† RSS is Intel-specific, depending on hardware another similar multi-queue receive
technology could also be used.

3370

Session 12C: Traffic Analysis and Side Channels

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

6 different base model classes which are variations of tree-based
methods, deep neural networks and neighbors-based classification.
The highest performing model for each problem we examine is an
ensemble of the base model classes.
AutoGluon has a presets parameter that determines the speedof the training and size of the model versus the overall predictive quality of the models trained. We set the presets parameterto high_quality_fast_inference_only_refit, which produces
models with high predictive accuracy and fast inference. There isa quality preset of “best quality” which can create models withslightly higher predictive accuracy, but at the cost of ~10x-200x
slower inference and ~10x-200x higher disk usage. We make thisdecision as we believe inference time is an important metric when
considering network traffic analysis. By selecting a high quality
model setting, each model is bagged with 10 folds, decreasing thebias of individual models. We note that the preset parameter for an
AutoML tool does not represent the training of a single model, but
an optimization of a set of models for a given task.
We set no limit on model training time, allowing AutoGluon tofind the best model, and split every dataset into 75% training and 25%
testing datasets. Finally, we set the evaluation metric to f1_macro,
which represents a F1 score that is calculated by calculating the
F1 Score for each class in a multi-class classification problem andcalculating their unweighted mean. This decision leads AutoGluon
to tune hyperparameters and ensemble weights to optimize theF1-macro score on validation data.

4.2

5

5.1

2

Active Device Fingerprinting

First, we examine the utility of nPrintML to extract features frompackets in the context of active device fingerprinting. We compare
the performance of models trained on the nPrint representation toNmap, perhaps the most popular device fingerprinting tool, which
has been developed for over 20 years.
Input (Packets). We use a dataset of labeled devices and fingerprints to compare nPrintML’s performance to Nmap’s handengineered features. Holland et al. previously used a subset of
Nmap’s probes to fingerprint network device vendors at Internet
scale [22]. They curate a labeled dataset of network devices throughan iterative clustering technique on SSH, Telnet, and SNMP banners,
which provides a list of labeled network devices to Nmap.
Although this previous work was concerned with fingerprinting
devices at scale, we are concerned with nPrintML’s classification
performance against Nmap’s full suite of features. As such, wedownsample the labeled network device dataset to create a set ofdevices to compare the classification performance of nPrintML with
Nmap. We further expand the types of devices we are testing totest the adaptability of nPrintML across a larger range of device
types. To this end, we add a new device category to the dataset:
Internet of Things (IoT) devices. We gather labels for four types of
IoT devices, two IoT cameras and two IoT TV streaming devices
through Shodan [49]. Table 2 shows the final dataset.
Holland et al.’s dataset includes the Nmap output and raw packet
responses for each label in the dataset. We modify Nmap to output
the raw responses for each probe and Nmap each IoT device added
through Shodan labeling. We only have access to the responses
to the probes that Nmap sends, not the actual sent probes, as wecannot re-scan the labeled router dataset due to the chance that thedevice underneath the IP address may have changed.

nPrintML Implementation

We have implemented nPrintML in Python, directly combining
nPrint and AutoGluon-Tabular AutoML. nPrintML enables researchers to create full traffic analysis pipelines using nPrint ina single program call. Below is an example of fully recreating thepassive OS detection case study found in Section 5.
1

CASE STUDIESIn this section, we highlight the versatility and performance ofnPrint. Table 1 presents the breadth of problems that nPrintML
can be leveraged for, summarizing eight case studies tested with
nPrintML. This section examines those case studies in more depth:
We show that nPrintML can match or beat the performance ofexisting bespoke approaches for each of these problems.

$ nprintml -L os_labels . txt -a index \-P traffic . pcap -4 -t -- sample_size 10

A full tutorial of nPrintML is supplied in Appendix A.2. At thetime of writing, nPrintML can create machine learning pipelines
from a single traffic trace or an entire directory of traffic traces.

Packet Transformation and Data Representation. Nmap transforms the responses to each probe into a fingerprint using a seriesof varyingly complex tests developed for decades. Table 10 in Appendix A.3 outlines the full set of tests Nmap performs on the probe
responses, which includes complex features computed across setsof packets. We use this fingerprint for our evaluation in two ways.
First, Nmap compares the fingerprint generated to its database,making a classification of the remote device. Second, we take thefingerprint and encode each test as a categorical feature, creating afeature vector to be used with machine learning methods. Holland
et al. show that this technique is effective using Nmap’s closed-port
probes, which comprise only 6 of the 16 probes Nmap sends to eachdevice. We consider every test Nmap conducts when transforming
each fingerprint into a feature vector.
nPrintML uses the raw responses generated from the modified
version of Nmap to build a nPrint for each device. nPrintML uses

Metrics. nPrintML outputs a standard set of metrics for each classification problem, which we define here. We define a false positive
for class C as any sample that is not of class C, but misclassified asclass C by the classifier. A false negative for class C is any sample ofclass C that is not classified as class C. We then evaluate each trainedmodel using multiple metrics including balanced accuracy, ROC
AUC, and F1 scores. We use a balanced accuracy score to account for
any class imbalance in the data. In the multi-class classification case
we present macro AUC ROC scores in a “one vs rest” manner, where
each class C is considered as a binary classification task between
C and each other class. The ROC AUC is computed by calculating
the ROC AUC for each class and taking their unweighted mean.
F1 scores represent a weighted average of precision and recall. For
multi-class classification, we report a macro F1 score, calculated in
the same manner as optimized during training.

3371

Session 12C: Traffic Analysis and Side Channels

Nmap Direct

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

Nmap Aggressive

VendorDeviceType

LabeledDevices

Precision

RecallPrecision

RecallAdtranAvtechAxis
Chromecast
Cisco
Dell
H3C
HuaweiJuniper
LancomMikrotikNEC
Roku
Ubiquoss
ZTE

Network
IoT
IoT
IoT
Network
Network
Network
Network
Network
Network
Network
Network
IoT
Network
Network

1,449
2,152
2,653
2,872
1,451
1,449
1,380
1,409
1,445
1,426
1,358
1,450
2,403
1,476
1,425

0.95
0.00
0.00
0.00
0.99
0.11
0.00
0.76
0.99
0.05
0.00
0.00
0.00
0.00
0.00

0.24
0.00
0.00
0.00
0.56
0.27
0.00
0.24
0.48
0.00
0.00
0.00
0.00
0.00
0.00

0.70
0.00
0.01
0.00
0.95
0.11
0.31
0.55
0.72
0.15
0.04
0.60
0.00
0.00
0.00

1.00
0.03
0.52
0.33
0.99
0.87
0.85
0.87
0.98
0.84
0.47
0.99
0.00
0.00
0.00

Average Precision

Table 2: The active device fingerprinting set and Nmap’s heuristic performanceon the devices. Nmap’s heuristic only excels on Cisco devices, with significantly
lower performance across IoT devices.

Vendor

ML-Enhanced
Nmap

nPrintAdtranAvtechAxis
Chromecast
Cisco
Dell
H3C
HuaweiJuniper
LancomMikrotikNEC
Roku
Ubiquoss
ZTE

1.00
0.87
0.93
1.00
0.97
0.85
0.95
0.94
0.99
0.99
0.88
1.00
0.92
0.99
0.99

1.00
0.95
0.98
1.00
0.99
0.99
0.96
0.95
0.99
0.99
0.91
1.00
0.99
0.99
0.99

Table 3: Models trained on the nPrint representation match or outperform
models trained on Nmap’s hand-engineered features for every class in thedataset.

the same response packets that Nmap uses to build each nPrint.
Further, while Nmap computes many of its features across both sent
and received packets, we only have access to the received packets
in each nPrint.
We configure nPrintML to include IPv4, TCP, and ICMP headersin each nPrint, resulting in 1,024 features for each individual packet.
Here we note that while configuring nPrint and nPrintML requires
some user intervention, namely indicating the type of traffic to
include in the output via a command-line flag, this configuration
requires significantly less work when compared with manually
defining, extracting, and representing features from multiple traffic
protocols. nPrintML aggregates all responses from each device
to create a 21-packet nPrint for each device. We point out that
there are 21 packets in each nPrint, while Nmap sends only 16
probes to the device. Nmap re-sends probes that do not garnera response up to three times. It uniquely re-names these probes.
Rather than disambiguate the names, which is unreliable due to theunique naming scheme, we consider each uniquely named Nmap
response as a packet in the nPrint. This does not give us access toany information that Nmap does not use, and at worst duplicatesdata already in the nPrint. nPrintML fills any probe response with
-1s if the device did not respond to the probe. Finally, as each probe
is specifically named, we sort each aggregated nPrint created bynPrintML by the probe name to ensure consistent order across all
samples.

Representation
nPrintNmap

Balanced
Accuracy

ROC AUC

F1

95.4
92.7

99.7
99.3

95.5
92.9

Table 4: Models trained on the nPrint representation outperform Nmap’s
hand-engineered features across every metric considered.

Representation
Nmap
Nmap
nPrintnPrint# Models in
Ensemble

Training Time
(Seconds)

Inference Time
(Seconds)

F1

2
3
3
2

497
630
755
188

12
32
267
3

92.8
92.9
95.4
95.2

Table 5: Ensembles of models trained on the nPrint representation can outperform Nmap in both F1 score and inference time.

recall. We notice a trend in the devices that Nmap is accurate atclassifying: older, North American based routers. Of all of the IoT
devices in the dataset, Nmap’s highest heuristic precision is .01.
nPrintML outperforms an AutoML-enhanced Nmap. To directlycompare Nmap’s long-developed techniques for extracting differentiating features from network protocols to nPrintML, we enhanceNmap by replacing its heuristic with AutoML. We take each one-hotencoded Nmap fingerprint and run the AutoML pipeline, creating
optimized models for the extracted features. We also run nPrintML
on the nPrint representation generated for each device.
Table 3 shows the average precision score, used to summarize aprecision-recall curve, of the highest performing classifier for Nmap
and nPrintML [46]. Immediately, we see that not only can models
trained on the nPrint representation differentiate the devices without manual feature engineering, nPrintML outperforms Nmap’s

Nmap’s heuristics perform poorly for some devices. Nmap compares its generated fingerprint to a hand-curated fingerprint database using a carefully-tuned heuristic. Table 10 in Appendix A.3
shows weights of this heuristic. Nmap either outputs a direct guessof the device, which it is more confident in, or an aggressive guess,where there is less confidence in the output. Table 2 shows theperformance of this heuristic on the entire dataset using both direct
and aggressive metrics. We see that Nmap’s performance is low
across the entire dataset with the exception of Cisco, Juniper, andAdtran devices, for which it has high precision with relatively low

3372

Session 12C: Traffic Analysis and Side Channels

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

p0f

nPrint1 Packet

10 Packets

100 Packets

1 Packet

10 Packets

100 Packets

Host

p0f Label

P

R

P

R

P

R

P

R

P

R

P

R

Mac OSMac OS x 10.x

1.00

0.05

1.00

0.28

1.00

0.88

0.99

0.99

1.00

0.99

1.00

0.99

Linux 3.11 and newer

1.00
1.00
1.00
1.00
1.00
1.00

0.01
0.04
0.04
0.05
0.04
0.05

1.00
1.00
1.00
1.00
1.00
1.00

0.25
0.20
0.20
0.19
0.24
0.25

1.00
1.00
1.00
1.00
1.00
1.00

0.74
0.69
0.65
0.68
0.79
0.74

0.99

0.99

0.99

1.00

0.99

1.00

Windows 7 or 8

0.99
0.99
1.00
0.99
1.00

0.00
0.01
0.04
0.05
0.01

0.98
0.98
1.00
0.99
1.00

0.02
0.04
0.23
0.25
0.27

0.98
1.00
1.00
0.99
1.00

0.09
0.14
0.71
0.77
0.71

1.00

1.00

1.00

1.00

1.00

1.00

No output

-

-

-

-

-

-

-

-

-

-

-

-

Web Server
Ubuntu 14.4 32B
Ubuntu 14.4 64B
Ubuntu 16.4 32B
Ubuntu 16.4 64B
Ubuntu Server
Windows 10
Windows 10 Pro
Windows 7 Pro
Windows 8.1
Windows Vista
Kali Linux

Table 6: Performance of nPrintML vs. p0f for passive OS fingerprinting. nPrintML is capable of finer granularity for OS fingerprinting and achieves near perfectprecision and recall when compared directly to p0f. While p0f never provides any OS estimate for Kali Linux, we include it for testing fine-grained OS classification
with nPrintML.

0.000

Version

IHL

TTL

0.005

ID

0.010

0.015

TOS
R DM
ProtocolSource IP
Destination IP

model for both Nmap and nPrintML. We see that nPrintML outperforms Nmap’s long-developed features in every metric without
access to the sent probes.
Table 5 examines the training and inference time of the twohighest performing models for both Nmap and nPrintML. We seethat although the highest performing nPrintML model takes longer
to train than on Nmap’s features, models trained on the nPrint
representation can achieve higher F1 scores than models trainedon Nmap’s features with a lower training time and faster inference
speed. Figure 4 shows a heatmap of the feature importance gatheredfrom the random forest model trained with nPrintML. We find that
the TCP source port of the response probes are one of the more
important features in classifying the device vendor. Upon further
inspection, we find that Nmap does a port scan to find an open port
to sent its open-port probes to. The IoT devices each have a specificport that is found to be open during the port scan that identify theclass of devices from the routers. We also see that the IP TTL and
TCP window size are helpful in identifying the device.

0.020

Total Length
Frag Offset
Checksum

Options(a) IPv4
Source Port

Sequence
Ack
Offset Rsvd N C E U A P R S F
Checksum

Destination Port
Window Size
Urgent PointerOptions

5.2(b) TCPType

CodeRest of Header

Passive OS Fingerprinting

We study nPrint in the context of passive OS fingerprinting: determining the operating system of a device from network traffic.
We compare the performance of nPrintML to p0f, one of the mostwell-known and commonly used passive OS fingerprinting tools.
Ultimately, we find that nPrintML can perform OS detection withmuch higher recall rate than p0f, using much smaller packet sequences. Further, we find that nPrintML can uncover fine-grained
differences in OSes. We highlight that nPrintML is able to be used
in a variety of manners including single-packet OS detection andusing sequences of packets.
p0f uses an array of traffic fingerprinting mechanisms to identify
the OS behind any TCP/IP communication. p0f relies on a usercurated database of signatures to determine the operating system

Checksum(c) ICMP

Figure 4: Per-bit feature importance for active fingerprinting, computed bysumming the feature importance for each individual nPrint in the 21 packet
nPrint created for each device. Brighter colors are more important. Given the
nPrint representation, ML models can learn important features (e.g., IP TTL,
window size), as opposed to relying on manual engineering.

hand-engineered features. Table 4 examines the best performing

3373

Session 12C: Traffic Analysis and Side Channels

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

of any given device. p0f generates semantic fingerprints from device traffic and looks for direct matches in its database in order
to identify the OS. We note that p0f’s fingerprinting capabilities
include much more breath than is considered in this case study,
as it contains signatures for not only OS detection, but browser
identification, search robots, and some commonly used applications. We specifically aim to test if models trained on the nPrint
representation can exceed the performance of p0f on a common
task for the tool and test if nPrintML can be used for automated,
fine-grained OS detection.

0.000 0.005 0.010 0.015 0.020 0.025 0.030

Version
TTL

IHL

ID

TOS
R DM
ProtocolSource IP
Destination IP

Total Length
Frag Offset
Checksum

Options(a) IPv4

Input (Packets). We use the CICIDS2017 intrusion detection evaluation dataset, which contains PCAPs of over 50 GB of network
traffic over the course of 5 days [48]. The traffic contains labeled
operating systems ranging from Ubuntu to Windows to MacOS.
There are 17 hosts in the IDS dataset, but we find only 13 withusable traffic. The usable devices are seen in the first column ofTable 6.

Source Port

Sequence
Ack
Offset Rsvd N C E U A P R S F
Checksum

Destination Port
Window Size
Urgent PointerOptions

Packet Transformation and Data Representation. We vary theamount of traffic available to p0f and nPrintML to compare performance in different settings. We use the first 100,000 packets seenfor each device and split them into sets of 1, 10, and 100 packet
samples (i.e., 100,000 1-packet PCAPs, 10,000 10 packet PCAPS, etc.)
to be used with both. This results in three separate classification
problems, each problem using the same traffic, but varying amounts
of information the classification technique accesses in each sample.
p0f extracts several fields from each packet and compares extracted values against a fingerprint database to find a matching
OS for the extracted fields. We run p0f, without modifications, oneach of the 1, 10, and 100 packet samples created from each device’s
traffic. We create separate nPrintML pipelines for the 1, 10, and100 packet traffic samples, configuring it to remove the IP source
address, IP destination address, IP identification, TCP source anddestination ports, and TCP sequence and acknowledgement numbers from each nPrint to avoid learning direct identifiers of specificdevices rather than general operating system characteristics. Further, we configure nPrintML to only use IP and TCP headers tofairly compare nPrintML with p0f.
p0f outputs an operating system guess only on packets that
directly match a fingerprint in p0f’s database, so the number of
estimates varies between samples. We treat each estimate as a vote.
For each sample, we tally the number of correct votes, incorrect
votes, and cases where p0f offered no estimate. Using these values
we calculate the precision and recall for each experiment. Table 6
shows the precision and recall for each separate experiment.
Table 6 illustrates the relatively coarse-grained output generated
by p0f. For example, p0f classifies all Ubuntu devices, and the web
server, as “Linux 3.11 and newer”, and all of the Windows devices
as “Windows 7 or 8”. Table 6 shows that p0f generally increases
in performance when given access to more packets in a sample,
and that it is precise when it does make an estimate. Unfortunately,
p0f’s recall is generally quite low until given access to 100 packet
samples for a device, as p0f does not offer any estimates for many
samples. Finally, p0f never outputs a single vote for the Kali Linux
traffic samples.

(b) TCP

Figure 5: Per-bit feature importance for passive OS detection. Brighter colors
are more important. Given the nPrint representation, ML models can automatically discover important features (e.g. IP TTL, window size), as opposed torelying on manual engineering.

nPrintML can perform OS classification with high recall and few
packets. To directly compare nPrintML to p0f, we use the “p0fLabel” in Table 6 as a label for each host and train each classifier
using the 3 coarse-grained classes on the same samples that p0f
was executed on. Table 6 shows the results of this classification.
As shown, models trained on the nPrint representation achievenearly the same precision as p0f while drastically improving upon
p0f’s recall. Interestingly, with only one packet, a model trained on
the nPrint representation can outperform p0f when given accessto 100 packet samples. We also see very slight improvements inmodel performance when increasing the number of packets ineach nPrint. Figure 5 shows a heatmap of the feature importancegathered from the highest performing random forest model trainedwith nPrintML. For the IPv4 header, the most important features
are in the time-to-live (TTL) field and, to a lesser degree, the IPID
field. These results confirm past observations that TTL IPID canbe used for OS detection, because different operating systems use
different default values for those fields [7, 33]. In the TCP header, thewindow size field is the most important feature. We also observe thatcertain bits in the TCP options can help determine OS as some OSes
include particular options by default such as maximum segment
size, window scaling, or selective acknowledgement permitted.
nPrintML can reveal fine-grained differences in OSes. Next, we
aim to discover if models trained on the nPrint representation canenable automated passive OS detection at a more fine-grained level.
We take the same 100-packet samples and train each model using
the “Host” column of Table 6 as a label for each sample, resulting

3374

Session 12C: Traffic Analysis and Side Channels

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea(a) nPrintML PR.

Figure 7: nPrintML Confusion Matrix using only initial TTL values.

select pairs from the two lists to train a model, and test against theremaining hosts in the lists.
nPrintML differentiates between Ubuntu and Windows machineswith perfect balanced accuracy, ROC AUC scores, and F1 scoresno matter which device pair was used for training. This is due tothe different initial IP time-to-live that is set by the two operating
systems which the model immediately learns. This experimentillustrates that models can successfully identify operating systems
generally from the nPrint representation, as opposed to memorizing
individual device characteristics.
Finally, we ensure that nPrint is not memorizing the locationsof the hosts in a network via the TTL of the devices‡ . In this experiment, we seek to limit nPrint to initial TTLs. The CICIDS2017
traffic was captured on end hosts. Given this, we filter and split the
PCAPs to only include packets whose source was also the capture
host. This ensures that nPrint only has access to the initial TTL
values for each host rather than a decremented value that may
indicate the location of the device. Figure 7 shows the results ofthe experiment using 10-packet samples. We see that the confusionof nPrint still lies within individual operating systems. nPrint still
differentiates the Web Server and Windows Vista from the otherclasses. Kali Linux is omitted from this experiment as it was notfound to initiate any traffic flows in the traffic examined.

(b) nPrintML Confusion Matrix.

Figure 6: Passive OS fingerprinting PR and confusion matrix for nPrintML.
Models trained on the nPrint representation learn to identify operating systems
at a finer granularity (e.g., versions) than p0f.

in a 13-class classification problem. Figure 6a shows the PR curveof the highest performing model.
Figure 6b examines the performance of the model trained on
the nPrint representation in more detail. The classifier separates
operating system characteristics, with almost all of the confusion
being within the same operating system. The classifier can separate
the devices into more fine-grained classes than p0f’s original output,
finding differences in Windows Vista and Kali Linux that are notencoded into p0f. Figure 6b also illustrates a challenge in finegrained OS detection, as bit version differences of the same OS tend
to lead to confusion. This result is unsurprising, as we anticipate
the network stack for an OS would be similar between different bit
versions.

5.3

DTLS Application Identification

We test the ability of nPrintML to identify a set of applications
through their DTLS handshakes. We aim to automatically identify
the application and browser that generated a DTLS handshake with
nPrintML when provided with the handshake traffic. MacMillan etal. examined the feasibility of fingerprinting Snowflake, a pluggable
transport for Tor that uses WebRTC to establish browser-to-browser
connections [32], which is built to be indistinguishable from other
WebRTC services. They collect almost 7,000 DTLS handshakes
from four different services: Facebook Messenger, Discord, Google

nPrintML differentiates OSes, not simply devices. We seek to verify
that nPrintML detects operating systems generally, rather thanlearning to identify specific devices on the network. We construct anexperiment where we compare sets of devices that share a common
OS. We take the five Ubuntu hosts and five Windows hosts in thedataset and set up a binary classification task where we iteratively‡ Note that p0f does use location along the path as a feature. p0f attempts to guessthe initial TTL by assuming that the value is one of 64, 128, or 255. Decremented TTLs
are then used to calculate “distance” (e.g., location on the path) from the initial TTL.

3375

Session 12C: Traffic Analysis and Side Channels

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

Application Handshakes
Browser

Snowflake

Facebook

GoogleDiscordFirefox
Chrome

991
0

796
784

1000
995

992
997

Table 7: The application identification dataset [32].

Hangouts, and Snowflake, across two browsers: Firefox and Chrome.
They then write a protocol parser and extract and engineer semantic
features from the handshakes to show that Snowflake is identifiablewith perfect accuracy.
We are interested demonstrating nPrintML’s ability to automate
this process entirely. This DTLS classification problem highlights
both nPrintML’s ability to adapt to entirely new tasks that involve
packet payloads, and nPrintML’s performance in a noisy environment, as the packet traces vary in length due to retransmission in
the UDP-based handshakes.

Model Architecture

Fit Time(Seconds)

Total
Inference Time
(Seconds)

F1

Random Forest
ExtraTrees
KNeighbors
LightGBM
Catboost
Weighted Ensemble
Neural Network

3.69
3.89
3.90
5.21
9.00
46.1
85.58

0.37
0.43
8.95
0.15
0.38
0.45
29.9

99.8
99.9
96.0
99.8
99.7
99.9
99.7

Table 8: The nPrint representation performs well across models, with training
and inference times varying depending on the type of modle.

0.000

Version
TTL

Input (Packets). Table 7 shows an overview of the nearly 7,000
DTLS handshakes in the dataset. While MacMillan et al. examine
the classification task solely at the application level, we further
split the classification task into the specific (browser, application)
pair created the handshake, increasing the number of classes in thetask from four to seven. Snowflake traffic is specific to the Firefox
browser and did not contain any Chrome instances.

IHL

0.005

ID

0.010

0.015

0.020

TOS

R DM
ProtocolSource IP
Destination IP

0.025

0.030

Total Length
Frag Offset
Checksum

Options(a) IPv4

Packet Transformation and Data Representation. We configure
nPrintML to transform each handshake, which was captured andfiltered as a PCAP file, into a nPrint. We vary the configuration ofnPrintML to test nPrintML under a variety of constraints. Table 1
shows these configurations. The number of packets in the packet
captures varies from 4 to 43 due to both client behavior and packet
retransmissions. nPrintML pads each fingerprint to the maximum
capture size with -1s and allows the models trained on the nPrint
representation to identify important features in noisy traffic.

Source Port
LengthDestination Port
Checksum(b) UDPDTLS(c) DTLS

Figure 8: Per-bit feature importance for (browser, application) identification,
computed by summing the feature importance for each individual nPrint
in the 43 packet nPrint created for each handshake. Brighter colors are more
important. Given the nPrint representation, ML models can automatically learn
important features (e.g., length), as opposed to relying on manual engineering.

nPrintML can automatically detect features in a noisy environment. We run nPrintML on the handshakes in nPrint format. The
weighted ensemble classifier trained on the nPrint representationachieves a perfect ROC AUC score, 99.8% accuracy, and 99.8% F1score. nPrintML can almost perfectly identify the (browser, application) pair that generates each handshake. While in prior work
manually engineered features achieved the same accuracy on aneasier version of the problem, nPrintML avoids model selection andfeature engineering entirely, matching the performance of manualfeatures and models on a more difficult instance of the problem [32].
Finally, we examine models trained on the nPrint representation
using different combinations of headers in each packet in the traffic.
Table 1 shows that models trained on the nPrint representation can
perform this task with high accuracy using only the UDP headersin each packet.

is important as some environments require different optimizations.
For example, classification in streaming environments may preferfaster inference time over the highest performing model.
nPrintML can be used to understand semantic feature importance.
We use nPrintML to map feature importance to their semantic
meaning. Figure 8 shows a heatmap of the feature importancegathered from the highest performing random forest model trainedon the nPrint representation. In this instance, the successive lengthsof the headers (steps in the handshake) and the information in the
DTLS payload drive much of the performance in nPrintML.

5.4

nPrintML performs well across models and trains quickly. Table 8shows the F1 score, training time, and total inference time on thetesting dataset for each non-ensemble classifier trained and theweighted ensemble classifier with the highest overall performance.
The nPrint representation works across models, with inference andtraining time varying. nPrint working well across different models

Additional Case Studies

In this section, we apply nPrintML across several additional casestudies, as summarized in Table 1. For brevity, we describe eachstudy and show the nPrintML command associated with the performance results shown in the table. In each case study we ran
nPrintML with a range of packet counts. We include the best performing commands here.

3376

1
2
3
4
5

1
2
3
4
5
1
2
3
4
5
1
2
3
4
5

Session 12C: Traffic Analysis and Side Channels

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

5.4.1 netML Challenge Examples. We study the breadth of nPrint
and nPrintML in context of the netML challenge, a public traffic
analysis challenge that consists of three flow-level traffic analysis
problems: malware detection for IoT devices, intrusion detection,and traffic identification[6]. Each task is further divided into easierand harder versions of the task by changing the granularity of
the labels, increasing or decreasing the number of classes to be
identified. The creators released a set of hand-engineered features
for each problem, equating to a semantic representation of eachflow. Participants are invited to train and tune any type of modelwith the goal of achieving the highest performance for each task.
We aim to see if nPrintML can outperform the hand-engineeredfeatures and hand-tuned models published on the netML leaderboard [37]. Unfortunately, the authors only released the features foreach traffic flow in the dataset, not the raw traffic data. As such, wefaithfully recreated the raw datasets with help from the original authors. In the vast majority of cases, we exactly match the number ofsamples for each class in each task. However, due to this recreation
effort we do not necessarily have the same training and testing split
as the original challenge dataset, and as such our results do notreflect a perfect comparison. Finally, we remove the IP addresses
from each packet in our sample when running nPrintML to matchthe original challenge.
Table 1 shows the results for each of the three netML challengecase studies. In every task but one nPrintML outperforms the highest performing hand-tuned model on the netML leaderboard [37]without manual feature engineering. Below we show the nPrintML
commands used to create the full traffic analysis pipeline for eachtask.

accuracy using the IPv4, TCP, and UDP headers of each packet. The
nPrintML command to create the traffic analysis pipeline is shown
below.
1
2
3
4$ nprintml -L labels . txt -a pcap \-- pcap - dir application_traces /\-4 -t -u -p 50 -c 25\-x " ipv4_src .*| ipv4_dst .*"5.4.3 Streaming Video Providers. Last, we demonstrate how nPrint
and nPrintML can enable rapid hypothesis testing for traffic analysis. Bronzino et al. examined the feasibility of training a model toinfer the streaming video quality of different video services (Netflix,
YouTube, Amazon, and Twitch) [10]. Rather than video quality, wewish to train a model that can differentiate the video services. Our
hypothesis is that each streaming video service player may exhibit
individualistic flow behavior to deliver video traffic, which may
then enable the identification of each service. We filter each individual video traffic trace for only SYN packets (to capture flow creation)and create a full traffic analysis pipeline using the command below. As shown in Table 1, the highest performing model trained on
nPrint can identify the streaming service with 98% accuracy.
12
3
4$ nprintml -L labels . txt -a pcap \-- pcap - dir video_traces\-4 -t -u -R -c 25\-f " tcp [13] == 2"

6

RELATED WORKThis section explores past work on manual and automated fingerprinting techniques and how they relate to nPrint.
Machine learning-based traffic analysis. Machine learning techniques have been applied to network traffic classification and fingerprinting [3, 5, 57, 63]. Wang et al. developed an ML-based classification model to detect obfuscated traffic [58]. Sommer and Paxson
demonstrated that using machine learning to detect anomalies canhave significant drawbacks, as network anomalies can exhibit different behavior than other problems solved by ML [53]. Trimananda etal. used DBSCAN to identify smart home device actions in network
traffic [55].
Other work has used machine learning to identify websites visited through the Tor network [20, 42, 59, 60]. Deep learning techniques have recently garnered attention as they have proven to
be applicable to the task for inferring information from encrypted
network traffic. Various work has used machine learning models tofingerprint websites visited through the Tor network [35, 39, 45, 51].
These works differ from this work, due to their focus on the Tor
setting. In Tor, all packets are the same size and encrypted, meaning network traffic in Tor can be represented by a series of -1s
and 1s that represent the direction of the traffic. This work in contrast considers traffic over any network that can vary in size andprotocol.
Deep learning techniques have become popular for network traffic classification problems [1, 23, 34, 62, 64]. Yu et al. used convolutional autoencoders for network intrusion detection [64]. Mirksy
et al. use an ensemble of autoencoders and human-engineeredfeatures on the intrusion detection problem in an unsupervised setting [34]. Our work differs as we aim to explore whether a variety
of machine learning models can automatically extract important# Intrusion Detection$ nprintml -L labels . txt -a pcap \-- pcap - dir traffic_flows /\-4 -t -u -c 5\
-x " ipv4_src .*| ipv4_dst .*"# Malware Detection$ nprintml -L labels . txt -a pcap \-- pcap - dir traffic_flows /\-4 -t -u -c 10\-x " ipv4_src .*| ipv4_dst .*"# Traffic Identification ( No Payload bytes )$ nprintml -L labels . txt -a pcap \-- pcap - dir traffic_flows /\-4 -t -u -c 5\
-x " ipv4_src .*| ipv4_dst .*"# Traffic Identification (10 Payload bytes )$ nprintml -L labels . txt -a pcap \-- pcap - dir traffic_flows /\-4 -t -u -p 10 -c 5\
-x " ipv4_src .*| ipv4_dst .*"5.4.2 Mobile Country of Origin. We examine the ability ofnPrintML to determine the country of origin of mobile application traces using the Cross Platform dataset [44]. Specifically, we
use nPrintML to train a model that can determine if the mobile
application trace originated in China, the United States, or India.
Table 1 shows the performance of the best model trained on thetask. Ultimately, nPrintML can perform this task with over 96%

3377

Session 12C: Traffic Analysis and Side Channels

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

features from network traffic in a supervised setting. Wang et al.
applied off-the-shelf deep learning techniques from image recognition and text analysis to intrusion detection; in contrast, we focus
specifically on creating a general representation for network trafficthat can be used in a variety of different models, across a broadclass of problems[61]. Our results also suggest that Wang et al.’smodel could be more complex than necessary, and that better input
representations such as nPrint may result in simpler models.

knowledge, we are the first to explore the combination of AutoML
and network traffic classification.
We specifically use AutoGluon-Tabular, which performs feature
selection, model selection, and hyperparameter optimization by
searching through a set of base models [14]. These models include
deep neural networks, tree based methods such as random forests,
non-parametric methods such as K-nearest neighbors, and gradient boosted tree methods. Beyond searching the singular models,AutoGluon-Tabular creates weighted ensemble models out of thebase models to achieve higher performance than other AutoML
tools in less overall training time [14].

TCP-based host fingerprinting. Idiosyncrasies between TCP/IP
stack implementations have often been the basis of networked
host fingerprinting techniques. Actively probing to differentiate between TCP implementations was introduced by Comer and Lin [11].
Padhye and Floyd identified differences between TCP implementations to detect bugs in public web servers [41]. Paxson passively
identified TCP implementations using traffic traces [43].
Past work has developed techniques to fingerprint host operating systems and devices. There are multiple tools and methods forhost OS fingerprinting, using both active and passive techniques.
Passive OS identification aims to identify operating systems frompassively captured network traffic [9, 30]. P0f passively observes
traffic and determines the operating system largely based on TCP
behavior [40]. Another common tool is Nmap [31], which performs
active fingerprinting. Nmap sends probes and examines the responses received from a target host, focusing on TCP/IP settingsand Internet Control Messaging Protocol (ICMP) implementationdifferences between different operating systems and devices. Nmap
is widely considered the “gold standard” of active probing tools. In
contrast, nPrint does not focus on heuristics and a priori knowledgeof implementation differences between host networking stacks. Instead, nPrint relies on the model to learn these differences during
training.
Remote fingerprinting can be used to characterize aspects of
the remote system other than its operating system or networking
stack. Clock skew information determined from the TCP timestamp
option was used to identify individual physical devices by Kohno
et al. [26]. Formby et al. passively fingerprint industrial control
system devices [18].

7

CONCLUSION

The effectiveness of applying machine learning to traffic analysis
tasks in network security depends on the selection and appropriate
representation of features as much as, if not more than, the model
itself. This paper creates a new direction for automated traffic analysis, presenting nPrint, a unified packet representation that takes as
input raw network packets and transforms them into a format thatis amenable to representation learning and model training, thereby
automating a part of the machine learning process which until now
has been largely painstaking and manual.
This standard format makes it easy to integrate network traffic
analysis with state-of-the-art automated machine learning (AutoML) pipelines. nPrintML, the integration of nPrint with AutoML,
automatically learns the best model, parameter settings, and feature
representation for the corresponding task. We applied nPrintML
to eight common network traffic analysis tasks, improving on thestate of the art in many cases. nPrint has demonstrated that many
network traffic classification tasks are amenable to automation,
though many open problems exist such as automated timeseries
analysis and classification involving multiple flows. nPrint should
ultimately be applied to a larger set of classification problems. To
this end, we have released nPrint, nPrintML, and all datasets as abenchmark for the research community, to make it easy to both
replicate and extend the results from this work.

8

Automated machine learning. While the use of deep learningtechniques can help automate feature engineering, a separate line ofresearch has examined how to perform automated machine learning(AutoML). The work examines the use of optimization techniques
to automate not only feature engineering and selection, but modelarchietecture and hyperparameter optimization as well [15, 16, 24,
27, 29]. These tools have recently been used for model compression,
image classification, and even bank failure prediction [2, 21]. To our

ACKNOWLEDGEMENTS

We thank our shepherd Katharina Kohls and the anonymous reviewers for their helpful comments. We also thank Vitaly Shmatikov
for guidance and feedback on early versions of this work, and Jesse
London for collaborating on the development of nPrintML. This research was supported in part by the Center for Information Technology Policy at Princeton University and by NSF Award CPS-1739809
under a cooperative agreement with the Department of Homeland
Security, DARPA and AFRL under Contract FA8750-19-C-0079, andNSF Awards CNS-1553437 and CNS-1704105.

3378

Session 12C: Traffic Analysis and Side Channels

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

REFERENCES[23] Ren-Hung Hwang, Min-Chun Peng, Van-Linh Nguyen, and Yu-Lun Chang. 2019.
An LSTM-Based Deep Learning Approach for Classifying Malicious Traffic at
the Packet Level. Applied Sciences 9, 16 (2019), 3414.
[24] Haifeng Jin, Qingquan Song, and Xia Hu. 2019. Auto-keras: An efficient neuralarchitecture search system. In Proceedings of the 25th ACM SIGKDD International
Conference on Knowledge Discovery & Data Mining. 1946–1956.
[25] Thomas Karagiannis, Konstantina Papagiannaki, and Michalis Faloutsos. 2005.
BLINC: Multilevel Traffic Classification in the Dark. In ACM SIGCOMM. 229–240.
[26] Tadayoshi Kohno, Andre Broido, and Kimberly C Claffy. 2005. Remote physicaldevice fingerprinting. IEEE Transactions on Dependable and Secure Computing 2,
2 (2005), 93–108.
[27] Lars Kotthoff, Chris Thornton, Holger H Hoos, Frank Hutter, and Kevin LeytonBrown. 2017. Auto-WEKA 2.0: Automatic model selection and hyperparameter
optimization in WEKA. The Journal of Machine Learning Research 18, 1 (2017),
826–830.
[28] Stratosphere Lab. [n. d.]. Stratosphere Lab Malware Capture Facility Project. https://www.stratosphereips.org/datasets-malware iot malware.
[29] Erin LeDell and S Poirier. 2020. H2o automl: Scalable automatic machine learning.
In Proceedings of the AutoML Workshop at ICML, Vol. 2020.
[30] Richard Lippmann, David Fried, Keith Piwowarski, and William Streilein. 2003.
Passive operating system identification from TCP/IP packet headers. In Workshop
on Data Mining for Computer Security, Vol. 40.
[31] Gordon Fyodor Lyon. 2009. Nmap Network Scanning: The Official Nmap Project
Guide to Network Discovery and Security Scanning. Insecure, USA.
[32] Kyle MacMillan, Jordan Holland, and Prateek Mittal. 2020. Evaluating Snowflakeas an Indistinguishable Censorship Circumvention Tool. arXiv preprint
arXiv:2008.03254 (2020).
[33] Tony Miller. 2020. Passive OS Fingerprinting: Details and Techniques. http://www.ouah.org/incosfingerp.htm.
[34] Yisroel Mirsky, Tomer Doitshman, Yuval Elovici, and Asaf Shabtai. 2018. Kitsune:an ensemble of autoencoders for online network intrusion detection. In Network
and Distributed System Security Symposium, NDSS. San Diego, CA, USA.
[35] Milad Nasr, Alireza Bahramali, and Amir Houmansadr. 2018. DeepCorr: Strong
Flow Correlation Attacks on Tor Using Deep Learning (CCS ’18). Toronto, Canada.
[36] SC Nayak, Bijan B Misra, and Himansu Sekhar Behera. 2014. Impact of datanormalization on stock index forecasting. International Journal of Computer
Information Systems and Industrial Management Applications 6, 2014 (2014), 257–
269.
[37] NetML. [n. d.]. NetML Network Traffic Analytics Challenge 2020. https://eval.ai/
web/challenges/challenge-page/526/leaderboard/1473 netML.
[38] ntop. [n. d.]. PF_RING, High-speed packet capture, filtering and analysis. https://www.ntop.org/products/packet-capture/pf_ring/ pfring.
[39] Se Eun Oh, Saikrishna Sunkam, and Nicholas Hopper. 2019. p1-FP: Extraction,
Classification, and Prediction of Website Fingerprints with Deep Learning. Proceedings on Privacy Enhancing Technologies 2019, 3 (2019), 191–209.
[40] p0f 2016. p0f v3 (version 3.09b). http://lcamtuf.coredump.cx/p0f3.
[41] Jitendra Padhye and Sally Floyd. 2001. On Inferring TCP Behavior. SIGCOMM
Comput. Commun. Rev. 31, 4 (Aug. 2001), 287–298. https://doi.org/10.1145/964723.
383083[42] Andriy Panchenko, Fabian Lanze, Jan Pennekamp, Thomas Engel, Andreas Zinnen, Martin Henze, and Klaus Wehrle. 2016. Website Fingerprinting at Internet
Scale.. In NDSS.
[43] Vern Paxson. 1997. Automated Packet Trace Analysis of TCP Implementations.
SIGCOMM Comput. Commun. Rev. 27, 4 (Oct. 1997), 167–179. https://doi.org/10.
1145/263109.263160[44] Jingjing Ren, Daniel J. Dubois, and David Choffnes. 2019. An International View ofPrivacy Risks for Mobile Apps. (2019). https://recon.meddle.mobi/papers/crossmarket.pdf[45] Vera Rimmer, Davy Preuveneers, Marc Juárez, Tom van Goethem, and Wouter
Joosen. 2018. Automated Website Fingerprinting through Deep Learning. In
Network and Distributed System Security Symposium, NDSS. San Diego, CA, USA.
[46] scikit learn. [n. d.]. Computing the Average Precision Score. https://scikit-learn.org/
stable/modules/generated/sklearn.metrics.average_precision_score.html average
precision score.
[47] scikit learn. [n. d.]. Tuning the hyper-parameters of an estimator. https://scikitlearn.org/stable/modules/grid_search.html grid search.
[48] Iman Sharafaldin, Arash Habibi Lashkari, and Ali A Ghorbani. 2018. Toward generating a new intrusion detection dataset and intrusion traffic characterization..
In ICISSP.
[49] Shodan. 2020. Shodan. https://www.shodan.io/.
[50] Dalwinder Singh and Birmohan Singh. 2019. Investigating the impact of datanormalization on classification performance. Applied Soft Computing (2019),
105524.
[51] Payap Sirinam, Mohsen Imani, Marc Juarez, and Matthew Wright. 2018. Deep Fingerprinting: Undermining Website Fingerprinting Defenses with Deep Learning.
arXiv preprint arXiv:1801.02265 (2018). arXiv:1801.02265 [cs.CR][52] Matthew Smart, G Robert Malan, and Farnam Jahanian. 2000. Defeating TCP/IP
Stack Fingerprinting.. In Usenix Security Symposium.

[1] G. Aceto, D. Ciuonzo, A. Montieri, and A. Pescapé. 2019. Mobile Encrypted Traffic
Classification Using Deep Learning: Experimental Evaluation, Lessons Learned,
and Challenges. IEEE Transactions on Network and Service Management 16, 2(2019), 445–458.
[2] Anna Agrapetidou, Paulos Charonyktakis, Periklis Gogas, Theophilos Papadimitriou, and Ioannis Tsamardinos. 2020. An AutoML application to forecastingbank failures. Applied Economics Letters (2020), 1–5.
[3] Mashael AlSabah, Kevin Bauer, and Ian Goldberg. 2012. Enhancing Tor’s Performance Using Real-Time Traffic Classification. In Proceedings of the 2012 ACM
Conference on Computer and Communications Security (Raleigh, North Carolina,
USA) (CCS ’12). Association for Computing Machinery, New York, NY, USA,73–84. https://doi.org/10.1145/2382196.2382208[4] Blake Anderson and David McGrew. 2017. Machine Learning for Encrypted
Malware Traffic Classification: Accounting for Noisy Labels and Non-stationarity.
In Proceedings of the 23rd ACM SIGKDD International Conference on knowledge
discovery and data mining. 1723–1732.
[5] J. Barker, P. Hannay, and P. Szewczyk. 2011. Using Traffic Analysis to Identify
the Second Generation Onion Router. In 2011 IFIP 9th International Conference on
Embedded and Ubiquitous Computing. 72–78. https://doi.org/10.1109/EUC.2011.
76[6] Onur Barut, Yan Luo, Tong Zhang, Weigang Li, and Peilong Li. 2020. NetML: A
Challenge for Network Traffic Analytics. arXiv preprint arXiv:2004.13006 (2020).
[7] Steven M. Bellovin. 2002. A Technique for Counting Natted Hosts. In Proceedingsof the 2nd ACM SIGCOMM Workshop on Internet Measurment (Marseille, France)(IMW ’02). Association for Computing Machinery, New York, NY, USA, 267–272.
https://doi.org/10.1145/637201.637243[8] Laurent Bernaille, Renata Teixeira, and Kave Salamatian. 2006. Early Application
Identification. In Proceedings of the 2006 ACM CoNEXT Conference. 1–12.
[9] Robert Beverly. 2004. A Robust Classifier for Passive TCP/IP Fingerprinting. In
Proceedings of the 5th Passive and Active Measurement (PAM) Workshop.
[10] Francesco Bronzino, Paul Schmitt, Sara Ayoubi, Guilherme Martins, Renata Teixeira, and Nick Feamster. 2019. Inferring streaming video quality from encrypted
traffic: Practical models and deployment experience. Proceedings of the ACM on
Measurement and Analysis of Computing Systems 3, 3 (2019), 1–25.
[11] Douglas E Comer and John C Lin. 1994. Probing TCP implementations. In Usenix
Summer. 245–255.
[12] Gerard Draper-Gil, Arash Habibi Lashkari, Mohammad Saiful Islam Mamun,and Ali A Ghorbani. 2016. Characterization of encrypted and vpn traffic using
time-related. In Proceedings of the 2nd international conference on information
systems security and privacy (ICISSP). 407–414.
[13] Zakir Durumeric, Eric Wustrow, and J Alex Halderman. 2013. ZMap: Fast Internetwide scanning and its security applications. In Proceeedings of the 22nd USENIX
Security Symposium. 605–620.
[14] Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy,
Mu Li, and Alexander Smola. 2020. AutoGluon-Tabular: Robust and Accurate
AutoML for Structured Data. arXiv preprint arXiv:2003.06505 (2020).
[15] Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, ManuelBlum, and Frank Hutter. 2015. Efficient and robust automated machine learning.
In Advances in neural information processing systems. 2962–2970.
[16] Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Tobias Springenberg,
Manuel Blum, and Frank Hutter. 2019. Auto-sklearn: efficient and robust automated machine learning. In Automated Machine Learning. Springer, Cham,
113–134.
[17] David Formby, Preethi Srinivasan, Andrew M Leonard, Jonathan D Rogers, andRaheem A Beyah. 2016. Who’s in Control of Your Control System? Device
Fingerprinting for Cyber-Physical Systems. In Network and Distributed Systems
Security Symposium.
[18] David Formby, Preethi Srinivasan, Andrew M. Leonard, Jonathan D. Rogers,
and Raheem A. Beyah. 2016. Who’s in Control of Your Control System?
Device Fingerprinting for Cyber-Physical Systems. In 23rd Annual Network
and Distributed System Security Symposium, NDSS, 2016, San Diego, California, USA, February 21-24, 2016.
http://wp.internetsociety.org/ndss/wpcontent/uploads/sites/25/2017/09/who-control-your-control-system-devicefingerprinting-cyber-physical-systems.pdf[19] João Gama, Indrė Žliobaitė, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid
Bouchachia. 2014. A Survey on Concept Drift Adaptation. ACM Computing
Surveys (CSUR) 46, 4 (2014), 1–37.
[20] Jamie Hayes and George Danezis. 2016. k-fingerprinting: A Robust Scalable
Website Fingerprinting Technique. In 25th USENIX Security Symposium (USENIX
Security 16). USENIX Association, Austin, TX, 1187–1203. https://www.usenix.
org/conference/usenixsecurity16/technical-sessions/presentation/hayes[21] Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. 2018. Amc:
Automl for model compression and acceleration on mobile devices. In Proceedingsof the European Conference on Computer Vision (ECCV). 784–800.
[22] Jordan Holland, Ross Teixeria, Paul Schmitt, Kevin Borgolte, Jennifer Rexford,
Nick Feamster, and Jonathan Mayer. 2020. Classifying Network Vendors at
Internet scale. arXiv preprint arXiv:2006.13086 (2020).

3379

Session 12C: Traffic Analysis and Side Channels

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea[53] Robin Sommer and Vern Paxson. 2010. Outside the closed world: On using
machine learning for network intrusion detection. In 2010 IEEE symposium on
security and privacy. IEEE, 305–316.
[54] tcpdump. [n. d.]. Man page of PCAP. https://www.tcpdump.org/manpages/pcap.
3pcap.html libpcap.
[55] Rahmadi Trimananda, Janus Varmarken, Athina Markopoulou, and Brian Demsky. 2020. Packet-Level Signatures for Smart Home Devices. In Network andDistributed System Security Symposium, NDSS. San Diego, CA, USA.
[56] Thijs van Ede, Riccardo Bortolameotti, Andrea Continella, Jingjing Ren, Daniel J
Dubois, Martina Lindorfer, David Choffnes, Maarten van Steen, and AndreasPeter. 2020. Flowprint: Semi-supervised Mobile-app Fingerprinting on Encrypted
Network Traffic. In Network and Distributed System Security Symposium. InternetSociety.
[57] Shobha Venkataraman, Juan Caballero, Pongsin Poosankam, Min Kang, and Dawn
Song. 2007. Fig: Automatic Fingerprint Generation.. In Network and Distributed
System Security Symposium, NDSS.
[58] Liang Wang, Kevin P. Dyer, Aditya Akella, Thomas Ristenpart, and Thomas
Shrimpton. 2015. Seeing through Network-Protocol Obfuscation. In Proceedingsof the 22nd ACM SIGSAC Conference on Computer and Communications Security(Denver, Colorado, USA) (CCS ’15). Association for Computing Machinery, New
York, NY, USA, 57–69. https://doi.org/10.1145/2810103.2813715[59] Tao Wang, Xiang Cai, Rishab Nithyanand, Rob Johnson, and Ian Goldberg. 2014.
Effective Attacks and Provable Defenses for Website Fingerprinting. In 23rd

USENIX Security Symposium (USENIX Security 14). USENIX Association, San
Diego, CA, 143–157. https://www.usenix.org/conference/usenixsecurity14/
technical-sessions/presentation/wang_tao[60] Tao Wang and Ian Goldberg. 2017. Walkie-Talkie: An Efficient Defense against
Passive Website Fingerprinting Attacks. In Proceedings of the 26th USENIX Conference on Security Symposium (Vancouver, BC, Canada) (SEC’17). USENIX Association, USA, 1375–1390.
[61] Wei Wang, Yiqiang Sheng, Jinlin Wang, Xuewen Zeng, Xiaozhou Ye, Yongzhong
Huang, and Ming Zhu. 2017. HAST-IDS: Learning hierarchical spatial-temporalfeatures using deep neural networks to improve intrusion detection. IEEE Access
6 (2017), 1792–1806.
[62] W. Wang, M. Zhu, J. Wang, X. Zeng, and Z. Yang. 2017. End-to-end encrypted
traffic classification with one-dimensional convolution neural networks. In 2017
IEEE International Conference on Intelligence and Security Informatics (ISI). Beijing,
China.
[63] Nigel Williams, Sebastian Zander, and Grenville Armitage. 2006. A Preliminary
Performance Comparison of Five Machine Learning Algorithms for Practical IP
Traffic Flow Classification. SIGCOMM Comput. Commun. Rev. 36, 5 (Oct. 2006),
5–16. https://doi.org/10.1145/1163593.1163596[64] Yang Yu, Jun Long, and Zhiping Cai. 2017. Network intrusion detection throughstacking dilated convolutional autoencoders. Security and Communication Networks 2017 (2017).

3380

Session 12C: Traffic Analysis and Side Channels

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

A APPENDIX
A.1 Option Representation Evaluation
Section 2 outlines a set of representation requirements and pitfallsof strawman representations. One issue with a semantic representation that renders it unusable as a standard representation is that
option ordering is not preserved. We now aim to exhibit the performance degradation that can occur when option ordering is notpreserved.
We set up a classification problem using the dataset fully explained in Section 5.1. At a high level, the dataset consists of fingerprints for 15 classes of devices probed with Nmap. We take the TCP
responses from each fingerprint and generate two representationsof the TCP options in each packet, one using nPrint, and one using asemantic representation. For the semantic representation we parse
all of the options and consider each TCP option as a continuous
valued feature, with the name of the feature being the TCP option
and the value being its corresponding value in the packet. nPrint
fingerprints are the bitmap representation of the options, which
preserves ordering.

A.2

nPrintML Example

Section 3 introduces nPrintML, our pipeline to connect nPrint andautoML. nPrintML enables researchers with network traffic andcode to perform state-of-the-art traffic analysis without writingcode in minutes. Here, we present an example of reproducing our
results in Section 5.3 from scratch.
(1) Clone the repository to get the data.
(2) Uncompress the traffic traces.
(3) Write a small script to generate labels. In cases where wehave labels apirori, no code needs to be written.
(4) Run the label generation script(5) run nPrintML to perform traffic analysis.
These steps, instantiated.
1
2
3
4
1
2
3

#
$
#
$

1. clone
git clone repo_name2. uncompress
tar - xvf dataset . tar . gz

# 3. Generate Labels , not necessary if labels exist
import sys
import pathlib

4

Model
Catboost
ExtraTrees
LightGBM
Neural Network
Random Forest
Weighted Ensemble
Catboost
Extra Trees
LightGBM
Neural Network
Random Forest
Weighted Ensemble

Representation

F1

Semantic

75.1
72.9
74.4
68.2
73.6
75.6

nPrint

85.2
83.3
83.1
82.3
83.3
85.9

Table 9: Option ordering increases performance across all models.

5
6
7
8
9
10
11
12

# Example file name : dataset / facebook /windows_chrome_facebook_1383 . pcap
# Get file paths
paths = list ( pathlib . Path ( sys . argv [1]) . rglob ( '*. pcap ') )for path in paths :# Build label
tokens = str ( path . stem ) . split ( '_ ')label = ' {0} _ {1} '. format ( tokens [0] , tokens [1])print ( ' {0} ,{1} '. format ( path , label ) )2# 4. Generate Labels$ python gen_labels . py dataset / > labels . txt

1# 5. run nPrintML with different configurations

1

2
3
4
5

# IPv4 , UDP , first 10 payload bytes of each packet :$ nprintML -L labels . txt -a pcap \-- pcap_dir dataset / -4 -u -p 10

67

Table 9 shows the performance degradation that occurs whenordering is lost across a wide array of models. In general, we see
over 10% increase in F1 scores for nPrint over the semantic representation.

89# First 100 payload bytes of each packet :$ nprintML -L labels . txt -a pcap \-- pcap_dir dataset / -p 100

10
11
12
13

3381

# UDP headers only :$ nprintML -L labels . txt -a pcap-- pcap_dir dataset / -u\

Session 12C: Traffic Analysis and Side Channels

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

Test Name

Summary

Nmap Weight

Explicit Congestion Notification
ICMP Response CodeIntegrity of returned probe IP Checksum
Integrity of returned probe UDP Checksum
IP ID Sequence Generation Algorithm
IP Total Length
Responsiveness
Returned probe IP ID value
Returned Probe IP Total Length
TCP Timestamp Option Algorithm
Unused Port unreachable Field Nonzero
Shared IP ID Sequence Boolean
TCP ISN Greatest Common Divisor
Don’t Fragment ICMPTCP Flags
TCP ISN Counter RateTCP ISN Sequence Predictability Index
IP Don’t Fragment Bit
TCP Acknowledgment Number
TCP Miscellaneous Quirks
TCP Options Test
TCP Reset Data Checksum
TCP Sequence Number
IP Initial Time-To-Live
TCP Initial Window Size

TCP Explicit Congestion control flag.
ICMP Response Code.
Valid checksum in an ICMP port unreachable.
UDP header checksum received match.
Algorithm for IP ID.
Total length of packet.
Target responded to a given probe.
IP ID value.
IP Length of an ICMP port unreachable.
TCP timestamp option algorithm.
Last 4 bytes of ICMP port unreachable message not zero.
Shared IP ID Sequence between TCP and ICMP.
Smallest TCP ISN increment.
IP Don’t Fragment bit for ICMP probes.
TCP flags.
Average rate of increase for the TCP ISN.
Variability in the TCP ISN.
IP Don’t Fragment bit.
TCP acknowledgment number.
TCP implementations, e.g, reserved field in TCP header.
TCP header options, preserving order.
Checksum of data in TCP reset packet.
TCP sequence number.
IP initial time-to-live.
TCP window size.

Table 10: Nmap’s highly complex device detection tests, which are used to generate a fingerprint for each device.

A.3

Nmap Tests

3382

100
100
100
100
100
100
100
100
100
100
100
80
75
40
30
25
25
20
20
20
20
20
20
15
15

Session 12C: Traffic Analysis and Side Channels

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

-4 ,
-6 ,
-A ,
-c ,
-C ,
-d ,
-e ,
-f ,
-F ,
-h ,
-i ,
-N ,
-O ,

CCS ’21, November 15–19, 2021, Virtual Event, Republic of Korea

include ipv4 headers
include ipv6 headers
include absolute timestmap field
number of packets to parse ( if not all )csv ( hex packets ) infile
device to capture from if live capture
include eth headersfilter for libpcap
integer to fill missing bits withprint regex possibilities
include icmp headersnPrint infile
Output file Index ( first column ) Options :0: source IP ( default )1: destination IP
2: source port
3: destination port
4: flow (5 - tuple )-p , -- payload = PAYLOAD_SIZE include n bytes of payload-P , -- pcap_file = FILE
pcap infile-R , -- relative_timestamps include relative timestamp field-S , -- stats
print stats about packets processed when finished-t , -- tcp
include tcp headers-u , -- udp
include udp headers-V , -- verbose
print human readable packets with nPrints-W , -- write_file = FILE
file for output , else stdout-x , -- nprint_filter = STRING regex to filter bits out of nPrint . nprint -h fordetails-? , -- help
Give this help list-- usage
Give a short usage message-- version
Print program version

A.4-- ipv4-- ipv6-- absolute_timestamps-- count = INTEGER-- csv_file = FILE-- device = STRING-- eth-- filter = STRING-- fill_int = INT8_T-- nprint_filter_help-- icmp-- nPrint_file = FILE-- write_index = INTEGER

nPrint Configuration Options

3383