## 1 Introduction
### Network Packet and Flow Datasets  
Machine learning is a powerful tool in cybersecurity, particularly in its ability to detect anomalies, and cybersecurity researchers in network threat identification and intrusion detection are particularly interested in the analysis of large network packet and flow datasets. A survey of the composition of publicly available intrusion datasets is provided in [PERSON] (1948). Figure 1: Simplified illustration of information entropy for a fixed set of eight symbols. Lowest entropy is achieved with a monotonic set of repeating symbols (each with probability 1 of being predicted). Highest entropy is achieved when the full symbol set is used, with each symbol appearing randomly with equal probability. Packet datasets are typically large, high-dimensional, time-series data, often containing tens of millions of discrete packet events. Due to memory constraints and temporal complexity, it is common to abstract packet data into lower-dimensional containers called flows. Flows capture the essential details of packet streams in a compact, extensible format, without the inherent complexity of raw packets [5]. Packet flows offer a convenient lower-dimension sample set with which to do cyber research. A flow can be created using a simple tuple, to create a unique fingerprint with which to aggregate associated packets over time, based on the following attributes:  
* Source and Destination IP Address;
* Source and Destination Port identifier;
* Protocol ID. Flows can be stateful (for example, TCP flows have a definite lifecycle, controlled by state flags), with additional logic and timeouts required to capture the full lifecycle of a flow. Flows may also be directional (i.e., unidirectional, or bidirectional), and flows may be unicast, multicast, or broadcast (one-to-one, one-to-group, one-to-all). While flows are essentially unique at any instant, they may not be unique across time--based solely on the tuple--since some attributes are likely to be reused in the distant future. (For example, port numbers will eventually 'wrap around' once they reach a maximum bound--so large packet trace could contain two identical flows, but these are likely to be separated by a substantial time interval, unless there is a bug in the port allocation procedure). Modern public datasets used in network threat research often include high level flow summaries and metadata, but rarely include payload content in these summaries (notably, the UNB 2012 intrusion dataset did include some payload information encoded in Base 64; however, subsequent updates did not, due to the size implications). Since packet payload typically represents the largest contributor to packet size (typically, an order of magnitude larger than the protocol headers), it tends to be removed during the creation of flow datasets. Payload also adds complexity in that it requires reassembly and in-memory state handling across the lifecycle of a flow. Payload is often encrypted (see Figure 2), which means that many potentially useful features are not accessible. There are also potential privacy and legal concerns, given that payload may contain confidential, sensitive, or personal information [5]. For these reasons, we rarely see much information on packet payload within flow datasets and metadata summaries other than simple volumetric metrics. As such, we have very few insights of what the actual content of the data being transferred looks like, at any point in time, and this lack of visibility can impair the detection of anomalous and suspicious activity that might exploit this feature. Specifically, the omission of such metrics in flow and packet data may inhibit the detection of certain types of attacks, and as discussed in Section 2. Figure 2: Common well-known TCP and UDP,’ well-known’ ports for plaintext and cryptographic services. Here, y = yes, n = no, and p = partial. Client applications that wish to use encrypted services typically start by exchanging cryptographic keys so that the rest of the conversation is secure.