## 2 Related Work  
Existing research demonstrates that information gain metrics, using techniques such as entropy, can be useful in detecting anomalous activity. Encrypted traffic tends to exhibit a very different entropy profile to unencrypted traffic (unencrypted but compressed data may also show high entropy, depending on the compression algorithm and underlying data); specifically, it tends to have much higher entropy values due to the induced unpredictability (randomness) of the data. Entropy has been widely used as a method to detect anomalous activity, and so intrusion detection, DDoS detection, and data exfiltration are of interest in research. A major challenge in detecting anomalies at the service layer (i.e., packet payload) is that there are no published 'ground truth' metrics on data composition with which to compare live results--unless the researcher is prepared to perform their own 'ground truth' analysis. Furthermore, publicly labelled intrusion datasets, frequently used in anomaly detection research, do not provide any useful metadata on payload composition other than size and flow rate metrics [5]. This effectively means that information on higher level service anomalies is largely hidden. One of the key contributions of this work is the publication of baseline payload entropy values across common network services, achieved by sampling normal (i.e., benign) network activity from multiple large packet flow datasets. This metadata can be used to compare deviations in live traffic characteristics. We would also suggest that payload entropy should be added to public intrusion dataset flow metadata, so that it can be employed to assist in anomaly detection and machine learning research. Early work by [6] characterises the entropy of several common network activities, as shown in Figure 3. As discussed in [6], with standard cryptographic protocols (such as SSH, SSL, and HTTPS), it is feasible to characterise which parts of traffic should have high entropy after key exchange has taken place. Therefore, significant changes in entropy during a session may indicate malicious activity. During an OpenSSL or OpenSSH attack, entropy within an encrypted channel is likely to drop below expected levels as the session is perturbed; the authors of [6] suggest entropy scores would dip to approximately six bits per byte during such a compromise (i.e., entropy values of around six, where we would normally expect them to be closer to eight). In [7], the authors use static analysis across large sample collections to detect compressed and encrypted malware, using entropy analysis to determine statistical variations in malware executables. In [8], the authors use methods that exploit structural file entropy variations to classify malware content. In [9], the authors use visual entropy graphs to identify distinct malware types. In [10], the authors propose a classifier to differentiate traffic content types (including text, image, audio, video, compressed, encrypted, and Base64-encoded content) using Support Vector Machine (SVM) on byte sequence entropy values. Analysis of the DARPA2000 dataset in [11] lists the top five most important features as TCP SYN flag, destination port entropy, entropy of source port, UDP protocol occurrence, and packet volume. In [12], the authors describe how peer-to-peer Voice over IP (VOIP) sessions can be identified using entropy and speech codec properties with packet flows, based on payload lengths. In [13], the authors use graphical methods for detecting anomalous traffic, based on entropy estimators. Figure 3: Early analysis of entropy values from several content types, derived from [6]. \(H_{N}^{MLE}\) is the sample entropy of a word of length N, MLE stands for Maximum Likelihood Estimator, and \(H_{N}\) is the sample entropy. As a point of reference, Figures 4 and 5 provide a more recent analysis of similar content types, where for example, email has an average entropy ranging between 5.40 (POP3) and 5.92 (SMTP).