Approaching Anomaly Detection in Transactional Data | by Ivan Senilov | Towards Data Science

Usually, people mean financial transactions when they talk about transactional data. However, according to Wikipedia, “Transactional Data is data describing an event (the change as a result of a transaction) and is usually described with verbs. Transaction data always has a time dimension, a numerical value and refers to one or more objects”. In this article, we will use data on requests made to a server (internet traffic data) as an example, but the considered approaches can be applied to most of the datasets falling under the aforementioned definition of transactional data.
Anomaly Detection, in simple words, is finding data points that shouldn’t normally occur in a system that generated data. Anomaly detection in transactional data has many applications, here are a couple of examples:
Fraud detection in financial transactions.
Fault detection in manufacturing.
Attack or malfunction detection in a computer network (the case covered in this article).
Recommendation of predictive maintenance.
Health condition monitoring and alerting.
The data we will use as an example looks like this:
with columns described below:
timestamp: When the event occurred.
serverId: A unique identifier of a server being accessed.
ip: Either IPv4 address or IPv6 address of the client.
timeMs: Time in milliseconds that passes between moments of request and server’s response.
clientSSLProtocol: SSL Encryption protocol used for the request.
clientHTTPMethod: POST, GET, PUT, PATCH, and DELETE.
clientRequestHTTPProtocol: Version of HTTP Protocol, e.g. HTTP/1, HTTP/2.
clientRequestPath: The path requested, ex. /v3/article/.
userAgent: A characteristic string that lets servers and network peers identify the application, operating system, vendor, and/or version of the requesting user agent.
We are not performing classic EDA (Exploratory Data Analysis) here but rather try to understand how the data can be used from the several given examples.
The existing features (columns) fall into 3 categories complemented with ways to use them:
Real numbers (timeMs, ...). These can be used straight away (in the case of tree-based models) or normalized before feeding to a model.
Categorical variables with low cardinality (clientSSLProtocol, clientHTTPScheme, clientHTTPMethod, ...) Can be encoded (one hot, target, etc..) or embedded.
Categorical variables with high cardinality and textual data (clientRequestPath, userAgent, ...). These are trickier, one way is to parse the texts and try to infer categories that have lower cardinality or, for example, use text embeddings.
2.3.1 Time features
The timestamp can be encoded to features like:
Minute of hour
Hour of day
Day of week
etc...
To get rid of implicit ordering (i.e. distance between Monday and Sunday is the same as between other consecutive days but if they are encoded as 1 and 7 respectively this property is not true), often Sine and Cosine of normalized values are used.
2.3.2 Processing of IP addresses
IP addresses can be translated to a geolocation using dictionary lookup. This location can be then treated as a categorical variable.
Additionally, the IP addresses can be checked for being well-known proxy servers and being flagged as such (useful for applying one of the rules from 3.1.1)
2.3.3 userAgent flags
userAgent values can be parsed to extract information about the presence of particular substrings like “torrent”, “iphone”, “chrome”, etc... and then the corresponding flags “uaTorrent”, “uaIphone”, “uaChrome”, etc... are set to 1.
However, the parsing rules might need maintenance as the relevant keywords tend to change with time.
2.3.4 clientRequestPath flags
The same as userAgent, but searching for patterns like “login”, “admin”, etc...
The records in the dataset are not labeled in terms of “bad” and “good” actors but we assume the dataset can have those labels for supervised modeling approaches in the next section.
In this section, several approaches to modeling the anomalous traffic detector are reviewed, from simplest to more advanced considering their pros and cons.
3.1.1 Simple rules set
In this trivial detector, a set of rules is applied to each individual record, and alerting happens if, for example:
timeMs > X
uaTorrent == 1
And the rules’ logic combinations (AND, OR, ...)
Pros:
Simple to reason about
Easy to implement and test
Alerts are explainable
A reasonably complex ruleset is the fastest option
Cons:
Hard to maintain, especially when the ruleset becomes complex
Is not (quickly) adaptable to changing behavior
Might miss non-obvious cases hence potentially low recall
3.1.2 Supervised classification of records
This method assumes labels are available for the data and the problem can be framed as a binary classification. Almost all classification models can be applied here having the features (2.1) as an input vector. The models can be something simple like logistic regression as well as complex like neural networks outputting an “anomaly score” between 0 and 1. However, tree ensembles seem like an optimal choice here, combining natural explainability, good expected prediction performance, support of categorical variables without explicit encoding, and acceptable inference speed. The latter is very important if volumes of the data are big.
Pros:
Classification is a well-known ML problem with many examples of successful applications
(Potentially) high detection performance (see cons as well)
Most of the models are explainable
Cons:
Requires labels and the detection quality is only as good as labels’ quality
Labels are most probably imbalanced (much less anomalous traffic) which requires applying special measures like undersampling of the majority class, oversampling of the minority class, data augmentation, SMOTE, etc.
Requires infrastructure for model (re-)training, validating and storing
3.1.3 Unsupervised anomaly detection
High-quality labels are not always available so unsupervised methods might be handy. The anomaly detection methods/models usually learn the most popular patterns in data then predicting the “anomalousness” of new examples.
Pros:
No need for labels for model training
Cons:
Labels are still handy for model validation
Requires infrastructure for model (re-)training and storing
Using time would allow models to continuously adapt to the changing behavior of potential malicious actors but it would require more preprocessing of the data. This preprocessing would be aggregations over a rolling time window and could be done with something like this pseudo-Spark-code:
df ... .groupBy(window(col(“timestamp”), 1.minute), col(“ip”)) .agg(   count(col(“*”)) as ‘num_reqs,   avg(col(“timeMs”)) as ‘avgTimeMs,   skew(col(“timeMs”)) as ‘skewTimeMs,   count(when(col(“clientRequestPath”).contains(“login”)) as ‘num_logins), ...)
Note: the data is grouped by “ip” but it might be better to group by, for example, a combination of country and/or region (decoded from IP address), depending on the use case.
With this query, we can transform the data into a set of time series and apply the time series specific methods described below.
3.2.1 Comparing with historical statistics
To add adaptability to the rules “model”, we can use the temporal characteristics of the data with the simple rules set modifying the rules like this:
currentTimeMs > avgTimeMs + someConstant
where “avgTimeMs” is a continuously updated average value of the metric over the last N timestamps.
For categorical features, comparing distributions of categorical variables for the current moment of time and some aggregated distribution from the same last N timestamps can be used.
Pros:
Does not require model (re-)training but still provides adaptability
Cons:
Needs additional preprocessing step
Note: other pros/cons are the same as for simple rules set.
3.2.2 Forecasting time-series for the task of anomaly detection
This method would work as follows: based on a certain time window, the model predicts the next value(s) and compares the prediction with the real observation(s). Underlying forecasting methods may vary from something more traditional like SARIMA to deep learning. Probabilistic forecasting is preferred due to factors described in the cons list.
Pros:
Is a self-supervised method per se so does not requires labels
Can be explainable for a case of multivariate forecasting when metrics that fall out of normal values are returned as anomalous, optionally with distance from normality being anomalous score and data columns used for grouping by
In the case of probabilistic models like provided in gluon-ts for example, does not require setting thresholds, only percentiles, falling out of which the observation would be considered anomalous
Cons:
Requires infrastructure for model (re-)training, validating and storing
Inference time can be high, especially for probabilistic models
Explainable predictions are important for identifying root causes of the detections which might be very useful for debugging and investigating attacks. Therefore, models that are not easily explainable like neural networks are not recommended.
In terms of tooling, there is for example SHAP which is one of the best packages for inferring feature importances even from the black-box models.
These two repositories are great sources of information on research and tooling for anomaly detection and time-series forecasting:
github.com
github.com
In this document, we reviewed and summarised several options for data preprocessing, feature engineering, and modeling for the task of anomaly detection applied to transactional data. Depending on system requirements, the number of options can be narrowed down: for example if inference time should be microseconds — the deep learning models are most probably out of the game. The feasible options should be then tested offline as well as online and the optimal one identified based on success metric(s) selected at the beginning of the project. The metric can be precision if false alarms are the concern or recall if missed attacks are costly or even some integral metric that can be weighted combination of those two with various metrics like inference time, training time, compute load, cost of error, etc...