# Traceable Transformer-Based Anomaly Detection for a Water Treatment System

Shenzhi Qin, Yubo Lang, Kam-Pui Chow

<figure><img src="image_1.png" /></figure>

## To cite this version:

Shenzhi Qin, Yubo Lang, Kam-Pui Chow. Traceable Transformer-Based Anomaly Detection for a Water Treatment System. 19th IFIP International Conference on Digital Forensics (DigitalForensics), Jan 2023, Arlington, VA, United States. pp.219-234, 10.1007/978-3-031-42991-0_12. hal-05517478

HAL Id: hal-05517478

https://inria.hal.science/hal-05517478v1

Submitted on 18 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
<figure><img src="image_3.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Traceable Transformer-Based Anomaly Detection for a Water Treatment System

Shenzhi Qin¹, Yubo Lang², and Kam-Pui Chow¹

¹ University of Hong Kong, Hong Kong, China
chow@cs.hku.hk

² Criminal Investigation Police University of China, Shenyang, China

**Abstract.** As industrial control system malfunctions caused by attacks become more complex and frequent, anomaly detection and subsequent forensic analyses are more important than ever. When an anomaly is detected, security professionals need to accurately identify the components that are under attack. However, traditional methods do not provide enough traces, which makes it difficult to identify the targeted components.

This chapter describes a traceable anomaly detection method that leverages unsupervised learning using industrial control system component time series data. The method generates customized transformer-encoder classifiers for industrial control system components. The final detection result is ensembled from all the classifier outputs. Experiments with water treatment testbed data indicate that the method achieves good performance with low false positive rates and delays, and strong traceability.

**Keywords:** Industrial Control Systems · Time Series Data · Anomaly Detection · Transformer-Encoder Classifier

## 1 Introduction

The umbrella term “industrial control systems” covers various control systems and their instrumentation, including systems, devices, networks and controls that collectively operate industrial processes. Industrial control systems are widely used in critical infrastructure sectors such as electricity, water and wastewater, oil and gas, transportation, chemicals, pharmaceuticals, aerospace and automotive manufacturing [12]. Internet connectivity has increased the potential of attacks on industrial control systems. Meanwhile, the scale and complexity of industrial control systems renders system exceptions more frequent. The anomalies induced by attacks and exceptions can cause serious harm to infrastructure assets. Reliable detection of anomalies is vital because there can be no mitigation without detection.

Deep learning is a promising approach for anomaly detection. A deep learning model can identify anomalies that are not present in training data. Due to the absence of high-quality attack data and the difficulty obtaining data pertaining to normal operation, unsupervised training is more practical than unsupervised training for industrial control systems.
Traditional anomaly detection methods can also be effective at detecting anomalies in industrial control systems. Methods that rely on data distributions may be inflexible and find it difficult to model the features of complex patterns encountered in industrial control systems. However, they are suitable for detecting anomalies in components such as backup pumps that are rarely enabled and/or whose data values are very stable during normal operation. Once these components are enabled or the data fluctuates, the system is likely to be abnormal. In these situations, traditional methods based on data distributions are more suitable than deep learning methods because they have very low training costs and provide fast anomaly detection with low false positive rates.

A key requirement imposed on anomaly detection systems is reporting the sources of anomalies, a feature referred to as traceability. Traceability is important to handle anomaly alerts and mitigate negative impacts as well to conduct forensic investigations. However, the vast majority of anomaly detection solutions focus on detection metrics such as the F1-score and do not provide adequate traceability.

This chapter describes a traceable time series prediction anomaly detection (TTPAD) method that performs unsupervised real-time anomaly detection based on time series data from industrial control system components. Anomalies reported by TTPAD enable forensic practitioners to pinpoint the components that act abnormally so that investigations can focus on specific components instead of the entire industrial control systems.

The proposed TTPAD method combines data distribution and deep learning approaches to perform anomaly detection. Industrial control system components are categorized into two groups according to their stability. Components with very stable values are assigned static thresholds to identify anomalies whereas components with less stable values are monitored by customized transformer-encoder classifiers. The final detection result is ensembled from all the classifier outputs. Experiments using water treatment testbed data demonstrate the effectiveness of the TTPAD method with respect to anomaly detection (low false positive rates and delays) and traceability.

## 2 Related Work

Conventional anomaly detection techniques employ statistical methods, time series analysis and signal processing. Krishna et al. [6] presented an autoregressive integrated moving average (ARIMA) model [5] for detecting electricity theft perpetrated by tampering with smart meter readings. Autoregressive integrated moving averaging is a classic time series analysis method. However, the time series data should be stationary. In the case of industrial control system components, time series data may not be stationary.

Data-driven techniques such as machine learning and deep learning have been used in anomaly detection. Wei et al. [14] designed an unsupervised multi-autoencoder that detects insider threats by analyzing system log files. The multi-autoencoder increased the model depth, which enhanced detection performance.
Zhou et al. [17] employed an informer model to perform long-sequence time series forecasting. Informer is an adjusted version of a transformer model that decreases the complexity of the transformer via self-attention and self-attention distilling. The adjusted transformer-decoder can output entire long-sequence predictions instead of one prediction at a time.

Malhotra et al. [7] proposed a long short-term memory (LSTM) network-based encoder-decoder scheme for anomaly detection. The training goal involved reconstructing normal time series behavior. The approach assumes that a model trained using normal data is unable to reconstruct the sequence well when an anomaly occurs. Therefore, the reconstruction error is used to detect anomalies.

Chauhan and Vig [3] proposed an LSTM method for detecting anomalies in electrocardiogram signals. An LSTM is created as a predictive model of electrocardiogram signals and the predicted errors are indicators of anomalous signals.

Real-world industrial control system data is hard to obtain. This is a problem because anomaly detection techniques need high-quality real data for model training and testing. Secure Water Treatment (SWaT) [4, 8] is an open-source water treatment testbed that has been employed by several industrial control system researchers.

Wijaya et al. [15] have implemented supervised learning models using traces of normal and abnormal SWaT operations. They compared several classifiers such as one-class support vector machine (OCSVM), multilayer perceptron and decision tree classifiers, noting that supervised methods yielded lower false positive rates compared with unsupervised methods.

Raman et al. [11] proposed a supervised probabilistic neural network for detecting anomalies. Balaji et al. [2] created ensemble classifiers using Random Forest, bagging with k-nearest neighbor and Gaussian Naive Bayes algorithms to detect anomalies in SWaT data. However, these supervised learning methods need high-quality labeled industrial control systems data, including real-world attack data, that are hard to obtain. As a result, unsupervised methods are better suited to anomaly detection in industrial control systems.

Raman et al. [10] proposed a method that identifies anomalies in the process dynamics of industrial control systems. Complex relationships between components are modeled in operational data using a deep convolutional neural network. However, the work only considers a small subset of SWaT testbed components.

Aoudi et al. [1] proposed a process-aware stealthy attack detection (PASAD) method that monitors time series sensor measurements in real-time for structural changes in process behavior. PASAD detects attacks by checking if current sensor values diverge from previous values due to generating mechanism changes. However, in its SWaT experiments, PASAD produced numerous false positives on AIT-202, one of the two demonstration components.

Yau et al. [16] utilized OCSVMs to detect attacks in the SWaT testbed. Six models were trained for six filtration processes so that alerts could be traced at the filtration process level. However, the approach is not detailed enough because each filtration process has many components and a forensic practitioner would have to investigate all the components without any guidance provided by
<figure><img src="image_4.png" /><figcaption>Fig. 1. TTPAD classifier generation workflow.</figcaption></figure>

the model. The training process also did not engage time sequence information, which affected detection performance.

Raman et al. [9] proposed deep autoencoder based anomaly detection using reconstruction error as the anomaly indicator. Different depths in the hidden layer were investigated to enhance performance. The best performance model was achieved when each process stage had an independent autoencoder classifier. The method only provides traces at the filtration process level and, thus, exhibits the same problems as the approach of Yau et al. [16].

The methods described in this section do not satisfy the strong traceability requirement that is crucial in forensic investigations. Additionally, they tend to use metrics such as accuracy, precision, recall and F1-score that are not generally employed by the digital forensics community. In contrast, the TTPAD method described in this chapter is verified using the SWaT dataset with forensic metrics such as detection delay, detection rate and traceability. As described later in this chapter, the experimental results are also very satisfying.

## 3 TTPAD Method

The traceable time series prediction anomaly detection (TTPAD) method generates a classifier for each industrial control system component, which enhances traceability. Figure 1 shows the classifier generation workflow for a single component.
TTPAD first filters the stable features. Some components in industrial control systems, such as backup pumps in water treatment systems, are rarely enabled or have very stable values during normal system operation. These stable components are special because when their values change, an industrial control system is almost certain to have anomalies.

Component data sampled during normal industrial control system operation is input to the TTPAD method. After minimum-maximum normalization, the standard deviation of each component is computed. The standard deviation reflects the stability of the component. If the component standard deviation is less than a threshold, the component is considered to be stable. The threshold is determined based on prior knowledge about the system and the standard deviation distribution (discussed in Section 4). This data distribution based detection method is adopted for stable industrial control system components. Less stable and unstable components are fitted using transformer-encoder models.

## 3.1 Data Distribution Based Detection

Classifier generation for a stable industrial control system component involves three steps. In the first step, component data collected during normal operation is normalized using the z-score method. Next, the maximum value $Max$, minimum value $Min$ and standard deviation $SD$ of the normalized data distribution are computed. In the third step, the data distribution based method learns the upper and lower bounds from the normal data to generate the classifier. The upper and lower bounds are computed as:

$$ \text{Upper Bound} = Max + SD $$

$$ \text{Lower Bound} = Min - SD $$

When a detection classifier is testing or monitoring a stable component, the real-time component value is normalized by the z-score. The mean and standard deviation used to evaluate normalization are computed from normal system operation data, not testing data. An anomaly is detected when the normalized data value is not within the upper and lower bounds.

## 3.2 Transformer-Encoder Based Detection

Transformer-encoder models are employed to detect anomalies in unstable industrial control system components. Figure 2 shows the transformer-encoder detection model. If an industrial control system reads a component value once a second, the model input is the component value for $n$ consecutive seconds and the output is the component value predicted by the model for time $n + 1$. The input embedding layer is a fully-connected network that maps the input value to an eight-dimensional vector for each second. Positional encoding information
<figure><img src="image_5.png" /><figcaption>Fig. 2. Transformer-encoder detection model.</figcaption></figure>

is added to the output of the input embedding layer. This step encodes the sequence of time in the embedding so that it incorporates timing information.

Inputting the embedding results of *n*-second time series data to two transformer-encoder [13] layers produces *n* outputs. The training process of the encoder causes the output generated each second to be based on the inputs during the previous *n* seconds. Thus, the encoder learns the pattern of the *n*-second time series data corresponding to its component. The output of the encoder layer is passed to a global average pooling layer that reduces the data dimension from eight to one. Next, the reduced output is sent to a fully-connected network layer to produce the final output, which is the component value at time *n*+1 predicted by the encoder.

Data collected during normal industrial control system operation is employed for model training. The training goal is to make the model output and the real value at time *n* + 1 as close as possible. Mean squared error is employed as the loss function:
$$Loss = (y_{n+1} - y'_{n+1})^2$$

where $y_{n+1}$ and $y'_{n+1}$ are the real and predicted values at time $n+1$, respectively.

Components with low training loss are well fitted by their detection models. Components with high training loss are not used for real-time detection. During the detection phase, real-time data is input to the transformer-encoder model. An anomaly is detected when the difference between the output predicted by the model and the real value is greater than a threshold; otherwise, the component value at time $n+1$ is normal.

A transformer-encoder uses self-attention to process time sequence information. The mechanism computes the attention score for all the inputs in a sequence that could encode contextual information. The encoder is adopted because the self-attention mechanism can fit the time series well, which benefits the task of predicting the next value.

The original transformer model designed for natural language processing has six encoder layers and an embedding dimension of 512. However, the model has high complexity and too many parameters, which render it unsuitable for time series predictions for a single component. Therefore, the model structure was customized to the time series data characteristics. The resulting slimmed-down model speeds up training and avoids overfitting. Additionally, the time complexity of the model inference process is reduced to enable faster predictions that meet real-time industrial control system requirements. The slimmed-down model also handles longer input sequences, which increases its scalability and adaptability.

For detailed adjustments, the embedding dimension was reduced to eight in the input layer. The depth of the encoder layer was reduced to two. Global average pooling was used to reduce the output dimension of the encoder layer concisely and efficiently. The replacement of multiple fully-connected layers by the combination of the global average pooling layer and a single fully-connected layer reduced the number of model parameters.

# 4 TTPAD Method Evaluation

This section describes the evaluation of the TTPAD method using the SWaT dataset. The metrics employed to assess TTPAD performance include detection delay, precision, recall, F1-score, detection rate and traceability.

## 4.1 SWaT Dataset

The SWaT testbed is a scaled-down version of a real-world industrial water treatment plant that filters five gallons of water per minute. SWaT has six sub-processes: Process 1 stores raw water, Process 2 assesses water quality, Process 3 performs membrane-based ultrafiltration, Process 4 performs dechlorination using ultraviolet lamps, Process 5 reduces inorganic impurities using reverse osmosis and Process 6 distributes the treated water. SWaT has a total of 51 components. Table 1 shows the five main SWaT component types and their functions.
<table><caption>**Table 1.** Main SWaT component types and functions.</caption><thead><tr><th colspan="2">Component Function</th></tr></thead><tbody><tr><td>LIT</td><td>(Sensor) Level Transmitter</td></tr><tr><td>AIT</td><td>(Sensor) Chemical Composition Analyzer</td></tr><tr><td>FIT</td><td>(Sensor) Flow Meter</td></tr><tr><td>MV</td><td>(Actuator) Motorized Valve</td></tr><tr><td>P</td><td>(Actuator) Pump</td></tr></tbody></table>

<table><caption>**Table 2.** Process 1 components.</caption><thead><tr><th>Name</th><th>Description</th></tr></thead><tbody><tr><td>FIT-101</td><td>Flow Meter: Measures inflow into raw water tank</td></tr><tr><td>LIT-101</td><td>Level Transmitter: Measures raw water tank level</td></tr><tr><td>MV-101</td><td>Motorized Valve: Controls water flow to raw water tank</td></tr><tr><td>P-101</td><td>Pump: Pumps water from raw water tank to second stage</td></tr><tr><td>P-102</td><td>Pump: Serves as the backup pump for P-101</td></tr></tbody></table>

Table 2 shows the Process 1 component names and their descriptions. Process 1 has a raw water tank and uses pumps to transfer the raw water to Process 2. A component name includes its component type and three digits. The first digit is the process index, the second digit is 0 and the third digit denotes the index within the same component type in the current process. For example, P-102 is the component name of the backup pump in Process 1. The component type is P for pump, the first digit is one because it is in Process 1 and the third digit is two because it is the second pump in Process 1. Readers are referred to the original SWaT paper [4] for details about the other five processes and their components.

SWaT operates 24 hours a day without stoppages. Data is recorded each second and it is assumed that an attack on SWaT cannot be launched in less than one second. All the tanks in SWaT are empty when data collection starts. It takes about five hours for SWaT to become stable after it starts operating.

Data was recorded over two periods. The first period lasted seven days with data recorded from 12/22/2015 16:30:00 to 12/28/2015 9:59:59. This period included SWaT initialization and stable operation without any attacks. The second period lasted six days with data recorded from 12/28/2015 10:00:00 to 1/2/2016 14:59:59. During the second period, 36 attacks were launched intermittently.

The data collected during the first period comprises 495,001 timestamped records of 51 component values, all with normal labels. The data collected during the second period comprises 449,919 timestamped records of 51 component values, 54,621 with attack labels and 395,298 with normal labels.

The first attack in the dataset (Attack 1) targeted Process 1 to cause a tank overflow. The process uses a motorized valve MV-101 to control flow to the
<table><caption>**Table 3.** Effective and discarded SWaT components.</caption><thead><tr><th>Components</th><th></th><th>Count</th></tr></thead><tbody><tr><td>Effective Stable Components</td><td>P-102, P-202, P-301, AIT-401, P-401, P-402, P-403, P-404, UV-401, P-501, P-502, P-601, P-603</td><td>13</td></tr><tr><td>Discarded Stable Components</td><td>Mechanical Interlock: P-201, P-204, P-206</td><td>3</td></tr><tr><td>Effective Unstable Components</td><td>LIT-301, AIT-203, AIT-402, LIT-101, AIT-502, AIT-503, AIT-202, FIT-501, FIT-401, FIT-503, PIT-501, PIT-503, DPIT-301, AIT-501, FIT-504, LIT-401</td><td>16</td></tr><tr><td>Discarded Unstable Components</td><td>Huge Training Loss: AIT-504, FIT-201, FIT-101, FIT-301, FIT-502, PIT-502, FIT-601, P-203, MV-101, P-101, P-602, MV-201, MV-301, MV-303, P-205, P-302, MV-302, MV-304<br>Data Drift After Attack: AIT-201</td><td>19</td></tr></tbody></table>

raw water tank. MV-101 was closed before the attack. The attack opened MV-101 and kept water flowing into the tank. Another attack, Attack 4, targeted an unknown component MV-504 and had no impact on SWaT operation. Therefore, the labels of the corresponding records were changed from attack to normal. As a result, the final dataset used in the experiments had 35 attacks. Readers are referred the original SWaT paper [4] for a link to the SWaT dataset with details about all the attacks.

## 4.2 Evaluation and Results

The normal data collected during the first period was used for TTPAD model training. The data collected during the second period was used for TTPAD model testing.

The first step in the TTPAD method is to identify the stable and unstable components. After min-max normalization, the standard deviation of each component was computed and sorted in ascending order. The 16th standard deviation (Component: AIT-401, SD: 0.000025) was much smaller than the 17th standard deviation (Component: AIT-504, SD: 0.0038). Since the lower standard deviation means that a component is more stable, the 16 components with low standard deviations were designated as stable components.

Table 3 shows the effective and discarded SWaT components. The first two rows in the table list the 16 stable components. Thirteen components are effective stable components classified by their upper and lower bounds. Three stable components, P-201, P-204 and P-206, were discarded due to mechanical interlock logged in the attack list.
<figure><img src="image_6.png" /><figcaption>Fig. 3. Stable backup pump P-102 state visualization.</figcaption></figure>

Stable components have unique advantages with regard to anomaly detection. Figure 3 shows the state visualization of stable component P-102, the backup pump for P-101. During normal operation, the backup pump P-102 remained off (state = 1). During the testing phase, after P-102 was turned on (state = 2), it is clear that the system came under attack. Redundancy is a basic rule in industry and industrial control systems always have backup components. The TTPAD method leverages the redundancy in industrial control systems to improve its detection capabilities.

Transformer-encoder models were created for the 35 unstable components. During the training process, a sliding window (window size = 19 and step size = 1) was used to process component data to generate the training samples. The sliding window contained 19 consecutive component readings as model input and the model predicted the component value for the next second. Mean squared error was used as the loss function to evaluate the prediction error, perform gradient backpropagation and update the model parameters. During the training process for each component, the training epoch was set to 50, batch size to 512, optimizer to Adam and learning rate to 0.01. Training was terminated when the training error did not decrease over five consecutive epochs and the model with the smallest training error was selected as the final model for the component.

After model training for all the unstable components was completed, the effective models were selected based on their training losses. A large training loss indicates that the information learned by a model is incomplete, which results in a high false positive rate and reduced traceability. Therefore, components whose
models had small training losses were selected as effective components and were used for anomaly detection. Components whose models had large training losses were discarded.

After sorting the training losses, 17 component models had training losses on the order of $10^{-6} \sim 10^{-5}$ and 18 components had training losses on the order of $10^{-4} \sim 10^{-3}$. Finally, a threshold of $10^{-4}$ was selected and components with training losses on the order of $10^{-4} \sim 10^{-3}$ were discarded. The last two rows in Table 3 list the effective and discarded unstable components. Component AIT-201 was discarded due to data drift after attack. This is because the data pattern changed dramatically and it would be difficult for the encoder to predict values based on normal data patterns.

The testing phase used the same sliding window as the training phase. When the model prediction error (i.e., absolute value of the distance between the predicted and real values) exceeded a certain threshold, the model reported an anomaly in the associated component. If one or more models were to report anomalies at the same time, a system-level alert would be raised and the list of components reporting anomalies would be provided for traceability.

The prediction error threshold for detecting anomalies was set based on the training statistics. After model training was completed, the maximum and standard deviation of the prediction error were computed for the training data. The prediction error of effective components was on the order of 0.01 and the maximum prediction error was on the order of 10. Therefore, the standard deviation was multiplied by 100 and added to the maximum value to obtain the prediction error threshold. Each effective unstable component was assigned its own threshold. A prediction error exceeding the prediction error threshold resulted in an anomaly being reported during the testing phase.

Figure 4 shows the TTPAD method performance. The horizontal axis is time and the vertical axis is system state. The system state may be normal or under attack. The first (top) trace shows the attacks detected by the effective stable components. The second trace shows the attacks detected by the TTPAD encoder models. The third trace, which collects the attacks in the first and second traces, expresses the TTPAD method performance. The bottom trace shows the real attacks. The traces demonstrate that the stable and unstable features work well and they collectively improve TTPAD method performance.

Detection delay is a useful anomaly detection metric. It is defined as the latency between when an attack occurs and when the attack is reported. The delay is greater than or equal to zero by default. Figure 5 shows the detection delay statistics. The TTPAD method detects 26 attacks, 13 of them within ten seconds after the attack were launched.

Experiments were conducted to compare the TTPAD method performance against the autoencoder (AE), LSTM, PASAD and OCSVM baseline models. All the models executed in the same computing environment with an Intel I7-7700 CPU, 16 GB memory, Nvidia 3060 12 GB memory GPU, Ubuntu 20.04 LTS system, Python and the PyTorch deep learning framework.
<figure><img src="image_7.png" /><figcaption>Fig. 4. TTPAD method performance.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Fig. 5. Detection delay statistics.</figcaption></figure>

The same training and testing data were used to create the TTPAD, AE, LSTM and OCSVM baseline models. The training data used for the PASAD baseline model comprised the last 5,000 data items intercepted from the training data.
The AE and LSTM models employed a sliding window size of 19 and a step size of one to process the training data. The encoder in the AE baseline model incorporated three fully-connected layers with output dimensions of 64, 24 and eight, respectively, and its decoder incorporated three fully-connected layers with output dimensions of 24, 64 and 19, respectively. The LSTM baseline model incorporated two LSTM layers and a hidden layer dimension (hidden size) of eight. The hyper parameters were: training epoch = 50, batch size = 512, loss function = mean squared error, optimizer = Adam and learning rate = 0.1. Training was terminated when the training error did not decrease for five consecutive epochs. The AE loss function computed the reconstruction error whereas the LSTM loss function computed the prediction error.

The PASAD model employed the official implementation [1]. As in the original paper, the length of the initial part of the time series used for training was 30,000 and the statistical dimension was 10.

OCSVM was implemented using scikit-learn, a machine learning toolset for Python. The implementation employed the rbf kernel function with a nu value of 0.0005. Since the results were too poor to be used when OCSVM models were trained by components, all 51 component values at one second were used as input (i.e., feature length = 51). Thus, traceability was not possible at the component level. After the input data was uniformly scaled, the model was trained to produce normal and attack outputs.

The precision, recall and F1-score metrics were used to evaluate anomaly detection performance:

$$ 
\begin{aligned}
\text{Precision} &= \frac{\text{TP}}{\text{TP} + \text{FP}} \\
\text{Recall} &= \frac{\text{TP}}{\text{TP} + \text{FN}} \\
\text{F1-Score} &= 2 \times \text{Precision} \times \frac{\text{Recall}}{\text{Precision} + \text{Recall}}
\end{aligned}
$$

where true positives TP is the number of attacks identified as attacks, false positives FP is the number of normal conditions identified as attacks and false negatives FN is the number of attacks identified as normal.

Table 4 compares the TTPAD method performance against the AE, LSTM, PASAD and OCSVM baseline models. It also presents the results of an ablation experiment on the TTPAD encoder model (Encoder in Table 4). OCSVM has the best precision whereas TTPAD has the best recall and F1-score. The customized encoder in TTPAD has the best F1-score among the basic classifiers. In the ablation experiment, the contribution of the stable components was incorporated, which improved the recall significantly.

Experiments were also conducted to compare the detection rate and traceability of the TTPAD method against the other baseline models. An attack was deemed to be detected successfully if, during an attack period, a model raised at least one anomaly alert. An attack was deemed to be traceable if the component
<table><caption>**Table 4.** TTPAD vs. baseline model performance comparisons.</caption><thead><tr><th>Model</th><th>Precision</th><th>Recall</th><th>F1-Score</th></tr></thead><tbody><tr><td>AE</td><td>0.8654</td><td>0.6684</td><td>0.7542</td></tr><tr><td>LSTM</td><td>0.8673</td><td>0.6761</td><td>0.7598</td></tr><tr><td>PASAD</td><td>0.7392</td><td>0.6727</td><td>0.7044</td></tr><tr><td>OCSVM</td><td><strong>0.9940</strong></td><td>0.6270</td><td>0.7689</td></tr><tr><td>Encoder</td><td>0.9233</td><td>0.6863</td><td>0.7873</td></tr><tr><td>TTPAD</td><td>0.9278</td><td><strong>0.7536</strong></td><td><strong>0.8317</strong></td></tr></tbody></table>

models that reported the anomalies corresponded to the components that were, in fact, attacked. The detection rate and traceability were computed as follows:

$$ 
\begin{aligned}
\text{Detection Rate} &= \frac{\text{Detected Attacks}}{\text{Total Attacks}} \\
\text{Traceability} &= \frac{\text{Traceable Attacks}}{\text{Detected Attacks}}
\end{aligned}
 $$

<table><caption>**Table 5.** TTPAD vs. baseline model performance comparisons.</caption><thead><tr><th rowspan="2">Baseline</th><th rowspan="2">Total<br>Attacks</th><th colspan="2">Detected<br>Attacks</th><th colspan="2">Traceable<br>Attacks</th></tr><tr><th>Rate</th><th>Rate</th><th>Rate</th><th>Rate</th></tr></thead><tbody><tr><td>AE</td><td>35</td><td>13</td><td>37.14%</td><td>4</td><td>30.77%</td></tr><tr><td>LSTM</td><td>35</td><td>12</td><td>34.29%</td><td>7</td><td>58.33%</td></tr><tr><td>PASAD</td><td>35</td><td>2</td><td>5.71%</td><td>0</td><td>0.00%</td></tr><tr><td>OCSVM</td><td>35</td><td>5</td><td>14.29%</td><td>0</td><td>0.00%</td></tr><tr><td>TTPAD</td><td>35</td><td><strong>26</strong></td><td><strong>74.29%</strong></td><td><strong>21</strong></td><td><strong>80.77%</strong></td></tr></tbody></table>

Table 5 compares the attack detection and traceability results for the TTPAD method against the four baseline models. The results reveal that the TTPAD method has by far the best attack detection ability and traceability.

# 5 Conclusions

The TTPAD method leverages unsupervised learning using industrial control system component time series data. Customized transformer-encoder classifiers are created for unstable industrial system components and the classifier outputs are ensembled to identify anomalies. Features of stable components such as backup pumps are also leveraged to enhance anomaly detection. Experiments
with SWaT testbed data indicate that the method achieves good performance compared with baseline models with respect to false positive rates, detection delays and traceability. However, the current transformer-encoder model design is not good at fitting discrete data. Future research will attempt to address this issue.

## References

1. W. Aoudi, M. Iturbe and M. Almgren, Truth will out: Departure-based process-level detection of stealthy attacks on control systems, *Proceedings of the ACM SIGSAC Conference on Computer and Communications Security*, pp. 817–831, 2018.

2. M. Balaji, S. Shrivastava, S. Adepu and A. Mathur, Super detector: An ensemble approach for anomaly detection in industrial control systems, *Proceedings of the Sixteenth International Conference on Critical Information Infrastructures Security*, pp. 24–43, 2021.

3. S. Chauhan and L. Vig, Anomaly detection in ECG time signals via deep long short-term memory networks, *Proceedings of the IEEE International Conference on Data Science and Advanced Analytics*, 2015.

4. J. Goh, S. Adepu, K. Junejo and A. Mathur, A dataset to support research in the design of secure water treatment systems, *Proceedings of the Eleventh International Conference on Critical Information Infrastructures Security*, pp. 88–99, 2017.

5. J. Hamilton, *Time Series Analysis*, Princeton University Press, Princeton, New Jersey, 1994.

6. V. Krishna, R. Iyer and W. Sanders, ARIMA-based modeling and validation of consumption readings in power grids, *Proceedings of the Tenth International Conference on Critical Information Infrastructures Security*, pp. 199–210, 2016.

7. P. Malhotra, A. Ramakrishnan, G. Anand, L. Vig, P. Agarwal and G. Shroff, LSTM-Based Encoder-Decoder for Multi-Sensor Anomaly Detection, arXiv: 1607.00148v2 (arxiv.org/abs/1607.00148), 2016.

8. A. Mathur and N. Tippenhauer, SWaT: A water treatment testbed for research and training in ICS security, *Proceedings of the International Workshop on Cyber-Physical Systems for Smart Water Networks*, pp. 31–36, 2016.

9. M. Raman, W. Dong and A. Mathur, Deep autoencoders as anomaly detectors: Method and case study in a distributed water treatment plant, *Computers and Security*, vol. 99, article no. 102055, 2020.

10. M. Raman and A. Mathur, A hybrid physics-based data-driven framework for anomaly detection in industrial control systems, *IEEE Transactions on Systems, Man and Cybernetics: Systems*, vol. 52(9), pp. 6003–6014, 2022.

11. M. Raman, N. Somu and A. Mathur, Anomaly detection in critical infrastructure using a probabilistic neural network, *Proceedings of the Tenth International Conference on Applications and Techniques in Information Security*, pp. 129–141, 2019.

12. K. Stouffer, J. Falco and K. Scarfone, Guide to Industrial Control Systems (ICS) Security, NIST Special Publication 800-82, Gaithersburg, Maryland, 2011.

13. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser and I. Polosukhin, Attention is all you need, *Proceedings of the Thirtieth Annual Conference on Neural Information Processing Systems*, pp. 5998–6008, 2017.
14. Y. Wei, K. Chow and S. Yiu, Insider threat detection using multi-autoencoder filtering and unsupervised learning, in *Advances in Digital Forensics XVI*, G. Peterson and S. Shenoi (Eds.), Springer, Cham, Switzerland, pp. 273–290, 2020.

15. H. Wijaya, M. Aniche and A. Mathur, Domain-based fuzzing for supervised learning of anomaly detection in cyber-physical systems, *Proceedings of the Forty-Second IEEE/ACM International Conference on Software Engineering Workshops*, pp. 237–244, 2020.

16. K. Yau, K. Chow and S. Yiu, Detecting attacks on a water treatment system using one-class support vector machines, in *Advances in Digital Forensics XVI*, G. Peterson and S. Shenoi (Eds.), Springer, Cham, Switzerland, pp. 95–108, 2020.

17. H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong and W. Zhang, Informer: Beyond efficient transformer for long-sequence time-series forecasting, *Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence*, pp. 11106–11115, 2021.