SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbf {X} ^ {c} = \left\{\mathbf {x} _ {1} ^ {c}, \dots , \mathbf {x} _ {T} ^ {c} \right\} \approx \hat {\mathbf {X}} ^ {c} = \left\{\hat {\mathbf {x}} _ {1} ^ {c}, \dots , \hat {\mathbf {x}} _ {T} ^ {c} \right\}, \quad \hat {\mathbf {x}} _ {t} ^ {c} = \hat {\mathbf {x}} _ {p t, t} = \mathcal {M} _ {p t} \left(\mathbf {x} _ {t} ^ {0}\right) \tag {15}
$$
$\mathcal{M}_{pt}$  can be any model that has the ability to reconstruct  $\mathbf{X}^0$  point-wise. To our surprise, the best performance can be achieved by using a simple Performer-based autoencoder ([34]) that actually has the potential to discover temporal information. Despite having such a possibility, it only learned to reconstruct point-by-point during training. We demonstrated this fact by shuffling the input time points and observing that the result will be the same after reordering the output sequence. One possible explanation is that during training, it is a lot easier to individually reconstruct single time points than to find complex time-dependent relationships; and since the Performer-based autoencoder tries to optimize over a batch of time points, this reduces the effect of overfitting and allows the model to better generalize to unseen data. However, the exact reason for this remains an open question. For the rest of the study, we will use  $\mathcal{M}_{pt}$  to refer to the Performer-based autoencoder model. Details for the architecture of  $\mathcal{M}_{pt}$  are shown in Appendix B.1.
3.6 Sequence-based Reconstruction Models
Contrary to  $\mathbf{x}_t^c$ ,  $\mathbf{x}_t^*$  should not only be in  $\mathcal{X}^*$  but also obey the time-dependent relationships. Therefore, it is necessary that the model  $(\mathcal{M}_{seq})$  for approximating  $\mathbf{x}_t^*$  takes a sequence of time points as input.
$$
\mathbf {X} ^ {*} \approx \hat {\mathbf {X}} ^ {*} = \left\{\hat {\mathbf {x}} _ {1} ^ {*}, \dots , \hat {\mathbf {x}} _ {T} ^ {*} \right\} \triangleq \mathcal {M} _ {s e q} \left(\mathbf {X} ^ {0}\right) \tag {16}
$$
How close  $\mathcal{M}_{seq}(\mathbf{X}^0)$  approximates  $\mathbf{X}^*$  depends on the amount of training data and the model capacity. In practice, for computational reasons, only a section of  $\mathbf{X}^0$  is input and reconstructed at a time. We found that given a subsequence  $\mathbf{X}_{ab}^0 = \{\mathbf{x}_a^0,\dots,\mathbf{x}_b^0\}$  as input for reconstruction, a model tends to simply reconstruct individual points and do not take temporal information into account (as discussed in section 3.5). Therefore, we use a Performer-based stacked encoder as  $\mathcal{M}_{seq}$ , which predicts the middle  $\delta$  points from its surrounding  $2\gamma$  points to force the learning of time-dependent relationships. We concatenate all the predicted time points output by  $\mathcal{M}_{seq}$  to construct  $\hat{\mathbf{X}}^{*}$ . For the rest of the study, we will use  $\mathcal{M}_{seq}$  to refer to the Performer-based stacked encoder model. Details for the architecture of  $\mathcal{M}_{seq}$  are shown in Appendix B.2. Fig. 3 gives an illustration of the architecture for  $\mathcal{M}_{pt}$ ,  $\mathcal{M}_{seq}$ , and the overall scheme. By obtaining  $\hat{\mathbf{X}}^c$  and  $\hat{\mathbf{X}}^{*}$ , we can calculate  $N(\cdot)$  and select some  $A(\cdot)$  for calculating  $\hat{A} (\cdot)$ . The algorithm for evaluating a trained  $\mathcal{M}_{pt}$  and  $\mathcal{M}_{seq}$  using the soft gate function is shown in Algorithm 1.
Figure 3: (a) Performer-based autoencoder  $\mathcal{M}_{pt}$ , (b) Performer-based stacked encoder  $\mathcal{M}_{seq}$ , and (c) main scheme for NPSR. GELUs are used as the activation function for each layer.
Algorithm 1 NPSR F1* Evaluation (soft gate function)
function NPSR(Mpt, Mseq, X0 = {x1, ..., xT}, y = {y1, ..., yT}, θN, d)
Construct Xc = {x1c, ..., xTc} with xtc← Mpt(xt0)	▷ (15)
Construct X* = {x1*, ..., xT*} ← Mseq(X0)	▷ (16)
Construct A(·) with A(t) ← ||xtc - xt0||2/2	▷ section 3.5
Construct N(·) with N(t) ← ||xtc - xt*||2/ ||xt0 - xt*||2	▷ (1)
Construct gθN(N(·)) with gθN(N(t)) ← max(0, 1 - N(t)/θN)	▷ (8)
Construct A(·; ·) with A(t; τ) ← A(τ) ∏k=min(τ+1,t)max(t-1t=τ,τ-1)gθN(N(k))	▷ (7)
Constructˆ(·) withˆ(t) ← ∑τ=min(τ+1,t-d)A(t; τ)	▷ (6)
return F1* ← maxθa F1(ˆ(ˆ(·), θa); y)
We evaluate NPSR on the following datasets:
- SWaT (Secure Water Treatment) [35]: The SWaT dataset is collected over 11 days from a scaled-down water treatment testbed with 51 sensors. During the last 4 days, 41 anomalies were injected using diverse attack methods, while only normal data were generated during the first 7 days.
- WADI (WAter Distribution testbed) [36]: The WADI dataset is acquired from a reduced city water distribution system with 123 sensors and actuators operating for 16 days. The first 14 days contain only normal data, while the remaining two days have 15 anomaly segments.
- PSM (Pooled Server Metrics) [37]: The PSM dataset is collected internally from multiple application server nodes at eBay. There are 13 weeks of training data and 8 weeks of testing data.
- MSL (Mars Science Laboratory) and SMAP (Soil Moisture Active Passive) [38, 13]: The MSL and SMAP datasets are public datasets collected by NASA, containing telemetry anomaly data derived from the Incident Surprise Anomaly (ISA) reports of spacecraft monitoring systems. The datasets have 55 and 25 dimensions respectively. The training set contains unlabeled anomalies.
- SMD (Server Machine Dataset) [9]: The SMD is collected from a large internet company, comprising 5 weeks of data from 28 server machines with 38 sensors. The first 5 days contain only normal data, and anomalies are injected intermittently for the last 5 days.
- trimSyn (Trimmed Synthetic Dataset) [24]: The original synthetic dataset was generated using trigonometric functions and Gaussian noises. We obtained the dataset from [39] and trimmed the test dataset such that only one segment of anomaly is present.
The statistics for the datasets are summarized in Table 1. For multi-entity datasets, Train#/Test# corresponds to the number of train/test time points summed over all entities, and the anomaly rate is calculated from the ratio between the sum of all anomaly points and sum of all test points.
Table 1: Datasets used in this study before preprocess.
Dataset	Entities	Dims	Train #	Test #	Anomaly Rate (%)
SWaT	1	51	495000	449919	12.14
WADI	1	123	1209601	172801	5.71
PSM	1	25	132481	87841	27.76
MSL	27	55	58317	73729	10.48
SMAP	55	25	140825	444035	12.83
SMD	28	38	708405	708420	4.16
trimSyn	1	35	10000	7680	2.34
4.2 Baselines
We evaluate the performance of NPSR against several deep learning algorithms and simple heuristics using F1*. Due to the exhaustive nature of optimizing for all datasets and algorithms, we follow a three-step approach to populate Table 2. Firstly, we reference values from the original paper, and if unavailable, we search for the highest reported values among other publications. Finally, if no reported values are found, we modify and run publicly available code. We cannot find any reported F1* of the PSM dataset using THOC or any publicly available code, hence leaving the value blank. For multi-entity datasets (MSL, SMAP, and SMD), we compare the performance using two methods - (1) combining all entities and training them together; (2) training each entity separately and averaging the results. Moreover, some literature attempt to find a threshold  $(\theta_{a})$  and then calculate the performance conditioned on it [24, 40]. Since  $\theta_{a}$  can simply be a one-value parameter, we assume that other studies have already optimized this value, and regard their reported F1 as F1*. More information on the sources of data can be found in Appendix E.
In Table 2, we report the results for  $\mathrm{F1}^*$  on several datasets. Detailed preprocessing steps and training settings are reported in Appendix C. NPSR almost consistently outperforms other algorithms, only being slightly inferior to TranAD on the PSM dataset. NPSR makes use of  $\mathcal{M}_{pt}$  to precisely capture point anomalies with low false-positive rates (given the best threshold). Moreover, it acquires the ability to detect contextual anomalies by incorporating  $\mathcal{M}_{seq}$  through the calculation of  $\hat{A}(\cdot)$ , without compromising its capability of detecting point anomalies. We observe that recent studies do not necessarily have higher  $\mathrm{F1}^*$  scores than older ones. Interestingly, simple heuristics can even perform fairly well on  $\mathrm{F1}^*$ , with NPSR being the only algorithm consistently outperforming them.
Table 2: Best F1 score  $(\mathrm{F1}^{*})$  results on several datasets, with bold text denoting the highest and underlined text denoting the second highest value. The deep learning methods are sorted with older methods at the top and newer ones at the bottom.
Algorithm \ Dataset	SWaT	WADI	PSM	MSL	SMAP	SMD	trimSyn
Simple Heuristic [11, 30, 31]	0.789	0.353	0.509	0.239	0.229	0.494	0.093
DAGMM [26]	0.750	0.121	0.483	0.199	0.333	0.238	0.326
LSTM-VAE [22]	0.776	0.227	0.455	0.212	0.235	0.435	0.061
MSCRED [24]	0.757	0.046	0.556	0.250	0.170	0.382	0.340
OmniAnomaly [9]	0.782	0.223	0.452	0.207	0.227	0.474	0.314
MAD-GAN [23]	0.770	0.370	0.471	0.267	0.175	0.220	0.331
MTAD-GAT [27]	0.784	0.437	0.571	0.275	0.296	0.400	0.372
USAD [28]	0.792	0.233	0.479	0.211	0.228	0.426	0.326
THOC [18]	0.612	0.130	-	0.190	0.240	0.168	-
UAE [11]	0.453	0.354	0.427	0.451	0.390	0.435	0.094
GDN [12]	0.810	0.570	0.552	0.217	0.252	0.529	0.284
GTA [41]	0.761	0.531	0.542	0.218	0.231	0.351	0.256
Anomaly Transformer [40]	0.220	0.108	0.434	0.191	0.227	0.080	0.049
TranAD [25]	0.669	0.415	0.649	0.251	0.247	0.310	0.282
NPSR (combined)	-	-	-	0.261	0.511	0.227	-
NPSR	0.839	0.642	0.648	0.551	0.505	0.535	0.481
We suggest using these simple heuristic values as a strong baseline for future studies to compare against. In light of recent publications highlighting the limitations of using the point-adjusted F1 score ([11, 30, 31, 42]), and yet the large amount of work still using it, we also report the results using point-adjustment in Appendix D.
For multi-entity datasets, we observe that the standard method (training one point-based and sequence-based model per entity) outperforms the combined method for MSL and SMD datasets. This is not surprising, given that entity-to-entity variations might be large. However, for the SMAP dataset, we observe that the combined method performs better. We attribute such results to the fact that the SMAP spacecraft are routine, hence the resulting telemetry between entities can have similar underlying distributions. This contributes to additive learning from the increased training data [13].
The ablation study conducted in this section sheds light on several aspects of the proposed method. In Table 3, we compare the performance of five methods that yield different anomaly scores (either  $A(\cdot)$  or  $\hat{A}(\cdot)$ ). For the first two methods, the reconstruction errors of  $\mathcal{M}_{pt}$  and  $\mathcal{M}_{seq}$  are used, respectively. Since  $\mathcal{M}_{pt}$  mostly performs better than  $\mathcal{M}_{seq}$ , we use the point-based reconstruction error as  $A(\cdot)$  to calculate  $\hat{A}(\cdot)$  for the last three methods. The third method corresponds to an unnormalized simple smoothed value of  $A(\cdot)$  (cf. section 3.4). The fourth and fifth methods use gate functions (11) and (8), respectively, but the same  $\theta_N$  that corresponds to the 98.5 percentile of the nominality score from the training data ( $N_{trn}$ ). This method for setting  $\theta_N$  works well enough and can be applied across different datasets. Illustrations of the distribution of nominality scores for the SWaT and WADI datasets, along with the 98.5% threshold value are shown in Fig. 4. We can observe that the distributions are close to appropriate (cf. section 3.3). The results for the last three methods are averaged over induction lengths  $d = 1, 2, 4, 8, 16, 32, 64, 128$ , and 256. Each entity is trained separately for multi-entity datasets and the results are pooled together.
Firstly, we observe that  $\mathcal{M}_{pt}$  outperforms  $\mathcal{M}_{seq}$  and achieves competitive results on its own, despite not modeling the time-dependent relationships. Secondly, by smoothing  $A(\cdot)$  (third row), the AUC and  $\mathrm{F1}^*$  are increased for most datasets. This means smoothing is generally an effective method for improving performance. Moreover, our experiments show that the soft gate function along with an appropriate  $\theta_N$  performed the best on average in terms of  $\mathrm{F1}^*$ . This suggests that the distribution of nominality scores is predominantly overlapped, and a soft gate function will be more appropriate to prevent excessive accumulation of anomaly scores on normal time points, reducing false-positives (cf. Appendix F). This method will also have a generally stable AUC and  $\mathrm{F1}^*$  (low  $\sigma_d$ ) across a wide range of  $d$ . This makes sense - when time point  $\tau$  is farther away from time point  $t$ , more gate function outputs are multiplied onto  $A(\tau)$ , hence  $A(t,\tau)\rightarrow 0$ . However, the results also suggest that the best choice of gate function and  $\theta_N$  may depend on the specific dataset at hand.
4.5 Detection Trade-off Between Point and Contextual Anomalies
We elaborate on the trade-off between detecting point and contextual anomalies and relate them with the performance of  $\mathcal{M}_{pt}$  and  $\mathcal{M}_{seq}$ . It may seem intuitive that finding temporal information in
Table 3: AUC and  $\mathrm{F1}^*$  for different methods and datasets, with bold text denoting the highest and underlined text denoting the second highest value. The mean  $(\mu_d)$  and standard deviation  $(\sigma_d)$  of the performance metrics evaluated across  $d = 1,2,4,8,16,32,64,128,256$  are shown.
Dataset	SWaT	WADI	PSM	MSL	SMAP	SMD	trimSyn
Method	AUC	F1*	AUC	F1*	AUC	F1*	AUC	F1*	AUC	F1*	AUC	F1*	AUC	F1*
Mpt( ||xt-c - xt0||2)	0.908	0.839	0.819	0.629	0.790	0.626	0.640	0.366	0.647	0.329	0.820	0.485	0.721	0.100
Mseq( ||xt-xt0||2)	0.899	0.755	0.843	0.559	0.766	0.576	0.621	0.351	0.611	0.292	0.820	0.482	0.832	0.345
Mpt+Hard (11)(θN=∞)	μd	0.912	0.813	0.827	0.630	0.775	0.621	0.708	0.451	0.665	0.389	0.835	0.492	0.785
σd	0.005	0.034	0.007	0.037	0.023	0.020	0.032	0.038	0.010	0.036	0.025	0.052	0.037
Mpt+Hard (11)(θN=98.5%Ntrn)	μd	0.912	0.820	0.844	0.625	0.779	0.624	0.718	0.467	0.659	0.386	0.833	0.495	0.791
σd	0.005	0.024	0.007	0.023	0.017	0.015	0.041	0.051	0.012	0.034	0.024	0.050	0.069
Mpt+Soft (8)(θN=98.5%Ntrn)	μd	0.909	0.837	0.856	0.639	0.804	0.636	0.698	0.465	0.656	0.388	0.840	0.525	0.862
σd	0.000	0.001	0.011	0.008	0.005	0.004	0.031	0.061	0.005	0.039	0.003	0.011	0.063
Figure 4: Histograms of the nominality scores for the SWaT and WADI dataset.
time series would lead to better performance. However, the modeling complexity increases with the number of time points being considered. This leads to the difficulty of focusing on the reconstruction of single time points. Fig. 5(a) shows  $A(\cdot)$  calculated using either  $\mathcal{M}_{pt}$  or  $\mathcal{M}_{seq}$ , and  $\hat{A}(\cdot)$  calculated by NPSR using the WADI dataset. Given the fact that point  $t$  is predicted anomalous if  $A(t) \geq \theta_{a}$ , we can see that  $\mathcal{M}_{seq}$  has a higher false positive rate due to the spikes. In comparison,  $\mathcal{M}_{pt}$  more accurately detects anomalies in a point-wise fashion. This highlights the superiority of  $\mathcal{M}_{pt}$  over  $\mathcal{M}_{seq}$  for this dataset. Furthermore, the induced anomaly score calculated by NPSR has very low false positive rates, and can sometimes even learn to identify anomalies that are not detected by  $\mathcal{M}_{pt}$  or  $\mathcal{M}_{seq}$  (Fig. 5(b), the anomaly subsequence at  $t = 15200$  and the subsequence to its left, pointed by the black arrows). This suggests that  $\hat{A}(\cdot)$  is superior to the original  $A(\cdot)$  for this dataset. However,  $\mathcal{M}_{pt}$  might not always perform superior to  $\mathcal{M}_{seq}$ . False negatives can be visualized between  $t = 14800$  and  $t = 14900$ , where  $\mathcal{M}_{pt}$  struggles to recognize the anomaly but  $\mathcal{M}_{seq}$  effectively detects anomalous time-dependent relationships. This suggests that the anomaly segment contains relatively more contextual than point anomalies. Since the reconstruction error of  $\mathcal{M}_{pt}$  is used as  $A(\cdot)$ , we lose the advantage of effectively utilizing the reconstruction error of  $\mathcal{M}_{seq}$ . This results in  $\hat{A}(\cdot)$  not high enough to reach  $\theta_{a}$  within this segment. An important future direction would be to explore how to appropriately select  $A(\cdot)$  among multiple models (cf. Appendix F).
Figure 5: (a) Anomaly scores using  $\mathcal{M}_{pt}$ ,  $\mathcal{M}_{seq}$  and NPSR (soft gate function,  $\theta_N = 99.85\% N_{trn}$ , and  $d = 16$ ), and the true labels of the WADI dataset. (b) Magnification for  $t \in \{14500, \dots, 15600\}$ .
In conclusion, we introduce an improved framework for unsupervised time series anomaly detection. We specify the relationships between point and contextual anomalies and derive the nominality score and induced anomaly score to provide a theory-based algorithm with provable superiority. NPSR captures both point and contextual anomalies, resulting in a high combined precision and recall. Our results show that NPSR exhibits high performance, is widely applicable, and has a relatively straightforward training process. It has the potential to decrease labor needs for fault monitoring and correspondingly accelerates decision making and can also contribute to AI sustainability by preventing energy waste or system failure.
We thank Piyush Desai, Raphael Schutz, and Nikhil Deshmukh from Turntide Technologies for their helpful discussions and insights.
A The Best F1 Score
This section describes the method for calculating the best F1 score  $(\mathrm{F}1^{*})$  from a set of anomaly scores  $\mathbf{a} = \{a_{1},\dots,a_{T}\}$  and a set of labels  $\mathbf{y} = \{y_1,\dots,y_T\}$ . Firstly, given  $\mathbf{a}$  and some arbitrary threshold  $\theta_{a}$ , we can calculate  $\hat{\mathbf{y}} = \{\hat{y}_1,\dots\hat{y}_T\}$ , where  $\hat{y}_{t}\triangleq \mathbb{1}_{a_{t}\geq \theta_{a}}$ . Secondly,  $\hat{\mathbf{y}}$  is used to calculate TP, FP, and FN, which corresponds to the sets of time points for true positives, false positives, and false negatives.
$$
\mathrm {T P} \triangleq \{t | \hat {y} _ {t} = 1, y _ {t} = 1 \}, \quad \mathrm {F P} \triangleq \{t | \hat {y} _ {t} = 1, y _ {t} = 0 \}, \quad \mathrm {F N} \triangleq \{t | \hat {y} _ {t} = 0, y _ {t} = 1 \} \tag {17}
$$
Thirdly, we calculate the precision (P) and recall (R), and then calculate the F1 score, which is the harmonic mean between R and P.
$$
\mathrm {F} 1 \triangleq \frac {2 \mathrm {P R}}{\mathrm {P} + \mathrm {R}}, \quad \mathrm {P} \triangleq \frac {\mathrm {n} (\mathrm {T P})}{\mathrm {n} (\mathrm {T P}) + \mathrm {n} (\mathrm {F P})}, \quad \mathrm {R} \triangleq \frac {\mathrm {n} (\mathrm {T P})}{\mathrm {n} (\mathrm {T P}) + \mathrm {n} (\mathrm {F N})} \tag {18}
$$
Finally, we calculate  $\mathrm{F1}^*$  by using the threshold that yields the highest F1.
$$
F 1 ^ {*} (\mathbf {a}; \mathbf {y}) \triangleq \max  _ {\theta_ {a}} F 1 \left(\hat {\mathbf {y}} \left(\mathbf {a}, \theta_ {a}\right); \mathbf {y}\right) \tag {19}
$$
Performers (an improved variant of Transformers) are competitive in terms of execution speed compared with other Transformer variants [34, 43], hence we use them as the basic building block of our models. For both  $\mathcal{M}_{pt}$  and  $\mathcal{M}_{seq}$ , we use a linear layer with input and output dimensions equal to  $D$  as the token embedding layer, a fixed positional embedding layer at the beginning, a feature redraw interval of 1, and a tanh activation function immediately before the output. GELUs [44] are used as the activation layer for all linear layers. We do not change any other predefined activation layer inside Performers.
B.1 Performer-based autoencoder
For the Performer-based autoencoder  $\mathcal{M}_{pt}$ , the input with the shape  $(\text{batch}, W, D)$  is passed through a plain Performer with  $N_{\text{perf}}$  layers after the positional embedding step, where  $W$  represents the input window size. This is followed by a linear encoding layer that transforms the dimensionality from  $D$  to  $D_{\text{lat}}$ , resulting in the shape  $(\text{batch}, W, D_{\text{lat}})$  for the latent variables. In the case of  $\mathcal{M}_{pt}$ , there is no compression along the time domain. The latent variables then pass through another linear decoding layer that transforms the dimensionality from  $D_{\text{lat}}$  back to  $D$ , followed by another Performer with  $N_{\text{perf}}$  layers.
B.2 Performer-based stacked encoder
For the Performer-based stacked encoder  $\mathcal{M}_{seq}$ , the input with shape  $(\text{batch}, W_0, D)$  is passed through a plain Performer with one layer after the positional embedding step, followed by a linear encoding layer that transforms the window size from  $W_0$  to  $W_1$ , where  $W_0 = 2\gamma$  (cf. section 3.6). It should be noted that unlike  $\mathcal{M}_{pt}$ , compression is done along the time domain for  $\mathcal{M}_{seq}$ . The one-layer Performer and linear layer are stacked  $N_{enc}$  times, where in the  $i$ -th stack, the window size is compressed from  $W_{i-1}$  to  $W_i$ .  $W_{N_{enc}}$  is equal to the target output window size  $\delta$ . For both  $\mathcal{M}_{pt}$  and  $\mathcal{M}_{seq}$ , we optimize  $N_{perf}$ ,  $D_{lat}$ ,  $W$ ,  $N_{enc}$ ,  $W_i$  for  $i \in \{0, \dots, N_{enc}\}$ , and  $\delta$  for the best performance. Note that  $\mathcal{M}_{seq}$  isn't capable of reconstructing the first and last  $\gamma$  time points due to its architecture, hence we discard the first and last  $\gamma$  points reconstructed by  $\mathcal{M}_{pt}$  so that rest of the time points have exactly two reconstructed values corresponding to using  $\mathcal{M}_{seq}$  and  $\mathcal{M}_{pt}$ , respectively.
C Data Preprocessing and Training Details
Table 4 shows the hyperparameters used for implementing NPSR on the experimented datasets. To ensure fair comparison, the same preprocessing method is applied to all algorithms for the same dataset. The search for hyperparameters is done manually, starting from some reasonable value (e.g. a learning rate of  $10^{-4}$ ). The authors believe that there is still room for improvement by fine-tuning these hyperparameters. To speed up training, we load all training inputs and outputs, and testing inputs onto the GPU before training. We use a local GPU, which can be either GeForce RTX 3070 (8GB), 3080 (12GB) or 3090 (24GB). For an individual experiment using a single dataset and training method, the training time ranges from approximately 2 minutes to 12 hours. For single-entity datasets and multi-entity datasets that use the combined training method, we run the experiments for at least 3 times and confirm that the results are stable given different random seeds. For multi-entity datasets with entities trained individually, the results are averaged across all entities. Generally, datasets with single entities train faster than those with multiple entities. There are some additional remarks regarding the preprocessing of the datasets.