DilanjanaSDK/api-anomaly-detection-research-model

LSTM-Autoencoder for REST API Anomaly Detection

Reconstruction-based LSTM-Autoencoder for detecting anomalous REST API traffic. Trained exclusively on normal (benign) traffic, so it needs no labelled attack data and can generalize to sequence-dependent attacks — such as Broken Object Level Authorization (BOLA) — that request-isolated, signature-based defences (WAFs) typically miss.

Model Details

• Architecture: 2-layer LSTM encoder (hidden size 64) → 64-dimensional bottleneck → 2-layer LSTM decoder + linear projection
• Parameters: ~124,000
• Framework: PyTorch 2.x
• Input modes:
  • Embedding mode (for HTTP-log/URL traffic, e.g. CSIC 2010): abstracted URL-template tokens → 32-d learned embedding → LSTM
  • Numeric mode (for network-flow traffic, e.g. CIC-IDS2018, UNSW-NB15): standardized float feature vectors fed directly to the LSTM
• Sequence length: sliding-window sessions of W = 5 consecutive requests/flows
• Dropout: 0.2 between LSTM layers
• Training objective: minimize mean-squared reconstruction error on normal traffic only
• Detection rule: a session is flagged anomalous if its reconstruction error exceeds a threshold calibrated at the 95th percentile of normal-traffic reconstruction errors; the threshold can be further refined via an Active Learning (uncertainty-sampling) loop without retraining the model

This repository provides separate checkpoints per dataset, since each uses a different input mode / feature space (see "Checkpoints" below).

Intended Use

Detecting anomalous/attack sessions in REST API or network-flow traffic in settings where labelled attack data is unavailable (the common zero-day scenario). Designed for real-time or near-real-time deployment: inference latency is ~0.044–0.050 ms per session on CPU.

Not intended for: single-request (non-sequential) anomaly detection, traffic types substantially different from the training distributions below without retraining/fine-tuning, or as a sole security control (best used alongside, not instead of, a WAF).

Training Data

Trained separately, per dataset, on normal traffic only:

Dataset	Type	Normal sessions (train)
CSIC 2010	HTTP logs	35,996
CIC-IDS2018	Network flows	1,676,966
UNSW-NB15	Network flows	74,396

See the accompanying dataset card for full dataset details and licensing.

Evaluation

Dataset	Precision	Recall	F1	FPR	Latency (ms/session)
CSIC 2010	0.946	1.000	0.972	0.040	0.050
CIC-IDS2018	0.037†	0.988	0.071†	0.057	0.044
UNSW-NB15	0.979	0.667	0.794	0.128	0.044

† CIC-IDS2018 precision is limited by extreme session-level class imbalance (~454:1 normal:attack) in the test set, not a model deficiency — recall stays high (0.988).

On UNSW-NB15, the model outperforms an Isolation Forest baseline (F1 = 0.663) while running 6.1–6.8× faster.

Ablation: why normal-only training?

A supplementary ablation trained the same architecture on balanced normal/attack data (reconstruction objective and direct supervision). Balanced reconstruction training degraded F1 by 43–78% relative to the normal-only model across all three datasets, confirming that training exclusively on normal traffic is necessary — not incidental — for reconstruction-based anomaly detection to work as intended. Direct supervision achieved near-perfect scores only because it requires labelled attack data at training time, which defeats the zero-day use case this model targets.

How to Use

import torch
from model import LSTMAutoencoder  # see repo for model.py

model = LSTMAutoencoder(mode="numeric", input_dim=23, hidden_size=64)
state_dict = torch.load("lstm_ae_cicids2018.pt", map_location="cpu")
model.load_state_dict(state_dict)
model.eval()

# x: tensor of shape (batch, W=5, input_dim), pre-scaled with the matching StandardScaler
with torch.no_grad():
    x_hat = model(x)
    error = ((x - x_hat) ** 2).mean(dim=(1, 2))  # per-session MSE anomaly score

threshold = 0.0136  # example — use the calibrated threshold shipped for each checkpoint
is_anomalous = error > threshold

For CSIC 2010 (embedding mode), inputs are integer token sequences produced by the matching vocabulary/abstraction pipeline in the source repository — using the model without that exact preprocessing will produce unreliable results, since the abstraction step is load-bearing for this model (see Limitations).

Limitations

• CIC-IDS2018 precision is fundamentally constrained by the dataset's extreme class imbalance at the session level, independent of model quality.
• Preprocessing-sensitive: the URL-abstraction / feature-scaling pipeline used at inference time must exactly match the one used to build the training vocabulary/scaler. Divergent implementations of the same logic (observed during development) silently inflate false positives.
• Session ordering matters for flow-based datasets: sessions must be constructed in timestamp order, not raw file order, to match training conditions.
• BOLA is a motivating attack class for the sequence-aware design but was not evaluated with a dedicated, labelled BOLA dataset.
• Overlapping sliding-window sessions carry a theoretical train/test leakage risk (adjacent windows share requests), acknowledged as a limitation rather than resolved.
• Latency figures are per-session batched inference, not single live-request latency.

Citation

Kandanaarachchi, K.A.D.S.D. "Detecting Anomalous REST API Traffic."
BSc thesis, University of Vavuniya, 2020/ICT/19.
Supervised by Dr. S. Kirushanth.

License

MIT (code/weights). Underlying datasets retain their original licenses — see the dataset card.