AskarT/flight-delay-catboost

Flight delay risk: CatBoost pipeline

Binary classifier that estimates the probability that a US domestic flight arrives more than 15 minutes late (the FAA definition of a delayed flight) from information known at scheduling time only: airline, origin and destination airports, date, scheduled departure hour, distance and scheduled flight time.

This repository holds the two model artifacts used by the Flight Delay Prediction & Risk Analysis service (Streamlit UI with SHAP explanations and an FAA-based expected-loss estimate). Live demo: https://flight-delay-askar.streamlit.app · project page: https://huggingface.co/spaces/AskarT/flight-delay-risk

Files

File	Size	Contents
model_pipeline_compact.pkl	114 MB	compact variant, served by the live demo: the same pipeline trained with max_ctr_complexity=3
model_pipeline_main.pkl	433 MB	full variant, the diploma reference model (max_ctr_complexity default = 4)

Both files are sklearn.pipeline.Pipeline = FlightFeatureEngineer (custom transformer) + CatBoostClassifier, saved with joblib.dump(..., compress=3), and share data, split, features and every other training parameter. CatBoost stores counter (CTR) tables for combinations of categorical features (ROUTE ≈ 4 200 values × DAY × MONTH × ...); limiting the combinations to three features cuts the model from 433 MB / ~1.5 GB RAM to 114 MB / ~0.6 GB RAM at a cost of 0.007 ROC-AUC. The full model was kept unchanged; the compact one was added so that the public demo fits the 1 GB free tier of Streamlit Community Cloud.

	Full model	Compact model
Training script	src/train_final.py	src/train_compact.py
ROC-AUC (408 907 hold-out flights)	0.786	0.779
Recall / precision @ 0.37	0.818 / 0.306	0.822 / 0.297
Recall / precision @ 0.50	0.660 / 0.396	0.657 / 0.386
Pipeline on disk	433 MB	114 MB
RAM after joblib.load / peak while loading	~1.5 GB / ~2.8 GB	~0.6 GB / ~0.9 GB

Smaller configurations (max_ctr_complexity 2 and 1, 5–52 MB) were also trained and lost 0.03–0.05 ROC-AUC; the comparison is in reports/metrics/compact_experiments.txt of the service repository.

Intended use

Portfolio / educational demonstration of an end-to-end ML service. The model was trained on 2015 US data; airports, airlines and traffic patterns are those of 2015. It is not an operational tool.

Training data

2015 Flight Delays and Cancellations (US Department of Transportation). First 2.1 M rows of flights.csv, 80/20 stratified split. Target: ARRIVAL_DELAY > 15 minutes, positive rate 18.9 %. Only pre-departure columns are used: MONTH, DAY, DAY_OF_WEEK, AIRLINE, ORIGIN_AIRPORT, DESTINATION_AIRPORT, SCHEDULED_DEPARTURE, SCHEDULED_ARRIVAL, SCHEDULED_TIME, DISTANCE.

Features (built inside the pipeline)

hour_sin, hour_cos (cyclic departure hour), traffic_density (departures from the origin in that month/day/hour), AIRLINE_FLIGHT_COUNT (carrier departures from the origin that day), ROUTE (origin–destination pair), plan_speed (distance / scheduled time), IS_NIGHT, IS_WEEKEND. Categorical features passed to CatBoost: MONTH, DAY, DAY_OF_WEEK, AIRLINE, ORIGIN_AIRPORT, DESTINATION_AIRPORT, ROUTE, IS_WEEKEND, IS_NIGHT.

Training

CatBoost 1.2.8 on GPU: 2500 iterations, learning rate 0.03, depth 7, l2_leaf_reg=12, scale_pos_weight=4.2, Bernoulli subsample 0.8, early stopping on AUC (100 rounds). The compact variant adds max_ctr_complexity=3; nothing else differs.

Evaluation (408 907 hold-out flights)

Metric	Full model	Compact model
ROC-AUC	0.786	0.779
Recall @ 0.37 (chosen threshold)	0.818	0.822
Precision @ 0.37	0.306	0.297
Recall / Precision @ 0.50	0.660 / 0.396	0.657 / 0.386

The decision threshold 0.37 was chosen to favour recall: a missed delay costs the airline more than a false alarm. Raw predict_proba is not calibrated.

How to use

The pickle references the custom class as src.feature_engineering.FlightFeatureEngineer, so run the snippet from the root of the service repository (or put that root on sys.path). Versions matter: Python 3.10, catboost 1.2.8, scikit-learn 1.7.2, numpy 2.2.6, pandas 2.3.3.

git clone https://github.com/Askar-Turanazarov/FlightDelay-Service.git
cd FlightDelay-Service
pip install -r requirements.txt

import os, sys, joblib, pandas as pd
from huggingface_hub import hf_hub_download

sys.path.insert(0, os.getcwd())    # repository root, so that `src.feature_engineering` resolves
path = hf_hub_download("AskarT/flight-delay-catboost", "model_pipeline_compact.pkl")  # or model_pipeline_main.pkl
pipe = joblib.load(path)

flight = pd.DataFrame([{
    "MONTH": 7, "DAY": 15, "DAY_OF_WEEK": 3, "AIRLINE": "AA",
    "ORIGIN_AIRPORT": "JFK", "DESTINATION_AIRPORT": "LAX",
    "SCHEDULED_DEPARTURE": 1800, "SCHEDULED_ARRIVAL": 2115,   # HHMM as in the source data
    "SCHEDULED_TIME": 375, "DISTANCE": 2475,
    "DEP_HOUR": 18, "ARR_HOUR": 21, "IS_WEEKEND": 0,          # derived by the caller, see src/app.py
}])
prob = pipe.predict_proba(flight)[0, 1]
print(f"P(delay > 15 min) = {prob:.3f}, flagged = {prob >= 0.37}")

DEP_HOUR / ARR_HOUR are the scheduled departure and arrival hours (0–23); IS_WEEKEND is 1 for Saturday and Sunday. The service computes them in src/app.py before calling the pipeline. The two variants give slightly different probabilities for the same flight (this example: 0.469 compact, 0.375 full); the service selects the variant with the MODEL_VARIANT environment variable (compact by default, full for the diploma model).

Limitations

• 2015 US data only; no weather, no real-time information.
• Probabilities are not calibrated; the service shows them next to the 18.9 % base rate.
• Traffic features fall back to 1 for airport/day/hour combinations unseen in training.

Author

Askar Turanazarov, Data Science diploma project, 2026. MIT license.