Skill: mark-stadtmueller/superwise-drift-detection-skill (version 1.0.5)

--- name: superwise-drift-detection-skill description: > Detects feature drift in tabular ML models using Superwise Compare Distribution policies (Jensen-Shannon divergence for categorical columns). Handles everything: Superwise dataset creation, training data upload, drift policy setup, inference ingestion, and Telegram alerts via OpenClaw's existing Telegram connection. Use when: user says "set up drift detection", "detect model drift", "monitor my model for drift", "check if my model is drifting", or "add drift detection to my model". --- # Superwise Drift Detection Skill When the user wants to set up drift detection for their model, guide them through the steps below in order. Complete each step before moving to the next. ## What You Need From the User Ask the user to provide the following before starting: 1. **Superwise credentials** — `SUPERWISE_CLIENT_ID` and `SUPERWISE_SECRET_TOKEN` (found in their Superwise account under Settings → API Keys) 2. **Training CSV** — the CSV file their model was trained on (feature columns only; a `row_id` column will be added automatically) 3. **Model name** — a short label with no spaces (e.g. `my_churn_model`); used to name datasets in Superwise 4. **Inference endpoint** — one of two options: - **Option A**: They already have a running model endpoint. Ask for the URL. It must return JSON shaped as `{"records": [{...}, ...]}` where each record contains the same columns as the training CSV. - **Option B**: They need to deploy one. Use the dc-bikeshare example in `examples/dc-bikeshare-drift/app.py` as a template. Guide them to adapt `predict.py` for their model, then run `python app.py` locally. Set `INFERENCE_ENDPOINT_URL=http://localhost:5001/predict` for local testing. ## Step 1 — Install dependencies ```bash pip install -r requirements.txt ``` Create a `.env` file in the skill root directory with the following contents, filling in the values the user provided: ``` SUPERWISE_CLIENT_ID=<from user> SUPERWISE_SECRET_TOKEN=<from user> MODEL_NAME=<model name> INFERENCE_ENDPOINT_URL=<from user's answer to question 4> SUPERWISE_AUTO_TRIGGER=true SUPERWISE_TRAINING_DATASET_ID= SUPERWISE_TRAINING_DATASET_NAME= SUPERWISE_TRAINING_CUBE_NAME= SUPERWISE_INFERENCE_DATASET_ID= SUPERWISE_INFERENCE_DATASET_NAME= SUPERWISE_INFERENCE_CUBE_NAME= SUPERWISE_DRIFT_POLICY_ID= # Additional per-column policies — add one line per column printed by setup_drift_policy.py: # SUPERWISE_DRIFT_POLICY_ID_<column>=<policy ID> SCHEDULE_HOUR_UTC=6 SCHEDULE_MINUTE_UTC=0 PORT=5000 ``` The dataset and policy ID fields will be filled in after Steps 2 and 3. ## Step 2 — Create Superwise datasets and upload training data ```bash python setup_dataset.py \ --training-csv path/to/training.csv \ --model-name their_model_name ``` This creates a training dataset and an inference dataset in Superwise, uploads the training CSV row by row, and prints the dataset IDs and cube names. Copy the printed values into `.env`: ``` SUPERWISE_TRAINING_DATASET_ID=<printed value> SUPERWISE_TRAINING_DATASET_NAME=<printed value> SUPERWISE_TRAINING_CUBE_NAME=<printed value> SUPERWISE_INFERENCE_DATASET_ID=<printed value> SUPERWISE_INFERENCE_DATASET_NAME=<printed value> SUPERWISE_INFERENCE_CUBE_NAME=<printed value> ``` Note: training CSV column names must use only letters, numbers, and underscores, and must start with a letter. Warn the user if any column names violate this. ## Step 3 — Create drift policies ```bash python setup_drift_policy.py \ --training-csv path/to/training.csv \ --policy-name their_model_drift ``` This creates one Jensen-Shannon divergence policy per categorical (string/boolean) column in the training CSV, comparing the training dataset against the inference dataset. Numeric columns are skipped with a warning — this is a known Superwise platform limitation (wasserstein distance for numeric columns is not yet supported). Copy the primary printed policy ID into `.env`: ``` SUPERWISE_DRIFT_POLICY_ID=<primary policy ID> SUPERWISE_DRIFT_POLICY_ID_<column2>=<policy ID> SUPERWISE_DRIFT_POLICY_ID_<column3>=<policy ID> # ... one line per additional column policy printed by setup_drift_policy.py ``` Copy all printed policy lines (not just the first) — `_check_drift()` will trigger and check every `SUPERWISE_DRIFT_POLICY_ID*` env var automatically. If the user wants to monitor only specific columns, add `--columns col1 col2 col3`. ## Step 4 — Run an end-to-end drift check ```bash python -c " import os; os.chdir('.') from dotenv import load_dotenv; load_dotenv() from skill import run result = run() print(result) " ``` This will: 1. Fetch inference records from `INFERENCE_ENDPOINT_URL` 2. Ingest them into the Superwise inference dataset 3. Trigger all configured drift policy evaluations 4. Poll until each evaluation completes 5. Send an alert via OpenClaw's Telegram connection if drift is detected 6. Print a summary **Note:** The full check can take 5-10 minutes (inference fetch + ingest + policy polling across all columns). Run it in a background or PTY session to avoid exec timeouts: ```bash nohup python -c "..." &> drift_check.log & ``` ## Step 5 — Schedule recurring drift checks Register the skill with OpenClaw using the metadata in `skill.py`: - **Trigger command:** `/drift_check` - **Schedule:** `0 6 * * *` (06:00 UTC daily — adjust to the user's inference cadence) - **Alerts:** routed through OpenClaw's existing Telegram connection For a production deployment, guide the user to deploy `scheduler.py` to Render using the included `render.yaml`. Set all `.env` values as Render environment variables. ## Telegram Alerts This skill uses OpenClaw's existing Telegram connection — no separate bot setup needed. The `_send_telegram()` function in `skill.py` uses `TELEGRAM_BOT_TOKEN` and `TELEGRAM_CHAT_ID` from the user's OpenClaw environment. If the user does not have Telegram set up in OpenClaw yet, point them to OpenClaw's Telegram setup docs. Drift alerts look like: - **Unhealthy:** lists features with high drift (score > 0.2), flags that predictions may be unreliable - **Healthy:** confirms no significant drift with per-feature scores ## Reference Example: DC Bikeshare `examples/dc-bikeshare-drift/` is a complete working example the user can run to verify the skill end-to-end before connecting their own model: ```bash cd examples/dc-bikeshare-drift pip install -r requirements.txt python collect_training_data.py # fetch real station data, generate synthetic history python train.py # train RandomForest classifier python app.py # start local Flask inference server on port 5001 ``` Then run the drift check from the skill root with: ``` INFERENCE_ENDPOINT_URL=http://localhost:5001/predict ``` This example uses only categorical features (station size, e-bike presence, hour of day, day type, season) to predict bike availability — all suitable for JSD drift detection. ## Retraining When the user retrains their model: ```bash python setup_dataset.py \ --training-csv path/to/new_training.csv \ --model-name their_model_name \ --retrain ``` This creates a versioned replacement training dataset and preserves the inference history. Update `.env` with the new training dataset ID and cube name, then re-run `setup_drift_policy.py` to create a new policy pointing at the updated baseline. ## Troubleshooting - **Policy stays `pending` forever:** Check that inference records were successfully ingested before triggering. The inference dataset must have data for the policy to evaluate. - **`unhealthy` on first run:** Expected if inference records are a single time-slice snapshot (all same hour/day/season). Accumulate inference records over time for a more representative distribution. - **Numeric columns skipped:** Wasserstein distance for numeric dimensions is a known Superwise platform bug. Use categorical bucketing as a workaround, or wait for the Superwise platform fix. - **CubeJS cube not found after upload:** New datasets take a few minutes to register in Superwise's CubeJS layer. Wait 2–3 minutes and retry verification. - **Drift check killed / no output (SIGTERM):** The full check takes 5-10 minutes and will be killed by any exec timeout in your environment. Run it in a background session: `nohup python -c "from skill import run; run()" &> drift_check.log &` Then `tail -f drift_check.log` to follow progress.


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)