Nyoro012/data-on-call-demo-model

Model Card: Data On-Call v2 Risk Predictor (DEMO)

DEMO MODEL: trained on SYNTHETIC data generated by a documented rule. It has NOT been validated on real player behavior. A model trained on 54 real rows from 3 participants failed the pre-registered evaluation gate (accuracy 0.444 vs majority baseline 0.574, ROC AUC 0.42), so no model learned from real players is published. This demo exists to show the train, package, publish, consume and parity pipeline.

1. Model Summary & Purpose

• Purpose: Demonstrates the end-to-end technical pipeline (training, serialization, export to ONNX runtime, packaging, downstream serving in Node.js/web, and cross-framework probability parity).
• Model Architecture: Scikit-learn Pipeline(StandardScaler, LogisticRegression(C=1.0, max_iter=1000, random_state=42))
• Format: ONNX runtime artifact (model.onnx, float32 input input [1, 7], output probabilities)
• Artifact SHA256: 91bca9e20ad87e43f5eec7b71356a4953feaef31c13ce129fd0665d078fb5c57
• Packaging Timestamp: 2026-11-06T08:06:12.692774+00:00
• Version: v2.0-demo
• Training Source: Synthetic data (data/synthetic_dev.csv)

2. Intended Use & Non-Use

• Intended Use: Demonstration of the technical pipeline, packaging integrity, and numerical probability parity across environments.
• Intervention Trigger: Evaluated strictly at the start of a decision scene before the player acts. Predicted probability $P(\text{chose_risky} = 1) \ge 0.50$ serves as a demonstration trigger for proactive tutoring scaffolding.
• NOT FOR USE (Explicit Prohibition):
  • NOT for any real adaptive decision in actual gameplay or production.
  • NOT for any claim about real players or human decision-making behavior.
  • NOT for high-stakes grading, formal student evaluation, or automated user profiling.

3. Training Data & Features (Synthetic)

• Data Source: Synthetic dataset (data/synthetic_dev.csv) generated by scripts/make_synthetic.py with random seed 42.
• Generative Rule: Base log-odds intercept -0.85 (~46% risky rate). Risk probability rises with Experienced level (+1.3 log-odds), previous risky choices (+0.9 log-odds), and previous timeouts (+1.1 log-odds), and falls with prior hints requested (-1.3 log-odds), subject to Gaussian noise (mean 0.0, std 0.45).
• Total Samples: 300 synthetic decisions across 100 simulated sessions.
• Target Label: chose_risky (1 if the simulated player chose the risky option or timed out, 0 otherwise).
• Model Features (Known strictly at scene start):
• level
• goal
• scene_no
• prev_hints_used
• prev_risky_count
• prev_timeout_count
• prev_decision_time_sec
• Strict Anti-Leakage Rule: In-scene interaction metrics (hint_used_this_scene, decision_time_sec) describe actions during the scene and are strictly excluded from model features.

4. Evaluation & Synthetic Recovery Check

• Validation Protocol: leave-one-session-out
• Synthetic Recovery Check (Pipeline Sanity Verification):
  • Out-of-Fold Accuracy: 0.6600
  • Majority-Class Baseline: 0.5400
  • Sanity Threshold: 0.60
  • Status: PASSED (confirms that the training pipeline successfully recovers the synthetic generator's mathematical rule).
  • Crucial Note: This check only proves pipeline recovery of the known synthetic rule; it does NOT indicate real-world predictive validity.
• Real-Data Pre-Registered Gate Record:
  • Result: FAIL
  • Real Accuracy: 0.4444 vs Baseline 0.5741 (ROC AUC: 0.4215)
  • Protocol: leave-one-participant-out (54 rows, 18 sessions, 3 participants)
  • Audit Record: docs/training_run1.txt

5. Limitations & Ethical Considerations

• Synthetic Origin: Synthetic labels come entirely from our own mathematical generative rule. The model can only reproduce that programmed rule and cannot reflect genuine human learning or cognitive nuances.
• Composite Label & Timeout Dynamics: The target label chose_risky counts timeout expirations as risky choices (1), so it partly reflects timer expiration rather than purely intentional decision risk.
• Mode-Specific Features:
  • Timeouts only occur for Experienced players (level = 1), as Beginner mode has no active countdown timer.
  • Hint usage only varies for Beginner players (level = 0), as hint requests are disabled in Experienced mode.
• Small Real Sample: The real player dataset (54 rows from 3 participants) failed the pre-registered gate, demonstrating that a simple linear model cannot be claimed to predict real player choices without more data and deeper feature engineering.