alina-afanaseva/rl-mistake-analysis-models

Rocket League mistake detection models

Versioned models for identifying potential player mistakes in Rocket League replays. This repository stores releases for different mistake kinds, with separate versions and evaluation reports for each model.

The rl-mistake-analysis project identifies candidate incidents, extracts features, and runs inference. Replay-analysis applications can use the predictions for review and coaching.

A candidate is an incident worth reviewing. Features are measurements describing it. A model uses those measurements to estimate whether it is a mistake. Different mistake kinds can use different models.

Model catalog

The root manifest.json lists available mistake kinds and model versions. Each release has its own directory:

manifest.json
models/
  <kind>/
    <version>/
      model.onnx
      evaluation.json

The manifest records candidate and feature contracts, thresholds, evaluation paths, checksums, source provenance, and any separate ONNX weight files. New versions get new directories; released paths stay stable. Catalog and model versions are independent. Downstream applications can choose which release to activate.

Recommended models

Mistake kind	Version	Model	Features	Validation F1	Test F1
Teammate bumps	0.1.0	XGBoost	90	69.30%	73.22%
Banging the ball away with time	0.3.0	Extra Trees	141	64.84%	68.47%

See the changelog for release details, classifier comparisons, and earlier versions.

Using a model

Choose a mistake kind and version from the catalog. With the Hugging Face CLI installed, download that release:

hf download alina-afanaseva/rl-mistake-analysis-models \
  --include "manifest.json" \
  --include "models/<kind>/<version>/*" \
  --local-dir "/path/to/model-repository"

Replace <kind>, <version>, and /path/to/model-repository with your chosen release and destination. This downloads the catalog and all files for that release. Pass the directory, kind, and version to the rl-mistake-analysis inference library. It uses ONNX Runtime and the release's decision threshold.

Mistake kinds

Teammate bumps (bumping_teammate)

This mistake kind uses the play around a teammate collision to judge it from each player's perspective. Each participant is scored separately. Being the physical initiator does not automatically make a player responsible.

Features describe motion, positioning, ball touches, possession, and surrounding events. See the teammate-bump documentation for the mistake definition and feature details.

Banging the ball away with time (bang_with_time)

This mistake kind judges whether a player hit the ball away despite having time to control it or choose a more useful touch. Features describe the candidate touch, nearby players, possession, and subsequent touches and events. Each release's evaluation report records its ordered feature definitions and measurement sources.

Release 0.3.0 uses candidates whose next distinct contact includes an opponent and excludes fifties at both contacts. Its candidate contract is bang_with_time.candidates.v2; its numeric input schema is bang_with_time.context.v1.touch_context.v1. Candidate eligibility and measurement definitions have independent versions. Use rl-mistake-analysis 0.2.0 or later to read the candidate metadata and extract its features. The catalog format remains schema version 1.

See the bang documentation for the mistake definition and feature details.

The dataset combines current manual reviews with legacy reviews; current reviews take precedence where an incident has been reviewed again.

Training and evaluation

Human-reviewed incidents provide binary mistake labels for the relevant player. Ten classifier families are compared: CatBoost, XGBoost, Random Forest, Extra Trees, an Explainable Boosting Machine, two logistic regression variants, and linear, RBF, and polynomial SVMs.

Cross-validation keeps whole replays together, including both participants in teammate collisions. Preprocessing is learned within training folds. Parameters, class weighting, and probability thresholds from 0.40 to 0.95 are selected by mean replay-grouped validation F1. Reserved test replays do not select parameters or thresholds. SVM probability calibration also keeps replays together.

Unweighted training is compared with mild and full inverse-frequency class weighting. Mild weighting uses the square root of inverse class frequency; full weighting uses inverse class frequency. Weights are computed separately from each fit's training labels and normalized to mean one.

Precision measures how many flags are correct; recall measures how many labeled mistakes are found. F1 balances the two. Each release's evaluation.json records its scores and evaluation setup. DVC preserves data snapshots, and MLflow records experiments. ONNX probabilities are checked against the original models on training and test rows.

Limitations

• Both models use events after the candidate incident and require retrospective replay analysis.
• Both models' labels come from one human annotator.
• Replay data and annotations are private.

License

The model artifacts and documentation in this repository are released under the MIT License.