KneeSplatNet (Res-B): a 3D knee-frame branch on a 2.5D ConvNeXt

KneeSplatNet is a research architecture from a solo entry to the Kaggle competition RSNA 2026 Knee Abnormality Detection. It gives a 2.5D slice model an explicit 3D frame of the knee. This repository holds the residual variant "Res-B": 8 fold models, weights and inference code.

It is not part of the scored final. It matched its 2.5D twin overall and failed its pre-registered inclusion gate (see Results). The scored ensemble is Volkopat/rsna-knee-2026-c9cu, which also documents the shared base model, preprocessing and training data in more detail.

Research use only. Not for clinical use or diagnosis. Not reviewed or cleared by the FDA or any other regulator. Trained on labels produced by language models, not by radiologists. The competition data terms (RSNA MIRA) allow research use only.

The idea

A 2.5D model looks at each 3-slice stack on its own. It has to learn from data that a sagittal slice and a coronal slice can show the same piece of anatomy. KneeSplatNet uses the scanner geometry stored in the DICOM headers to place every image feature at its 3D position in the knee, and then reads the knee as one small 3D volume.

The design is inspired by Gaussian splatting, which builds a 3D scene from many small elements placed in space. Here features from every slice of every series are "splatted" (spread with trilinear weights) into one shared 3D grid.

1. A canonical frame per knee. The origin is a joint-centre landmark: the centre of the 3D overlap of the sagittal and coronal fat-suppressed volumes, moved 11.3 mm inferior (a calibrated offset). The axes are x = lateral, y = anterior, z = superior. Left knees are mirrored, so lateral is +x for both sides and left and right knees line up. Every feature cell gets its 3D position from the slice's ImagePositionPatient, ImageOrientationPatient and PixelSpacing.
2. Neck. For each 3-slice stack, ConvNeXt-Tiny stage-3 features (stride 16) and upsampled stage-4 features (stride 32) are projected to 384 channels per stride-16 cell (24 x 24 cells per 384-px image). A geometry embedding (3D position, Fourier terms, slice normal, fat-suppression flag, slice index, slice spacing) and a slice-position embedding are added.
3. Splat into a 3D grid. Each cell's feature, reduced to 64 channels, is splatted into a 12 x 12 x 12 grid with 8 mm cells (-44 mm to +44 mm around the joint centre on each axis). There are separate banks for fat-suppressed and other series, a medial-lateral mirror-difference volume, and hit masks (194 input channels in all).
4. A small 3D network. A 1 x 1 x 1 convolution to 128 channels and two 3D ConvNeXt-style blocks (depthwise 3 x 3 x 3 convolution, LayerNorm, MLP) read the grid.
5. Per-finding readout. For each finding, four Gaussian sub-queries centred on anatomical anchors (a fixed table of positions in mm, plus learned offsets) pool the grid into one vector, and a linear head gives the 3D logit z_3d.
6. Res-B: a residual branch. The unchanged 2.5D model of the c9cu release (ConvNeXt-Tiny trunk with per-finding attention pooling) gives z_prod. The output is z = (1 - a_f) * z_prod + a_f * z_3d with a learned weight per finding, a_f = 0.65 * sigmoid(theta_f). After training, a_f is 0.12-0.14 for ACL, Medial Meniscus, Medial OA, Effusion and Baker's, and 0.30-0.34 for MCL, Lateral Meniscus, Lateral OA, PF OA, Synovitis, Contusion and Fracture (range over the 8 folds).

Each member has 29.81 M parameters: 28.25 M in the 2.5D base model and 1.56 M in the KneeSplatNet modules. (The full KneeSplatNet design also has an anatomical-anchor pooling path over the 2D cells; Res-B skips it.)

What it predicts and its input

The same 12 findings as the competition, as probabilities: ACL, MCL, Medial Meniscus, Lateral Meniscus, Medial OA, Lateral OA, PF OA, Effusion, Synovitis, Baker's, Contusion, Fracture.

Input: a knee MRI study as DICOM series, organised like the competition data, with the same 6 series slots and preprocessing as the c9cu models (24 slice triplets per series, 384 px, 130 mm central crop). In addition the model takes positions, a (batch, 6, 24, 10) tensor per study: for each slice triplet, the 3D affine of its feature grid in the canonical frame (units of 65 mm) and a validity flag. model/ks_infer.py computes it from the DICOM headers.

Training

• Same competition training set as the c9cu models (4,407 studies), no external imaging data, trunk initialised from timm's ImageNet convnext_tiny weights.
• Labels: table h13m, soft labels produced by large language model readers of the radiology reports and image views: the staged per-finding reader's table (h6v2) with Synovitis, Lateral OA, Lateral Meniscus, PF OA and MCL replaced or blended with dedicated re-reads, and Contusion blended with out-of-fold CNN predictions. Labels are not released.
• Recipe: the production recipe of the c9cu base models (15 epochs, weight EMA 0.999, stochastic depth 0.2), seed 13, 8 folds (one model per fold). The KneeSplatNet modules train at 1.5 x the head's learning rate.

Results

Measured on held-out folds 0-2 (795 studies) with within-fold ROC AUC against language-model labels (table h13m), not radiologist labels. For 8 findings only studies whose report explicitly states the finding are scored; ACL, Effusion, Synovitis and Contusion use all studies. The control is the 2.5D twin: the same base model trained with the same labels, folds, seed and recipe, without the 3D branch.

Model (folds 0-2)	Macro AUC
KneeSplatNet Res-B (as released)	0.9408
Same checkpoints, 3D branch switched off (2.5D head only)	0.9391
Same checkpoints, 3D branch only	0.9281
2.5D twin (control)	0.9381

Res-B minus control, per finding:

ACL	MCL	Medial Meniscus	Lateral Meniscus	Medial OA	Lateral OA	PF OA	Effusion	Synovitis	Baker's	Contusion	Fracture
-0.030	-0.003	-0.004	-0.005	-0.001	+0.003	+0.013	-0.009	+0.005	+0.014	+0.027	+0.022

What this shows:

• Overall it matched its 2.5D twin (+0.0027 macro here; +0.0045 when scored against the mean of four language-model label tables).
• Switching the 3D path off lowered the score (-0.0017 macro), so the model does use it.
• It moved accuracy between findings: better on PF OA, Baker's cyst, contusion and fracture; worse on the ACL, effusion, and slightly on the menisci and MCL.
• Mixed at 15 % into the ensemble of that time, it added nothing over the same mix with the twin (-0.0001 macro).

It therefore failed its pre-registered inclusion gate: at least +0.0010 macro and +0.0020 on five weak findings (Synovitis, Contusion, Lateral OA, Lateral Meniscus, PF OA) over its own 2.5D head (observed +0.0017 and +0.0010), and at least +0.0008 macro in the ensemble mix (observed -0.0001). It was not used in the scored final. Folds 4-8 were trained after the gate; all 8 folds are released.

How to run

Download (about 0.96 GB):

pip install -U huggingface_hub
hf download Volkopat/kneesplatnet --local-dir kneesplatnet
cd kneesplatnet
pip install -r requirements.txt

One member, dummy input (CPU, ~12 s). Prints the blended output, z_prod, z_3d and the mixing weight per finding:

python examples/load_and_predict.py --cpu

All 8 members on competition-style data (GPU recommended):

python examples/load_and_predict.py --data-dir /path/to/data --out ks_predictions.csv

In code:

import json, os, sys, torch
sys.path[:0] = ["model", "code"]                 # model/ first: kneesplat_patch, geo_patch, closeup_infer, ks_infer
for k, v in json.load(open("model/KS_MANIFEST.json"))["env"].items():
    os.environ[k] = str(v)                       # KNEE_KS_RESB=1, KNEE_KS_D=384, ... (read when the model is built)
import kneesplat_patch
from model import KneeNet
Net = kneesplat_patch.make_ks_class(KneeNet)
ck = torch.load("model/w_ksresb8_f0.pt", map_location="cpu", weights_only=True)
net = Net(ck["backbone"], ck["n_targets"], ck["n_slots"], pretrained=False, img_size=ck["size"], k=ck["k"],
          slice_pos=ck["slice_pos"], target_query=ck["target_query"], in_chans=3, pool=ck["pool"], spatial=ck["spatial"])
net.load_state_dict(ck["state_dict"], strict=True); net.eval()
# logits = net(x, mask, positions=pos)[0]   x: (B, 6, 24, 3, 384, 384) in [0, 1]; mask: (B, 6); pos: (B, 6, 24, 10)
# net._ks_eval then holds (z_prod, z_3d) from the same forward pass

model/ks_infer.py renders a study with the base pipeline, computes the canonical positions and averages the 8 members (predict_ks). code/ks_infer_v2.py is the hardened multi-GPU version that ran in the Kaggle kernels; when a study has no valid canonical position it falls back to the 2.5D head output.

Files

Path	Content
model/w_ksresb8_f<fold>.pt	8 fold models (folds 0, 1, 2, 4, 5, 6, 7, 8), ~119 MB each
model/KS_MANIFEST.json	members, sha256, the environment settings that select the Res-B variant
model/kneesplat_patch.py	the KneeSplatNet model (make_ks_class); it also holds training-time patch text for a training script that is not included
model/geo_patch.py, model/closeup_infer.py	geometry helpers (joint-centre landmark, per-series DICOM geometry)
model/ks_infer.py	study-level inference
code/	fast_infer2.py, loader.py, decode_v2.py, model.py (identical to code/ in the c9cu release), ks_infer_v2.py, CODE_MANIFEST.json
examples/load_and_predict.py	portable example
SHA256SUMS.txt	sha256 of every file

Checkpoints are plain PyTorch dictionaries that load with torch.load(path, weights_only=True): state_dict plus the 9 settings the loader reads (backbone, n_targets, n_slots, size, k, slice_pos, target_query, pool, spatial). Training metadata was removed before release; the tensors are byte-identical to the trained checkpoints.

Requirements

Python 3.10+, torch (tested 2.8.0), timm (tested 1.0.22), numpy, pandas, opencv-python-headless, pydicom==3.0.2. See requirements.txt.

Limitations

• Research model that did not beat its baseline overall. Its value is the idea and the per-finding shifts, not a better score.
• Labels: language-model readings of reports and images, not radiologist annotations. All numbers above are against such labels.
• Geometry: needs correct DICOM position and orientation tags and both sagittal and coronal fat-suppressed series for the joint-centre landmark (there are fallbacks). Errors in the landmark or the knee side shift every feature in the 3D grid.
• Data: a single challenge dataset; behaviour elsewhere is unknown. Not evaluated for any clinical task.

Intended use

Research on knee MRI analysis and on geometry-aware architectures. Not for clinical use, diagnosis or patient management.

Data access

No images, DICOM files, reports or labels are included. Get the data from the competition page under its rules and the RSNA data terms (research use only; no redistribution).

License

CC BY-NC 4.0 (Creative Commons Attribution-NonCommercial 4.0 International), for the weights and code in this repository. This is the license type of the competition's winner license, and the RSNA data terms do not allow monetization. The timm library and the ConvNeXt ImageNet weights the trunk started from are Apache-2.0.

Citation

@misc{patil2026kneesplatnet,
  author       = {Digvijay Patil},
  title        = {73rd Solo Silver: The KneeSplatNet},
  howpublished = {Kaggle write-up, RSNA 2026 Knee Abnormality Detection},
  year         = {2026},
  url          = {https://www.kaggle.com/competitions/rsna-knee-abnormality-detection/writeups/kneesplatnet}
}

Author

Volkopat

Acknowledgments

• RSNA and Kaggle, for the challenge and the data.
• University at Buffalo Information Technology (UBIT), for GPU compute.
• AI coding agents (Claude Code) helped build and run this project under the author's direction.