PhysioWave-T-ECG: multi-route pretrained ECG encoder

One of the three PhysioWave-T encoders (EEG, ECG, sEMG); the RALF fusion model combines them. Code: https://github.com/ForeverBlue816/PhysioWave.

Masked dual-reconstruction pretraining on 12-lead and single-lead ECG. One shared transformer serves two lead configurations through route-specific wavelet frontends, and lead identity enters as a learned embedding of the lead NAME rather than of its index.

Trained for 57 epochs of 60 configured (57000 optimizer updates) on a mixture of public ECG corpora, 30.1 M parameters. The files here are built around pretrain_ecg_c1_moe_best.pth, epoch 33.

Objective

0.5*L_spec + 0.5*L_raw + 0.001*L_foldKL

Two heads at equal weight. One predicts folded wavelet patches, the other predicts the preprocessed ECG waveform of the same masked patches. Masking is applied BEFORE the wavelet frontend (True), at ratio 0.7, and the spec target is per-patch normalised (True).

Training curves

Pretraining ran for 60 epochs (60,000 optimizer updates) on 16 GPUs over the seven corpora. The validation loss reaches its plateau by about 20,000 steps and stays there while training continues to improve; the released checkpoint (epoch 33, the lowest validation total loss) lies in that plateau, and the four per-criterion checkpoints published beside it are within validation noise of one another.

Train and validation per epoch: the total loss, the two reconstruction terms (folded wavelet patch, raw waveform) and the masked spectral correlation.

Validation, at the released checkpoint (epoch 33)

metric	value	what 1.0 would mean
total loss	0.1827	—
spec MSE	0.2562	—
spec NMSE	0.256	predicting zero
spec Pearson r	0.851	perfect
raw SmoothL1	0.1092	—
raw NMSE	0.255	predicting zero
raw Pearson r	0.845	perfect
route-macro total	0.2041	—

NMSE is dimensionless: MSE(prediction, target) / MSE(0, target), so 1.0 is what predicting zero scores on that head's own target. The two losses are NOT comparable to each other -- one is an MSE on normalised wavelet coefficients, the other a SmoothL1 on the window-normalised waveform. The columns that compare the heads are the correlations and the NMSEs.

The route-macro row weights the two routes equally. The plain rows weight by sample count, and validation is dominated by the largest corpora, so a model specialising toward the smaller ones moves the two rows in opposite directions.

Files

Which file do you want

your situation	download
fine-tune on any montage	ecg_c1_encoder.pth
your montage IS one of the two routes below	pretrain_ecg_c1_moe_best.pth, then cut the route out (one command, below)
continue pretraining, or take the model apart yourself	pretrain_ecg_c1_moe_best.pth

pretrain_ecg_c1_moe_best.pth is the complete model -- all two wavelet frontends, the patchers, both reconstruction decoders. Nothing is held back; only the optimiser state was stripped, which no downstream use needs. Take it if you want to keep or strip pieces yourself.

ecg_c1_encoder.pth is the same weights minus everything route-bound and pretraining-only. It exists because it cannot be loaded wrong.

file	what it is
ecg_c1_encoder.pth	encoder alone: channel embedding, gate, transformer
pretrain_ecg_c1_moe_best.pth	complete model, lowest validation total loss (epoch 33)
pretrain_ecg_c1_moe_final.pth	complete model, last epoch (epoch 57)
pretrain_ecg_c1_moe_best_spec.pth	complete model, lowest spec MSE (epoch 19)
pretrain_ecg_c1_moe_best_raw.pth	complete model, lowest raw SmoothL1 (epoch 37)
pretrain_ecg_c1_moe_best_macro.pth	complete model, lowest route-macro total (epoch 28)
metrics_epoch.jsonl	the full training curve
config_resolved.yaml	every resolved hyperparameter

Why several checkpoints

Each is the minimum of a different validation metric, and they are published together so a downstream comparison can ask which criterion transfers instead of assuming the one that names best.pth. Reconstruction loss cannot answer that question; a downstream task can.

best.pth is selected on the TOTAL loss -- the loss that was trained (epoch 32 counted from 0). It is not the best spec loss, and not the best raw loss.

What is in the complete model

component	parameters	share
shared transformer	28.38 M	94.3%	encoder, always transfers
channel-name embedding	0.00 M	0.0%	always transfers
channel projection	0.03 M	0.1%	always transfers
wavelet frontends (all two routes)	0.03 M	0.1%	route-bound; downstream usually builds its own
patchers (both rates)	0.20 M	0.6%	route-bound
spec decoder	0.72 M	2.4%	pretraining only
raw decoder	0.72 M	2.4%	pretraining only
total	30.08 M	100%

The shared transformer is 94% of the model and the two frontends together are 0%, which is why there is no per-route encoder file here: two of them would duplicate that transformer two times to deliver frontends that are a small fraction of it. Cut the one you want out of the complete checkpoint instead.

What transfers, and what does not

downstream.py splits a checkpoint into two sets. The channel embedding, its gate and the shared transformer always load. The wavelet frontend and the patcher load only when your montage is the route they were trained for -- same channel count, same patch length -- and are skipped otherwise rather than reshaped.

A 12-lead recording at 500 Hz in 10 s windows IS the L12_500 route, so on the most common ECG input the pretrained frontend and patcher transfer as well: cut that route out (below) and pass route_id="L12_500". Any other lead set, rate or window length builds its own.

The two routes the shared transformer was pretrained across:

route	channels	rate
L12_500	12	500 Hz
L1_250	1	250 Hz

To fine-tune ON one of them exactly, take that route's frontend and patcher out of the complete checkpoint:

python scripts/export_eeg_pretrained_encoder.py \
    --checkpoint pretrain_ecg_c1_moe_best.pth --route L12_500 \
    --output encoder_L12_500.pth

The exporter reads the modality from the checkpoint; its name is historical. Omit --route and you get ecg_c1_encoder.pth again. Both pretraining decoders are dropped either way: fine-tuning against the pretext objective by accident is the failure that prevents.

The released checkpoint and the last epoch differ by 2.6% in validation total loss (0.1827 against 0.1877). Which transfers better is a downstream question, not a reconstruction-loss question -- measure both.

The channel vocabulary travels with the weights

An embedding row means whatever channel held that id when the row was learned. Every checkpoint here carries the vocabulary and its hash. Resolving lead names against a different vocabulary silently trains on relabelled channels, which is why the hash is checked rather than assumed.

Architecture

embed_dim   512      depth 9      heads 8
norm        rmsnorm     ffn swiglu     qk_norm True
wavelet     max_level 3, kernel 16, bank db4, db6, sym4, sym8, coif2
channel     id embedding, injected at the token site
mask        frequency_guided, ratio 0.7

Loading

ECGC1Downstream builds the frontend and patcher for your montage and loads this file into everything else. Nothing here is route-bound, so the montage, the window length and the patch length are yours to choose.

import torch
from physiowave.ecg_c1.downstream import ECGC1Downstream

ENC = "ecg_c1_encoder.pth"
ck = torch.load(ENC, map_location="cpu", weights_only=False)
mc = ck["model_config"]                  # the width the weights were trained at
CH = ["I", "II", "III", "aVR", "aVL", "aVF",
      "V1", "V2", "V3", "V4", "V5", "V6"]   # your leads

model = ECGC1Downstream(
    in_channels=len(CH), window_samples=5000, sampling_rate=500.0,
    patch_samples=250, num_classes=2, channel_names=CH,
    embed_dim=mc["embed_dim"], depth=mc["depth"], num_heads=mc["num_heads"],
    channel_embed_dim=mc.get("channel_embed_dim", 64),
    channel_vocab_size=ck["channel_vocab_size"])

report = model.load_pretrained(ENC)
print(model.describe_transfer(report))
# loaded: transformer N, channel encoder N, gate yes, frontend 0, patcher 0

class Meta:                               # or your dataset's ChannelMeta
    channel_names = CH
    channel_mask = None

out = model(torch.randn(2, len(CH), 5000), Meta())   # {"logits": [2, 2]}

frontend 0, patcher 0 is the expected line, not a warning: those are the two things your montage supplies.

The frontend you build is not randomly initialised. Its filters are seeded from real wavelet families -- db4, db6, sym4, sym8, coif2 -- and learned from there during fine-tuning. What a fresh frontend costs you is the adaptation those filters underwent during pretraining, not the wavelet prior itself.

Names, not indices, are what this encoder and your data agree on. Pass real lead names and check channel_vocab_sha256 -- load_pretrained refuses a mismatch rather than silently training on relabelled channels.

Code: https://github.com/ForeverBlue816/PhysioWave

---

Prepared by scripts/prepare_hf_release.py from pretrain_ecg_c1_moe on 2026-10-29.