Marraffini-Giovanni/fbert-1m

Model Card for F-BERT-1M

F-BERT is a small Transformer encoder for parcellated fMRI timeseries, pretrained on about 4,000 hours of resting-state and task fMRI from 162 openly available datasets. It is trained by kernel alignment to the flattened connectome: the pairwise similarities between the embeddings of short recording windows are aligned with the pairwise similarities between the recordings' functional-connectivity matrices after a spectral filter, FC^alpha = V diag(lambda^alpha) V^T with alpha = 0.35, which recalibrates the eigenvalues of each subject's connectome. The encoder reads a window of a scan and produces one embedding per recording that can be used with a frozen linear probe for cognition, sex, age, diagnosis and subject identification.

This is the 1M-parameter model. The 8M model is at Marraffini-Giovanni/fbert-8m. The code, the transform and every table of the paper are at github.com/GioMarraffini/fc-power.

Model Details

Model	Parameters	Layers	Width	Heads	FF width	Regions	Pretraining windows	Released step
F-BERT-1M	0.851M	4	128	4	512	450	80 timepoints	56,000
F-BERT-8M	8.070M	8	384	8	512	450	80 timepoints	12,000

• Input: a parcellated fMRI recording at Schaefer-400 + Tian-S3 (450 regions), band-passed to 0.01–0.08 Hz and z-scored per region, shape (batch, 450, time). Positional encoding is sinusoidal, so a model pretrained on 80-timepoint windows reads a recording of any length; a whole 57.6-minute HCP-YA scan (4,800 timepoints) is encoded as one sequence.
• Output: one embedding per recording (128-dimensional for this model), the output of a prepended summary token.
• Objective: one minus the centred kernel alignment (CKA) between the batch Gram matrix of the embeddings and the batch Gram matrix of the teacher vectors, where the teacher of a window is vech(FC^0.35) of its whole recording, centred and unit-norm.
• Developed by: the authors of Flattening the Connectome Spectrum: A Spectral Filter for FC Induces a Pretraining Target for fMRI Encoders (under review).
• License: CC BY-NC 4.0. The pretraining corpus inherits the licence of each source dataset and one of them is CC BY-NC 4.0, so the weights are released for non-commercial use.

Uses

Inference

pip install git+https://github.com/GioMarraffini/fc-power

import torch
from huggingface_hub import hf_hub_download
from fcpower.encoder import load_checkpoint

path = hf_hub_download("Marraffini-Giovanni/fbert-1m", "fbert_1m.pth")
model = load_checkpoint(path)                      # eval mode, CPU

x = torch.randn(2, 450, 80)                        # (batch, regions, timepoints), z-scored per region
with torch.no_grad():
    embeddings = model(x)                          # (2, 128)

The embedding of a recording is then used with a frozen linear probe (ridge regression or logistic regression on z-scored features); scripts/probe_checkpoint.py in the code repository is the probe used in the paper. No GPU is needed: one 57.6-minute recording takes 0.46 s on four CPU threads with this model.

Intermediate checkpoints

Every checkpoint written during pretraining (one every 2,000 steps, batch 1,024 windows) is available as a branch of this repository, named by step and by the number of windows seen, in the manner of the OLMo releases. main holds the checkpoint used in the paper (step 56,000, selected on a held-out downstream probe).

path = hf_hub_download("Marraffini-Giovanni/fbert-1m", "fbert_1m.pth", revision="step10000-windows10M")

The list of branches is under "Files and versions". Each checkpoint is a torch.save dictionary with state_dict, config (the pretraining configuration), step and loss; optimiser state is not included.

Evaluation

Frozen linear probe on HCP-YA (cognitive composite, Pearson r; sex and age, AUC) and CoRR test–retest identification accuracy, 20 x 10 family-aware cross-validation (Table 3 of the paper). FC^alpha* is the transform the encoder is distilled from; FC is the raw connectome under the same probe.

representation	input	composite	sex	age	fingerprint
FC^0.35 (kernel ridge)	full scan, 57.6 min	0.642	0.99	0.63	0.895
FC (kernel ridge)	full scan, 57.6 min	0.559	0.97	0.62	0.762
F-BERT-1M (this model)	full scan, 57.6 min	0.488	0.94	0.61	0.670
F-BERT-8M	full scan, 57.6 min	0.502	0.96	0.61	0.841
Brain-Semantoks (63M)	full scan	0.441	0.96	0.59	0.624
FC^0.35 (kernel ridge)	first 2.4 min	0.460	0.91	0.55	0.863
FC (kernel ridge)	first 2.4 min	0.359	0.85	0.52	0.569
F-BERT-1M (this model)	first 2.4 min	0.418	0.86	0.57	0.513
F-BERT-8M	first 2.4 min	0.412	0.87	0.56	0.705

On short scans and in small cohorts the encoder is above kernel ridge regression on the raw connectome for the cognitive composite; the full grid over scan length and number of subjects is in the paper's appendix.

Training

Data

The pretraining corpus is Marraffini-Giovanni/fbert-corpus: 162 OpenNeuro datasets, 9,578 subjects, 27,835 recordings (resting state and task), preprocessed with fMRIPrep, parcellated at 450 regions, band-passed and z-scored, cut into 106,397 windows of 80 timepoints. None of the evaluation cohorts (HCP-YA, AOMIC-ID1000, ABIDE-I, ADHD-200, CoRR) is in the corpus.

Hyperparameters

objective	1 − CKA between embedding and teacher Gram matrices
teacher	vech(FC^0.35) of the whole recording, centred, unit norm
optimiser	AdamW, lr 8.5e-4, weight decay 1e-4, betas 0.9/0.999
schedule	100 warm-up steps, cosine decay over 300,000 steps
batch	1,024 windows from 1,024 distinct recordings
window	80 timepoints
precision	fp16 autocast, gradient clipping at 1
seed	0
checkpoint	every 2,000 steps; step 56,000 released as main

Bias, Risks, and Limitations

The corpus and the evaluation cohorts over-represent Western, healthy, well-educated adults; the encoder inherits those biases and needs re-validation before any clinical use. The connectome-derived embedding is a strong fingerprint of a subject across sessions; identification was evaluated only within consenting research cohorts and the model cannot link a scan to any external identity. The model is meant to be used frozen, on small cohorts and short scans; fine-tuning is outside the scope of its evaluation.

Citation

A preprint is in preparation; the citation will be added here when it is available.

@article{fbert2026,
  title   = {Flattening the Connectome Spectrum: A Spectral Filter for FC Induces a Pretraining Target for fMRI Encoders},
  year    = {2026},
  note    = {Preprint to appear}
}

Model Card Contact

Open an issue on the code repository.