synthea-bert

An encoder-only, BERT-style masked model over sequences of health events, trained on a synthetic Synthea population. Teaching material for Transformer Architectures for Computational Social Science (ICSC 2026, Oxford).

Do not load this with AutoModelForMaskedLM

The model adds Time2Vec time encodings to the input embeddings. Loading the inner BertForMaskedLM directly skips them, and the model then returns confident nonsense while the loss still looks fine. Use the loader shipped in this repo:

import sys

import pandas as pd
from huggingface_hub import snapshot_download

model_dir = snapshot_download("carlomarxx/synthea-bert")
data_dir = snapshot_download("carlomarxx/synthea-workshop-data", repo_type="dataset")

sys.path.insert(0, model_dir)  # event_bert.py ships alongside the weights
from event_bert import EventBertForMaskedLM

vocabulary = pd.read_csv(f"{data_dir}/vocabulary.csv")
model = EventBertForMaskedLM.from_pretrained(
    model_dir, expected_vocab_size=len(vocabulary)
)

Vocabulary

1036 tokens: 5 special, 46 background and the rest event types. The background tokens are a person's static attributes, placed at the head of the sequence following life2vec rather than bolted on downstream as features: 6 age bands, 2 sex, 6 race, 2 ethnicity, 5 marital status, and 25 equal-frequency income bins. The vocabulary lives in carlomarxx/synthea-workshop-data and the two are only valid as a pair — expected_vocab_size asserts this.

What it learned

Synthea is a rule-based simulator, so structure recovered from it reflects generator rules, not epidemiology. Measured consequence: on scalar outcomes this model ties or loses to a tabular baseline of age plus record volume. On next-event prediction it wins decisively (0.619 top-1 against 0.303 for a bigram baseline), because the generator wrote care pathways — vaccination schedules, dental bundles, screening batteries — and pathways are what it learned.

Published from carlomarxdk/workshop-transformers at commit ffae814.