BiscuitBobby/word-to-eeg-generator

word-to-EEG generator (THINGS-EEG)

This generates 63-channel EEG from text: a word, a one-sentence description of a scene, or both. The models are ridge regressions from CLIP ViT-B/32 text vectors to the EEG of 10 subjects in THINGS-EEG (Haitao999/things-eeg, Preprocessed_data_250Hz_whiten), with realistic noise models on top.

Files

file	size	contents
wte.py	8 KB	inference code. Needs only numpy + sentence-transformers
model.npz	104 MB	word → EEG means (pooled and per subject), training-session noise
model_test.npz	122 MB	test-session noise adapter (4-session mixture)
photo_model.npz	14 MB	description → EEG and word+description → EEG means

Use it directly from Hugging Face

1. Install (Python 3.9+):

pip install numpy sentence-transformers huggingface_hub

2. Log in. This repo is private, so you need to log in once per machine (or set the HF_TOKEN environment variable):

hf auth login

3. Load and generate. This is copy-paste ready. The first run downloads the code and models (240 MB) and the CLIP text encoder (600 MB); both are cached afterwards. A CPU is enough.

import sys
from huggingface_hub import snapshot_download

root = snapshot_download("BiscuitBobby/word-to-eeg-generator")   # code + models, cached
sys.path.insert(0, root)
from wte import Generator, CHANNELS

g = Generator(root)

# word only
eeg = g.sample(word="dog", level="mean")

# word + scene description: the most informative input
desc = "A black dog lies on green grass in sunlight, facing left, a wooden fence behind it."
eeg = g.sample(word="dog", description=desc, level="mean")

# description only, in subject 3's style, with realistic single-trial noise
eeg = g.sample(description=desc, subject="sub-03", level="trial_test", n=5, seed=0)

print(eeg.shape)      # (5, 63, 80) = samples x channels x time points
print(CHANNELS[:5])   # ['Fp1', 'Fp2', 'AF7', 'AF3', 'AFz']

4. Use the output.

import numpy as np

np.save("dog_eeg.npy", eeg)                          # float32, (n, 63, 80)
times_ms = (np.arange(18, 98) / 250 - 0.2) * 1000    # dataset time axis (see Output below)
oz = eeg[0, CHANNELS.index("Oz")]                    # channel Oz of the first sample

Optional plot:

import matplotlib.pyplot as plt

plt.plot(times_ms, oz)
plt.xlabel("time (ms)"); plt.ylabel("whitened units"); plt.title("Oz, 'dog'")
plt.show()

Command line. Download once, then run the script:

hf download BiscuitBobby/word-to-eeg-generator --local-dir word-to-eeg-generator
python word-to-eeg-generator/wte.py dog -d "A black dog lies on green grass in sunlight, facing left, a wooden fence behind it." -s sub-03 -l trial_test -n 5 --seed 0 -o dog.npy

The same files are also on Kaggle as the private dataset lemondobby/word-to-eeg-generator.

Inputs

• word: an object name such as "dog", "baseball bat" or "fire_truck". It is encoded as the average CLIP vector of 4 prompts ("a photo of a {word}.", ...).
• description: one concrete sentence, ideally 15–23 words, naming the object with its colour, pose and setting. It should read like the training captions, e.g. "A tan antelope with long ridged horns stands in a dry golden grassy field, facing left in profile." Short, terse descriptions measurably lose accuracy (see below).
• subject: 'sub-01' … 'sub-10' (that subject's style) or None (the 10-subject average). With a description-only input the subject sets the noise only; the subject's mean offset is fitted on word vectors, so it needs a word.
• level:

level	adds	subject	use it for
mean	no noise	optional	maximum word / scene information
erp	averaged-response noise (training session)	optional	realistic averaged ERPs
trial	single-trial noise (training session)	required	single trials like the training recordings
erp_test	40-repetition average noise (test session)	required	averaged ERPs like the test recordings
trial_test	single-trial noise (test session)	required	most realistic single trials

Output

A float32 array of shape (n, 63, 80): n samples × 63 channels (wte.CHANNELS, 10-10 system, Fp1 … O2) × 80 time points at 250 Hz. These are samples 18–97 of the dataset's 250-sample epoch; by the dataset's own times field that is about −0.128 s to +0.188 s. That field has 300 entries for 250 samples, though, so the absolute timing is unverified. Values are in the dataset's whitened units, not microvolts.

Accuracy: which setup is best

Accuracy is identification on the 200 THINGS-EEG test objects. Neither these objects nor their photos were seen in training. For each test object the generator's noise-free output is correlated with the real EEG of all 200, and it scores a hit if its own object ranks first. Chance is 0.5% top-1 and 2.5% top-5. 2-way is a gentler score: how often an object's output matches its own EEG better than it matches another object's EEG, averaged over all pairs (chance 50%).

input	target	top-1	top-5	2-way
description only, detailed (~19 words)	10-subject average	15.0%	34.0%	88.3%
word + description, detailed (~19 words)	10-subject average	13.0%	33.5%	87.6%
word + dataset's original caption (~8 words)	10-subject average	10.0%	31.0%	87.1%
word only	10-subject average	9.5%	28.5%	83.9%
dataset's original caption only (~8 words)	10-subject average	8.5%	31.5%	87.3%
word + description, subject set	that subject's average	7.0% (3.5–9.5%)
word only, subject set	that subject's average	6.0% (3.5–8.5%)
word only, pooled (no subject)	each subject's average	4.9%
real photo's CLIP image vector (upper bound, not in this package)	10-subject average	25.5%	50.0%

The "dataset's original caption" rows use THINGS-EEG's own short test captions (e.g. "Tan antelope with curved horns standing in dry grass field"). The "detailed" rows use the same 200 photos re-described in the ~19-word style the models were trained on.

What has the highest accuracy: give a detailed description (with or without the word) and use level='mean'.

• Description only (15.0%) and word + description (13.0%) tied on the development data (6.8% each). Word + description was chosen before the test run, and the 2-point gap on test is within noise (one standard error is about ±2.4 points on 200 items). Treat them as equivalent.
• The dataset's own short captions (~8 words) barely beat the word on top-1 (10.0% and 8.5% vs 9.5%), though they gain on top-5 and 2-way. Detailed ~19-word descriptions are what make the difference.
• For one subject's EEG, pass subject together with a word: it beats the pooled prediction (6.0% vs 4.9% for word only on test: better for 8 of 10 subjects, 1 tie; 10 of 10 on development data).
• Noise lowers identifiability. A single trial / trial_test sample identifies its object at chance (1.2% on a 100-way test with 1% chance; a real single trial gets 10.5%). That is how noisy real single-trial EEG is. Use mean for information and the noise levels for realism, or average many noisy samples.

How this compares to real EEG (why top-1 is low)

The test EEG averages 80 repetitions per person, so it is very consistent between people:

predictor → target	top-1	2-way
real EEG, average of 5 people → average of the other 5 (100 random splits)	76.1%	99.5%
real EEG, one person → average of the other 9	45.5%
this model, word + detailed description → the same 5-person averages	10.6%	86.7%
this model, word only → the same 5-person averages	8.1%	83.0%

The brain responses are clearly distinct between objects, so the gap is in the model and its input, not in the test data:

• The main metric is strict. Exact top-1 out of 200 punishes near misses. The same outputs pick the right object over a random alternative 84–88% of the time.
• Text lacks what drives this EEG. This early visual window responds to what the particular photo looks like (colours, contrast, layout, object size). A word says none of that, and a description says only some of it.
• The model is linear and trained on noisy EEG. It is a ridge map from CLIP vectors, trained on 4 repetitions per training photo. Even the real photo's CLIP image vector only reaches 25.5%.

Adding visual detail (images generated from the text) or a better image→EEG model could raise the score. Neither is tested or included here.

Realism

A quadratic real-vs-generated classifier scores AUC 0.5 when the two are indistinguishable.

level	generated vs real	same-session real vs real (reference)
trial_test	0.567	0.534
erp_test	0.649	0.482
trial (vs held-out training-session trials)	0.550
erp (vs held-out training-session averages)	0.555

Amplitude ratio (generated / real) is 0.99–1.00. Single trials are close to real; test-session averaged responses are still detectable. Noise was fitted around the word-only prediction, so with a description it is slightly over-dispersed.

Limits

• The models know about single objects (the 1,854 THINGS concepts and their photos). Other text runs, but it hasn't been validated.
• The accuracy numbers above are for the 200 THINGS test objects with descriptions in the training-caption style.
• Derived from THINGS-EEG (Gifford et al., 2022). Check the dataset's licence before redistributing.