ChordBERT

ChordBERT is a compact DeBERTa-v2 masked language model for symbolic chord sequences. It can predict masked chords or act as an encoder that maps a chord progression to a 256-dimensional embedding.

Model developed by Lameusiwe.

Model details

Property	Value
Architecture	DeBERTa-v2 masked language model
Transformer layers	4
Attention heads	4
Hidden size	256
Intermediate size	1,024
Vocabulary size	3,230
Parameters	4,713,886
Recommended maximum length	256 tokens including boundary tokens
Training objective	Masked language modeling
Training domain	Chordonomicon chord sequences
License	Apache-2.0

This repository contains the original Chordonomicon-pretrained ChordBERT checkpoint. It is not the separately adapted ChordBERT + WIR checkpoint.

Intended uses

ChordBERT is intended for research with harmonic sequences, including:

• chord and progression embeddings;
• harmonic similarity and retrieval;
• clustering and visualization of musical works;
• masked-chord prediction;
• feature extraction for downstream music-information-retrieval models.

The model was not designed for audio transcription, music generation, copyright detection, composer attribution, or high-stakes cultural and historical judgments.

Input format

Input must be a whitespace-separated sequence of tokens from the supplied tokenizer vocabulary:

C G Amin F
Cmaj7 Amin7 Dmin7 G7
Bb F Gmin Eb

Chord spelling follows the Chordonomicon convention:

• major triads use only the root, such as C;
• sharps use s, such as Csmin for C-sharp minor;
• flats use b, such as Bb;
• common suffixes include min, 7, maj7, min7, dim, dim7, and aug;
• some structural markers, such as <verse_1>, are present in the vocabulary.

Unknown or differently formatted symbols may become <unk>. Check token coverage before embedding a new corpus:

from transformers import AutoTokenizer

model_id = "YOUR_USERNAME/YOUR_MODEL_NAME"
tokenizer = AutoTokenizer.from_pretrained(model_id)

tokens = "C G Amin F".split()
unknown = [token for token in tokens if token not in tokenizer.get_vocab()]
print("Unknown tokens:", unknown)

Masked-chord prediction

from transformers import pipeline

model_id = "YOUR_USERNAME/YOUR_MODEL_NAME"
fill_mask = pipeline("fill-mask", model=model_id, tokenizer=model_id)

predictions = fill_mask("C G <mask> F", top_k=5)
for prediction in predictions:
    print(prediction["token_str"], prediction["score"])

Generate a progression embedding

The checkpoint does not include a trained sentence-pooling head. A practical default is mean pooling over non-special tokens:

import torch
from transformers import AutoModel, AutoTokenizer

model_id = "YOUR_USERNAME/YOUR_MODEL_NAME"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id).eval()

progressions = [
    "C G Amin F",
    "Dmin7 G7 Cmaj7",
]

batch = tokenizer(
    progressions,
    padding=True,
    truncation=True,
    max_length=256,
    return_tensors="pt",
)

with torch.inference_mode():
    hidden = model(**batch).last_hidden_state

pool_mask = batch["attention_mask"].bool()
for special_id in tokenizer.all_special_ids:
    pool_mask &= batch["input_ids"] != special_id

weights = pool_mask.unsqueeze(-1).to(hidden.dtype)
embeddings = (hidden * weights).sum(dim=1) / weights.sum(dim=1).clamp(min=1)

print(embeddings.shape)  # torch.Size([2, 256])

The resulting vectors are not normalized. Apply L2 normalization if cosine similarity is the downstream comparison:

embeddings = torch.nn.functional.normalize(embeddings, p=2, dim=1)

Long sequences

For progressions longer than 254 chord tokens:

1. split the sequence into chunks of at most 254 tokens;
2. embed each chunk with the pooling procedure above;
3. combine chunk embeddings using the number of chord tokens as weights.

This leaves room for the two boundary tokens added by the tokenizer and matches the evaluation setup used for this checkpoint.

Training information

The supplied checkpoint is recorded in the project artifacts as a Chordonomicon-pretrained masked language model. The original detailed training hyperparameters and a complete training-data statement are not included with this checkpoint; they should not be inferred from the later evaluation and adaptation scripts.

The model configuration identifies the architecture as DebertaV2ForMaskedLM. Weights are stored in safetensors format.

Reproducibility

Recommended dependencies:

torch
transformers
safetensors

For deterministic comparisons, keep the same token conversion, maximum length, special-token handling, pooling, chunk weighting, and vector normalization across all corpora.

Citation

@misc{he2026modelingstylisticcoevolutionsymbolic,
      title={Modeling Stylistic Co-evolution in Symbolic Music Heritage Collections}, 
      author={Yulong He and Ivan Smirnov and Yanming Li},
      year={2026},
      eprint={2607.23957},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2607.23957}, 
}