Hurwitz/RNGen-checkpoints

RNGen checkpoints

This collection provides pretrained MusicBERT, domain-fine-tuned RNBERT, and RNGen checkpoints for Roman numeral analysis. RNGen combines a music encoder with Qwen through a projector and is trained in three stages: Key → Mixed → RNA.

Contents

Path	Description
pretrained/checkpoint_last_musicbert_base.pt	Original Microsoft MusicBERT base checkpoint
rnbert/unconditioned_checkpoint_best.pt	Domain-fine-tuned RNBERT from experiment 1742830064
rnbert/1742830064_synced_metrics.csv	Reference evaluation metrics for that RNBERT experiment
rngen/RNGen-7B-Key/, rngen/RNGen-14B-Key/	Checkpoints after local-key training
rngen/RNGen-7B-Mixed/, rngen/RNGen-14B-Mixed/	Checkpoints after mixed intermediate-task training
rngen/RNGen-7B-RNA/, rngen/RNGen-14B-RNA/	Checkpoints after final RNA training

Each RNGen checkpoint stores its trained encoder, projector, Qwen LoRA adapters, and tokenizer files. The original Qwen base-model weights are downloaded separately through Transformers.

Setup and download

Install the RNGen code with Python 3.11 and activate its environment:

git clone https://github.com/Hurwitzzz/RNGen.git
cd RNGen
python3.11 install.py
source .venv/bin/activate

Run the commands below from that checkout's root. Full 7B/14B evaluation and training require a suitable GPU platform. The core installer includes the musicbert_hf dependency needed to load the encoder; the other preprocessing submodules are not needed to use the released JSONs.

Download these weights and the matching RNGen-data annotations:

huggingface-cli download Hurwitz/RNGen-checkpoints \
  --local-dir ckpts/releases
huggingface-cli download Hurwitz/RNGen-data --repo-type dataset \
  --local-dir datasets/releases/rngen

These commands download the current repository files. For fixed release revisions, use the code repository's download instructions. For repositories requiring authentication, first run huggingface-cli login with an account that has access.

Keep the downloaded directory layout intact. Each RNGen checkpoint resolves ../../rnbert/unconditioned_checkpoint_best.pt relative to its own directory. The loader uses that base file to construct the encoder, then restores the stage-specific encoder weights saved inside the selected RNGen checkpoint. Loading Mixed or RNA does not require a separate copy of its preceding RNGen stage. The pretrained MusicBERT control is also unnecessary for these six checkpoints.

The Qwen model name is read from the checkpoint: 7B uses Qwen/Qwen2.5-7B-Instruct, and 14B uses Qwen/Qwen2.5-14B-Instruct. Transformers downloads and caches those base weights on first use. Each RNGen checkpoint supplies its own tokenizer and LoRA adapter; keep all its files together.

Evaluate a checkpoint

Choose a model size and run the command for the stage you want to evaluate:

SIZE=7B  # Use 14B for the other model size.
MODEL_ROOT="$PWD/ckpts/releases"
DATA_ROOT="$PWD/datasets/releases/rngen"
EVAL_ROOT="$PWD/outputs/evaluation"

# Local-key prediction.
python -m rngen.sft_eval --config configs/rngen/evaluation.yaml \
  --model-path "$MODEL_ROOT/rngen/RNGen-${SIZE}-Key" \
  --task key_sft --json-path "$DATA_ROOT/key/test.json" \
  --max-new-tokens 256 \
  --output-dir "$EVAL_ROOT/RNGen-${SIZE}-Key"

# Six-task intermediate stage: reports task losses.
python -m rngen.sft_eval --config configs/rngen/evaluation.yaml \
  --model-path "$MODEL_ROOT/rngen/RNGen-${SIZE}-Mixed" \
  --task rna_curriculum_stage2_sft --json-path "$DATA_ROOT/mixed/test.json" \
  --output-dir "$EVAL_ROOT/RNGen-${SIZE}-Mixed"

# Final Roman numeral analysis.
python -m rngen.sft_eval --config configs/rngen/evaluation.yaml \
  --model-path "$MODEL_ROOT/rngen/RNGen-${SIZE}-RNA" \
  --task rna_sft --json-path "$DATA_ROOT/rna/test.json" \
  --output-dir "$EVAL_ROOT/RNGen-${SIZE}-RNA"

The evaluation configuration enables the spelling-aware encoder, keeps spelling_downsampling_scale: 1.0, and uses BF16 with evaluation batch size 1. Key and RNA generation use deterministic beam search with four beams; the generation limits are 256 and 2,048 tokens, respectively. The evaluation CLI overrides YAML values with --flag value arguments.

Metrics are printed with an eval_ prefix even when evaluating the test split. Key and RNA also save prediction snapshots with metrics in the selected output directory; RNA saves classwise reports. Mixed evaluation writes stage2_eval_loss_summary.json, containing six task losses and stage2_selection_loss. To measure full-RNA performance of a Mixed checkpoint, use its model path with --task rna_sft and the RNA test.json, writing to a separate output directory.

Load from Python

Use RNGen's custom loader for these composite checkpoints:

from rngen.cli.entrypoint_helpers import load_pretrained_rngen_checkpoint

model, tokenizer = load_pretrained_rngen_checkpoint(
    model_path="ckpts/releases/rngen/RNGen-7B-RNA",
    use_spelled_musicbert=True,
    spelling_downsampling_scale=1.0,
    is_lora_trainable=False,
)
model.eval()

The helper restores the trained encoder, projector, and Qwen LoRA adapter. It handles the saved encoder vocabulary size for both 7B and 14B. For complete input preparation, generation, and scoring, use the evaluation entrypoint above, which supplies RNGen's dataset and collator.

Train or continue training

The training guide provides the full Key → Mixed → RNA sequence and validation-based best-checkpoint selection. Public recipes are in configs/rngen/. Key initializes from the independent RNBERT file and the selected Qwen model; Mixed and RNA use finetuned_rngen_checkpoint to load the previous stage. The model size must match that checkpoint.

For example, start a new Mixed run from the released 7B-Key checkpoint:

python -m rngen.sft_ft --config configs/rngen/curriculum_stage2.yaml \
  llm_model_name=Qwen/Qwen2.5-7B-Instruct \
  finetuned_rngen_checkpoint="$PWD/ckpts/releases/rngen/RNGen-7B-Key" \
  jsonl_dir="$PWD/datasets/releases/rngen/mixed" \
  output_dir="$PWD/outputs/training/7B-Mixed" \
  report_to=none wandb_project=null

Training uses key=value overrides. This starts a new stage with the previous stage's model weights; it does not restore an optimizer or resume an interrupted training run. The public recipes use effective training batch size 4; see the configuration guide for gradient accumulation and distributed settings.

Component licensing and attribution

The collection has multiple licensing sources. The full upstream notices and their scope are recorded in LICENSE.

Component	Source and attribution	Licensing information
Original MusicBERT	Microsoft; official MusicBERT project	Redistributed using the upstream project's MIT license as the licensing basis, as described below
Domain-fine-tuned RNBERT and RNGen encoders	MusicBERT-based adaptations trained by Hewei Gao; RNBERT method and implementation by Malcolm Sailor	Derived MusicBERT material; the upstream component basis and training-data scope notes in LICENSE apply
7B Qwen-related material	Qwen/Qwen2.5-7B-Instruct	Apache License 2.0
14B Qwen-related material	Qwen/Qwen2.5-14B-Instruct	Apache License 2.0
RNGen projectors and adapter training	Hewei Gao; RNGen	Upstream component notices and applicable training-data rights remain relevant to the trained models
Companion RNGen training annotations	DCML corpus creators, with RNGen preprocessing and added labels	CC BY-NC-SA 4.0 for the data release

MusicBERT redistribution basis

The upstream project license is MIT, and its official README provides the pretrained checkpoints. This collection uses that project license as the basis for MusicBERT weight redistribution and retains Microsoft's complete copyright and MIT notice.

This basis is an interpretation of the available upstream materials; the MusicBERT authors have not separately confirmed it for this collection. The source-code MIT license is not presented as a blanket license for all weights and data here.

Adaptations and training data

The independent RNBERT checkpoint is from our domain-fine-tuning experiment. RNGen then trains its encoder, projector, and Qwen LoRA adapters through the staged tasks. RNGen's tokenizers add music boundary tokens to the Qwen tokenizers.

The companion data release uses CC BY-NC-SA 4.0 and records the original corpus credits and RNGen preprocessing changes. Its conditions apply to use and redistribution of those data. Their applicability to particular trained weights is not settled by this component notice. This collection makes no blanket assurance of unrestricted commercial use.

Citation

If you use RNGen or these checkpoints in a research publication, please cite:

@inproceedings{gao2026rngen,
  author = {Gao, Hewei and Beyer, Tim and Wang, Xiaoxuan and Rohrmeier, Martin},
  title = {{RNGen}: A Large Language Model Approach to Roman Numeral Analysis Toward Explainable Harmonic Analysis},
  booktitle = {Proceedings of the 27th International Society for Music Information Retrieval Conference},
  year = {2026},
  url = {https://ismir2026program.ismir.net/poster_405.html}
}

Please also credit and cite the upstream MusicBERT, RNBERT, and Qwen projects for the components used.