Maxilicious20/Instrumenta-1

Instrumenta

Instrumenta is a small text-to-music model trained from scratch on a single consumer GPU (RTX 4060, 8GB VRAM). It generates short instrumental audio clips from text prompts describing genre, mood, key, tempo, and instrumentation.

This is a hobby/personal project, not a research release or a commercial product. It is shared as-is.

Model summary

• Architecture: autoregressive Transformer decoder over EnCodec discrete audio tokens, using a delay pattern across codebooks (MusicGen-style) and cross-attention to a frozen T5 text encoder for prompt conditioning.
• Audio tokenizer: pretrained EnCodec (frozen, used only to tokenize/detokenize audio — not fine-tuned).
• Text encoder: T5-small (frozen).
• Trainable parameters: ~30M (the Transformer decoder only; EnCodec and T5 are frozen and not counted).
• Model dimensions: 448 embedding dim, 8 attention heads, 8 layers.
• Codebooks: 4 (from EnCodec), 1024 tokens per codebook vocabulary.
• Sample rate: 24 kHz.
• Output length: 10-second clips (750 frames at 75 frames/sec).
• Conditioning: free-text prompt, with classifier-free guidance (CFG) supported at inference time.

Training data

Trained on a personal/self-assembled dataset of roughly 10,000 short instrumental audio clips with paired text captions (genre, mood, key, tempo, instrumentation), plus a small held-out validation split. The dataset is not publicly released alongside this model.

Because the dataset is on the small side for training a generative audio model from scratch, expect more variance and less fidelity than larger models trained on much bigger datasets (e.g. MusicGen, Stable Audio). This project intentionally traded dataset/model scale for something trainable end-to-end on a single 8GB GPU.

Training procedure

• Phase 1: trained from scratch for 150 epochs on the full dataset.
• Phase 2 (fine-tune): continued training for a further ~230 epochs at a reduced learning rate, starting from the Phase 1 checkpoint, to further reduce validation loss. Both phases used the same dataset, architecture, and tokenization pipeline — Phase 2 did not introduce new data or change the model architecture.
• Precision: mixed precision (FP16 autocast + gradient scaling) for the trainable Transformer; EnCodec and T5 run in their pretrained precision.
• Hardware: single NVIDIA RTX 4060 (8GB VRAM).

No objective benchmark scores (e.g. FAD, CLAP score) have been computed for this model yet. Any quality claims below are based on informal listening, not a scored evaluation.

Files in this repository

File	Description
instrumenta_best.pt	Checkpoint with the lowest validation loss reached during training.
instrumenta_last.pt	Final checkpoint at the end of training (last completed epoch).

Both checkpoints are full training checkpoints — in addition to model weights, they also contain optimizer/scaler state, epoch/step counters, and the training config, so they can be used to resume training as well as for inference.

To use only the model weights for inference, load the "model" key from the checkpoint dictionary.

Usage

This repository contains only weights, not a packaged inference pipeline. To generate audio you need the accompanying model code (Transformer architecture, delay-pattern encode/decode, EnCodec + T5 loading) — this model is not yet wrapped for the transformers/diffusers pipeline() API.

import torch

checkpoint = torch.load("instrumenta_best.pt", map_location="cpu")
model_state_dict = checkpoint["model"]
config = checkpoint["config"]  # training configuration used for this checkpoint

Load model_state_dict into the matching model architecture from the training code, then use it together with the frozen EnCodec and T5-small encoders to condition on a text prompt and autoregressively generate audio tokens, which EnCodec decodes back into a waveform.

Example prompts the model was trained on

Prompts describe genre/mood, key, tempo, and instrumentation, e.g.:

"a punchy dark instrumental arrangement, ambient-style, in D minor, 86 bpm, no vocals"

Limitations

• Small training dataset. ~10k clips is modest for training a generative audio model from scratch; expect inconsistent quality across prompts, and weaker performance on prompt combinations underrepresented in training.
• Short, fixed output length. Generates fixed 10-second clips; no long-form structure (intro/verse/chorus) is modeled.
• No formal evaluation. No FAD/CLAP/KL scores or human evaluation have been run. Quality assessment so far is informal listening only.
• English-language prompts only (T5-small was used as-is, not fine-tuned on non-English captions).
• Not intended for production or commercial use without further evaluation.

License

Apache 2.0. Note that this model was built using pretrained EnCodec and T5-small as frozen components at both training and inference time — check their respective licenses if redistributing a pipeline that includes them.

Acknowledgements

Built on top of pretrained EnCodec (audio tokenizer) and T5-small (text encoder), both used frozen. Architecture inspired by MusicGen-style delay-pattern autoregressive audio generation.