SteerSound

SteerSound is an instruction-guided audio editing model supporting sound generation, addition, removal, replacement, pitch modification and volume modification. This repository provides the final SteerSound weights and the components needed for inference.

Model components

The audio VAE and DiT editor are trained by the SteerSound team. The text encoder/tokenizer and vocoder use pretrained weights from the sources below.

Component	Source	Files in this repository
Audio VAE (encoder and decoder)	Trained by us	vae/
DiT audio editor	Trained by us	dit/
T5 text encoder and tokenizer	google-t5/t5-base, using its encoder	text_encoder/
HiFi-GAN vocoder	cvssp/audioldm2 — vocoder	vocoder/

Quick start

Use Linux, Python 3.10 or 3.11, and an NVIDIA GPU. Install a CUDA-enabled PyTorch build and the inference package:

git clone https://github.com/GeWu-Lab/SteerSound.git
cd SteerSound
python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install torch==2.7.1 torchaudio==2.7.1 torchvision==0.22.1 --index-url https://download.pytorch.org/whl/cu126
pip install -r requirements.txt
pip install -e . --no-deps

For private model access, run huggingface-cli login with an account that has access. Access to the GitHub repository is also required while the code is private.

import soundfile as sf
from steersound import SteerSoundPipeline

pipe = SteerSoundPipeline.from_pretrained("GeWu-Lab/SteerSound", device="cuda")

# Edit an existing recording.
audio = pipe("Remove the sound of a dog barking.", audio="input.wav")
sf.write("edited.wav", audio, pipe.sample_rate, subtype="PCM_16")

# Generate audio without an input recording.
audio = pipe("Generate the sound of rain.")
sf.write("rain.wav", audio, pipe.sample_rate, subtype="PCM_16")

Command-line inference:

python inference/infer.py \
  --model GeWu-Lab/SteerSound \
  --instruction "Remove the sound of a dog barking." \
  --input input.wav --output edited.wav

Instruction examples

Task	Instruction
Generation	Generate the sound of rain.
Addition	Add the sound of a dog barking.
Removal	Remove the sound of a dog barking.
Replacement	Replace the sound of a dog barking by the sound of a cat meowing.
Volume modification	Increase the volume of the sound of a dog barking by 3.0 dB.
Pitch modification	Reduce the pitch of the sound of a dog barking by 2.0 semitones.

Supply a plain English instruction. The inference code identifies the task and change amount and adds model-control tokens internally. Editing tasks require input audio; generation does not.

For pitch and volume, use Increase or Reduce with an explicit magnitude. Volume changes support ±5 dB; pitch changes support ±4 semitones in increments of 0.5. A temporal instruction can specify a region, for example: Increase the volume of the sound of a dog barking from 2.0 to 4.0 seconds by 3.0 dB. Temporal regions are interpreted by the model from the instruction. Unsupported or ambiguous instructions raise an error.

Audio format and inference settings

Input files are downmixed to mono, resampled to 16 kHz, and truncated or right-padded to 10 seconds. Output is a 10-second mono waveform at 16 kHz. The default runtime uses BF16, PyTorch SDPA and 50 Euler steps, with guidance scale 8.5 for Addition and 3.5 for the other tasks. Use --seed with the CLI or seed=... with the Python API for repeatable sampling.

Model size and GPU memory

Component	Parameters
DiT editor	0.673B
Audio VAE	0.509B
T5 encoder	0.110B
HiFi-GAN vocoder	0.055B
Total	1.348B

GPU memory for inference: approximately 8 GB (BF16, batch size 1, 10-second audio).

Model files

The DiT and VAE weights are stored in BF16. The T5 encoder and vocoder files are stored in FP32 and cast to the runtime dtype when loaded. Architecture and audio settings are in config.json.