FineAudio-Judge

A Fine-Grained Evaluation Framework for Complex Text-to-Audio Generation

Code | Benchmark

FineAudio-Judge is a fine-grained evaluation framework for assessing whether audio matches text descriptions or generation instructions for complex audio scenes.

Given an audio–text pair, the framework first structurally decomposes the text into events, attributes, and inter-event relations, then converts them into atomic binary rubric questions. The audio judge answers each question with yes or no based on the audio, assessing whether the audio matches the corresponding aspect of the text description. The proportion of yes answers forms the overall audio–text consistency score.

To train the audio judge, we construct a multi-route corpus covering real, composed, and generated audio. Rubric-guided prompt/audio perturbations and cross-sample mismatches provide fine-grained supervision across varying levels of audio–text consistency.

This repository releases the resulting LoRA adapter, obtained by supervised fine-tuning of Qwen3-Omni-30B-A3B-Instruct. The adapter needs to be loaded together with the base model. Rubric decomposition uses a separately configured language-model backend in the code repository, while the released adapter performs audio-based rubric judgments.

Files

• adapter_config.json: PEFT LoRA configuration.
• adapter_model.safetensors: FineAudio-Judge LoRA adapter weights.

Training

• Base: Qwen3-Omni-30B-A3B-Instruct
• Corpus: 12,508 constructed audio–prompt pairs, with 278,199 rubric-level question–answer examples used for training
• Stage: LoRA supervised fine-tuning with a 1-epoch training schedule
  • 8 × NVIDIA A800 80GB GPUs
  • per-device batch size 1, gradient accumulation 8
  • learning rate 5e-5
  • LoRA rank 64, alpha 64, dropout 0.05

Usage

The evaluation scripts are provided in the FineAudio-Judge code repository. They use ms-swift's vLLM inference engine to load Qwen3-Omni-30B-A3B-Instruct together with the released LoRA adapter.

Clone the repository and enter the FineAudio-Judge directory:

git clone https://github.com/QwenAudio/FunResearch.git
cd FunResearch/FineAudio-Judge

Follow the setup instructions in the code repository's README to install the project and configure the inference environment. Run the evaluation commands below from the FunResearch/FineAudio-Judge/ directory.

Replace /path/to/benchmark with the downloaded benchmark repository root and /path/to/adapter with the directory containing the released adapter files.

Rubric-QA evaluation

Each audio–rubric question pair is evaluated independently.

python -m judge.cli.reproduce_rubric_qa \
  --dataset-root /path/to/benchmark \
  --model Qwen/Qwen3-Omni-30B-A3B-Instruct \
  --adapter /path/to/adapter \
  --output outputs/rubric_qa.jsonl

Human-preference evaluation

Rubrics are evaluated jointly for each candidate audio, and the proportion of yes predictions forms its prompt–audio consistency score.

python -m judge.cli.reproduce_preference \
  --dataset-root /path/to/benchmark \
  --model Qwen/Qwen3-Omni-30B-A3B-Instruct \
  --adapter /path/to/adapter \
  --output outputs/preference.jsonl

License

The FineAudio-Judge LoRA adapter is released under the Apache License 2.0. See LICENSE for details.

The adapter is fine-tuned from Qwen3-Omni-30B-A3B-Instruct, which is also released under the Apache License 2.0. The base model weights are not included in this repository.

Base model copyright: Copyright 2025 Alibaba Cloud.