AGAV-Rater: Adapting Large Multimodal Model for AI-Generated Audio-Visual Quality Assessment
Authors: Yuqin Cao
Xiongkuo Min
Yixuan Gao
Wei Sun
Guangtao Zhai
Venue: ICML 2025 poster (2025)
Keywords: AI-Generated
Audio-visual
Quality Assessment
Large Multimodal Model
Primary area: applications->computer_vision
TL;DR: AGAV-Rater: Adapting LMM for AI-Generated Audio-Visual Quality Assessment

Abstract: Many video-to-audio (VTA) methods have been proposed for dubbing silent AI-generated videos. An efficient quality assessment method for AI-generated audio-visual content (AGAV) is crucial for ensuring audio-visual quality. Existing audio-visual quality assessment methods struggle with unique distortions in AGAVs, such as unrealistic and inconsistent elements. To address this, we introduce **AGAVQA-3k**, the first large-scale AGAV quality assessment dataset, comprising $3,382$ AGAVs from $16$ VTA methods. AGAVQA-3k includes two subsets: AGAVQA-MOS, which provides multi-dimensional scores for audio quality, content consistency, and overall quality, and AGAVQA-Pair, designed for optimal AGAV pair selection. We further propose **AGAV-Rater**, a LMM-based model that can score AGAVs, as well as audio and music generated from text, across multiple dimensions, and selects the best AGAV generated by VTA methods to present to the user. AGAV-Rater achieves state-of-the-art performance on AGAVQA-3k, Text-to-Audio, and Text-to-Music datasets. Subjective tests also confirm that AGAV-Rater enhances VTA performance and user experience. The dataset and code is available at https://github.com/charlotte9524/AGAV-Rater.

AGAV-Rater: Adapting Large Multimodal Model for AI-Generated Audio-Visual Quality Assessment
Yuqin Cao
Xiongkuo Min
Yixuan Gao
Wei Sun?
Guangtao Zhai
Abstract
Quediun:
The video is [vxdeo]. The
V2A-Mapper
audio [rudio 13 The audio
[sadio 2]. The andio i [sedio 3] The audio is (andio 4). Which audio
Many video-to-audio (VTA) methods have been proposed for dubbing silent Al-generated videos. An efficient quality assessment method for AI- generated audio-visual content (AGAV) is crucial for ensuring audio-visual quality. Existing audio- visual quality assessment methods struggle with unique distortions in AGAVs, such unrealis- tic and inconsistent elements. To address this, we introduce AGAVQA-3K, the first large-scale AGAV quality assessment dataset, comprising 3, 382 AGAVs from 16 VTA methods. AGAVQA- includes two subsets: AGAVQA-MOS, which provides multi-dimensional scores for audio qual- ity, content consistency, and overall quality, and AGAVQA-Pair, designed for optimal AGAV pair selection. We further propose AGAV-Rater, a LMM-based model that can score AGAVs, as well as audio and music generated from text, across multiple dimensions, and selects the best AGAV generated by VTA methods to present to the user. AGAV-Rater achieves state-of-the-art perfor- mance AGAVQA-3K, Text-to-Audis and Text- to-Music datasets. Subjective tests also confirm that AGAV-Rater enhances VTA performance and user experience. The dataset and code are avail- able at https//ggttub..oom////lottt22240 Rater.
TIVA
Frieren
best matches this video tonns of audio contens, quality and rhythm??
[vsken]
pudoo UNISTANZA (ando 2/40 Wes
Soate VidiieL
juudo 3/40 Fim (sudio 424 VTx-LMM VZA-ScencTeeeclee
Auswer:
* AGAA--Mttrr (Dand:
GPT-4e+ Andio Caption:
Audio
d5s
GPr- Caption:
Genteo-0 Flask::
Audio
Genssnd Plash-di:
Atdio X
STAVZA
ReWaS
2
Core:
Roku Plask:
SSVIA
Figure I. Comparison of answer accuracy between AGAV-Rater and proprietary LMMs on AGAV-Pair subset. construct question-answer annwerr pairs from demonstration AGAV' on VTA GitHub pages, prompting LMMs to identify the optimal AGAV pair. Fach dimension in the radar chart represents the answer accuracy with the correct answer corresponding to the current VTA method.
1. Introduction
The quality of Al-generated content needs to be con- trolled. Many researchers focus on developing video-to- audio (VTA) methods to add sound to silent Al Al-generated videos. Some researchers (Wang al., 2024f; Lu al., 2024) combine large multimodal models (LMMs) with dif- fusion models, empowering them with the ability to generate audio from videos. On the commercial side, companies like ElevenI.bbs have launched efficient VTA models to dub videos. Although VTA methods can significantly improve post-production efficiency and enhance the audio-visual (A/V) experience of Al-generated content (AIGC), they oc- casionally encounter issues such as poor A/V alignment low perceptual audio quality when generating audio. There- fore, there is a need for an automated model to evaluate Al-generated audio-visual content (AGAV), select the most user- uise--rrferred results to present to users, and provide feed- back for improving the generated content.
This work was supported in part by the National Natural Sci- ence Foundation of China under Grant 62271312. Grant 62132006, Grant 62225112 12 and Grant 62301316: in part by STCSM un- der Grant 22DZ2229005: and in part by the Oceanic Interdisci- plinary Program of Shanghai Jiao Tong University (project number SI2202ZZD102) 'Institute of Image Communication and Network Engincering, Shanghai Key Laboratory of Digital Media Process- ing and Transmissions, Shanghai Jiao Tong University, Shang- hai "School of Communication & Electronic Engineering, East China Normal University, Shanghai. Correspondence Guang- tao Zhai sirigguunnnao@suuuuuuu..oo iongkuo @ @eckuoosjtu.coo wwt..cdd..nn...
Traditional audio-visual quality assessment (AVQA) meth- ods (Cao et al., 2023a;b; Min et al., 2020) focus on dis- tortions caused during the capture and transmission stages, which makes it difficult to identify unique distortions in AIGC, such as inconsistent A/V content and unnatural audio. With the rise of LMMs, researchers (Wang et al., 2024b;c; Wu et al.) utilize the powerful content and language com-
Xiongkuo Min <minx-
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

AGAV-Rater: Adapting LMM for Al-Generated Audio-Visual Quality Assessment
prehension capabilities of LMMs evaluate the quality of AIGC images and videos more accurately. However, most quality assessment research focuses on the visual ca- pabilities of LMMs (Sun et al., 2024b;a; 2023), with little exploration of their A/V capabilities. This raises the ques- tion:
human-annotated multi-dimensional MOSs.
Our experimental results demonstrate that AGAV-Rater achieves of-e-oo-thhearr performance on three quality as- sessment datasets: AGAVQA-MOSS text-to-audio (TTA), and text-to-music (TTM) (Deshmukh et al., 2024). Since the video content and VTA methods in the AGAVQA-MOS and SEAVQA--Pir subsets do not overlap, we validate the gen- eralization ability and robustness of AGAV-Rater on the un- seen dataset AGAVQA-Pait as shown in Fig. 1. We further conduct subjective experiment and find that AGAV-Rater helps VTA methods select high-quality generated results to present to users, thereby enhancing user experience. Our core contributions are threefold:
Can LMMs be utilized to evaluate the quality of audio-visual content generated by VTA methods?
In this paper, to develop and refine methods for evaluating AGAV quality, we construct the first Al-generated audio- visual quality assessment dataset, AGAVQA-3K, which con- tains 3, 382 AGAVs generated by 16 VTA methods. Fig. 2 illustrates the dataset construction pipeline. We hope that AGAV quality assessment methods can score AGAVs from multiple dimensions, while also assisting VTA models in selecting the optimal result from multiple generated outputs. To meet the above two needs, our AGAVQA-3K dataset is divided into two subsets. In the AGAVQA-MOS sub- set, we utilize 8 VTA methods to generate 3, 088 AGAV from 386 AIGC videos. Then we conduct a subjective ex- periment to collect 9, 264 Mean Opinion Scores (MOSs) across three dimensions, including audio perceptual qual- ity, A/V content consistency, and overall A/V quality. In the AGAVQA-Pair subset, we collect 294 AGAVs from 8 VTA GitHub pages. The same video content forms group, resulting in 75 question-answer pairs. We evaluate 7 closed-source LMMs on the AGAVQA-Pair subset to assess accuracy in selecting the optimal AGAVs, as shown in Fig. The results indicate that LMMs still have significant room for improvement in evaluating AGAV quality. Moreover, existing closed-source LMMs have difficulty providing numerical score for AGAV quality as humans do. Therefore, this paper primarily focuses on:
A large-scale AGAV quality assessment dataset AGAVQA-3K. It labels AGAVs' quality in two ways: multi-dimensional numerical scores and the optimal
AGAV pair.
A novel LMM-based AGAV quality assessment model, AGAV-Rater. It can predict multi-dimennional quality scores for AGAVs, TTA, and TTM, and assist
VTA methods in selecting the optimal AGAV samples.
. Enhance the user experience of AGAVs generated by VTA methods. According to our experiment, 80% of users recognize that using AGAV Rater to select higher-quality AGAVs offers a better A/V experience, validating that AGAV-Rater can assist VTA methods in improving quality.
2. Related Works
2.1. Audio-Visual Quality Assessment Dataset
Early research focused on compression distortions during transmission, leading to the construction of several tradition- ally distorted AVQA datasets. The largest one is the LIVE- SJTU dataset proposed by Min et al. (Min et al., 2020), which contains 14 original high-quality A/V consequences and 336 degraded ones. With the development of streaming media, researchers (Cao et al., 2023b; Ying et al., 2022) found that user-uploaded videos have more complex and diverse distortions, thus constructing authentically distorted AVQA datasets. With the rise of generative models, AIGC exhibits unique distortions that do not occur in real-world scenarios. To explore the distortions and perceived quality of AGAVs, we establish the first AGAV quality assessment dataset, AGAVQA-3k. Compared to single-modal AIGC image (video) quality assessment datasets, AGAVQA-3K datasct tackles more complex multimodal challenges, such as A/V content inconsistency and synchronization issues.
How to adapt LMMs to score AGAV like humans?
We propose the first LMM-bbssd quality assessment method for AGAV, AGAV-Rater. AGAV-Rater is trained through two stages to perceive AGAV in a human-like manner and outputs numerical scores across three dimensions. Firstly, we create 50, 952 instruction-response pairs related to the perceived quality from 3 large-scale real-world audio- caption datasets, including audio-visual datasets VGGSound (Chen et al., 2020), audio captioning dataset AudioCaps (Kim et al., 2019), and music captioning dataset MusicCaps (Agostinelli et al., 2023). These instruction-respone pairs do not require human annotations. Instead, the responses are automatically labeled using two text-defined rating levels (excellent and bad). These labels are then utilized to pre- train the LMM, enabling it to roughly assess whether the quality is good or bad. This approach significantly reduces the labor and costs associated with dataset construction and allows the model to better predict numerical scores in sub- sequent stages. Finally, we fine-tune the pre-traines LMM
2.2. Quality Assessment Methods
Audio Quality Assessment. Most audio quality assessment
(AQA) methods focus on distortions in speech recordings
2

---

AGAV-Rater Adapting LMM for Al-Generated Audio-Visual Quality Assessment
Step 1: A-Generalad Video Collection
Step 2 Generate udio From Video
Step 3 Human Evaluation
(a) AGAAQA
Cordinatiion Process
Vidko Sourrsss
R
Procepts
CogViideo
8 Auuio (VTA)
Methools
3088 Videos
Instruction
AGAVS
Kuna N
VBNCH
SpacKORAN
306A16C Videos
VTA-LDM Alai
Modoveesse
Mainn Audioo Soarcs in Vidros
Funteyy
Scenery
Crafter
Tronsformer Based
B A/V Content Consivieney
Audi Quality
Tedting Hissson
AGAM
LLM Based
Difiusion
56
30
65
36
68
22
biff Folcr
Based
AGAVQU
Areanittryy
ElevenLabd
A. Overall Quualtty
B
9,264M0ss
15Particppants
(b) AGAVQA-Pall Pair
Constrattion Prooss
VIA Project Pages
AGAV Pairs
Part
SSVZA
ASAVQA
.Pai
Besed
Sonieiia WLM
T'e video Is (videol. The audio 15 laudiu 11 The sudiu 2 Leutio 21 The autio 3 loudio The audio 4 [audio
Question:
Componison with beseines
Diffusion on Based
audio
matches this video terms audio cuntent, quility and tigthm?
ReW
(video)
[aodlo 1)mb)) Proposed
(audio 2| 1 Method 1
TEVA AUAUCA
VZA-
[Boddo 3]400 M Method2 2
(ancho 4] 4) Method 3
APA Nagaor
STAV2A
Arsownt Audio
Stepl: Collect Pairs
8VTA Project
294 Videos
Step2: Form Question A nswer Pairs
Audio Quality
A/V Cunteni
AW Oveeall Quealtty
Consisteccy
AGAVQA-3 -33 -3k
AGAVQA-MOS
AGAVQA-Pai
3088 AGAVS
294 ASAVS
9264 3-Dimensional MO5s 75 Question-Answer Pairs
Ladien
(c) Comparison of MOS Across VTA Methods in A GAV QA- -MOS
Figure 2. The construction process of the AGAVQA-3k dataset. The AGAVQA-3 33 dataset is divided into two subsets: (a) AGAVQA-MOS and (h) AGAVQA-Pair, which involve multi-dimensional score prediction and optimal AGAV pair selection tasks. (c) We present the maximum, minimum, and average subjective scores for the 8 VTA methods across the three dimensions.
from modern communication networks, such noise and discontinuities. Early speech quality assessments (Rix et al., 2001; Beerends et al., 2013) used handcrafted metrics de- signed by speech experts to predict speech quality. How- ever, these methods rely on comparing degraded speech with clean references, severely limiting their application in real-world scenarios. As a result, researchers proposed machine learning-bai based methods to predict speech qual- ity using only degraded speech, eliminating the need for clean speech references during inference. Training a ro- bust speech quality evaluator requires large-scale listening tests to collect speech and MOS for training. For example, NORESQA -MOS (Manocha & Kumar, 2022) was trained on 7, 000 audio recordings, and NISQA (Mittag al., 2021) was trained 72, 903 audio files. Soham et al. (Desh- mukh et al., 2024) applied the original weight of audio- language models directly to predict TTA and TTM quality without additional training data, which led to limited perfor- mance. Despite the high cost of collecting training samples, most AQA methods have difficulty handling the unique distortions in Al-generated audio, which differ from real- world audio distortions. In this paper, construct 50, 952 Iestruction-responee pairs without human annotations for pre-training the AGAV-Rater, thereby alleviating the burden of large-scale subjective experiments while allowing the AGAV-Rater to capture distortions in AIGC.
Audio-Visual Quality Assessment. Compared to AQA, AVQA is more complex task as it requires handling the interaction between video and audio modalities. Early re- search (Min et al., 2020) utilized Support Vector Regression (SVR) to predict A/V quality scores by regressing hand- crafted features extracted from video and audio. Although this method is effective for evaluating compressed A/V con- tent, it performs poorly in real-world scenarios with mixed distortions. To address this issue, researchers (Cao et al., 2023b) have proposed deep dearning-base approaches to predict A/V quality in real-world environments. However, these methods are not suitable for AGAVs, as they do not consider A/V content consistency and fail to address unnat- ural audio often present in AIGC scenarios. Our proposed AGAV-Rater not only predicts the quality of AGAVs but also evaluates the quality of TTAs and TTMs. We validate the effectiveness of AGAV-Rater on the AGAVQA-3K, TTA, and TTM datasets.
3. Dataset Construction
Our proposed AGAVQA-33 dataset consists of two subsets: the AGAVQA-MOS and AGAVQA-Pair. designed for multi- dimensional score prediction and optimal AGAV selection, respectively. In this section, we introduce the construction process of two subsets and analyze the subjective scores in the AGAVQA-MOS subset, as illustrated in Fig. 2.
3

---

R.A-Ratter: Adapting LMM for Al-Generated Audio-Visual Quality Assessment
Stage 1: Pre Training via Text- Defined Levels
Stage 2 Fine-T Taiinn via Numerical Scores
Audio-Tar Text Thrisse
Construct
50,952
Instrctionn
Cocsudancy Bakdel
Cut you meaiaade Ene eadn quality andiovindd quirry glees content one?
The les tan spedk and the hoom Nows How wodld min
Aud
Vo0 MusicCaga
Renquree Pains
Instractun:
Audio Qualtty Reale
Phirs
areadl
4g boda Earam Buroma dodutiin dias cleckic
Hit sprii hor Kewe Haddi quality 72 sodti fest condisinnc
qualits toe sodoo contect cocaistacc
the test?
Can you orchast the zadio visal
oseant sandiang
Instruction Response Fales
gives
liakdel
Raa you eeteaaaa the
Insircction Resparne Palks
eadio quaity of Bariooo
Quellty Hekeed
nodi eitaa conslence 50
woall addio sisud quilas 64
Nedio quality
vocaa lits tko yeu wass ou quiryy the
Escrii galle
qualiy
aceil
hoppe Cliqueo
virsa comndorcyy exoeleck
Aadio qualit: ecoeline
Peppl Chipses
Modd Sfratture AGAV-Rater
Flyingg
Unt you wallitee the andis vissa
cuzant is uno wood?
costent consitrery theggag
Ardlossinal conside Sel
Rosn you enciliate the
axtio ity of gloon
Asadin pu one went?
qusliy bei
tudlo
Slage E:
USER: Can you wallzate the acdio-tiaui content
coasstoccy of tte qiven cottent in oze wand?
Iraeruction (Contnttnninnnn as etample) ]
USER: Can you walussee the satho quality, audio-risua custer constieane and overall audio-visu qualits the giren content one by ond? Assiitan::
Stage 2 Instruction (A (Auiio-isual Pair os otampla)
quality: [Mask] sudio overall
sastannt A adie-viassi consituacyy (Mast).
quality; [Mask].
The wetle Muter munay Wisdisyan opintee radio contint constnenccy wusi?
Le you vedi
gfrem cedio qualiity woulf
40 Carmacng
Spotisl Corvolation
++ N (+)
Trainabie
Paszon
Audasacc condidagg: selllien
Asali qudiyy moseilles.
Syotial Teerpori Downeampling
Insdudied di las bonkgrondd Hoee th you sevte las
speaking then
Hio aesess cedio cuilts of the giras
Towared Heiks bestetboll dge spessng Mor
Syctill Convolation
Froxxeer
Tolonizer
one wars?
sadi an soeled oxnobbay of the gives oa
codlast in dee wood?
M..-Poo
loves
Aurtle commiree ey bed
Audia quality bed
Video Projection
Adio Pejection
playnng an
acnudied sdth
modd- coedent ormooberg of the give oo
This unattarr cocasting How would you jadge thc
and ceeppie
How wosld joogen the consl qualit the girea
coobent in die word.
Grux..
comples
Pro tminod I Lang l. language Model
now ward.
Labak Audas quality 6J,
complex Kusdo teet conditeage lentlee
Husiic quslity
LLM Fliiden Stales
aadaerbaad aenobaday 50,
Response visui consstroxy: <LEVEL>
overd bndle siteal quality
emplayyng romben coneiivery the coeast in see wood?
aapapsp csactili dlap lag qucc
Plaase eraiaale mosk conked, wuxl?
essemble Buggape
F.
Label: Audio visuol consisoonyy accellent
CrocExtrops Los
Regresoon
Quality
pvenil budlo viccal quallty 45CORE>
andascona seacy -SCOREX
PLOC Loss
<SCORES
parz organ
Moste jest conetreecyy bed
Mysic quatity bed
Staga Teas Handior ('onerieliee resepla)
Slage 2 Less Finction IAudio
ecample
Figure 3. Training process and architecture of AGAV- Rater. The training process of AGAV-Rater consists of two steps: first, pre-training AGAV-Rater via text-defined levels, and then fine-tuning it via numerical scores.
3.1. AGAVQA-MOS Subset Generation
assesses whether the audio aligns with the corresponding visual elements in the video. Overall A/V quality evaluates the overall quality of the audio and video, including video quality, audio quality, and the compatibility between audio and video. These three quality dimensions are related yet distinct, offering a comprehensive evaluation of AGAVs from multiple perspectives.
AIGC Video Collection. We first collected nigh-quality AIGC videos from AIGC video display websites (Sora (sor, 2024), KLing (kli, 2024), and Gen3 (Gen, 2024)) and the video generation benchmark Vbench (Huang et al., 2024). Additionally, we gathered prompts from FETV (Liu et al., 2024b) and generated AIGC videos using closed-source text-to-video platforms (Pika 1.0 (pik, 2023) and Gen3 (Gen, 2024)). From these AIGC videos, manually selected 386 high-quality videos with clear audio source information, covering 11 types of audio sources. The distribution of main audio sources in AIGC videos is shown in Fig. 2(a).
We invited 15 subjects to participate in our subjective ex- periment. We designed a user interface that allows subjects to watch, listen, and rate the AGAVs across three dimen- sions. The interface displayed 3 continuous quality rating bars, each labeled with a 1-5 Likert scale for rating. We first explained the experiment requirements and the three rating dimensions to each subject. Then, subjects entered a brief training phase to familiarize themselves with the user interface and scoring rules by watching 24 AGAVs. After- ward, they proceeded to the official testing phase. Finally, we normalized the three-dimensional raw scores to Z-scores ranging from 0 to 100 and calculated the mean Z- Z-scores to obtain the mean opinion scores.
Audio Generation from Video. We utilized 8 latest VTA methods, including Diff-Foley (Luo et al., 2024), Foley- Crafter (Zhang et al., 2024a), VTA-LDM (Hu al., 2024), 1m2wav (Sheffer & Adi, 2023), Spec Rahtu, 2021), ModaVerse (Wang et al., 2024f), SVA (Chen et al., 2024), and ElevenLabs (ele, 2023), to generate audio from AIGC videos, thus producing AGAVs. For the Eleven- Labs closed-source platform, we used its API! I, while for the other 7 VTA methods, we used their default weights and code to generate audio. As a result, we obtained a total of 3, 088 AGAVs (8 VTA methods 386 AIGC videos).
(lashin &
3.2. MOSs Analysis
In Fig. 2(c), the maximum, minimum, and average sub- jective scores for the VTA methods are presented across the three dimensions. We can observe that the AGAVs gen- erated by SVA (Chen et al., 2024) exhibit the best audio quality, which can be attributed to the use of proprictary TTA tools, i.e., AudioGin (Kreuk et al., 2022) and Music- Gen (Copet et al., 2024), to generate high-quality sound
Human Evaluation. We asked subjects to rate AGAVs across three dimensions: audio quality, A/V content con- sistency, and overall A/V quality. Audio quality evaluates the perceived quality of the audio, including clarity, natural- ness, and pleasantness. A/V content consistency primarily
hhtts://vidooookk.eeeeenabbbb
4

---

AGAV-Rater: Adapting LMM for Al-Generated Audio-Visual Quality Assessment
effects and background music. ElevenL achieves the best A/V content consistency and overall quality by using ChatGPT-4 to extract key sound information from videos and generate high-quality audio with its proprietary TTA tool, ensuring seamless coherence between audio and video. Krippendorff's ss a (Hayes & Krippendorff, 2007) can be used to measure the quality of the subjects' ratings. We calculate Krippendorff's a for audio quality. A/V content consis- tency, and A/V overall quality, which are 0.6814, 0.7343, and 0.7143, respectively, indicating appropriate variations among subjects. We also randomly divide subjects into two groups and calculate the SRCC of average scores between the two groups. After ten repetitions, the average SRCC for audio quality, A/V content consistency, and A/V overall quality are 0.8043, 0.8318, and 0.8297, validating rating consistency.
4. The AGAV-Rater
In this section, we provide a detailed description of the train- ing process and architecture of AGte-Rater, as illustrated in Fig. 3. We first construct 50, 952 instruction-response pairs for AGAV-Rater pre-training, where no human annota- tions are required, and two text-defined levels are utilized as labels. Then, we utilize the stree-dimensional numerical scores from the AGAVQA-MOS subset as labels to finetune the AGAV-Rater.
4.1. Pre- Pre-Training via Text-Defined Levels
To alleviate the burden of constructing large-scale quality assessment datasets, we first construct 50, 952 instruction- response pairs from real-world audio-caption related datasets for AGAV-Rater pre-training, as shown in Fig. 3. The responses are automatically labeled with two text- defined levels (excellent and bad). We select the VGGSound, AudioCaps, and MusicCaps datasets to cover 3 different scenarios: audio-video, audio-text, and music-text. In the 50, 952 instruction-ressonse pairs, the audio-video, audio- text, and music-text scenarios contain 25, 592, 19, 000, and 6, 000 pairs, respectively. Instruction-responee pairs are designed from two perspectives: content consistency and audio quality. Under each scenario, half of the pairs focus on content consistency, and the other half on audio quality. Take the A/V scenario as an example, the instruction for content consistency related pairs is as follows:
3.3. AGAVQA-Pail Subset Collection
Most VTA GitHub pages exhibit superior performance by dubbing the same video using both their approach and other VTA methods. As shown in Fig. 2(b), collected 294 pub- licly available AGAVs from 8 VTA GitHub pages, including SSV2A (Guo et al., 2024a), WeWaS (Jeong et al., 2025), TIVA (Wang et al., 2024d), V2A-Mapper (Wang et al., 2024a), STAV2A (Ren et al., 2024), "2A-SceneDetector (Yi Li, 2024), Frieren (Wang et al., 2024g), and Sonic Vi- sionLM (Xie et al., 2024). These AGAVs, sourced from third-party platforms, offer more objective and impartial dataset. These VTA GitHub pages are all released in the past year, representing the latest technology in VTA methods. AGAVs with the same video content are grouped, with the optimal AGAVs already labeled on the GitHub pages. We manually verified the accuracy of the optimal AGAVs, and then formed 75 instruction-response pairs as follows:
#User: <audio><<ideo> Can you evaluate the audio-visual content consistency of the given content in one word? #As- sistant: Audio-visual consistency: [Mask]
In the VGGSound dataset, we consider the A/V content consistency of the original video to be excellent. After re- placing the original audio with audio from other categories, the consistency quality becomes bad. In the AudioCaps and MusicCaps dataset, we replace the original caption with another caption and ensure no overlapping nouns between the captions to create audio-text samples with bad audio-text consistency.
Instruction The video is <video>, Audio 1 is <audio 1>, Audio 2 is <audio 2>, Audio 3 is <audio 3>... Which audio best matches this video in terms of audio content, quality, and rhythm? Response: Audio 1.
Due to the inherent subjectivity in human evaluations of AGAVs, different subjects may provide varying scores to the same AGAV. However, when selecting the optimal AGAV, most subjects tend to give the same choice, leading to more reliable quality labels. In some application scenarios, busi- nesses only need to select the highest-quality result from multiple generated options to present to users, without re- quiring detailed quality scores for each AGAV. Therefore, we construct the AGAVQA-Pass subset. This subset evalu- ates the performance of AGAV quality assessment methods by measuring their accuracy in selecting the optimal AGAV. More details of the AGAVQA-3k dataset are provided in Appendix A.
Since the unnatural distortions in AIGC audio are difficult to simulate with real-world distortions like white noise or Gaussian noise, we simulate the unnaturalness by reversing the audio. In the VGGSound dataset, the audio quality of the original A/V sample is labeled as excellent. Videos with reversed audio marked as having bad audio quality. Similarly, for AudioCaps and MusicCaps datasets, if the audio in the audio-caption samples is reversed, the audio quality is labeled as bad. We utilize the content consistency and audio quality related instruction to pre-train the AGAV-Rater, allowing it to develop a basic level of quality perception. The AGAV-Rater utilizes the standard loss function of LMMs, which is the cross-entropy between the labels and output logits.
pairs together
5

---

AGAV-Rater: Adapting LMM for Al-Generated Audio-Visual Quality Assessment
Table 1. Performance comparisons on the AGAVQA-MOS subset from three dimensions. The best performance results are shown in bold,
and the socond-best performance results are underlined.
Dimension
Audio Quality
Content Consistency
Overail Quality
Model Type
Model/Metrics
SRČÖ f KRÖCT "FLCCT C
RMST:]
[
KRCC 7 PrCCT RMSF [
SRCC
FL KRČCf
f RMSE"""
PandaGPT (Arxiv 2023)
0.1326
0.08877
0.1697
10.7643
0.2739
0.1861
0.2943
10.8103
0.1272
0,0844
0.1922
11.1854
Audio Visual
NeXIGPT (ICML 2024)
0.1523
0.1029
0.0962
10.8685
0.0076
0.0047
0.0827
11.2758
0.0418
0.0278
0,0576
11.3874
LMMs
VIIA-1,0 (Arxiv 2024)
0.0717
0.0484
0.0980
10.8600
0.0603
0.0403
0.1015
11.2522
0.1284
0.0859
0.1760
11.2200
Vid2 (Arxiv 2024)
0,4079
0.2787
0.4384
9,8146
0.2326
0.1559
0.2706
10.8631
0,2438
0.1652
0.2657
10.9716
MOSNet (INTERSPEECH 2019)
0.4182
0.2888
0.4926
9.5064
0.2567
0.1713
0.2759
10.8737
0,2723
0.1829
0.2963
10.8916
STOI-Net (APSIPA ASC 2020)
0.2071
0.1364
0.3285
10.2281
0.0408
0.0239
0.1280
11.2099
0.1692
0.1179
0,2157
11.1291
AQA
NISQA (INTERSPEECH 2021)
0,5258
0.3701
0.5875
8,8416
0.3839
0.2641
0.4064
10.3339
0.3374
0.2286
0,3461
10,7026
PAM (INTERSPEECH 2014)
0.3180
0.2149
0,3608
10.0233
-0.0183
-0.0102
0.1441
11 8682
0,1421
0.0950
0.1876
11,3385
AVID-CMA (CVPR 2021)
0.6986
TO.TTOT
0.7350
74310
0,5486
D.385T
'03666
9,3384
0.6
0,4350
0.6246
"8366T
Audio- Video
Alignment
VAST (NIPS 2023)
0.7640
0.5682
0.7848
6.7285
0.6811
0.4944
0.6958
8.1110
0.7094
0.5166
0.7180
7.8624
VALOR (TPAMI 2024)
0.7474
0.5549
0.7773
6,8629
0,6471
0.4662
0.6635
8.4625
0,6888
0,4975
0.7034
8.0904
DNN-RNT 2023]
0.5228
'03666
0,5447
9.1389
0.4348
0,2970
0.4460
10.T23[
0.4940
0,3406
0,5031
9.8488
AVQA
DNN-SND (TIP 2023)
0,5582
0.3932
0.5782
8.9103
0.4686
0,3225
0.4821
9,9072
0.5457
0.3815
0.5548
9.4669
GeneralAVQA (TIP 2023)
0.6102
0.4346
0.6458
8.3252
0,4658
0.3219
0.4768
9.9448
0.6007
0.4249
0.6160
8.9777
AGAV-Rater (Ours)
0.7909
0.5980
0.8108
6,3894
0.7553
0.5639
0.7645
7.2956
0.7458
0,5516
0.7552
7,4611
Table 2. Performance comparisons on the TTA and TTM datasets from two dimensions. The best performance results are shown in bold.
and the second-best performance results are underlined.
Dataset
Text-to-Audio
Text-to-Music
Dimeension
"Audio Quality
Conteni Consistency"
Audio Quality
Cunient Consistency
Model Type
Maidd/Metries
SRCCT
KRCC. i
7
SRCC
KRČČ
PCCC7
SRCC1 1 RRCC 7 1
TSRCC KRCČ7
C
PandaGPT (Arxiv 2023)
0.0888
0.0616
0.2049
0.1976
0.1327
0.3436
0.1722
0.1162
0,2560
0.0748
0.0494
0.2240
Audio- Visual LMMs
NexIGPT (ICML 2024)
-0.0160
-0.0115
0.1677
-0.0097
-0,0068
0.1599
0.0498
0.0334
0.1648
-0.0586
-0.0397
0.2149
VITA-110 (Arxiv 2024)
0.1324
0.0915
0.2589
-0.0116
-0.0092
0.1521
-0.9961
-0.0682
0.1526
0818
0.1224
0.3325
VideoL.LaM (Arxiv 2024)
0,4698
0.3277
0.5061
0.5472
0.3880
0,5514
0.5046
0.3510
0,5258
0,1402
0.0957
0.2754
MöSNet ANTEERSPEEC 9)
0,4858
0.3328
0.4866
0.4222
0.2987
70.4537
0.4646
0.3176
00,4000
0.5206
T.2215
"03364
AQA
STOI-Net (APSIPA ASC 2020)
0.4327
0.3032
0.4603
0,4080
0,2858
0.4425
0.2760
0.2084
0.3346
0,2924
0.2298
0.3734
NISQA (INTERSPEECH 2021)
0.4262
0,3007
0.4603
0.4072
0,2830
0,4353
0,6264
0,4376
0.6394
0.5724
0.4036
0,5859
PAM (INTERSPEECH 2024)
0,5165
0.3655
0.5264
0,4100
0,2833
0.4217
0.6435
0.4630
0.6448
0.3465
0,2354
0.3813
CLAP (ICASSP 2023)
0,7040
0.5231
'077149
0.6700
0,4876
0.6782
0.8103
0.6265
'088096
0.7474
0.3533
"0.744
Audio (Music)
TTM-Retrievl (ICASSP 2023)
0.6121
0.4465
0.6455
0,5818
0,4173
0,5995
0,7586
0.5649
0.7649
0.6974
0.5102
0.7063
-Text
Alignment
VAST (NIPS 2023)
0.7255
0.5421
0.7312
0.6879
0.5021
0.6956
0,8289
0.6412
0.8175
0.7441
0.5557
0.7494
VALOR (TPAMI 2024)
0.6711
0.4972
0.6879
0.5948
0.4250
0.5963
0.2619
0.1838
0.3146
0,2350
0.1653
0.3013
AGAV-Rater (Ours)
0.7390
0.5566
0.7495
0.7330
0.5427
0.7367
0,8322
0.6500
0.8277
0.7719
0,5819
0.7811
4.2. Fine-Tuning via Numerical Scores
vert the MOSs into five text-defined rating levels (excellent, good, fair, poor, and bad) and subsequently format them into entrucctionresponns pairs for visual instruction tuning. However, converting MOSs to five rating levels discards significant amount of information. We directly regress the LMM's last hidden states output three-dimensional numerical scores. The loss between the predicted scores and the ground truth is calculated using PLCC Loss. By predicting numerical scores instead of text-defined levels, AGAV-Rater can more precisely understand human subjec- tive perception.
Evaluation Instruction. During the subjective experiment, we observe that subjects typically first evaluate audio quality, followed by A/V content con- sistency, and finally the overall A/V quality. To mimic the human thought process, we design a multi-dimensional instruction to fine-tune the LMM evaluate the three di- mensions sequentially, c.g.,
sequentie
Can you evaluate the audio quality, audio-visual content consistency, and overall audio-visual quality of the given content one by one? #Assistant: Au- dio quality: [Mask], ), audio-visual consistency: [Mask], overall audio-visual quality: [Mask].
#User: soudioo<<vidos::
4.3. Model Structure
As shown in Fig. 3, the structure of the AGAV-Rater is based on the recently released open-source LMM, VideoLLaMA2 (Cheng et al., 2024), which exhibits excellent A/V percep- tion abilities and strong language comprehension. The video is first converted into ffps image sequence. Then, the im- age sequence and audio signal are separately encoded by the video encoder and audio encoder. After encoding, they are projected into the same vector space through the video projection and audio projection, and input into the large language model together with text embedding. It enables us to process AGAVs, audio-text, and music-text quality assessment tasks within a unified model framework.
Masking Quality Levels. To further improve the model's evaluation capability, we mask the quality levels in instruc- tions, which can prevent the model from overly relying on the ground truth of other scores when predicting quality. It can encourage the model to utilize the inferred quality levels to guide subsequent score predictions, thereby deepening the understanding of the relationships between 3 quality dimensions.
Numerical Score Prediction. Most researchers (Wu et al.; Kou et al., 2024; Zhang et al., 2024b) consider it suboptimal to directly tune LMMs to output numerical scores. They con-
6

---

AGAV-Rater: Adapting LMM for Al-Generated Audio- Visual Quality Assessment
Table 3. Answer accuracy on AGAVQA-Pair subset. Based on the VTA method source, we divide the AGAVQA- subset into 8 categories, with All representing the entire dataset. AGAV-Rater is trained on the AGAVQA-MOS subset, demonstrating cross-datases performance on the AGAVQA- Pair subset. The best result is shown in bold, and the second-best is underlined.
Category
Sonie VisionL. Frieren V2AMapper
TIVA SeencDeeeete STAV2A
SSV2A
ReWaS
All
Random
0.33
0.20
0.41
0.20
0.50
0.25
0.20
0.25
0.32
Question Type: Iulti-Input Comparison, w/o fine-tuning the AGAVQA-MON subset.
PandaGPT (Su al., 2023)
0.29
0.20
0.43
0.22
0.43
0.23
0.20
0.22
0.26
NextGPT (Wu et al., 2024)
0.33
0.20
0.28
0.04
0.36
0.30
0.16
0.22
0.22
VITA-1.0 (Fu ct al., 2024)
0.33
0.12
0.26
0.08
0.43
0.18
0.04
0.19
0.17
Lade (Cheng al., 2024)
0.38
0.17
0.26
0.14
0.21
0.18
0.12
0.28
0.21
Gemini-1.5 Flash-8b (Team et al., 2024a)
0.10
0.05
0.20
0.06
0.29
0.10
0.04
0.17
0.11
Gemini-1.5 Pro (Team al., 2024a)
0.29
0.23
0.35
0.22
0.07
0.13
0.28
0.17
0.23
Gemini-2.0 Flash (Team al., 2024a)
0.10
0.18
0.43
0.26
0.64
0.25
0.24
0.19
0.27
Reka Core (Team et al., 2024b)
0.19
0.18
0.39
0.22
0.43
0.18
0.08
0.22
0.23
Reka Flash (Team et al., 2024h)
0.24
0.20
0.30
0.30
0.50
0.20
0.28
0.22
0.26
GPT-4o+Audio Caption (Hurst al., 2024)
0.29
0.20
0.46
0.18
0.57
0.30
0.20
0.36
0.29
GPT-4o+ Vidco Caption (Hurst et al., 2024)
0.29
0.18
0.48
0.16
0.64
0.40
0.12
0.36
0.30
Question Type: Single-input Scoring, we fine-tuning on the AGAVQA- subset. PandaGPT (Su et al., 2023)
0.29
0.17
0.67
0.10
0.57
0.20
0.20
0.33
0.35
NexIGPT (Wu et al., 2024)
0.43
0,08
0.28
0.30
0.57
0.30
0.40
0.33
0.31
VITA (Fu ct al., 2024)
0.50
0.17
0.39
0.10
0.29
0.30
0.20
0.22
0.27
VideoLLaMA2 (Cheng al., 2024)
0.71
0,25
0.67
0.30
0.29
0.20
0.40
Question Type: Single-input Scoring, with fine-tuning on the AGAVQA-MOS subsct.
AVID-CMA (Morgado et al., 2021)
0.29
0.58
0.61
0.50
0.71
0.50
0.40
0.44
0.52
VALOR (Liu al., 2024a)
1.00
0.75
0.72
0.70
0.71
0.70
0.40
0.44
0.55
VAST (Chen al., 2023)
0.86
0.83
0.78
0.80
0.40
0.64
AGAV-Rater (Ours)
1.00
0,92
0.83
0.80
0.71
0.70
0.60
0,56
0.78
5. Experiments
(Deshmukh et al., 2024).
AVQA: DNN-RNT (Cao et al., 2023a), DNN-SND (Cao et al., 2023a), and GeneralAVQs (Cao al., 2023b).
5.1. Experimental Settings
In this paper, we fine-tune the AGAV-Rater from the pre-
trained weights of VideoLLaMA2 (Cheng al., 2024). The AGAV-Rater model is implemented with PyTorch and trained on two 96GB H20 GPUs. The learning rate is set to
Multimodal alignment: AVID-CMA (Morgado et al., 2021) aligns video features with audio features. VAST (Chen et al., 2023) and VALOR (Liu et al., 2024a) map video, audio, and text into the same semantic space. CLAP (Elizalde et al., 2023) and TTM-Retrieval (Doh et al., 2023) align audio and music with text, respectively.
le 5, and the batch size is set to 9. During pre-training, the number of training epochs is set to 1, and optimization is performed. For fine-tuning, the number of training epochs set to 5 on the AGAVQA-MOS subset and 10 on the TTA and TTM datasets (Deshmukh al., 2024). Fine-tuning the AGAV-Rater model the AGAVQA-MOS subset for 5 epochs using two 96GB H20 GPUs takes approximately 5 hours. All experiments for each method are retrained on the AGAVQA-MOS subset using 5-fold cross-validation. The reported performance of the AGAV-Rater evaluated on the final weights after training.
Except for audio-visual LMMs, all methods are retrained on the AGAVQA-MOS, TTA, and TTM datasets after loading the default weights. Original multimodal alignment meth- ods extract audio and video features using their encoders, then align them into common vector space. We load the default parameters and use these encoders to extract au- dio and video features, which are then concatenated. The concatenated features are subsequently fed into a fully con- nected layer with an output dimension of 3 to predict the three-dimensional scores.
5.2. Compared Methods
Since no specific method has been proposed for evaluating AGAVs, we select state-of-the-art methods from four areas for comparison: audio-visual LMMs, AQA, AVQA, and multimodal alignment, including:
Audio-visual LMMs are directly tested on the dataset using their default weights. For quality-related questions, most audio-visun LMMs respond with text-defined quality levels and are unable to stably output numerical scores. Therefore, following the testing scheme designed in (Wu et al.), we prompt LMMs to receive quality level tokens. Then, ex- tract the probability distribution X' of the predicted level token and convert it into the final predicted scores SLMM as
PandaGPT (Su et al., 2023), NextGPT (Wu et al., 2024), VITA-1-0 (Fu et al., 2024), and VideoLLaMA2 (Cheng al., 2024).
Audio-visual LMMs:
AQA: MOSNet (Lo et al., 2019), STOI-Net (Zezario et al., 2020), NISQA (Mittag et al., 2021), and PAM

---

AGAV-Rater: Adapting LMM for Al-Generated Audio-Visual Quality Assessment
follows:
the optimal AGAV. For multi-input comparisons instruction type, we utilize instruction-response pairs in the AGAVQA- Pair subset to let LMMs answer the optimal audio number. To prevent the model from guessing, we shuffle the audio order and iterate through each audio number as the correct answer, inputting them into LMMs to calculate the response accuracy. This instruction type applies to both open-source and closed-source LMMs, with optimal AGAV judgment based on their textual responses. For the single-input scor- ing instruction type, we sequentially ask the overall quality of each AGAV and compute the final predicted scores by Eq. 1. The AGAV with the highest score is selected as the opti- mal AGAV. Since closed-source LMMs cannot access the token probability distribution, this method is only applied to open-source LMMs. The above 11 audio-visual LMMs use their original model parameters without fine-tuning on the AGAVQA-MOS subset. We also test the accuracy of the audio-video alignment methods which have been fine- tuned on the AGAVQA-MOS subset. As seen in Tab. 3, our method achieves the highest accuracy across all 8 VTA methods, demonstrating that our model has the best ability to judge the optimal AGAV and exhibits strong transferabil- ity.
5 ix
Xi Ej=1
STMM = Lix
where (411-=) = fexcellent, good, fair, poor, bad} are the standard text quality levels. A detailed introduction to com- pared methods can be found in Appendix C.3.
5.3. Evaluation on Multi-diimensoonl Scoring Tasks
We test the multi-dimensional scoring ability of AGAV- Rater on the AGAVQA-MOS, TTA, and TTM datasets, using Spearman Rank-order Correlation (SRCC), Kendall Rank-order Correlation (KRCC), Pearson Linear Correla- tion (PLCC), and Root Mean Squared Error (RMSE). The experimental results are presented in Tab. 1 and Tab. 2. AGAV-Rater demonstrates the best performance across all three datasets. Especially in the content consistency dimen- sion, the SRCC metric shows 1 11%, 7%, and 3% improve- ments on the AGAVQA-MOS, TTA, and TTM datasets, respectively. This highlights the powerful content under- standing capability of LMMs, which can significantly aid in evaluating the consistency of AIGC audio and video (text). Furthermore, the poor performance of audio-visual LMMs shows that they can not adapt well to quality assessment tasks, proving that our training process effectively enhances the LMM's understanding of human perception. Traditional AQA and AVQA methods perform weaker on the A/V con- tent consistency compared to the audio quality, as these methods have difficulty adapting the unique distortions in AIGC media. Audio-video alignment methods focus on the semantic alignment between audio and video, being ca- pable of recognizing semantic-level distortions in AGAVs and demonstrating suboptimal performance.
5.5. Ablation Study
Effects of Pretraining Procedure. The "w/o Pretrain" col- umn in Tab. 4 shows AGAV-Rater's performance with- out the pretraining step. Compared to the AGAVQA-MOS subset, the pretraining step significantly enhances AGAV- Rater's performance on the smaller TTA and TTM datasets. This is because pretraining provides rich prior knowledge, which effectively compensates for the limited scale and diversity of these smaller datasets. As shown in Tab. I, VideoLLaMA2 exhibits relatively weak performance on the content consistency dimension for the TTM dataset. The pre-training procedure, through the music-text instruction- response pairs, enhances the music perception ability of AGAV-Rater, thereby improving its performance the con- sistency dimension of the TTM dataset. On the AGAVQA- MOS subset, the pretraining step shows a more significant improvement in the overall A/V quality, compared to audio quality and A/V content consistency. We believe this is because AGAV-Rater relies on the scores of audio quality and A/V content consistency when predicting the overall A/V quality, and pretraining helps AGAV-Rater learn au- dio quality and A/V content consistency faster, reducing disturbances in learning overall A/V quality.
5.4. Evaluation on Optimal AGAV Selection Tasks
The AGAVQA-Pair subset was collected from 8 VTA web- pages, dividing it into 8 corresponding categories. For each category, the optimal AGAV is generated by the correspond- ing VTA method. Since there is no overlap between the video content in the AGAVQA-MOS and AGAVQA-Pair subsets, we perform cross-dataset validation experiments by training the AGAV-Rater on the AGAVQA-MOS sub- set and testing on the unseen AGAVQA-Pai subset. We collect 11 audio-visual LMMs as comparison methods, in- cluding 4 open-source LMMs and closed-source LMMs. For GPT-4o+ caption, we utilize GPT-4o-audio to gen- erate captions for audio, then feed the text, video, and audio captions into GPT-4o. Similarly, for + cap- tion, we utilize GPT-4o to generate captions for video, then feed the text, audio, and video captions into GPT-4o-audio. As shown in Tab. 3, we test their accuracy in answering optimal AGAV questions on the AGAVQA-Pain subset. We designed two types of instructions to prompt LMMs for
Effects of Scoring Method. We convert MOSs into text- defined levels and use text-defined levels as labels in the fine- tuning step to replace MOSs. As shown in the "Finetuning with Levels" column of Tab. 4, replacing numerical scores with text-defined levels for fine-tuning results in slight performance decrease. This demonstrates that the AGAV-
8

---

Rater:: Adapting LMM for Al-Generated Audio-Visual Quality Assessment
Table 4. Ablation study of the proposed AGAV-Rater.
Dataset
AGAVQA-MOS
TTA
TTM
Dinension
Audio Quality
Consistency
Overall Quality
Audio Quality
Consistency
Audio Quality
Consistency
Strategy
SRCC'
PLCC
FLCCf,SRCC1 f PLCC
SRCCT PLCC
SRCCC PLCC
SRCC f PLCCT f SRCCT 1
w/o Pretrain
0,7856
0,8030 0.7503
0.7599
0.7141
0.7202
0,7040
0.7117
0.7104
0.7215
0.8218
0.8233 0,7042
0.7188
Finetuning with Levels
0.7832
0.8043 0.7499
0.7451
0.7078
0.7167
0.7005
0.7051
0.7249
0.7324
0,8102
0.8121
0.7359
0.7532
Single-Dimension Instruction
0.7845
0.8035
0.7511
0.7599
0.6865
0.7007
0.7143
0.7152
0.7286
0.7277
0.8297
0.8253
0.7565
0.7759
AGAV-Ratee
0.7909
0.8108
0.7553
0.7645
0.7458
0.7552
0.7390
0.7495
0.7330
0.7367
0.8322
0.8277 0.7719 0.7811
Table 5. Ablation study the base models.
5.6. Enhancing ElevenI Results via AGAV-Rater
Dimension Matii
Quality
Conasiency
OierrilQQaalig
5
HLCC F SKCC" PLOC
SRCCT
PLCCTT
We finally conduct a subjective experiment to demonstrate that AGAV-Rater can help ElevenLabs select the high- quality audio to present to users. We collect 230 silent AIGC videos from T2V-CompBench (Ji et al., 2024) and Sora (sor, 2024), generating 2 AGAVs for each video us- ing ElevenLabs. We then utilize AGAV-Rater to select the higher-quality AGAVs. A total of 10 subjects are invited to watch and listen to AGAVs. 80% of them prefer the AGAVs selected by AGAV-Rater for its better quality. This demonstrates that AGAV-Rater can be applied in real-world scenarios to help improve the quality of outputs from VTA methods. On the project page", we display the AGAVs that AGAV-Rater considers high quality and low quality.
GroundingPP (ACL 3020
0.4387
0.1194
0.5067
0.4764
0.1915
0.3297
O (CVPR 2024)
0.6578
06184
0.6327
0.6388
AGAV-Rater
0.7909
0.8108
0.7583
0.7645
0.7458
0.7552
Rater, by using quality regression to map hidden states to numerical scores, allows the AGAV-Rater to more finely learn human subjective perception.
Effeets of multi-Dimensson Instruction. To guide LMMs to score according to the human thought process, we design a multi-dimensional instruction that enables AGAV-Rater to predict scores for all three dimensions simultaneously. For comparison experiments, we break this down into three single-dimensional instructions, where each instruction fo- cuses on one dimension's 's quality. As shown in the "Single- Dimension Instruction" column of Tab 4, we can see that the multi-dimensional instruction improves AGAV-Rater's performance on the overall A/V quality. This is because guiding AGAV-Rater to first consider audio quality and A/V content consistency helps better predict overall A/V quality. Additionally, the multi-dimensional instruction enables mu- tual enhancement between the two dimensions on the TTA and TTM datasets, further boosting performance.
6. Conclusion
In this paper, we construct AGAVQA-3k, the first AGAV quality assessment dataset, which labels AGAV quality in two ways: multi-dimensional score prediction and opti- mal AGAV selection. We propose novel LMM-bbseed AGAV quality assessment method, AGAV-Rater. AGAV- Rater demonstrates superior score prediction capabilities on the AGAVQA-MOS, TTA, and TTM datasets, and achieves the highest accuracy in identifying the optimal AGAV on the AGAVQA-Pair subset. AGAV-Rater enhances users a better audio-visud experience and enhance the quality of VTA method outputs.
Effects of the Base Models. We conduct ablation studies using GroundingGPT (Li al., 2024) and OneLLM (Han al., 2024) as base models the AGAVQA-MOS subset. For OneLLM and GroundingGPT, we first load the default weights, and then fine-tune them using the official training code on the AGAVQA-MOS subset. To ensure fairness, we also add the quality regression module, directly regressing the LLM's last hidden states to output three-dimensional nu- merical scores. As shown in Tab. 5, AGAV-Rater achieves the best performance. The main reason for this is that Vide- oLLaMA2 is designed for audio-video content understand- ing and pre-trained on more diverse audio-video datasets, making it more suitable for our quality assessment task. GroundingGPT focuses more on localization and visual un- derstanding and is not designed or trained to understand continuous audio-video content. Its ability to comprehend video quality may be weaker. OneLLM is a general mul- timodal model that, while supporting audio and video pro- cessing, is not specifically optimized or enhanced for video and audio aligmment. Its audio-related dataset only includes nudio-text data, and OneL.M is more suited to text-vision or tuxto matching and understanding, rather than specific audio-video content.
Impact Statement
The quality of Al-generated audio and video must align with human preferences. Among existing models, AGAV-Rater achieves the highest consistency with human perceptual evaluations of AGAVs, indicating its potential for supervis- ing and controlling the quality of AGAVs. We will continue to focus on and advance this research in the future.
References
Elevenlabs.
Prime Voice AI, 2023. e elevenlabs.
URL https://
Pika 1.0. Accessed December 28, 2023 [Online] https: /www:piik..ar.e.. 2023. URL ht https: //www/ aika.art/.
hmaps//agaa-matergihuhhi
9

---

AGAV-Rater: Adapting LMM for Al-Generated Audio-Visual Quality Assessment
Gen-3.
Accessed June 17, 2024
[Online]
Deshmukh, S., Alharthi, D., Elizalde, B., Gamper, H., Is- mail, M. A., Singh, R., Raj. B., and Wang, H. Pam: Prompting audio-language models for audio quality as- sessment. arXiv preprint arXiv:2402.00282 2024.
introducing 3-apha, 2024.
URL
httpss//rruuwayyllcoo////s introducing--ent apah.
Doh, S., Won, M., Choi, K., and Nam, J. Toward universal text-to-music retrieval. In ICASSP, 2023.
Kling. Accessed June 6, 2024 [Online] https:// klingga..kuishoou.co 2024. URL https: www.siu....kiihh.c.com com/.
Elizalde, B., Deshmukh, S., Ismail, M., and Wang, H. Clap learning audio concepts from natural language su- pervision. In ICASSP, pp. 1-5, 2023.
Sora. Accessed February 16, 2024 [Online] https: //wpp/ai....//////// com///aiccom/aee sora/, 2024. URL https://
comenai.commoon sora//
Fu, C.,, Lin, H., Long, Z., Shen, Y., Zhao, M., Zhang, Y., Dong, S., Wang, X., Yin, D., Ma, L., et al. Vita: Towards open-source interactive omni multimodal llm. arXiv preprint arXiv:2408.05211 2024.
Agostinelli, A., Denk, I., Borsos, Z., Engel, J., Verzetti, M., Caillon, A., Huang, Q. Jansen, A., Roberts, A., Tagliasacchi, M., al. Musiclm: Generating music from text. arXiv preprint
2023.
Guo, W., Wang. H., Cai, W., and Ma, Gotta hear them all: Sound source aware vision audio generation. arXiv preprint arXiv:2411.15441 2024a.
Beerends, J. G., Schmidmer, C., Berger, J., Obermann, M., Ullmann, R., Pomy, J., and Keyhl, M. Perceptual objec- tive listening quality assessment (polqa), the third genera- tion itu-t standard for end-to-ena speech quality measure- ment part temporal alignment. AES, 61663666,3 2013.
Guo, Y., Yang, C., Rao, A., Wang, Y., Qiao, Y., Lin, D., and Dai, B. AnimateDiff: Animate your personalized text-to- rext-to-image diffusion models without specific tuning. In ICLR, 2024b.
Cao, Y., Min, X., Sun, W., and Zhai, G. Attention-guided neural networks for full-reference and no-reference audio- visual quality assessment, IEEE TIP, 32:1 12:1882-1896. 2023a.
Han, J., Gong, K., Zhang, Y., Wang, J., Zhang. K., Lin, D., Qiao, Y., Gao, P., and Yue, X. Onellm: One framework to align all modalities with language. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26584-26595, 2024.
Cao, Y., Min, X., Sun, W., and Zhai, G. Subjective and ob- jective audio-visual quality assessment for user generated content. IEEE TIP, 2023b.
Hayes, A. F. and Krippendorff, K. Answering the call for standard reliability measure for coding data. Communi- cation methods and measures, 1(1):77-89, 2007.
Chen, G., Wang. G., Huang, X., and Sang. J. Semantically consistent video-to-aus tu-dioo generation using multimodal language large model. arXiv preprint arXiv:2404. 16305, 2024.
Hong. W., Ding, M., Zheng, W., Liu, X., and Tang, J. CogVideo: Large-scale pretraining for text-to- video generation via transformers. arXiv preprint wwXiv:2255.58666 2022.
Chen, H., Xie, W., Vedaldi, A., and Zisserman, A. Vg- gsound: A large-scale audio-visual dataset. In ICASSP. pp. 721-725, 2020.
Hu, Y., Gu, Y., Li, C., Chen, R., and Yu, D. Video-to-audio generation with fine-grained temporal semantics. arXiv preprint arXiv:2409. 14709, 2024.
Chen, S., Li, H., Wang, Q. Zhao, Z., Sun, M., Zhu, X., and Liu, J. Vast: caan--addo-uuuiill---eeeeee foundation model and dataset. NeurlPS, 36:72842-72866, 2023.
Huang, Z., He, Y., Yu, J., Zhang, F., Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q., Chanpaisit, N., et Vbench: Com- prehensive benchmark suite for video generative models. In CVPR, pp. 21807-21818, 2024.
Cheng. Z., Leng, S., Zhang, H., Xin, Y., Li, X., Chen, G., Zhu, Y., Zhang, W., Luo, Z., Zhao, D., et al. Vide- oLLaMA 2: Advancing spatial-temporal modeling and audio understanding in video-LLMs. arXiv preprint arXiv:2406.00 07476, 2024.
Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card. arXiv preprint arXi::2410.21276 2024.
Copet, J., Kreuk, F., Gat, I., Remez, T., Kant, D., Synnaeve, G., Adi, Y., and Défossez, A. Simple and controllable music generation. NeurIPS, 36, 2024.
lashin, V. and Rahtu, E. Taming visually guided sound generation, arXiv preprini arXiv:21 10.08791, 10.089,1, 2021.
10

---

AGAV-Rater: Adapting LMM for Al-Generated Audio-Visual Quality Assessment
Jeong, Y., Kim, Y., Chun, S., and Lee, J. Read, watch and scream! sound generation from text and video. In AAAI, 2025.
Mittag, G., Naderi, B., Chehadi, A., and Möller, S. Nisqa: A deep enn-self-attention model for multidimensional speech quality prediction with crowdsourced datasets. In INTERSPEECH, 2021.
Ji, P., Xiao, C., Tai, H., and Huo, M. T2VBench: Bench- marking temporal dynamics for toxt-to-video generation. In CVPR, pp. 5325-5333, 2024.
Morgado, P., Vasconcelos, N., and Misra, I. Audia-visual instance discrimination with cross-modal agreement. In CVPR, pp. 12475-12486 2021.
Kim, C. D., Kim, B., Lee, H., and Kim, G. Audiocaps: Generating captions for audios the wild. In NAACL- HILT, pp. 1 199-132, 2019.
Ren, Y. Li, C., Xu, M. Liang, W., Gu, Y. Chen, R., and Yu, D. STA-V2A: Video-to-audio generation with semantic and temporal alignment. arXiv preprint arXiv:2409.08601 2024.
Kou, T., Liu, X., Zhang, Z., Li, C., Wu, H., Min, X., Zhai, G., and Liu, N. Subjective-aligned dataset and metric for text-to-video quality assessment. In ACMM, pp. 7793- 7802, 2024.
Rix, A. W., Beerends, J. G., Hollier, M. P., and Hekstra, A. P. Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs. ICASSP, volume pp. 752, 2001.
Kreuk, F. Synnaeve, G., Polyak, A., Singer, U., Défossez, A., Copet, J., Parikh, D., Taigman, Y., and Adi, Y. Audio- gen: Textually guided audio generation. arXiv preprini arXiv:2209.15330 2022.
Sheffer, R. and Adi, Y. I hear your true colors: Image guided audio generation. In ICASSP, pp. 1-5, 2023.
Li, Z., Xu, Q., Zhang, D., Song, H., Cai, Y., Qi, Q., Zhou, R., Pan, J., Li, Z., Tu, V., al. Groundinggpt: Language enhanced multi-modal grounding model. In ACL, pp. 6657-6678, 2024.
Su, Y., Lan, T. Li, H., Xu, J., Wang, Y., and Cai, D. PandaGPT: One model to instruction-follow them all. arXiv preprint arXiv:2305. 16355, 2023.
Liu, J., Chen, S., He. X., Guo, L., Zhu, X., Wang, W., and Tang, VALOR: lssoo-audio-language omni- perception pretraining model and dataset. IEEE TPAMI, 2024a.
Sun, Y., Min, X., Duan, H., and Zhai, G. The influence of text- eext-guidance on visual attention. In IEEE ISCAS, pp. 1-5, 2023.
Sun, Y., Min, X., Duan, H., and Zhai, G. How is visual at- tention influenced by text guidance? database and model. IEEE TIP, 2024a.
Liu, Y., Li, L.., Ren, S., Gao, R., Li, S., Chen, S., Sun, X., and Hou, L. Fetv: A benchmark for fine-grained evaluation of open-domain text-to-video generation. NeurIPS, 36, 2024b.
Sun, Y., Zhang, Z., Wu, H., Liu. X.. Lin, W., Zhai, G., and Min, X. Explore the hallucination low-level perception for milms, arXiv preprint arXiv:2409.09744 09744, 2024b.
Lo, C.-C., Fu, S.WW., Huang, W.-C., Wang. X., Yamagishi, J., Tsao, Y., and Wang, H.-M. Mosnet: Deep learning based objective assessment for voice conversion. In IN- TERSPEECH, 2019.
Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint arXiv:2403.0553 2024a.
Lu, J., Clark, C., Lee, S., Zhang, Z., Khosla, S., Marten, R., Hoiem, D., and Kembhavi, A. Unified-IO 2: Scaling autoregressive multimodal models with vision language audio and action. In CVPR, pp. 26439-264555, 2024.
Team, R., Ormazabal, A., Zheng, C., d' Autume, C. d. M., Yogatama, D., Fu, D., Ong. D., Chen, E., Lamprecht, E., Pham, H., al. Reka core, flash, and edge: A series of powerful multimodal language models. arXiv preprint arXiv:2404.1233 12387, 2024b.
Luo, S., Yan, C., Hu, C., and Zhao, H. Diff-foley: Syn- chronized video-to-audio synthesis with latent diffusion models. NeurlPS, 36, 2024.
Manocha, P. and Kumar, Speech quality assessment through MOS using non-matching references. arXiv preprint ar Xiv:2206.. 12285, 2022.
Wang, H., Ma, J., Pascual, S., Cartwright, R., and Cai, W. V2a-mapper: A lightweight solution for -isidioo generation by connecting foundation models. In AAAI, volume 38, pp. 15492-15501, 2024a.
Min, X., Zhai, G., Zhou, J., Farias, M. C., and Bovik, A. C. Study of subjective and objective quality assessment of audio-visual signals. IEEE TIP, 29:6054 6068, 2020.
Wang. J., Duan, H., Zhai, G., and Min, X. Understanding and evaluating human preferences for ai generated images

---

AGAV-Rater: Adapting LMM for Al-Generated Audio- Visual Quality Assessment
with instruction tuning. arXiv preprint arXiv:2405.07733 2024b.
Zhang, Y., Gu, Y., Zeng, Y., Xing, Z., Wang, Y., Wu, Z., and Chen, K. Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds. arXiv preprint arXhv:440701448 2024a.
Wang, J. Duan, H., Zhai, G., Wang. J., and Min, X. AIGV- Assessor: Benchmarking and evaluating the perceptual quality of text-to-video generation with Imm. arXiv preprint arXiv2241..12222 2024c.
Zhang, Z., Wu, H., Zhou, Y., Li, C., Sun, W. Chen, C., Min, X., Liu, X., Lin, W., and Zhai, G. Lmm-peqa: Assisting point cloud quality assessment with Imm. In ACMM, pp. 7783-7792, 2024b.
Wang X., Wang, Y., Wu, Y., Song, R., Tan, X., Chen, Z., Xu, H., and Sui, G. Tiva: Time-aligned video-to-audio generation. In ACMM, pp. 573-582, 2024d.
Wang, X., Zhang, S., Yuan, H., Qing, Z., Gong, B., Zhang, Y., Shen, Y., Gao, C., and Sang, N. A recipe for scaling text-to-video generation with text-free vidcos. In CVPR, 2024e.
Wang, X., Zhuang, B., and Wu, Q. Modaverse: Efficiently transforming modalities with llms. In CVPR, pp. 26606- 26616, 2024f.
Wang. Y., Chen, X., Ma, X., Zhou, S., Huang, Z., Wang, Y., Yang. C., Hc, Y., Yu, J., Yang. P., et al. LAVIE: High- quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309.15110 2023.
Wang, Y., Guo, W., Huang, R., Huang. J., Wang, Z., You, F., Li, R., and Zhao, Frieren: Efficient video-to-aude generation with rectified flow matching. arXiv preprint arXiv:2406.00320, 2024g.
Wu, H., Zhang, Z., Zhang, W. Chen, C., Liao, L., Li, C., Gao, Y., Wang, A., Zhang, E., Sun, W., al. Q- Align: Teaching LMMs for visual scoring via discrete text-defined levels. In ICML.
Wu, S., Fei, H., Qu, L., Ji, W., and Chua, T.-S. NEXT-GPT: Any-to-any multimodal Ilm. In ICML, pp. 53366-53397, 2024.
Xie, Z., Yu, S., He, Q., and Li, M. Sonievisionlm: Playing sound with vision language models. In CVPR, pp. 26866- 26875, 2024.
Yi, M. and Li, M. Efficient video audio mapper with visual scene detection. arXiv preprint 2rXiv:2409.09823, 2024.
Ying, Z., Ghadiyaram, D., and Bovik, A. Telepresence video quality assessment. In ECCV, pp. 327-347. Springer, 2022.
Zezario, R. E., Fu, S.-W., Fuh, C.-S., Tsao, Y., and Wang, H.-M. STOI-Net: deep learning based non-intrusive speech intelligibility assessment model. In APSIPA ASC, pp. 482-486, 2020.
12

---

AGAV-Rater: Adapting LMM for Al-Generated Audio-Visual Quality Assessment
Table 6. Categorization of prompts for generating ATGC vidcos.
Category
Description
Key Words
birds fly, dogs sleep, dogs sneeze, penguins walk, fishes float, bees fly,
Animals
Describe animal behaviors and movements
ducks swim, turiles swim, dragons fly. camels walk, cats meow.
Include envirommental sounds
Flag waves, country road, restaurant, rocket goes off, fireworks explode,
Artificial
generated by human- madde objects
burn a candle, city road, fire.
pour whiskey, pack box, bread, coffee machine, make salad, preparation of vegetable soup, sparkling champagne, cook pasta coffee beans fall, pour into a cocktail glass.
Food
Describe food preparation and eating
cook, water ski, go boating. wash gravel, swim, walk, get lake, box, pour liquid, fight with swords.
People
Descrihe human action
flond water, ice melt, fallen leaves, water plants, mountain river, foresi, wind, beach, sea, snow all, waterfall, rain.
Nature
Include natural envirommental sounds
Vehícles
Describe vehicles
car, war vehicle. boat, airerfff bike.
A. More Details of AGAVQA-3k Construction.
A.1. Detailed Information of AIGC Video Collection
We begin by collecting AIGC videos from public display websites, including Sora', KLing", and Gen33, Additionally, we gather AIGC videos generated by AnimateDiff (Guo et al., 2024b), CogVideo (Hong et al., 2022), Gen3 (Gen, 2024), KLing (kli, 2024), LaVie (Wang et al., 2023), Pika (pik, 2023), and TF-T2V (Wang et al., 2024e) from the video generation benchmark Vbench (Huang et al., 2024). To diversify the types of audio sources in the videos. we also collect prompts containing audio information from FETV (Liu et al., 2024b). As shown in Tab. 6, these prompts animals, artificial, food, people, nature, and vehicles. We also list the key terms associated with each category. Finally, use these prompts to generate AIGC videos via closed-source text-to-video platforms (Pika 1.0 (pik, 2023) and Gen3 (Gen, 2024)).
categorized into 6 types:
A.2. Detailed Information of VTA Methods
We utilize 16 state-of-the-art VTA methods to construct the AGAVQA-3k dataset, encompassing both open-source and closed-source methods. For open-source models, we rely on official repositories and utilize default weights to generate audio. For closed-source models, we utilize publicly accessible APIs provided by open platforms. For models without publicly available code, we collect public AGAVs showeased on their GitHub pages.
1. DIFFUSION BASED VTA METHODS
Diff-Foley: Diff-Foley (Luo al., 2024) is a synchronized video-to-audio synthesis method that utilizes latent diffusion model (LDM) to generate high-quality audio with improved temporal synchronization and audio-visun relevance.
FoleyCrafter FoleyCrafter (Zhang et al., 2024a) is pluggable module integrated into a text-to-audio generator, enabling it to generate high-quality audio synchronized with video content. It primarily utilizes two key components: a semantic adapter for semantic aligmment and a temporal controller for temporal synchronization.
VTA-LDM. VTA-LDM (Hu et al., 2024) leverages the recently popular grounding segment anything model (Grounding SAM) extract fine-grained semantic features from video frames and then uses LDM to generate high-quality audio.
SSV2A. S (Guo et al., 2024a) is a Sound Source-Aware Video-to-Audio generator, which can locally perceive multimodal sound sources from a scene through visual detection and cross-modality translation.
ReWaS. ReWaS (Jeong et al., 2025) is video-and--ex--to-soumdd generation method, where video conditions control the text-to-audio generation model to create audio that matches the video.
TIVA. TIVA? (Wang et al., 2024d) a novel time-alige alignee aideo-to-audio generator that jointly achieves semantic matching and temporal synchronization when generating audio. TIVA encodes the semantic information of the video and predicts its
'haps://openii.com//// com/sora/
'ntts://Klingai.comm
13

---

AGAV-Rater: Adapting LMM for Al-Generated Audio-Visual Quality Assessment
rhythmie layout, then utilizes this information conditioning for latent diffusion-based audio generator to produce the audio.
V2A-Mapper. V2A-Mapper' (Wang et al., 2024a) employs CLIP, CLAP, and AudioLDM to design a lightweight VTA method. It maps the latent space from the visual CLIP model to the auditory CLAP model and then uses the pre-trained MudioLDM to generate high-fidelity, visually-aligned sound.
STAV2A. STAV2A" (Ren et al., 2024) is a semantic and temporal aligned video-to-audio method that generates audio by conditioning on both text and video features. STAV2A utilizes an LDM initialized with text-to-audio prior knowledge and guided by cross-modal features from both text and video.
V2A-Sceneed Detecoo. "2A=-cennDeetector" 10 (Yi & Li, 2024) combines LDM with scene detector to address the challenge of multiple visual scene transitions in videos. It can identify and handle multiple scenes in video, generating corresponding audio for each.
A.2.2. TRANSFORMER BASED VTA METHODS
Im2wav. Im2wav (Sheffer& Adi, 2023) is based on two transformer language models. First, language model generates a low-level audio representation. Then, additional language model upsamples the audio tokens generate high-fidelity audio samples.
Spec VGANN. Spec (lashin & Rahtu, 2021) is a disually-inducee audio generation method. It utilizes transformer to sample a new spectrogram from a pre-trained spectrogram codebook, given the set of video features, thereby generating the corresponding audio.
A.2.3. LLM BASED VTA METHODS
Moda Verse. Moda Verse (Wang et al., 2024f) is multi-modal large language model capable of understanding and converting content across various modalities, including images, videos, and audio. We leverage its iido-tt--addoo capability to add sound to silent vidcos.
SVA. SVA (Chen et al., 2024) is semantically consistent video-to-audio generation framework. SVA leverages the capabilities of multi-modal large language models (MLLMs) to understand the video semantics from key frames and generate creative audio plans. These plans are then used as prompts for text-to-audio models, simultancously generating sound effects and background music.
Visionissin.MM. VisionL (Xie et al., 2024) is new framework designed to generate wide range of sound effects by leveraging iision-languaee models (VLMs). It uses VLMs to recognize events in the video and generate sounds that match the video content.
A.2.4. FLOW MATHCING BASED VTA METHOD
Frieren. F 12 (Wang et al., 2024g) is a VTA model based on rectified flow matching. Frieren generates high-quality audio in just a few, or even single, sampling step through backflow and guided vector field distillation process.
A.2.5. PROPRIETORY VTA METHOD
LasvenLabs. ElevenLabs (ele, 2023) utilizes the ElevenLabs texts to sound effects API. It extracts sound source information from key frames using ChalGPT-40 and then inputs this data into ElevenLabs to generate the corresponding audio.
"htpss///2e-mappergiihbbbb
"hugss//fiireen-2aagggtuubo httt:s///vieeoooffeeeeeellaa
14

---

AGAV-Rater Rater: Adapting LMM for Al-Generated Audio- Visual Quality Assessment
Md Cathy
Teaing
Figure example of scoring interface. The audio-visual setup consists two 1080p monitors and headphone. One monitor displays the scoring interface, while the other is used for subjects to watch the AGAV videos and listen to the audio through the headphones.
B. More Details of Human Evaluation
B.1. Scoring Dimensions
During the subjective scoring process, participants are asked evaluate the AGAVs from three dimensions: audio quality, A/V content consistency, and overall A/V quality. These three dimensions provide a comprehensive and detailed assessment of the sGAVs. Audio quality mainly focuses the naturalness and clarity of the audio, minimizing the influence of the video content. Higher scores indicate that the audio is more natural, clear, and free of distortion, while lower scores reflect that distortions and noise in the sound degrade the listener's 's auditory experience. The scoring range is from 0 to 5, with scores accurate to one decimal place. The scoring criteria are as follows:
4-5 (Excellent): The audio is natural, clear, and free of distortion.
3-4 (Good): The audio generally clear but contains slight distortion or noise.
2-3 (Fair): The audio is somewhat muffled,, with noticeable distortion noise.
1-2 (Poor): The audio quite muffled, with severe distortion and noise that significantly affect the listening experience.
0-1 (Bad): The audio is completely distorted.
A/V content consistency independent of audio quality and primarily focuses on whether the sounds or music the audio align with the video content. Higher scores indicate a strong correlation between the audio and the video content, while lower scores indicate low correlation, with the audio lacking the sound elements intended to be conveyed by the video. The scoring criteria are follows:
4-5 (Excellent):
The audio content is highly consistent with the video content.
3-4 (Good): The audio content is generally consistent with the video content.
2-3 (Fair): The audio content has limited correlation with the video content.
1-2 (Poor): The audio content is mostly inconsistent with the video content.
0-1 (Bad): The audio largely distorted, and the audio content completely inconsistent with the video content.
Overall A/V quality mainly focuses on the overall perceptual experience of the audio and video, including audio quality, A/V content consistency, and A/V temporal synchronization. Higher scores indicate clear and natural audio, with both content and timing highly consistent with the video. Lower scores indicate obvious audio distortion and low correlation with the video content. The scoring criteria are follows:
15

---

Raterr Adapting LMM for Al-Generated Audio- Visual Quality Assessment
350
300
Video Number
250
Video Number
250
Video Number
250
150
100
100
100
50
50
30
30
40
50 MOS
60
70
80
20
40
50
60
70
80
20
60
MOS
MOS
(a) Audio Quality
(b) A/V Content Consistency
(c) Overall A/V Quality
Figure 5. Distribution of MOSs across three dimensions in the AGAVQA-MOS subset.
4-5 (Excellent): The overall quality of the audio and video excellent.
3-4 (Good): The overall quality of the audio and video is good, with slight distortion.
2-3 (Fair): The overall quality of the audio and video is average, with noticeable distortion.
1-2 (Poor): The overall audio and video quality is poor, with severe distortion or mostly inconsistent content.
0-1 (Bad): The overall audio and video quality is very poor, with completely inconsistent content and severely distorted audio or video.
We developed a subjective evaluation guideline that includes the scoring criteria for the three dimensions mentioned above. Before each participant begins subjective experiment, guide them through reading this document to familiarize themselves with the scoring criteria for each dimension, thereby ensuring accuracy and consistency of the ratings across different participants.
the
B.2. Scoring Interface
The example of the scoring interface is shown in Fig. 4. We designed the scoring interface using the Python Tkinter package. Before scoring, each subject is prompted to enter their username, which allows the retrieval of their scoring progress and subsequent presentation of the scoring interface. The interface includes three continuous quality rating bars, three navigation buttons, and a display of the number of AGAVs that have been scored. Each rating bar is labeled with 1-5 Likert scale, and participants can drag the slider to assign scores accurate to one decimal place. The navigation buttons, including "Prev", "Repeat", and "Next", enable participants to modify the previous AGAV's score, replay the current AGAV, and submit their score to proceed to the next AGAV, respectively, facilitating efficient scoring. All AGAVs are viewed at their original resolution without any scaling cropping.
B.3. Evaluation Environment
The official testing phase was conducted in a controlled laboratory setting with normal indoor lighting and a quiet environment. Subjects were seated at comfortable distance of approximately 60 cm from the screen. We used Redmi 23.8-inch monitor with a resolution of 1920 x 1080 and Sony WH-1000XM4 headphones to minimize distortions introduced by the viewing and listening devices during human evaluation.
B.4. Subject Selection
We invited subjects familiar with AVQA and AGAV to participate in on-site training sessions. Detailed explanations of the scoring criteria for each dimension were provided, along with additional AGAV samples for practice. Expert reviewers then evaluated the annotations and selected 15 qualified subjects. Each subject rated all 3, 088 samples in the AGAVQA-MOS subset in a randomized order. To prevent fatigue, each subject was limited to rating a maximum of 60 samples per day, completing the entire task in approximately two months. This protocol ensured the validity and robustness of each subject's ratings.
16

---

Rater:: Adapting LMM for Al-Generated Audio- Visual Quality Assessment
SVA
DiCT-Rley
Foley Crafter
Modavarse
SpecVQGAN
Eleveniaas
Em2wwa
VTA-LDM
Fiiz
File
Fire
Farrayy
Feezacy
Farrsy
Cooking
Cooking
Cooking
Ohjecs
Chris
Chija
Nainal
Acimal
Atimal
People
People
People
Sonny
Surrer
Seanery
Vahicle
Vahicl
Velicle
Soi
Quality
A/ Content Consutttnnnn
Quality
Figure 6. Comparison of mean MOSs of different VTA methods across various audio sources in vidcos. Results on audio quality, (b)
Results on A/V content consistency, (c) Results on overall A/V quality.
Table 7. Standard deviation of subjective scores for each category in the AGAVQA-MOS subset.
Animal
Water
People
Vehicle
Object
Scencry
Sca
Fantasy
Fire
Instrument
Cooking
12.56
12.16
12.20
11.79
12.57
12.74
12.37
13.11
12.35
11.93
12.23
B.5. More Subjective Scores Analysis
In the AGAVQA-MOS subset, we collected MOSs across three dimensions. SRCC between audio quality and A/V content consistency is 0.6860, indicating that the two dimensions are independent. SRCC between audio quality and A/V overall quality is 0876, and between content consistency and overall quality is 0.7926, suggesting that overall quality is influenced by both audio quality and content consistency.
The distribution MOSs for each dimension is shown in Fig. 5. MOSs are evenly distributed between 20 and 80 for all three dimensions. Additionally, as illustrated in Fig. 6, we compare the average MOSs of different VTA methods across various audio sources in videos. The audio generated by SVA consistently demonstrates superior quality across all source types. Most VTA methods show better audio quality in the instrument category. In the A/V content consistency dimension, most VTA methods struggle with the fantasy, cooking, people, and scenery categories, where it is challenging to generate audio that aligns with the content of these four types. In the overall A/V quality dimension, VTA methods underperform in the people and cooking categories.
The range MOSs [0, 100]. We categorize the video content into 11 main audio sources and then calculate the standard deviation the overall quality scores among 15 subjects. We compute the average standard deviation for each category, as shown in Tab. 7. The "Fantasy" category shows the highest standard deviation, as it represents unreal scenarios, leading to more diverse interpretations among subjects.
C. More Details of Experiments
C.1. Details of Datasets
We conduct experiments on our proposed AGAVQA-3K dataset, as well as on the TTA and TTM datasets. The TTA dataset (Deshmukh et al., 2024) generates 500 audio samples from 100 prompts using text-to-audio generation methods. Subjects were then invited to rate the quality of the audio and its relevance to the provided description. Similarly, the TTM dataset (Deshmukh et al., 2024) generates 500 music samples from 100 prompts using 5 text-to-mman musi generation methods. Subjects were asked to rate the quality of the music and its relevance to the provided description.
The TTM dataset does not include an evaluation of music's aesthetic quality. Since music aesthetic quality assessment is heavily influenced by personal preferences and the subject's taste in different music styles, it is difficult to achieve an objective evaluation. Therefore, in the AGAVQA- TTA, and TTM datasets, the audio quality dimension primarily focuses on the quality and realism of the audio.
C.2. Details of Loss Function
We first pre-train AGAV-Rater using text-defined levels. The model predicts a fixed-length text output of 1 token, representing either "Excellent" or "Bad". During pre-training, the cross-entropy loss is used as the objective function, which can be
17

---

AGtV-Rater: Adapting LMM for Al-Generated Audio-Visual Quality Assessment
Table 8. Instructions for Ausio-Visual LMMSs Ohtain Quality Levels for AGAV, TTA, and TTM.
Input Type
Dimension
Instruction
<viden> <audio>Can you evaluate the audio quality in terms of quality and realism?
Audio Quality
Response with excellent, good, fair, bad, or poor.
<video><<udio>> you evaluate the audio and video content consisteney?
AGAV
A/V Content Consistency
Response with excellent, good, fair, bad, or poor.
Fride>><audio>CCi you evaluate the overall Audio-visual quality in terms audio quality, audio and video content consistency? Response with excellent, good, fair, bad, or poor.
Overall A/V Quality
<audio> Can you evaluate the audio quality in terms of quality and realism? Response with excellent, good, fair, bad, or poor.
Audio Quality
TTA
<audio> The text is <text> Can you evaluate the audio-text content consistency of the audio and text? Response with excellent, good, fair, bad, or poor.
Audio-Texxt Consistency
<music> Can you evaluate the music quality in terms of quality and realism? Response with excellent, good, fair, bad, or poor.
Audio Quality
TTM
<music>The text <text>Can you evaluate the music-text content consistency of the music and text? Response with excellent, good, fair, bad, or poor.
Music-Text Consistency
defined as:
N C
LCE
1 N
ΣΣ
Σ
enng (ūn.e:
where N is the number of AGAVs in the batch, C' is the vocabulary size, Vn.cc is the ground-trute truth label for the n-th AGAV at vocabulary position c. fn,e is the predicted probability output by the model at position for the n-th AGAV. For fine-tuning, we train AGAV-Rater using numerical scores and employ the Pearson Linear Correlation Coefficient (PLCC) loss as the objective. The PLCCC loss defined as:
( mean( 8 mean( (s))
L (1
1 mean [1) - (uean(s))l: 1/2,
(3)
where s and @ are the vectors of MOSs and predicted scores of AGAVs in a batch respectively, (-) represents the inner product of two vectors, ||.|| denotes the norm operator for vector, and mean is the average operator for a vector.
C.3. Detailed Information of Compared Methods
For the multi-dimensional scoring task, we selected 16 compared methods, covering four types: visual LMMs, AQA, AVQA, and audio-video (text) alignment.
Audio- Visual LMMs. We chose 4 latest open-source audio-visual LMM models, including PandaGPT (Su et al., 2023), NexIGPT (Wu et al., 2024), VITA-1.0 (Fu et al., 2024), and VideoLLaMA2 (Cheng et al., 2024). We rely on official repositories and use default weights to initialize the models. Then, we input AGAVs, TTAs, and TTM into the models, sequentially asking the quality of each dimension and computing the final predicted scores by Eq. 1. The format of instruction for each dimension shown in Tab.8.
AQA. We selected 4 popular AQA methods, including MOSNet (Lo et al., 2019). STOI-Net (Zezario et al., 2020), NISQA (Mittag al., 2021), and PAM (Deshmukh al., 2024). PAM is based large language model and does not require training, allowing for direct testing. For all other models, we initialize them with default weights and train them on the AGAVQA-MOS, TTA, and TTM datasets. Since these AQA methods predict a single quality score based only on audio information, we modified the models to output multi-dimennsonal scores. Specifically, for the AGAVQA-MOS subset, we adjust the final fully connected layer output dimension from to 3, and for the TTA and TTM datasets, we change the output dimension from to 2, allowing the models to predict scores across multiple dimensions.
We selected 3 latest deep Borming--bsse AVQA methods: DNN-RNT (Cao et al., 2023a), DNN-SND (Cao et al., 2023a), and GeneralAVQA (Cao al., 2023b). DNN-RNT and DNN-SND are used to predict quality scores for compressed distorted A/V content, while GeneralAVQA is designed for predicting quality scores of real-world distorted audio-visual content, which includes issues such as jitter, overexposure, and motion blur caused during user capturing. All 3 AVQA methods extract features separately from both video and audio, then fuse these features and output a one-dimensional quality score via a fully connected layer. We modified the final fully connected layer's 's output dimension to 3, enabling the three AVQA methods to be trained on the dimensiinlsi MOS scores in the AGAVQA-MOT subset.
AVQA.
18

---

Rate-Rtter: Adapting LMM for Al-Generated Audio-Visual Quality Assessment
Table Inference latency and throughput of the AGAV-Rater on videos on RTX4090. As videos have variable lengths, we set the batch size as for them to avoid the padding cost.
Video Length (sec)
3
9
11
Latency (ms)
157
204
220
256
332
Throughput (video/sec)
6.36
4.91
4.55
3,90
3.01
Table 10. 230 AIGC videos content distribution and AGAV-Rate, filtering accuracy for each category. We collected these AIGC videos and then used ElevenLabs to generate AGAVs, which were subsequently filtered by AGAV-Rater to select high-quality AGAVs in Section 5.6.
Metric
Animal
Water
People
Vehicle
Object
Socnery
Sea
Fantasy
Fire
Instrument
Cooking
Video Numher
15
20
22
19
20
11
15
24
12
230
AGAV-Rater filtering accuracy
0.78
0.87
0.85
0.77
0.81
0.79
0.80
0.82
0.80
0.83
0.75
0.80
Audio- Video Alignment. AVID-CMA (Morgado et al., 2021) is self-supervised learning approach that learns audio- visual representations from video and audio, achieving highly competitive performance when fine-tuned on action recognition tasks. VAST (Chen et al., 2023) is an omni-modality video-text foundational model capable of processing vision, audio, and subtitles. VALOR (Liu et al., 2024a) is a Iision-Audio-Le Language Onne-pteRnetiion pretraining model that projects vision, language, and audio into a shared common space, enabling alignment across language, ladio-language, and audiovisual- languugg domains. All three models align video and audio the semantic level, so we fine-tune them on the AGAVQA-MOS subset. We use their encoders extract audio and video features, then concatenate them and apply fully connected layer to regress the three-dimensional scores.
(odiioMMuscc--aa Alignment. CLAP (Elizalde et al., 2023) utilizes two encoders and contrastive learning to map audio and text descriptions into a shared multimodal space. TeM-Retrieval (Doh et al., 2023) learns a text-music representation for universal text-to-music retrieval. CLAP, Tet-Retrieval, VAST, and VALOR are all capable of aligning audio and text features. We use their encoders to extract audio and text features, and then apply a fully connected layer to regress and predict audio quality and content consistency scores. We load their pre-trained weights and fine-tune them separately on the TTA and TTM datasets.
C.4. Details of Audio Preprocessing
AGAV-Rater uses default parameters from VideoLLaMA2 for audio preprocessing. Assuming T video frames are extracted, the steps are:
1. Divide the audio into T segments.
2. Concatenate all segments, then crop zero-pad to fixed length.
3. Transform into fbank spectrograms with 128 frequency bins.
4. Use BEAT's and an MI.P block to extract features from the spectrograms.
5. Concatenate audio, video, and text features, and input them into the LLM.
C.5. More Details of Enhancing ElevenLabs Results via AGAV-Rater
In Section 5.6, we collected 230 silent AIGC videos and used ElevenLabs generate audios, aiming verify that AGAV- Rater can assist ElevenLabs in selecting high-quality audio for users. We conducted a statistical analysis of the content distribution of these 230 AIGC videos, with the results presented in Tab. 10. The analysis shows that the video content is quite diverse, enabling a comprehensive evaluation of AGAv-Rater's performance across different types of video content. We further present the accuracy of AGAV-Rater in identifying higher-quality AGAVs across these categories. As shown in the Tab. 10, AGAV-Ratee achieves over 75% accuracy in each category.
C.6. Cost Analysis of AGAV-Rater
In Tab. 9, we report the inference latency of AGAV-Rater on AGAVs. On a single RTX 4090 GPU, the model can predict scores for 6.36 videos of 3 seconds or 3.01 videos of 12 seconds per second. With a performance 20x faster than real-time, the low latency of AGAV-Rater paves the way for broader real-world applications L.MM-haedd A/V scoring.
19


Reviews:
[{"review_id": "72QfmDDcYq", "reviewer": "Reviewer_F7tt", "summary": "This paper addresses a challenging and important question for the VTA methods: whether LMMs can be utilized to assess the quality of audio-visual content generated by VTA methods. To tackle this problem, the authors first establish a large-scale AGAV quality assessment dataset, AGAVQA, which includes two subsets:\nAGAVQA-MOS: Contains 9,264 MOS scores for 3,088 AGAVs.\nAGAVQA-Pair: Contains 75 question-answer pairs for 294 AGAVs.\nThen, this work introduces the AGAV-Rater, a LMM-based quality assessment method for AI-generated audio-visual content. AGAV-Rater can provide multi-dimensional scores for AGAVs, TTAs, and TTMs. Extensive experiments validate the performance of the AGAV-Rater in predicting multi-dimensional quality scores for AGAVs, TTA, and TTM, and assisting VTA methods in selecting the optimal AGAV samples.\n\n### After rebuttal ###\nI have read the rebuttal, and my concerns have been well solved. Thus, I tend to keep my original score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the constructive and valuable feedback. We have addressed your concerns point-by-point below.\n\n**1. Details of human evaluation**\n\nWe invited subjects familiar with AVQA and AGAV for on-site training. We provided detailed explanations of the scoring criteria for each dimension and additional AGAV samples for practice. Experts then reviewed the annotations and selected 15 subjects. To prevent fatigue, each subject rated a maximum of 60 samples per day, completing the task in about two months.\n\nThe official testing phase was conducted in a controlled lab with normal indoor lighting, quiet surroundings, and subjects sitting at a comfortable distance of about 60 cm from the screen. The AGAVs were played at their original resolution. **The scoring interface consisted of three continuous quality rating bars and three navigation buttons.** Each rating bar was labeled with a 1-5 Likert scale. Navigation buttons, including \"Prev\", \"Repeat\", and \"Next\", allowed subjects to switch and replay AGAVs freely. In the final manuscript, we will add images of the scoring interface and detailed documentation of the scoring criteria provided to subjects.\n\n**2. Training duration**\n\nWe trained AGAV-Rater on two 96GB H20 GPUs, with training epochs set to $5$ on the AGAVQA-MOS subset, taking approximately **$5$ hours**.\n\n**3. Details of audio-video alignment methods**\n\nOriginal audio-video alignment methods extract audio and video features using their encoders, then align them into a common vector space. We use these encoders with default parameters to extract audio and video features and then concatenate features. **The concatenated features are fed into a fully connected layer with an output dimension of 3** to predict three-dimensional scores.\n\n**4. Distribution of AIGC videos in Section 5.6**\n\nDue to word limitations, the distribution of AIGC videos is described in Section 4 of the response to Reviewer h1V5. We apologize for any inconvenience. \n\n**5. Modification of Fig. 2**\n\nThank you for your suggestion. In the final manuscript, we will add (a), (b), and (c) to Fig. 2 to help readers quickly understand its content.\n\n**6. Introduction to the loss function**\n\nWe use the PLCC loss to optimize the AGAV-Rater:\n\n$L=(1-\\frac{\\left<\\widehat{s}-mean(\\widehat{s}), s-mean(s)\\right>}{\\lVert\\widehat{s}-mean(\\widehat{s})\\rVert_2\\lVert s-mean(s)\\rVert_2})/2$,\n\nwhere $s$ and $\\widehat{s}$ are the vectors of MOSs and predicted scores of AGAVs in a batch respectively, $\\left<\\cdot\\right>$ represents the inner product of two vectors, $\\lVert\\cdot\\rVert$ denotes the norm operator for a vector, and $mean$ is the average operator for a vector.\n\n**7. Details of instruction-response pairs**\n\nIn the 50,952 instruction-response pairs, the audio-video, audio-text, and music-text scenarios contain 25,592, 19,000, and 6,000 pairs, respectively. Under each scenario, half of the pairs  focus on content consistency, and the other half on audio quality.\n\n**8. Analysis of Table 4**\n\nAGAV-Rater uses VideoLLaMA2 as the base model, and the training set used by VideoLLaMA2 mainly focuses on audio, with relatively less on music. In Table. 2 (in the manuscript), it can be seen that its ability to perceive music-text quality is weaker compared to audio-video and audio-text. Although the music-text scenario has the fewest instruction-response pairs, we repeat these pairs twice during pre-training to increase the learning times of AGAV-Rater for the music-text scenario. Therefore, **the music-text instruction-response pairs enhance the music perception ability of AGAV-Rater, improving the performance of consistency dimension on the TTM dataset.**\n\n**9. Supplement to Table 3**\n\n**The AGAVQA-Pair subset was collected from 8 VTA webpages**, dividing it into 8 corresponding categories. For each category, the optimal AGAV is generated by the corresponding VTA method. We chose to collect the AGAVQA-Pair subset from VTA webpages because these AGAVs, sourced from third-party platforms, offer a more objective and impartial dataset. These VTA webpages are all released in the past year, representing the latest technology in VTA methods.\n\n**The compared methods in Table 3 use their original model parameters** without fine-tuning on the AGAVQA-MOS subset. **We further show the accuracy of the audio-video alignment methods which have been fine-tuned on the AGAVQA-MOS subset**:\n\nMethod |SonicVisionLM | Frieren | V2AMapper | TIVA | V2A-SceneDetector | STAV2A | SSV2A | ReWaS | All\n:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:\nAVIDCMA |0.29 | 0.58 | 0.61 | 0.50 | **0.71** | 0.50 | 0.40 | 0.44 | 0.52 \nVALOR | **1.00** | 0.75 | 0.72 | 0.70 | **0.71** | **0.70** | 0.40 | 0.44 | 0.55 \nVAST | 0.86 | 0.83 | 0.78 | **0.80** | 0.43 | 0.40 | 0.40 | **0.56** | 0.64 \nAGAV-Rater | **1.00** | **0.92** | **0.83** | **0.80** | **0.71** | **0.70** | **0.60** | **0.56** | **0.78** \n\nAs can be seen, AGAV-Rater achieves the highest accuracy in each category.", "claims_and_evidence": "Yes, the authors demonstrate the effectiveness of their proposed model AGAV-Rater in predicting multi-dimensional quality scores on three datasets. This provides clear evidence that their proposed model can adapt LMM for AI-generated audio-visual quality assessment. Additionally, the authors further validate their claims through subjective experiments, showing that their model, AGAV-Rater, can effectively assist video-to-audio methods in selecting the optimal AGAV samples.", "methods_and_evaluation_criteria": "Yes, the AGAV quality assessment dataset, AGAVQA, established by the authors, contributes significantly to the advancement of the AIGC audio-visual quality assessment field. The model proposed by the authors is theoretically sound and can be effectively applied to evaluate the quality of AI-generated audio–visual content, as well as to identify the optimal AIGC audio-visual samples.", "theoretical_claims": "Yes, the process of constructing the AGAVQA dataset is rigorous and well-justified. The proposed model, based on the large multimodal model VideoLLaMA2, is theoretically feasible.", "experimental_designs_or_analyses": "Yes, the authors validated their proposed model on three datasets: AGAVQA, TTA, and TTM, demonstrating its ability to provide multi-dimensional scores for AIGC audio-visual, audio, and music content. Additionally, the authors conduct cross-dataset validation experiments, comparing the accuracy of their proposed model with closed-source LMMs in the optimal AGAV selection task. These experimental designs are reasonable and effective. Furthermore, the authors invited participants to verify that the AGAV samples selected by their model enhance the viewing experience, providing a more rigorous validation of the model’s performance through user experience.", "relation_to_broader_scientific_literature": "The primary contribution of this paper lies in constructing a large-scale AGAV quality assessment dataset, AGAVQA, which significantly advances the field of AI-generated audio-visual (AIGC) quality assessment. Previous research has predominantly focused on AIGC images and videos, making this work a pivotal step in addressing the gap in quality assessment for AIGC audio-visual content. Furthermore, the authors propose a LMM-based quality assessment method for AI-generated audio-visual content, providing a novel solution for evaluating the quality of audio generated from video or text inputs. This contribution extends the broader scientific literature on AIGC quality evaluation.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\nThis paper is motivated by the development of VTA methods and LMMs. The authors propose a novel issue in current quality assessment.\n1.The authors establish a large-scale AGAV quality assessment dataset that includes AGAVs generated by 16 different VTA methods. The dataset is rich and diverse, facilitating the development of VTA methods.\n2.The novel LMM-based AGAV quality assessment method, AGAV-Rater, proposed in the paper, enables multi-dimensional scoring for AGAVs, TTA, and TTM. The authors also conduct extensive experiments, demonstrating that the model can be applied to real-world VTA methods to enhance user viewing experiences.\n3.The analysis of multi-dimensional instructions in the experiments is quite interesting. It provides insights for future multi-dimensional quality assessments and can be easily implemented to improve the performance of quality assessment methods with multi-dimensional scoring.\n\nWeaknesses:\nThe main weaknesses of the paper lie in some unclear explanations, which may confuse readers.\n1.The paper lacks some details in the human evaluation section, such as a more detailed display of the scoring interface and the instructions given to subjects in Fig. 2.\n2.The paper does not provide details on the time required to train the AGAV-Rater. Understanding the computational cost and training efficiency is crucial for practical implementation.\n3.The paper lacks an introduction to the comparison methods. How did the authors train the multimodal alignment-based methods on the AGAVQA-MOS subset in Tab. 1 and Tab. 2? These methods are not specifically designed for quality assessment, and their original structure cannot directly output a one-dimensional quality score.\n4.The paper lacks a detailed explanation of the 230 silent AIGC videos selected in Section 5.6. Although the authors demonstrate some AGAV samples on the project page, there is no description of the content distribution of these AIGC videos.", "other_comments_or_suggestions": "1.Labeling the boxes in Fig. 2 as (a), (b), and (c) would help readers better understand the figure.\n2.The paper should provide a detailed definition of the loss function used during the training of AGAV-Rater.", "questions_for_authors": "1.What specific instructions were given to the human subjects during the testing session? Additionally, how long did it take for the subjects to complete the testing phase?\n2.In the 50,952 instruction-response pairs designed in the paper, what is the proportion of each scenario (audio-video, audio-text, and music-text)? Can you also analyze why the consistency dimension shows a relatively large improvement in the TTM dataset during the pre-training step in Tab. 4?\n3.The description of the category in Tab. 3 is confusing. How did the authors classify the AGAVQA-Pair subset into 8 categories?\n4.AGAV-Rater is trained on the AGAVQA-MOS and demonstrated cross-dataset performance on the AGAVQA-Pair. Are the compared methods in Tab. 3 finetuned on the AGAVQA-MOS? Or are the original model parameters directly utilized?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KT9Zon6TJw", "reviewer": "Reviewer_R3k7", "summary": "This work introduces a new quality assessment dataset and network for the AI-Generated Audio-Visual task. The database additionally handles multimodal challenges like A/V content inconsistency, and the quality assessment model leverages LMM to predict multi-dimensional scores.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the constructive and valuable feedback. We have addressed your concerns point-by-point below.\n\n**1. Details of the auto-labeling process**\n\nWe manually verify 500 auto-labeling results. Among them, the accuracy for content consistency related instruction-response pairs is $100$%, while the accuracy for audio quality related instruction-response pairs is $92$%. In content consistency-related instruction-response pairs, when the consistency quality is labeled as \"bad\", we ensure that **audio (text) and video from different categories are paired to achieve high accuracy**. In audio quality-related instruction-response pairs, for noisy audio types, such as machine sounds or wind noise, the reverse operation has a minimal negative impact on audio quality. We have **utilized category labels to filter out certain audio quality-related instruction-response pairs**, such as hair dryer drying and pumping water, to minimize the error rate.\n\n**2. Anlysis of human scores**\n\nThe range of MOSs is $[0,100]$. We categorize the video content into 11 main audio sources and then calculate the standard deviation of the overall quality scores among 15 subjects. We compute the average standard deviation for each category:\n\nAnimal| Water | People | Vehicle | Object | Scenery | Sea | Fantasy | Fire | Instrument | Cooking\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n12.56 | 12.16 | 12.20 | 11.79 | 12.57 | 12.74 | 12.37 | **13.11** | 12.35 | 11.93 | 12.23\n\n**The \"Fantasy\" category shows the highest standard deviation**, as it represents unreal scenarios, leading to more diverse interpretations among subjects. Krippendorff's α [1] can be used to measure the quality of the subjects' ratings. We calculate **Krippendorff's α** for audio quality, content consistency, and overall quality, which are 0.6814, 0.7343, and 0.7143, respectively, indicating appropriate variations among subjects. We also randomly divide subjects into two groups and calculate the **SRCC of average scores between the two groups**. After ten repetitions, the average SRCC for audio quality, content consistency, and overall quality are 0.8043, 0.8318, and 0.8297, validating rating consistency.\n\n**3. Ablation study of the base models**\n\nWe conduct ablation studies using GroundingGPT [2] and OneLLM [3] as base models on the AGAVQA-MOS subset. For OneLLM and GroundingGPT, we first load the default weights, and then fine-tune them using the official training code on the AGAVQA-MOS subset. To ensure fairness, we also add the quality regression module, directly regressing the LLM's last hidden states to output three-dimensional numerical scores. The results are as follows:\n\nDimension | Audio | Quality | Content|Consistency| Overall|Quality\n|:-|-:|:-|-:|:-|-:|:-|\nMetric | SRCC | PLCC | SRCC | PLCC | SRCC | PLCC\nGroundingGPT (ACL 2024)|0.4387 | 0.4494 | 0.5067 | 0.4764 | 0.4975 | 0.5297 \nOneLLM (CVPR 2024)|0.6578 | 0.6879 | 0.6184 | 0.6297 | 0.6327  | 0.6388\nAGAV-Rater | **0.7909** | **0.8108** | **0.7553** | **0.7645** | **0.7458** | **0.7552**\n\nAs shown in the experimental results, AGAV-Rater achieves the best performance. The main reason for this is that VideoLLaMA2 is designed for audio-video content understanding and pre-trained on more diverse audio-video datasets, making it more suitable for our quality assessment task. GroundingGPT focuses more on localization and visual understanding and is not designed or trained to understand continuous audio-video content. Its ability to comprehend video quality may be weaker. OneLLM is a general multimodal model that, while supporting audio and video processing, is not specifically optimized or enhanced for video and audio alignment. Its audio-related dataset only includes audio-text data, and OneLLM is more suited to text-vision or text-audio matching and understanding, rather than specific audio-video content.\n\n**4. More cases displays**\n\nThank you for your suggestion. We have displayed more cases about the samples and the corresponding scores rated by the AGAV-Rater on the project page (https://agav-rater.github.io).\n\n**References**\n\nPlease refer to the Response to Reviewer h1V5.", "claims_and_evidence": "The claims made in the submission are supported by detailed experiments.", "methods_and_evaluation_criteria": "The proposed methods and evluation criteria make sense for the assessment of audio-visual quality.", "theoretical_claims": "I checked the proofs and formulas in the method section, and there are no issues.", "experimental_designs_or_analyses": "The overall experimental design is mostly fine, with only a few details that need to be confirmed. Also, the ablation experiments can include more base models.", "relation_to_broader_scientific_literature": "This work may provide some insightful implications for the evaluation of audio-visual quality consistency, the application of large multi-modality models in quality assessment methods, and the development of improved video-to-audio methods.", "essential_references_not_discussed": "From my opinion, there are no missing essential references.", "other_strengths_and_weaknesses": "I do not find major issues in this work overall, except for some minor details:\n\n1.Are there any failure cases during the data auto-labeling process? What is the error rate approximately?\n\n2.Please show the variance in human scores during the subjective experiment.\n\n3.The ablation section could be further expanded, such as by including experiments with different base models in the ablation study.\n\n4.More cases could be shown to further demonstrate the effectiveness, such as the sample and the corresponding score rated by the proposed QA model.", "other_comments_or_suggestions": "Please refer to the weakenesses.", "questions_for_authors": "Please refer to the weakenesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rf17RexbUI", "reviewer": "Reviewer_AFZv", "summary": "This paper introduces a AI-generated audio-visual (AGAV) quality assessment dataset (AGAVQA) and AGAV-Rater, a large multimodal model (LMM)-based approach for evaluating AGAV. The AGAVQA dataset containing two subsets: AGAVQA-MOS (multi-dimensional score prediction) and AGAVQA-Pair (optimal AGAV selection). AGAV-Rater is trained using a two-stage process—pre-training with automatically labeled text-defined quality levels and fine-tuning with human-annotated numerical scores. The model achieves state-of-the-art performance on AGAVQA, text-to-audio (TTA), and text-to-music (TTM) datasets, surpassing traditional AVQA and AQA methods as well as general-purpose LMMs.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the constructive and valuable feedback. We have addressed your concerns point-by-point below.\n\n**1. Correlation of 3-dimensional MOSs**\n\n**SRCC between audio quality and content consistency is 0.6860, indicating that the two dimensions are independent**. SRCC between audio quality and overall quality is 0.7876, and between content consistency and overall quality is 0.7926, suggesting that **overall quality is influenced by both audio quality and content consistency**. \n\n**2. Differences between AGAV-Rater and audio-video alignment methods**\n\nThe key difference between AGAV-Rater and VAST is that **AGAV-Rater utilizes the semantic understanding ability of LLM**, improving performance. Although audio-video alignment methods align features semantically, we fine-tuned them on the AGAVQA-MOS subset, mapping audio and video features to quality dimensions. After fine-tuning, these features contain both alignment information and quality information. Therefore, the suboptimal performance of alignment methods does not imply that AGAVQA-MOS emphasizes alignment.\n\n**3. More experiments**\n\nWe conduct ablation studies to compare AGAV-Rater with fine-tuned LMMs, with a detailed analysis in Section 3 of the response to Reviewer R3k7. We also test on the AVQA dataset, SJTU-UAV, focusing on real-world user capture distortions.\n\nMethod | SRCC | PLCC\n:-|-:|:-\nDNN-RNT (TIP 2023) | 0.7125 | 0.7253\nGeneralAVQA (TIP 2023)  | 0.7753 | 0.7827\nAGAV-Rater | 0.7955 | 0.8052  \n\nAGAV-Rater achieves the best performance on SJTU-UAV, proving **its superiority comes from the model framework and training, not from dataset adaptability**.\n\n**4. Details of audio processing**\n\nAGAV-Rater uses default parameters from VideoLLaMA2 for audio preprocessing. Assuming $T$ video frames are extracted, the steps are:\n1. Divide audio into $T$ segments.\n2. Concatenate all segments, then crop or zero-pad to a fixed length.\n3. Transform into fbank spectrograms with 128 frequency bins. \n4. Use BEATs and an MLP block to extract features from the spectrograms. \n6. Concatenate audio, video, and text features, and input them into the LLM.\n\n\n**5. Details of human evaluation**\n\nWe invited subjects familiar with AVQA and AGAV for on-site training. We provided detailed explanations of the scoring criteria for each dimension and additional AGAV samples for practice. Experts then reviewed the annotations and selected 15 subjects. To prevent fatigue, each subject rated a maximum of 60 samples per day, completing the task in about two months.\n\nWe used the **ITU-recommended MOS processing method** [4], and no subjects were identified as outliers. **Krippendorff's α** [1] for audio quality, content consistency, and overall quality are 0.6814, 0.7343, and 0.7143, respectively, indicating appropriate variations among subjects. We also randomly divide subjects into two groups and calculate the **SRCC of average scores between the two groups**. After ten repetitions, the average SRCC for audio quality, content consistency, and overall quality are 0.8043, 0.8318, and 0.8297, validating rating consistency.\n\n**6. Supplement to the AGAVQA-Pair subset**\n\nDue to the lack of public AGAVQA datasets, the AGAVQA-Pair subset was collected from 8 VTA webpages released in the past year. **Its significance lies in that, as a third-party platform, it offers a more objective and impartial dataset.**\n\n**Best-of-pair selection is more reliable than scoring tasks**. Subjects show greater consistency and confidence in determining the optimal sample. In practical applications, identifying the optimal AGAV may be enough. **We use 230 AGAV pairs collected in Section 5.6 (in the manuscript) to further validate generalization.** The accuracy of AGAV-Rater, along with fine-tuned VAST, VALOR, and AVID-CMA, is 80%, 74%, 69%, and 68%, respectively.\n\n**7. Modification of Fig.2**\n\nWe apologize for the error in Fig. 2. There are 45 Kling video sources in total, with 23 from the Kling official website, and 22 from the video generation benchmark Vbench. This will be corrected in the final manuscript.\n\n**8. Motivation of our paper**\n\nPrevious AVQA and AQA work focused on **real-world capture or compression distortions**. Audio-video alignment methods targeted **semantic alignment in real scenarios**. LMM-based quality assessment **primarily centered on visual quality, with a limited focus on audio**. As AIGC video technology advances, more research explores dubbing techniques. The motivation of our paper is that the quality of AGAVs needs to be monitored and controlled. In the AGAVQA-MOS subset, both audio and video content are AI-generated. We aim to use LMMs to evaluate AGAV quality, replacing human subjective scoring to enhance efficiency and enable automation.\n\n**9. Correction of grammar errors**\n\nThank you for pointing out our grammatical errors. We will correct these mistakes in the final manuscript.\n\n**References**\n\nPlease refer to the Response to Reviewer h1V5.", "claims_and_evidence": "The paper does not provide a detailed correlation analysis among Audio Quality, Content Consistency, and Overall Quality dimensions in the AGAVQA-MOS dataset. Such an analysis is crucial for understanding dependencies between these dimensions and would improve interpretability. Without this, it is unclear whether Overall Quality is primarily influenced by Audio Quality or Content Consistency (or both), limiting insights into how AGAV-Rater makes its predictions.", "methods_and_evaluation_criteria": "Audio-video alignment methods (e.g., VAST) achieve strong results on AGAVQA-MOS, raising questions about whether the dataset primarily measures alignment quality rather than broader quality aspects. If AGAVQA-MOS is dominated by alignment factors, then AGAV-Rater’s superior performance might be due to its ability to model alignment, rather than providing a generalizable AGAV quality metric. A deeper analysis of how AGAV-Rater differs from alignment-based methods (e.g., VAST, VALOR) is needed to clarify whether AGAVQA-MOS is capturing a diverse range of distortions beyond alignment.", "theoretical_claims": "I don't think there are any theoretical claims since the paper focuses on new dataset and how to evaluate the audio-visual quality via a proposed two-stage training processes.", "experimental_designs_or_analyses": "While the paper compares AGAV-Rater to prior LMMs (e.g., VideoLLaMA2), these models cannot be fine-tuned, making the comparison somewhat unfair. It remains unclear whether AGAV-Rater’s advantage comes from its model architecture or its dataset adaptation.\n\nIn addition, the audio quality or audio-video alignment could be related to how the audio is processed, e.g., how to encode the audio, how to mix audio and video. I am wondering whether the proposed methods could fit all different processing methods.", "relation_to_broader_scientific_literature": "The paper presents the first large-scale AGAV quality assessment dataset, comprising 3,382 AGAVs from 16 VTA methods, which is of great interest to both industry and academic research communities.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\nThis paper addresses the quality assessment of AI-generated audio-visual (AGAV) content by proposing a novel dataset (AGAVQA) and a multimodal model (AGAV-Rater), and experimental results show notable improvements over other related methods across multiple evaluation dimensions.\n\nWeakness:\n1. The paper lacks a detailed correlation analysis among Audio Quality, Content Consistency, and Overall Quality dimensions in the AGAVQA-MOS dataset, which limits the interpretability of results and insights into dimension dependencies.\n2. There is no adequate analysis of the distinctions from AGAV-Rater and alignment methods, even though audio-video alignment methods (e.g., VAST) demonstrate strong performance. Given the fact of VAST's competitive results, it raises questions about whether the AGAVQA-MOS dataset primarily emphasizes audio-video alignment quality rather than broader quality aspects.\n3. The human evaluation process are insufficiently described. Critical information such as annotator backgrounds and training procedures is missing, raising concerns about annotation robustness and representativeness. Specifically, the authors do not discuss the annotation quality clearly as well, such as the consistency of ratings across annotators, individual differences in subjective perception, or how potential biases and variances were controlled or mitigated.\n4. AGAVQA-Pair dataset evaluation suffers from a notably limited scale (only 75 question-answer pairs) and simplistic annotation (best-of-pair selection), undermining its effectiveness for reliably assessing model generalization.\n5. Fig. 2 step1 indicates a significant data issue: the \"Kling\" video source appears twice with different sample counts, raising severe concerns regarding dataset accuracy and reliability.\n6. The paper needs clearer motivation, reasoning, and a stronger discussion of how it differs from prior work. Overall, it is difficult to read.", "other_comments_or_suggestions": "1. Several grammatical errors were found and the whole paper could be improved for clarity and readability. Such as\n 'It label AGAVs quality in two ways' -> 'It labels AGAVs' quality in two ways'; \n'Our core contributions can be summarized as three-fold'->'threefold\", and better to use either 'Our core contributions are threefold' or 'Our core contributions can be summarized in three ways'.\n'AGAV-Rater demonstrates superior score prediction capabilities on the AGAVQA- MOS, TTA, and TTM datasets, and achieves the highest accuracy in identifying the optimal AGAV on the AGAVQA- Pair subset AGAV-Rater offer users a better audio-visual experience and enhance the quality of VTA method outputs.' -> 'offers' and 'enhances'. Moreover, the original combines multiple ideas without proper separation. It would be good to add a comma before the second AGAV-Rater.", "questions_for_authors": "n/a", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4aUjy3xBe0", "reviewer": "Reviewer_h1V5", "summary": "This paper studies the LMM to assess the quality of AI-generated audio-visual content, evaluating AGAVs from three dimensions: audio perceptual quality, A/V content consistency, and overall A/V quality. The authors introduce a novel AI-generated audio-visual quality assessment dataset, AGAVQA, and propose an LMM-based AGAV quality assessment method, AGAV-Rater. The AGAV-Rater demonstrates SOTA performance in multi-dimensional scoring tasks. Compared to 11 audio-visual LMMs (4 open-source LMMs and 7 closed-source LMMs), AGAV-Rater can better select the optimal AGAV.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the constructive and valuable feedback. We have addressed your concerns point-by-point below.\n\n**1. Definition of evaluation metrics**\n\nSRCC and KRCC measure the prediction monotonicity, while PLCC and RMSE measure the prediction accuracy. Better AGAVQA methods should have larger SRCC, KRCC, PLCC values, and smaller RMSE values. SRCC can be formulated as:\n\n$SRCC = 1-\\frac{6\\sum^N_{n=1}(v_n-p_n)^2}{N(N^2-1)}$,\n\nwhere $v_n$ and $p_n$ denote the rank of the MOSs and predicted score, respectively. For a pair of ranks $(v_i,p_i)$ and $(v_j,p_j)$, the pair is concordant if $(v_i-p_i)(v_j-p_j)>0$, and discordant if $<0$. KRCC is defined as:\n\n$KRCC=\\frac{C-D}{N(N-1)/2}$,\n\nwhere $N$ is the number of AGAVs, $C$ and $D$ denote the number of concordant and discordant pairs, respectively. PLCC and RMSE are calculated as:\n\n$PLCC = \\left(\\sum^N_{n=1}(y_n-\\overline{y})(\\widehat{y}_n-\\overline{\\widehat{y}})\\right)/$\n\n$\\left(\\sqrt{\\sum^N_{n=1}(y_n-\\overline{y})^2\\sum^N_{n=1}(\\widehat{y}_n-\\overline{\\widehat{y}})^2}\\right)$,\n\n$RMSE = \\sqrt{\\frac{1}{N} \\sum_{n=1}^{N} (y_n - \\hat{y}_n)^2}$,\n\nwhere $y$ and $\\widehat{y}$ denote the MOSs and predicted scores, respectively. $\\overline{y}$ and $\\overline{\\widehat{y}}$ are the mean of the MOSs and predicted scores, respectively.\n\n**2. Introduction to TTA and TTM datasets**\n\nTTA and TTM were proposed by [5]. The TTA dataset generates 500 audio samples from 100 prompts using 5 text-to-audio generation methods. Subjects were then invited to rate the quality of the audio and its relevance to the provided description. Similarly, the TTM dataset generates 500 music samples from 100 prompts using 5 text-to-music generation methods. Subjects were asked to rate the quality of the music and its relevance to the provided description.\n\n**3. More experiments**\n\nOn the AGAVQA-Pair subset, AGAV-Rater is tested using a single-input scoring approach. We present the accuracy of fine-tuned audio-video alignment methods on the AGAVQA-Pair subset and the accuracy of random selection:\n\nMethod |SonicVisionLM | Frieren | V2AMapper | TIVA | V2A-SceneDetector | STAV2A | SSV2A | ReWaS | All\n:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:\nRandom | 0.33 | 0.20 | 0.41 | 0.20 | 0.50 | 0.25 | 0.20 | 0.25 | 0.32 \nAVIDCMA |0.29 | 0.58 | 0.61 | 0.50 | **0.71** | 0.50 | 0.40 | 0.44 | 0.52 \nVALOR | **1.00** | 0.75 | 0.72 | 0.70 | **0.71** | **0.70** | 0.40 | 0.44 | 0.55 \nVAST | 0.86 | 0.83 | 0.78 | **0.80** | 0.43 | 0.40 | 0.40 | **0.56** | 0.64 \nAGAV-Rater | **1.00** | **0.92** | **0.83** | **0.80** | **0.71** | **0.70** | **0.60** | **0.56** | **0.78** \n\nAs can be seen, AGAV-Rater achieves the highest accuracy in each category.\n\n**4. Distribution of AIGC videos in section 5.6**\n\nThank you for your suggestion. We statistically analyze the distribution of the 230 AIGC video contents in Section 5.6 (in the manuscript), and the results are as follows:\n\nAnimal| Water | People | Vehicle | Object | Scenery | Sea | Fantasy | Fire | Instrument | Cooking\n:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:\n41 | 15 | 20 | 22 | 31 | 19 | 20 | 11 | 15 | 24 | 12\n\nAs shown in the table, the video content is quite diverse, allowing for a comprehensive evaluation of AGAV-Rater's performance across different types of video content.\n\n**5. Explanation of standard text quality levels**\n\nResearchers have found that \"good\" and \"poor\" are the most frequently predicted tokens by LMM models when addressing quality issues. We then use the standard text rating levels defined by ITU [6]—excellent, good, fair, poor, and bad—to further refine the quality levels corresponding to these tokens.\n\n**References**\n\n[1] Hayes, A. F. and Krippendorff, K. Answering the call for a standard reliability measure for coding data. Commun. Methods Meas., 1(1):77–89, 2007.\n\n[2] Han, J., Gong, K., Zhang, Y., Wang, J., Zhang, K., Lin, D., Qiao, Y., Gao, P., and Yue, X. Onellm: One framework to align all modalities with language. In CVPR, pp. 26584–26595, 2024\n\n[3] Li, Z., Xu, Q., Zhang, D., Song, H., Cai, Y., Qi, Q., Zhou, R., Pan, J., Li, Z., Tu, V., et al. Groundinggpt: Language enhanced multi-modal grounding model. In ACL, pp. 6657–6678, 2024.\n\n[4] R. I.-R. BT, Methodology for the subjective assessment of the quality\nof television pictures. ITU, 2002.\n\n[5] Deshmukh, S., Alharthi, D., Elizalde, B., Gamper, H., Ismail, M. A., Singh, R., Raj, B., and Wang, H. Pam:Prompting audio-language models for audio quality assessment. arXiv preprint arXiv:2402.00282, 2024.\n\n[6]Recommendation 500-10: Methodology for the subjective assessment of the quality of television  pictures. ITU-R Rec.BT.500,2000.", "claims_and_evidence": "The claims made in the paper are generally well-supported by clear and convincing evidence. The paper introduces AGAVQA, a large-scale audio-visual quality assessment dataset, and AGAV-Rater, an LMM-based model for evaluating AIGC audio-visual content. The evidence provided includes:\n1.\tThe detailed descriptions of AGAVQA dataset construction.\n2.\tThe performance comparisons of AGAV-Rater across multiple datasets (AGAVQA-MOS, TTA, TTM).\n3.\tThe ablation studies of AGAV-Rater, including pretraining, scoring methods, and multi-dimensional instructions. \n4.\tThe subjective experiments confirm the effectiveness of AGAV-Rater in improving user experience.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria in this paper are well-aligned with the problem of AI-generated audio-visual quality assessment. The proposed model AGAV-Rater outperforms prior approaches in terms of both correlation with human scores and optimal AGAV selection accuracy.", "theoretical_claims": "The paper relies on empirical validation through extensive experiments on the AGAVQA dataset, demonstrating performance improvements over baseline methods. The claims regarding the effectiveness of AGAV-Rater are supported by experimental results rather than theoretical derivations.", "experimental_designs_or_analyses": "The experimental design in the paper appears well-structured and methodologically sound, including the following aspects:\n1.Evaluation on multi-dimensional scoring tasks: The study evaluates AGAV-Rater using three dimensions: audio quality, audio-visual content consistency, and overall audio-visual quality.\n2. Evaluation on optimal AGAV selection tasks: The performance is measured by answer accuracy in selecting the optimal AGAV.\n3. Ablation study on pretraining, scoring method, and instruction design.\n4. The subjective experiments confirm that AGAV-Rater enhances the user experience of AGAVs.", "relation_to_broader_scientific_literature": "The paper makes several key contributions that are well-aligned with broader trends in audio-visual quality assessment and large multimodal models, including:\n1.\tAGAVQA is the first dataset specifically designed for AI-generated audio-visual quality assessment, distinguishing it from traditional AVQA datasets focused on compression and transmission artifacts.\n2.\tAGAV-Rater integrates LMMs, improving semantic-level quality assessment, an area where traditional AVQA models are weak.\n3.\tAGAV-Rater is the first LMM-based model specifically designed for AIGC AVQA, bridging the gap between general LMMs and perceptual quality assessment.", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "The paper’s strengths are as follows:\nS1) In terms of originality. The paper establishes an AI-generated audio-visual quality assessment dataset and, based on this, proposes the AGAV-Rater to evaluate the quality of AI-generated audio-visual content. This is a novel contribution to the field.\nS2) In terms of rationality. The paper thoroughly validates AGAV-Rater’s performance through extensive experiments, including multi-dimensional scoring, optimal AGAV selection, and enhancing the user experience of Elevenlabs.\nS3) In terms of importance. This work fills a gap in AI-generated audio-visual quality assessment, contributing to the advancement of AIGC quality assessment and VTA methods.\nS4) In terms of structure. The paper is well-written, well-organized, and clearly structured.\n\nThe paper’s weaknesses are as follows:\nW1) Lack of detailed introduction of evaluation metrics: The evaluation metrics utilized in the experiments, such as SRCC, PLCC, KRCC, and RMSE, are not introduced in the paper. Providing a brief explanation of these metrics would enhance clarity, especially for readers unfamiliar with them.\nW2) Insufficient description of the datasets: The use of datasets is crucial in the experiments, but I do not see a detailed introduction to the TTA and TTM datasets.\nW3) Additional experimental data would aid understanding: In Tab. 3, including the accuracy of random selection as a baseline would provide valuable context for understanding the model's performance.", "other_comments_or_suggestions": "My suggestions are already included in the weaknesses discussed earlier.", "questions_for_authors": "I have some questions about the paper：\nQ1) The authors conduct cross-dataset experiments on the AGAVQA-Pair subset by training the AGAV-Rater on the AGAVQA-MOS subset. However, the cross-dataset performance of the SOTA methods in Tab. 1 is not shown. For example, how does VAST perform on AGAVQA-Pair when it is trained on AGAVQA-MOS?\nQ2) The author selects optimal AGAVs for 230 silent AIGC videos to enhance the user experience. Could you provide more details of these 230 videos? More diverse videos could better validate the generalizability of AGAV-Rater.\nQ3) How is the AGAV-Rater method tested on AGAVQA-Pair? Is it evaluated using a multi-input comparison approach or a single-input scoring approach?\nQ4) In Equation 1, why do the authors utilize excellent, good, fair, poor, and bad as the standard text quality levels?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The recommendation is based on the reviewers' comments, the area chair's evaluation, and the author-reviewer discussion. 

This paper proposes a new dataset and methodology for AI-generated audio-visual quality assessment. All reviewers find the studied setting novel and the results provide new insights. The authors’ rebuttal has successfully addressed the major concerns of most reviewers. However, Reviewer AFZv replied on April 11th to share some persistent concerns on motivation and presentation, which I believe can be addressed in the final version. Overall, I recommend acceptance of this submission. I also expect the authors to include the new results and suggested changes (especially the feedback from Reviewer AFZv) during the rebuttal phase in the final version.
Decision: Accept (poster)