SpeakerVid-5M: A Large-Scale High-Quality Dataset for Audio-Visual Dyadic Interactive Human Generation

## Abstract

Abstract The rapid development of large-scale models has catalyzed significant breakthroughs in the digital human domain. These advanced methodologies offer high-fidelity solutions for avatar driving and rendering, leading academia to focus on the next major challenge: audio-visual dyadic interactive virtual human. To facilitate research in this emerging area, we present SpeakerVid-5M dataset, the first large-scale, high-quality dataset designed for audio-visual dyadic interactive virtual human generation. Totaling over 8 , 743 8 743 8,743 8 , 743 hours, SpeakerVid-5M contains more than 5.2 5.2 5.2 5.2 million video clips of human portraits. It covers diverse scales and interaction types, including monadic talking, listening, and dyadic conversations. Crucially, the dataset is structured along two key dimensions: interaction type and data quality. First, it is categorized into four types(dialogue branch, single branch, listening branch and multi-turn branch) based on the interaction scenario. Second, it is stratified into a large-scale pre-training subset and a curated, high-quality subset for Supervised Fine-Tuning (SFT). This dual structure accommodates a wide array of 2D virtual human tasks. In addition, we provide an autoregressive (AR)-based video chat baseline trained on this data, accompanied by a dedicated set of metrics and test data to serve as a benchmark ( VidChatBench ) for future work. Both the dataset and the corresponding data processing code will be publicly released. Project page: https://dorniwang.github.io/SpeakerVid-5M/ .

## 1 Introduction

In the era of rapid advancements in large-scale models, large video models have acquired the capability to generate high-fidelity video, presenting unprecedented advantages for the generation and driving of high-quality virtual humans. Pioneering works based on Generative Adversarial Networks (GANs) [55, 41, 53], established the foundation for realistic 2D virtual human driving and rendering. However, the advent of large video models exemplified by diffusion-based works [25, 10, 44, 27, 46] has achieved state-of-the-art realism, significantly elevating the authenticity of both driving and rendering.

This leap in fidelity has enabled broader industrial adoption, facilitating commercial-grade applications in areas such as automated lip-syncing, digital newscasting, and virtual actors. Nevertheless, a more ambitious goal has captured the attention of researchers in both academia and industry: the creation of proactive, interactive virtual humans. This endeavor is akin to equipping an avatar with a “brain”, advancing towards virtual beings that are not just passively driven but can engage autonomously. Such technology holds compelling potential for applications like more realistic virtual assistants, live-streaming e-commerce, and online education. Several works have approached this task from a system-building perspective [42, 58, 33, 26], have constructed functional interactive agents by integrating existing, off-the-shelf modules. In contrast to earlier systems that lack the ability for multimodal perception and understanding, BodyofHer [4] presented an end-to-end trained interactive agent based on autoregressive LLM paradigm that accepts a full suite of multimodal inputs. These works collectively represent pioneering explorations in this direction.

In the current research paradigm, which is increasingly centered on foundation models, the development of task-specific applications necessitates large-scale training datasets. However, a notable scarcity of open-source datasets focused on interactive virtual humans persists within the academic community.

Training foundation models for interactive virtual humans requires vast amounts of specialized data, yet such resources are scarce. To address this critical need, we introduce SpeakerVid-5M, the first large-scale, high-quality dataset for audio-visual dyadic interaction, featuring richly annotated and meticulously aligned audiovisual pairs.

SpeakerVid-5M features:
(1) Large-scale data:
The dataset contains 770K high-quality dynadic conversion audiovisual pairs (totaling 1.8K hours) and 5.2M high-quality single-speaker clips (totaling 8.7K hours).
(2) High resolution:
93% of the videos are in 1080P or higher, ensuring detailed and clear visual input for generation tasks.
(3) Rich annotation types:
Each clip is accompanied by structured textual annotations, skeletal sequences, ASR transcriptions, and blur scores, supporting a wide range of multimodal learning objectives.
(4) High-quality:
Precise synchronization between audio and video is ensured, with rigorous filtering applied to both modalities to guarantee clean and reliable training data.
(5) Body compositional diversity: Data instance is captured with labels spanning full-body, half-body, head-only, and side-view profiles, enabling fine-grained control over framing analysis.
In addition, we select $500$ videos from out-of-distribution speaker IDs to construct the VidChatBench benchmark for the audio-visual dyadic interactive virtual human task, which focuses on video quality, audio-visual consistency, dialogue coherence, and identity preservation for
model performance evaluation.

SpeakerVid-5M is designed to facilitate research in interactive virtual human task. The rich annotations also make it a valuable resource for a variety of related tasks, such as human animation, talking head generation, multimodal dialogue modeling, etc.  all of which face the challenge of a lack of large-scale, high-quality datasets.
In addition, we conduct an initial exploration of implementing audio-visual dyadic interactive virtual human generation under an autoregressive paradigm. Given input video and audio, the model jointly generates the speaker’s response in both audio and visual modalities.

Our contributions can be summarized as follows:

- •
We propose SpeakerVid-5M, the first large-scale dataset designed specifically for the audio-visual dyadic interactive virtual human task. It includes 1M high-quality dialogue audiovisual pairs, with supporting for multi-turn conversations. The VidChatBench is also provided for better evaluation.
- •
SpeakerVid-5M contains 5M single-speaker audiovisual clips, making it the largest talking human dataset. It covers a wide range of annotated visual formats, including talking heads, half-body, full-body, and side-view videos.
- •
We open-source the entire dataset, including the raw data, annotations, and data processing pipeline, providing full transparency and reproducibility for the community.

## 2 Related Work

### 2.1 Audio-Visual Human Video Generation.

Lip-sync [32, 20, 31] and talking head generation [55, 41, 39, 53, 49] are foundational tasks in audio-driven human video generation, among those, PD-FGC [41] was a pioneering work that first achieved the generation of talking heads with vivid facial expressions. Additionally, tasks such as learning to listen [30, 29] also fall under the category of audio-driven portrait generation, focusing on responsive non-verbal behaviors. Subsequently, with the rapid development of foundational video models, the field has bifurcated into two primary research directions. One branch has pushed for higher quality and broader scope [39, 46, 27, 44, 10, 25], with EMO [39] introducing diffusion video models to achieve state-of-the-art realism, and works like OmniHuman-1 [25], MoCha [46], and Veo3 [1] expanding generation to full-body, multi-agent performances, and direct text-to-audiovisual synthesis. The second branch focuses on creating interactive virtual humans [42, 58, 33, 26, 4]. This includes modular systems like AgentAvatar [42] that integrate LLMs as planners, and end-to-end paradigms like BodyofHer [4] that enable direct multimodal understanding for autonomous reactions. These pioneering efforts in interactivity shift the focus from pure driving alignment to agent autonomy.

### 2.2 Audio-Visual Human Video Datasets.

Early research in talking head and lip-sync generation initially leveraged datasets from related fields such as lip reading [3, 2, 12] and speaker recognition [28, 13], Subsequently, datasets specifically curated for these tasks began to emerge [52, 54, 45]. ViCo [56] introduced a dataset for the “learning to listen” task, though it was focused on monadic scenarios. However, these early datasets were often limited in scale and of lower quality, rendering them inadequate for the demands of current high-quality, data-intensive models. While massive datasets exist, they are either too general-purpose and of variable quality (e.g., ACAV-100M [19]) or proprietary and unreleased (e.g., the data used by OmniHuman-1 [25]). The recent release of OpenHumanVid [21] provided a valuable resource, but it was only partially released and remains focused on monadic, talking-head scenarios. This leaves a critical void for a public, large-scale dataset centered on audio-visual dyadic interactions for virtual human research. Our work, SpeakerVid-5M, is designed to fill this void.

## 3 Dataset Curation

Figure: Figure 2: The SpeakerVid-5M curation pipeline. The process consists: (1) Source data collection from YouTube; (2) Multi-step audio-visual pre-processing; (3) Rich multi-modal annotation using models like Qwen-VL; (4) Rigorous quality filtering stage for data fidelity.
Refer to caption: x2.png

The construction process of our SpeakerVid-5M dataset mainly includes four parts: Source data collection, Data pre-processing, Data annotation, and Quality filter. The entire construction process is illustrated in Figure [2](https://arxiv.org/html/2507.09862v1#S3.F2).

### 3.1 Source Data Collection

We manually collected high-quality videos featuring two-person dialogue from YouTube. The sources mainly include interviews, news reports, seminars, television programs, variety shows, debates, and educational videos. The collected videos exhibit diverse formats such as full-body, half-body, frontal, and side views, all with clear facial visibility and corresponding audio. In total, we gathered $153$K audio-visual videos, amounting to $64,386$ hours of raw data, with $93\%$ of the videos in 1080P resolution or higher. The temporal scope of the dataset extends from $2006$ to the present day (June $2025$).
This collection spans a diverse range of genres, mainly including entertainment, people and blogs, comedy, news and politics, education, science, and sports.

### 3.2 Data Pre-processing

Scene splitting. We employed SceneDetect [6] for scene splitting, which identifies visually significant transitions by analyzing changes in color and brightness to determine scene boundaries. Based on the initial segmentation results, we conducted further post-processing by discarding clips shorter than $3$ seconds and splitting those longer than $14$ seconds. As a result, we obtained video clips ranging from $3$ to $14$ seconds in length. We record the temporal order and the human ID of each segment, allowing clips to be easily concatenated to form longer sequences when needed. The segments resulting from this stage are denoted by $S_{sp}$.

Speaker diarization. We utilize 3D-Speaker [11] to perform speaker diarization, segmenting the original audio (extracted directly from video before scene splitting) into multiple segments and recording the timestamps and corresponding speaker IDs for each segment. For each original audio, we identified the two primary speaker IDs based on the frequency and duration of their speech. Segments associated with other speaker IDs were discarded in the subsequent processing steps. The segments resulting from this stage are denoted by $S_{sv}$.

Human detection. We apply YOLO [18] for human tracking within each video clip, and aggregate the tracking results over time to form a single spatiotemporal visual track for each individual. Because the scene boundaries provided by SceneDetect can be inaccurate, we further refine each clip by temporally and spatially cropping it based on YOLO detection results to obtain usable single-speaker video clips. This stage may yield multiple clips, each originating from different spatial regions within the same temporal window of a single video clip $S_{sp}$. The resulting segments from this stage are denoted by $S_{rsp}$.

Lip synchronization. For each video clip obtained from the last stage, we first calculate the temporal overlap between $S_{rsp}$ and $S_{sv}$, this yields the overlapped segment $S_{ol}$. We then employ SyncNet [36] to perform audio-visual synchronization within each segment $S_{ol}$. The SyncNet confidence score is then used to associate each speaker ID with a specific visual bounding box detected by YOLO. In cases where two individuals are present and conversing within the same temporal window, we assign speaker ID to the bounding box with the highest confidence score.

ID correction.
To further verify and refine the speaker IDs obtained through the collaboration of Speaker diarization and lip-sync analysis, we employ a vision-based method, ArcFace [15], for further correction. For multiple clips extracted from the same original video, the speaker IDs derived from audio segmentation should be consistent with those inferred from visual information.
We first assign speaker IDs based on audio segmentation, then compute facial cosine similarity across clips belonging to the same speaker ID. Outliers identified via low similarity scores are compared against other speaker IDs using ArcFace. If a higher similarity is found with another ID, the outliers’ ID is updated accordingly, completing the correction process.

### 3.3 Data Annotation

Structured Textual Caption.
Textual captions are provided in detailed structure formats to ensure a comprehensive representation of the video content.
We leverage the powerful multimodal model Qwen2.5-VL [5] to generate textual annotations for our video data, which demonstrates remarkable capabilities in visual understanding and instruction following.
The structured annotations include camera movement patterns, a list of entities present in the video, body orientation (front or side view), and whether the person is shown in a half-body or full-body. Additionally, we provide detailed descriptions of human body movements and facial expressions.
Furthermore, we use Qwen-3 [50] to summarize automatic speech recognition results across multiple clips originating from the same source video. Based on these summaries, we annotate the topic category of the two-person dialogue presented in the video.

Audio Annotation.
Each data sample in our dataset consists of a speaker video with well-aligned audio and visual modalities. To support tasks related to audio generation and control, we annotate the audio with several key features. First, we apply Whisper [35] for automatic speech recognition to obtain text transcriptions of the audio. Second, we annotate each audio segment with its corresponding SyncNet metrics, including the audio-visual offset, synchronization confidence score, and audio-visual embedding distance. Finally, we use 3D-Speaker to assign speaker IDs to multiple audio segments from a single original audio.
For each clip, in addition to the original audio, we provide a cleaned version in which segments not belonging to the target speaker ID are replaced with silent audio.

Skeleton Sequence.
Building upon the spatiotemporal human bounding boxes obtained from YOLO, we further utilize DWpose [51] for human pose estimation. The resulting skeletal sequences include keypoints of the face, hands, and body. Clips without a detected face are filtered out.

Face and Hand Blur Score.
We extract face and hand bounding boxes from the DWpose annotations. For each frame, we crop the pixel regions corresponding to the face and both hands, then resize these crops to a resolution of $128\times 128$. Next, we calculate the Laplacian variance of the cropped face and hand images using the Laplacian operator. A higher variance typically indicates a sharper, clearer image, which we use as a clarity score. Since videos containing limb movements often exhibit motion blur during rapid movements, incorporating this clarity score as a conditioning factor may enhance the model’s performance in handling such scenarios [24].

Motion score.
The range of human motion plays a crucial role in generating human-centered videos. Compared to actions in general videos, human movements are more subjective and varied. Therefore, we employed Qwen2.5-VL to simulate human judgment and score the motion magnitude of individuals in the videos. To better simulate the behavior of different annotators, we employ multiple prompts, each embodying a distinct persona, to rate each video. The ratings are on a $1$-to-$5$ scale, where a score of $1$ indicates minimal movement and a score of $5$ denotes a high motion amplitude. After excluding outliers, the average score is taken as the final motion magnitude annotation for the video.

### 3.4 Quality Filter

Luminance Filtering.
Following OpenHumanVid [21], we calculate the luminance score to filter overly dark or bright videos. The luminance score is calculated using the formula: $0.2126\texttimes R+0.7152\texttimes G+0.0722\texttimes B$, where $R$, $G$, and $B$ denote the pixel values of the red, green, and blue channels, respectively. Videos with a luminance score below $10$ or above $210$ are filtered out.

Video Quality Filtering.
To ensure the visual quality of the video data, we employed the DOVER [47] model for video quality assessment. DOVER decomposes each video into aesthetics-related and technical-related components and evaluates them separately. This approach enhances the consistency between the model’s assessment results and human subjective perception of video quality. We filtering out videos with fused scores below $0.25$.

Clear score Filtering.
Resolution alone is insufficient to assess the visual clarity of a video, as some high-resolution videos may still exhibit blurriness and other low-quality characteristics. Bitrate reflects the amount of information carried per frame and provides an informative quality measure.
We compute a clarity score as ${B}/\sqrt{{W}\times{H}}$, where $W$ and $H$ means the resolution of a video and $B$ is the bitrate.
Videos with clarity scores in the bottom $5\%$ are filtered out.

Blur Filtering.
Human-centered videos often suffer from motion blur, especially when the subject exhibits large movements. In our dataset, such blur typically affects the face and hands. To address this, we compute the average blur score over face and hand for each frame in a video. Videos with an average blur score(face or hand) below $0.1$ are filtered out to ensure data quality.

Audio Filtering.
Previous audio-visual datasets have typically focused solely on visual quality while neglecting the quality of the audio component.
We additionally focus on four key metrics in the ASR process, confidence score, no-speech probability, compression ratio, and detected language. These indicators reflect the reliability of the ASR output, whether the audio contains human speech, and whether there are corrupted or non-speech signals in the audio. At this stage, we filter out audio clips with a confidence score(average log probability) lower than $-1.5$, a no-speech probability greater than $0.8$, or a compression ratio exceeding $2.5$. To further ensure the quality of the detected results, we filter out samples in which the detected language mismatches the language labeled in the video’s metadata.

## 4 Dataset Statistics and Analysis

Figure: (a) Face and Hand Blur Score
Refer to caption: x3.png

**Table 1: Comparative analysis of SpeakerVid-5M with existing human video datasets. Person num indicates the number of individuals present in a given clip. For each clip, only one person is retained, ensuring a clean alignment between the audio and visual streams. Blur anno represents the degree of blurriness of the hands and face in each frame. Body composition means the fine-grained annotations for body composition (full-body, half-body, head-only) and camera perspective (frontal, side), features that are absent in most prior work.**
| Datasets | Domain | Clips | Duration (hours) | Generation | Person num | audio | pose | Speaker ID | Blur anno | Body composition | Caption type | IDs | Resolution |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| UCF-101 [38] | Human | 13.3K | 26.7 | – | N/A |  |  |  |  |  | Text | N/A | 240P |
| ActivityNet [7] | Human | 100K | 849 | – | N/A |  |  |  |  |  | Text | N/A | N/A |
| NTU RGB+D [37] | Human | 114K | 3.7 | Conditioned | single |  | ✓ |  |  |  | - | N/A | 1080P |
| TikTok-v4 [8] | Human | 350 | 1 | Conditioned | single |  | ✓ |  |  |  | - | N/A | N/A |
| Openhumanvid [21] | Human | 13.4M | 16.7K | Conditioned | multi | ✓ | ✓ |  |  |  | Structured | N/A | 720P |
| VoxCeleb [28] | Head | 21.2K | 352 | Conditioned | single | ✓ |  |  |  |  | - | 1.2k | 224P |
| VoxCeleb2 [13] | Head | 150.4K | 2.4K | Conditioned | single | ✓ |  |  |  |  | - | 6.1K | 224P |
| MEAD [43] | Head | 281.4K | 39 | Conditioned | single | ✓ |  |  |  |  | - | 60 | 1080P |
| CelebV-HQ [57] | Head | 35.6K | 68 | Conditioned | single | ✓ |  |  |  |  | Structured | 15.6K | 512P |
| CelebV-Text [52] | Head | 70K | 279 | Conditioned | single | ✓ |  |  |  |  | Structured | N/A | 512P |
| SpeakerVid-5M | Human | 5.2M | 8.7K | Conditioned | single | ✓ | ✓ | ✓ | ✓ | ✓ | Structured | 83K | 1080P |
| SpeakerVid-5M (Dialogue) | Human | 770K | 1.8K | Dyadic | single | ✓ | ✓ | ✓ | ✓ | ✓ | Structured | 16K | 1080P |

Figure: Figure 4: Examples of dyadic dialogue and body composition in SpeakerVid-5M. The top rows illustrate a typical dyadic human generation sample (initiator and responder). The bottom rows demonstrate the variety of body compositions annotated in our dataset, including close-up headshots, half-body, and full-body views, which are critical for controllable generation.
Refer to caption: x10.png

### 4.1 Dataset Statistical Comparison.

As shown in Table [1](https://arxiv.org/html/2507.09862v1#S4.T1), we present a comparative analysis of our dataset against previous general and human video datasets.
Compared to traditional audio- or text-driven digital human generation, our dataset is the first to extend this task into audio-visual dyadic interactive scenarios, featuring complete audio-visual pairs of both questions and responses. This provides rich, high-quality training data for end-to-end audio-visual dyadic interactive virtual human generation. Moreover, our dataset represents the largest collection of single-speaker audio-video pairs to date. Each clip features well-aligned audio and video, capturing a clear and uninterrupted instance of speech.

Figure. [3](https://arxiv.org/html/2507.09862v1#S4.F3) shows statistical analysis of SpeakerVid-5M dataset. The blur score distribution, resolution and DOVER quality distribution demonstrate the high quality of our proposed dataset. Video topic and year distribution and the caption words distribution shows that the dataset exhibits diversity in monologue or conversation content.

Key features of our dataset include:
(1) Large-scale data.
The dataset comprises 5.2M single-speaker audio-video clips totaling 8.7K hours, and 770K two-person conversational audiovisual pairs totaling 1.8K hours.
(2) High resolution.
$93\%$ of the videos are in 1080P or higher, and $98\%$ exceed 720P, ensuring high visual fidelity.
(3) Rich annotations.
Each clip is accompanied by fine-grained structured textual annotations, ASR transcriptions, human pose keypoint sequences, motion magnitude scores, and motion blur scores, enabling detailed modeling and analysis.
(4) Diverse data formats.
The dataset includes single-speaker monologues, two-person dialogues, multi-turn conversational dialogues, and listening-human scenarios, supporting a wide range of human-centric generation tasks.
(5) High-quality audio-visual data:
Multiple filtering strategies across both audio and video modalities ensure that all clips exhibit clean, well-synchronized audiovisual pairs.
(6) Tiered dataset design:
The dataset is organized into a large-scale pretraining subset and a curated high-quality subset ($1.3$K hours), based on multiple quality indicators, facilitating research under varying levels of training resources and computational constraints.
(7) Body compositional diversity: Data instance is captured with labels spanning full-body, half-body, head, and side-view profiles, enabling fine-grained control over framing analysis.

### 4.2 SpeakerVid-5M Dialogue Branch

To support the training of audio-visual dyadic interactive virtual human generation, we introduce a two-person dialogue branch within the dataset, where each sample consists of two audio-visual pairs, one serving as the input and the other as the target response. Unlike condition-controlled generation tasks such as talking head, or monadic generation tasks such as learning-to-listen, dyadic generation requires the model to generate both audio and video responses based on a comprehensive understanding of the input multi-modal content, enabling interaction with the environment in both listening and behaving.
This setting goes beyond conventional modality-aligned generation, demanding stronger comprehension and reasoning capabilities from generative models.
Our two-person dialogue branch is primarily sourced from real-world conversational scenarios such as interviews, podcasts, news segments, educational videos, and debates. It consists of 770K clip pairs (totaling 1.8K hours) and includes 16K unique speaker IDs. The dialogue topics span a wide range of categories(Fig. [3(c)](https://arxiv.org/html/2507.09862v1#S4.F3.sf3) [3(d)](https://arxiv.org/html/2507.09862v1#S4.F3.sf4), including entertainment, people and blogs, comedy, news and politics, education, and science.
To ensure dialogue coherence, we only select temporally continuous clips and extract data exclusively from original videos in which the two main speakers account for over $80\%$ of the total speaking time.

### 4.3 SpeakerVid-5M Single Branch

The single-speaker video branch consists of 5.2M clips with 83K unique speaker IDs, totaling 8.7K hours. It covers a diverse range of camera framings and angles, including full-body, half-body, frontal, and side-view shots. Each clip is annotated with skeletal keypoint sequences, structured textual descriptions, and automatic speech recognition (ASR) transcriptions, making it well-suited for a wide array of conditioned generation tasks, such as audio-skeleton driven portrait animation, lip synchronization, and talking head generation.
In terms of scale, this branch represents the largest speaker-specific dataset to date, with a volume comparable to that of more general-purpose video datasets.

### 4.4 SpeakerVid-5M Listening Branch

Considering the real-world applications of digital humans, models are expected not only to generate meaningful responses but also to exhibit appropriate listening behaviors. To support this capability, we specifically collect two types of data representing listening states:
(1) Co-present listening dialogue (e.g., live interviews, news commentary, seminars, podcasts).
For naturally occurring two-person on-screen dialogues, we filter clips based on audio segmentation results and SyncNet confidence scores. Given a clip with two individuals $A$ and $B$ (before cropping), if only one speaker is active and the SyncNet scores between the two differ larger than a predefined threshold, if $A$ is the lower one, $A$ is determined to be in a listening state.
(2) Non-co-present listening dialogue.
For dialogue scenarios where speakers are not simultaneously visible, we similarly rely on SyncNet scores. A person (suppose as $A$) is classified as being in a listening state if the following conditions are met: the ASR result is valid, the transcription confidence is above a given threshold, and the detected SyncNet score for that person’s video is below a pre-defined threshold.
In both cases, the resulting listening pair is composed of the speaker’s audio track and the listener’s silent video track.

### 4.5 SpeakVid-5M Multi-turn Branch

To enable multi-turn dialogue capabilities in dyadic interactive scenarios, we select multiple clips extracted from the same original video, assigning them with sequential index, and preserve their temporal order. We specifically collect and organize two types of data for multi-turn dialogue. For a given pair of two-person dialogue clips, we define the dialogue start timestamp as $x$ and the maximum history temporal length as $T$. All clips occurring within the interval $[x-T,x]$ are considered as preceding turns of the current dialogue:
(1) Contextual multi-turn dialogue.
The corresponding ASR transcriptions of these preceding turns are aggregated to form the multi-turn dialogue context for the current response prediction.
(2) Sequential multi-turn dialogue.
We exam the temporal gap between consecutive clips. If the gap between two adjacent clips from preceding turns is less than a predefined threshold $\delta{t}$, we consider them part of the same continuous conversation. The audio-visual data from all such clips are then utilized for predicting the current response. (We discard clips shorter than 3s during pre-processing.)
To increase the amount of multi-turn data, we relax the filtering criteria for clips that fall between two valid adjacent dialogue segments. Relaxed criterion facilitates the construction of longer and more natural conversational sequences, enabling the training of dialogue models capable of coherent, multi-turn interactions.

### 4.6 Data Stratification

Beyond the four categorical divisions, we also graded the data by quality. We created a high-quality supervised fine-tuning (SFT) subset (571K clips, $1368$ hours) by selecting samples with a hand motion blur score above $0.5$ and face blur score above $0.7$, a DOVER score above $0.6$. We also constrain the motion score to above 2 and filtering the asr quality with confidence score above $-1$. The remaining data forms the large-scale pretraining subset ($7375$ hours).

## 5 Autoregressive Talking Human Generation

We design a baseline method based on an autoregressive framework tailored for audio-visual dyadic human generation. As illustrated in Fig [5](https://arxiv.org/html/2507.09862v1#S5.F5), we incorporate Qwen2.5-Omni [48] to enable multimodal understanding of the input video and audio. Subsequently, a next-chunk prediction autoregressive model is employed to jointly generate audio and video tokens. These tokens are then used as conditioning signals for a shallow diffusion MLP, which produces videos with enhanced detail and realism.

### 5.1 Autoregressive generation of video and audio

Following Qwen2.5-Omni, we feed both the hidden states produced by the Qwen2.5-Omni thinker and the embeddings of the original audio-video inputs into the autoregressive generation head.
Besides, we employ open-source 3D variational autoencoder (VAE) with a temporal stride of 4 and a spatial stride of 8 to encode the video frames to the latent space [23]. These latents are then divided into patches and encoded as token embeddings. For audio, we borrow audio tokenizer from CosyVoice2 [17] to encode raw audio into discrete tokens.
We consider a latent map from the 3D-VAE and its corresponding audio tokens to constitute a single chunk. During attention computation, both audio and video tokens attend to all preceding tokens and the tokens within their current chunk. Video tokens are augmented with a combined 1D temporal and 2D spatial positional encoding. Audio tokens utilize a dual-level 1D positional encoding scheme, which encodes both the token’s position within its chunk and the chunk’s position in the overall sequence. The reference image is also encoded by 3D VAE. Reference image tokens are appended after the input audio and video and before visual generation tokens. To mitigate error accumulation in autoregressive generation, we introduce random noise [40] to the visual tokens during training, which encourages the model to learn more robust representations and leads to improved generation quality.

### 5.2 Visual Optimization

Inspired by MAR [22], we also employ diffusion loss and masked-token prediction to enhance visual generation. For tokens in each generated chunk, another spatial transformer from NOVA [14] takes them as input and generate a detailed visual token set-by-set. The simultaneous generation of audio-video tokens via next-chunk prediction, combined with a carefully designed frame-wise audio injection into the spatial transformer, ensures high audio-visual consistency. The visual tokens generated by the spatial transformer are temporally and spatially aligned with the latents encoded by the 3D VAE, each token corresponds to a specific patch of visual latent feature and is used as a condition signal for diffusion MLP, which performs a denoising process to generate refined visual latents that can be directly decoded by the 3D VAE. CosyVoice flow matching vocoder are used to decode the generated audio tokens into audio.

Figure: Figure 5: Our autoregressive audio-visual generation method.
Refer to caption: x11.png

### 5.3 Progressive Training Process

The model training process is divided into three stages: visual pretraining, audio-visual joint training, and high-quality dyadic dialog fine-tuning.
In the visual pretraining stage, we utilize single-speaker data, where the ASR transcription of the target audio and textual captions (including both motion and expression captions) are used as condition signals to generate video. This stage aims to train the model’s basic capability in visual content generation.
In the joint audio-visual training stage, after filtering the audio data, we continue to use the ASR transcription and captions of the target audio as inputs, but extend the generation targets to include both video and audio modality. This stage enables the model to learn synchronized audio-visual generation.
In the high-quality dyadic dialog fine-tuning stage, we select premium dialogue audio-video pairs to further fine-tune the model. The goal of this stage is to enhance its multimodal understanding capabilities and its ability to generate coherent, emotionally aligned conversations.
During training, the visual objective is optimized using a diffusion loss, while the audio objective is supervised with a cross-entropy loss for next-chunk prediction.

## 6 Experimental Results and Analysis

Figure: Figure 6: Qualitative results of our dyadic generation model. From left to right, the input video of the initiator, the reference image providing the target identity, and the mode’s generated audio-visual response.
Refer to caption: x12.png

**Table 2: Quantitative results of our baseline method on VidChatBench benchmark. Audio means generating audio jointly. Spatial means the set-by-set prediction spatial transformer. Noise means the addition of noise in training the autoregressive audio-visual generator.**
| Method | Audio | Spatial | Noise | FID $\downarrow$ | FVD $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | ArcFace $\uparrow$ | $\text{CLIP}_{dialog}$ $\uparrow$ | $\text{Sync}_{conf}$ $\uparrow$ | $\text{FID}_{Emotion}$ $\downarrow$ | SIM-o $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Conditioned |  |  |  | 56.82 | 55.06 | 15.26 | 0.62 | 0.638 | – | – | 3.45 | – |
| Conditioned | ✓ |  |  | 57.03 | 55.16 | 15.31 | 0.62 | 0.630 | – | 2.063 | 3.45 | 0.65 |
| Conditioned | ✓ | ✓ |  | 38.53 | 34.64 | 16.79 | 0.64 | 0.732 | – | 2.459 | 3.36 | 0.64 |
| Conditioned | ✓ | ✓ | ✓ | 34.72 | 30.43 | 17.39 | 0.65 | 0.758 | – | 2.655 | 3.23 | 0.65 |
| Dyadic |  |  |  | 49.97 | 47.23 | 15.74 | 0.62 | 0.637 | – | – | 3.48 | – |
| Dyadic | ✓ |  |  | 49.86 | 36.90 | 15.63 | 0.62 | 0.635 | 0.642 | 2.239 | 3.43 | 0.64 |
| Dyadic | ✓ | ✓ |  | 35.67 | 31.28 | 17.44 | 0.65 | 0.749 | 0.643 | 2.541 | 3.33 | 0.65 |
| Dyadic | ✓ | ✓ | ✓ | 32.35 | 28.82 | 17.55 | 0.66 | 0.772 | 0.643 | 2.698 | 3.22 | 0.65 |

### 6.1 The VidChatBench Benchmark.

The audio-visual dyadic interactive virtual human generation involves audio-visual understanding, response generation, and the synthesis of corresponding audio and video content. It requires models to possess both understanding and generative capabilities, making it significantly more challenging than traditional generation task. To effectively evaluate the quality and appropriateness of the generated content, we constructed the VidChatBench benchmark, consisting of 500 representative input-output pairs with unseen speaker IDs, accompanied by a set of tailored evaluation metrics.
Specifically, we assess the model performance along the following five dimensions:
(1) Video Quality. To assess the visual fidelity of the generated videos compared to the ground truth (GT), we adopt several widely used metrics in the video and audio domains, including FID, FVD, PSNR, and SSIM.
(2) Identity Preservation. The generated video is expected to maintain a consistent identity with the reference image throughout the sequence without temporal degradation. We employ ArcFace to extract facial features frame-by-frame from the video and compute the cosine distance between these features and the reference image. The average distance across all frames is used as the identity preservation score.
(3) Dialogue Coherence. The generated content should exhibit semantic relevance and appropriateness with respect to the input. For each evaluation sample in our test set, we construct $5$ candidate responses of varying quality, generated and ranked by a multimodal large language model [5]. A score is then assigned to each candidate according to its rank, using the ordered set $[0.2,0.4,0.6,0.8,1.0]$. During testing, we obtain the ASR transcription of the generated audio and compute its CLIP [34] distance to each candidate response. The final dialogue coherence score is the score of the closest-matching candidate.
(4) Audio-Visual Consistency. Synchronization between audio and lip movements is critical for the perceived realism of the generated video. We evaluate this alignment using the SyncNet confidence score, where a higher score indicates better synchronization between the generated audio and lip motion.
(5) Emotional Alignment. In human interaction, the generated portrait should exhibit emotionally appropriate and temporally coherent expressions. We utilize Deep3DFaceRecon [16] to extract 64-dimensional expression features from both the generated and GT videos. The FID of these expression features measures the emotional alignment between the generated and GT videos.
(6) Speaker Identity Preservation. Following [9], we compute the timbre similarity (SIM-o) for evaluating the timbre of generated audio. The timbre of our generated audio is controlled by the reference audio of the CosyVoice decoder.

### 6.2 Qualitative and Quantitative Results

Our primary quantitative evaluations and ablation studies are presented in Table [2](https://arxiv.org/html/2507.09862v1#S6.T2), conducted on VidChatBench. We assess our model under two distinct protocols:
The Conditioned Setting, where generation is conditioned on textual inputs, specifically the ground-truth Automatic Speech Recognition (ASR) transcript and a descriptive caption.
The Dyadic Setting, where the model generates a response directly from the audiovisual input of an initiator, thereby simulating a natural dyadic interaction.

The results reveal several key insights. First, the dyadic setting significantly outperforms the conditioned setting, which we attribute to the richer, fine-grained information preserved in direct audiovisual inputs compared to abstracted textual information. Second, our joint audiovisual generation approach successfully maintains high video quality, demonstrating that incorporating audio as an additional condition does not degrade visual fidelity. Furthermore, our ablation studies validate two crucial components: (1) the spatial transformer, which yields substantial improvements in visual metrics by refining frame-level visual tokens, and (2) the training noise injection strategy, which effectively mitigates error accumulation in the autoregressive process, enhance overall video generation quality.

## 7 Conclusion

We introduce the first large-scale high-quality dataset SpeakerVid-5M, which is designed for audio-visual dyadic interactive virtual human generation task as well as VidChatBench benchmark to evaluate performance of trained models. In addition, we provide a baseline dyadic interactive method trained on SpeakerVid-5M. We categorize SpeakerVid-5M into large-scale pretraining subset and a curated high-quality subsets for SFT. Experiments demonstrate the effectiveness of our proposed dataset and the high-quality Supervised Fine-Tuning (SFT) data.

## 8 Limitation and Future Work

While SpeakerVid-5M represents a significant contribution to the field of interactive virtual humans, we still discuss the current scope of this work and highlight several promising avenues for further development.
First, our baseline model is presented as a preliminary benchmark to validate the SpeakerVid-5M dataset. Its performance was constrained by limited computational resources and does not represent the state-of-the-art. We anticipate that future work using more advanced architectures and large-scale training will unlock the dataset’s full potential and yield higher-fidelity results.
Second, the scope of interaction within SpeakerVid-5M is focused on single-person and two-person (dyadic) scenarios. The dataset does not currently capture the complex dynamics of multi-party conversations, such as group turn-taking. Extending the data collection to include these multi-person interactions is valuable for future research.

Supplementary Material

## 9 Implementation Details

During both training and inference, we standardize the frame rate to 8 FPS and resize all video frames to 480×768 resolution. The VAE compresses the input with a temporal stride of 4 and a spatial stride of 8.
We treat each 4×4 feature patch as a token embedding, resulting in 360 tokens per frame in the latent space. The spatial transformer further refines each latent frame into 1440 tokens. For the audio modality, each chunk contains 12 audio tokens. To initiate generation, we introduce learnable special tokens as start-of-generation embeddings for both the audio and visual tokens.
The Qwen2.5-Omni Thinker is kept frozen throughout the training process, while all other components remain trainable, resulting in a total of 0.8 billion trainable parameters. The learning rate is set to 1e-4, with warm-up and decay strategies applied.
The visual pretraining and joint audio-visual training are conducted over 15 days on 128 NVIDIA L40S GPUs, with video clips in 3-7 seconds. The fine-tuning stage is carried out on 32 NVIDIA A800 GPUs over 5 days, using clips ranging from 3 to 14 seconds.

## 10 Annotation File Usage

In this section, we provide detailed explanation of the annotation files in our SpeakerVid-5M dataset to promote the application.
The basic annotation file serves as a central repository for essential metadata pertaining to each clip. This includes:
Source video ID and URL: Enabling direct access to the original video content.
Clip timestamps: The precise start and end times of the clip within the source video.
Human bounding box: Spatial localization of the primary subject within the clip’s frames.
Video resolution: The dimensions of the video content.
SyncNet confidence score: A metric assessing the lip-sync quality.
DOVER score: A quality score of overall video clip quality.
Clarity score: An indicator of visual clarity.
Speaker ID: Unique identification based on audio for the speaker present in the clip.
These comprehensive annotations are designed to empower users to directly retrieve the original videos from YouTube and efficiently extract high-quality single-person audiovisual clips using straightforward FFmpeg commands. Furthermore, we provide open-source code that facilitates the intelligent linking of these individual clips, thereby enabling the convenient construction of two-person dialogues and more complex multi-turn dialogue datasets.

The l_score Annotation.
The l_score annotation provides quantitative clarity metrics for salient regions within each video, namely the left hand, right hand, and face. For each region, we compute Laplacian-based scores on a frame-by-frame basis. These are provided as both absolute scores, which are comparable across the entire dataset, and relative scores, normalized within each clip. Furthermore, the frame-wise absolute scores are aggregated to produce a single, holistic clip-level score. This hierarchical annotation is designed for two primary use cases: the fine-grained, frame-level scores serve as dynamic conditioning signals for generative models, enabling more precise control over motion fidelity. In contrast, the clip-level scores offer an effective mechanism for dataset curation, allowing researchers to easily filter for high-clarity training samples to enhance model robustness.

The Caption Annotation.
The Caption annotation file provides a rich set of clip-level semantic labels, automatically generated using large multimodal model (qwen2.5 vl). These annotations offer a structured, multi-faceted description of each video’s content, enabling detailed analysis and control. The specific labels are as follows:

- •
Video Quality:
–
Clarity: A binary label indicating the absence or presence of significant motion blur or artifacts.
–
Camera Dynamics: Classification of camera motion (e.g., static, panning, zooming) and shot framing (e.g., close-up, medium shot).
- •
Subject & Scene Composition:
–
Subject Count: The number of individuals detected in the frame.
–
Framing View: A label indicating if the subject is framed in a full-body or upper-body view.
–
Head Pose: The estimated orientation of the subject’s head (e.g., frontal, side view).
–
Scene Entities: A list of recognized objects and persons present.
- •
Behavioral & Action Details:
–
Speech Activity: A binary label indicating active speech.
–
Motion Status: A binary label classifying the primary subject as either static or in motion.
–
Movement Intensity: A categorical rating of the subject’s motion intensity (e.g., low, medium, high).
–
Holistic Action Summary: A high-level textual description of the subject’s overall actions.
–
Fine-grained Action Description: A detailed textual account of specific, nuanced actions performed by the subject.
–
Facial Expression Summary: A textual description of the subject’s primary facial expression.

Collectively, these annotations furnish a structured representation of the video content. This enables both fine-grained conditioning for generative tasks and provides a robust framework for the objective evaluation of synthesized visual attributes and behaviors.

The Scene Annotation.
The Scene annotation provides the temporal metadata required for managing and assembling video clips from the original source footage. This annotation is generated through a two-stage process. First, an initial scene detection algorithm segments the source videos into coherent shots, recording their start/end timestamps and frame indices. Subsequently, these segments are further partitioned to enforce a maximum duration of 14 seconds per clip. The final annotation file catalogs each processed clip with a unique identifier and its precise temporal boundaries (start and end times). This structured metadata is crucial for downstream applications, enabling the seamless concatenation of clips for long-form video synthesis and providing a flexible framework for tasks involving multi-turn dialogue or extended narrative generation.

The Speaker Annotation.
Speaker annotation is performed on the original audio of video recordings to provide detailed speaker diarization results. This comprehensive annotation process involves several key components.
Initially, raw diarization results are generated, capturing the precise start and end times for each detected speaker turn, along with their respective assigned speaker IDs. These results provide a foundational temporal map of all spoken segments.
For applications focused on dyadic dialogue scenarios, the raw results undergo a filtering process. This step specifically identifies and prioritizes the two primary speakers (labeled as Speaker A and Speaker B). A critical criterion for this filtering is that these two speakers must collectively account for at least 80% of the total speech duration within the video. Furthermore, in the context of dyadic data filtering, only the conversational exchanges occurring exclusively between Speaker A and Speaker B are retained. This ensures the selection of highly relevant dialogue for subsequent model training and analysis.
The final stage yields a cleaned and refined list of speaker IDs and their corresponding start and end times for each segment. This provides clear and accurate speaker attribution of the video, ensuring high-quality data for downstream tasks.

The ASR Annotation.
ASR annotation is performed on unified single-person audiovisual clips, which are meticulously derived from praevia video and audio processing. For example, in dual-person co-present scenes, the segmentation process spatially divides the video into two distinct regions, each corresponding to an individual. Concurrently, the associated audio is temporally segmented to align with the speech of each respective person.
To achieve these single-person audiovisual clips—where the video prominently features only one person and the audio contains solely that person’s speech—we integrate YOLO for visual analysis with speaker diarization for audio segmentation. Upon obtaining these refined clips, automatic speech recognition is then applied to their corresponding audio segments. The ASR output encompasses several key components.
Transcribed speech text: The verbatim textual representation of the spoken content.
Confidence score: A metric indicating the reliability and accuracy of the transcription.
Speech compression ratio: The ratio between the original speech duration and its compressed form, relevant for efficiency analysis.
No-speech probability: The likelihood that a given audio segment contains no discernible speech.
Language information: Identification of the language spoken within the clip.

## 11 Visualization of Pretrain and Finetune Model

As illustrated in Figure [7](https://arxiv.org/html/2507.09862v1#S11.F7), the model resulting from our initial pre-training phase exhibits pronounced motion blur, particularly in the face and hands. These artifacts are significantly exacerbated during rapid movements, degrading perceptual quality. To address this, we curated a high-quality subset by filtering the training data based on facial and hand clarity. The figure demonstrates that fine-tuning on this refined data yields substantial improvements in visual fidelity, markedly reducing blur and enhancing detail in regions critical for human perception.
To enable this data refinement, we compute blur scores for both the face and hands in each frame using a Laplacian-based method. Recognizing their broader utility, we release these scores as part of our public dataset annotations. The value of such explicit quality signals is substantiated by prior work; for instance, CyberHost demonstrated that conditioning a model on hand blur scores can significantly enhance the clarity and fidelity of synthesized hand motions.

Figure: Figure 7: Impact of finetuning on generation quality. A comparison between the model after pretraining phase and after finetuning on our high-quality subset.
Refer to caption: x13.png

Figure: Figure 8: Visualization of multi-turn dialogue and listening scenarios. Top (Multi-turn Dialogue): We showcase a sequence of conversational turns between an initiator and a responder. By preserving temporal context, our dataset facilitates the training of models capable of coherent, long-form conversations. Bottom (Listening Case): A speaker is paired with a non-speaking listener.
Refer to caption: x14.png

Figure: Figure 9: Diverse examples of dyadic pairs from SpeakerVid-5M. This figure highlights the diversity of subjects, environments, and interaction styles captured in our dataset.
Refer to caption: x15.png

## 12 Prompt Used in Annotation

Tables [3](https://arxiv.org/html/2507.09862v1#S12.T3) and [4](https://arxiv.org/html/2507.09862v1#S12.T4) present the caption prompts used for Qwen-VL-2.5, which serve as effective guidance for generating high-quality responses during captioning.

**Table 3: Prompt Used in Video Annotation**
| # Your Role: Video Annotation Expert |
| --- |
|  |
| ## Objective:<br>As a professional annotator, your task is to evaluate and label video content based on clarity, camera motion, human presence and activity, and semantic understanding. Your annotations must be structured and follow a predefined JSON format. |
|  |
| ## Annotation Guidelines<br>For each input video, assess and annotate the following attributes:<br>1.<br>Visual Clarity: Assess whether the primary subject in the video is clearly visible and distinguishable. Options: "Yes" or "No".<br>2.<br>Camera Motion: Identify the type(s) of camera movement observed. Choose one or more from the following: ["Static Camera", "Dynamic Shooting", "Still Shot", "Left-to-Right Pan", "Right-to-Left Pan", "Zoom In", "Zoom Out"].<br>3.<br>Human Motion Presence: Determine whether the person in the video exhibits obvious and recognizable physical movement. Options: "Yes" or "No".<br>4.<br>Motion Intensity Level: Rate the level of physical activity on a scale from 1 (very minimal motion) to 5 (extremely intense movement). Aim to differentiate clearly across the full range rather than concentrating ratings in the middle.<br>5.<br>Entity List: List all visually prominent entities in the video in descending order of saliency, e.g., ["man", "teddy bear", "river", "traffic sign", "apple"].<br>6.<br>Speech Presence: Identify whether the person is clearly speaking in the video. Options: "Yes" or "No".<br>7.<br>Observed Actions: List the specific actions shown in the video, such as ["running", "dancing", "eating", "talking", "singing", "presenting"].<br>8.<br>Number of People: Count the number of distinct people visible in the video.<br>9.<br>Upper-Body Only: Indicate whether only the upper half of the person is visible (if the legs or lower body appear, select ”No”). Options: "Yes" or "No".<br>10.<br>Facing Direction: Specify the subject’s orientation relative to the camera. Options: "Front", "Side", or "Back". |
| ## Output Format:<br>Return the annotation results in the following JSON structure:<br>1.<br>Clarity: "Yes"<br>2.<br>Camera Motion: ["Zoom In", "Left-to-Right Pan"]<br>3.<br>Human Motion Presence: "Yes"<br>4.<br>Motion Intensity Level: [3]<br>5.<br>Entity List: ["man", "dog", "apple"]<br>6.<br>Speech Presence: "Yes"<br>7.<br>Observed Actions: ["talking"]<br>8.<br>Number of People: 1<br>9.<br>Upper-Body Only: "Yes"<br>10.<br>Facing Direction: "Side" |
| Note:<br>Please ignore any subtitles or on-screen text when performing the annotation. Focus solely on the video’s visual and auditory content. |

**Table 4: Prompt Used in Video Motion Intensity Level**
| # Your Role: Human Motion Expert |
| --- |
|  |
| ## Annotation Guidelines:<br>As a human motion expert, assess the intensity of movement in the video by focusing on the amplitude and frequency of body movements:<br>1. Movement Amplitude: Whether the person frequently makes large body movements like arm swings, body turns, etc.<br>2. Movement Frequency: Whether the person repeats similar movements frequently, such as nodding their head, gesturing, etc. Score range from 1 to 5.<br>1.<br>The person is nearly stationary, with only minimal head or small hand movements.<br>2.<br>The person makes occasional small gestures or minor body adjustments.<br>3.<br>The person uses moderate gestures with occasional body adjustments, such as slight forward leans.<br>4.<br>The person uses frequent gestures, with larger body movements and more frequent body shifts.<br>5.<br>The person’s movements are frequent and large, with extensive use of hand gestures and body shifts, indicating high intensity. |
|  |
| # Your Role: Audience Member |
|  |
| ## Annotation Guidelines:<br>As an audience member, assess the movement intensity of the speaker by focusing on their emotional expression and the resulting body movements:<br>1. Emotional Fluctuations: Whether the speaker shows large emotional fluctuations in their speech, and whether it is accompanied by body language.<br>2. Audience Reactions: Whether the speaker adjusts their body in response to audience reactions (e.g., applause, nodding). Score range from 1 to 5.<br>1.<br>The speaker speaks in a calm, even tone with minimal body movement.<br>2.<br>The speaker has slight emotional fluctuations, with occasional small gestures or head movements.<br>3.<br>The speaker shows moderate emotional fluctuations and occasional body movements like hand gestures or body shifts.<br>4.<br>The speaker shows significant emotional fluctuations, with frequent body movements, gestures, and emotional intensity.<br>5.<br>The speaker displays intense emotional fluctuations, with frequent and large gestures and body movements, indicating high intensity. |
|  |
| # Your Role: Labeling Expert |
|  |
| ## Annotation Guidelines:<br>As a labeling expert, assess the intensity of movement by considering the body language and interaction frequency:<br>1. Gesture Usage: Whether the person frequently uses hand gestures or body movements to emphasize their speech.<br>2. Interaction Frequency: Whether the person interacts frequently with others, especially through body language responses (e.g., nodding, smiling). Score range from 1 to 5.<br>1.<br>The person makes little to no gestures or body movements and interacts minimally.<br>2.<br>The person occasionally uses gestures or makes small body movements, with limited interaction.<br>3.<br>The person frequently uses gestures and makes moderate body adjustments, with moderate interaction with others.<br>4.<br>The person uses hand gestures frequently, with larger body movements and frequent interaction.<br>5.<br>The person has frequent and strong body movements, with highly frequent interactions and large gestures, indicating high intensity. |

## 13 Failed Case and Analysis

(1) Fine-grained Hand Dynamics: Prevailing talking-head datasets are predominantly constrained to rudimentary hand motions, such as resting, crossing, or waving. Consequently, the synthesis of more complex and expressive actions, including intricate gesturing or human-object interactions, remains a formidable challenge in digital human generation. This difficulty is compounded by the fact that textual descriptions typically lack the requisite granularity to supervise these fine motor skills, thereby hindering a model’s ability to learn rich hand dynamics. To address this limitation, we provide comprehensive annotations for each video clip, encompassing hand keypoints, bounding boxes, and a novel hand-blur score. These annotations furnish precise supervisory signals, enabling more effective control and rigorous evaluation of synthesized hand motion quality.

(2) Synthesis of Occluded or Out-of-Frame Body Parts: A core challenge in this domain is the synthesis of body parts that are occluded or absent in the reference image, for instance, rendering a newly raised arm or animating eyes from a closed to an open state. This task necessitates models endowed with robust generative priors and a comprehensive understanding of human anatomy and plausible motion. The challenge is often exacerbated by data bias; many existing datasets are heavily skewed towards front-facing, tightly-cropped portraits, which lack diverse viewpoints and full-body context. To counteract this, our dataset incorporates a wide array of perspectives, including full-body, upper-body, and profile views. This provides a more balanced distribution of human body configurations, thereby empowering models to generalize beyond the visible input and mitigate in-painting artifacts.

(3) Temporal Coherence in Long-Form Video Generation: Generating temporally coherent, long-form video presents two primary obstacles. First, the scarcity of high-quality, long-duration training data fundamentally limits model performance. Our dataset addresses this by providing a large corpus of high-fidelity clips (5–14 seconds) that are specifically designed for seamless concatenation into extended sequences (e.g., 20 seconds to 3 minutes). Second, autoregressive models are prone to error accumulation, which degrades generation quality over time. To mitigate this, we introduce a simple yet effective noise injection strategy during training. This technique enhances the model’s temporal stability, ensuring consistent quality over extended generative horizons.

Figure: Figure 10: Analysis of failure cases. These include generating facial artifacts or unnatural distortions and struggling with severe motion blur during rapid head or hand movements.
Refer to caption: x16.png

## References

- [1]
Veo3.
- Afouras et al. [2018a]
Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman.
Deep audio-visual speech recognition.
*IEEE transactions on pattern analysis and machine intelligence*, 44(12):8717–8727, 2018a.
- Afouras et al. [2018b]
Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman.
Lrs3-ted: a large-scale dataset for visual speech recognition.
*arXiv preprint arXiv:1809.00496*, 2018b.
- Ao [2024]
Tenglong Ao.
Body of her: A preliminary study on end-to-end humanoid agent.
*arXiv preprint arXiv:2408.02879*, 2024.
- Bai et al. [2025]
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al.
Qwen2. 5-vl technical report.
*arXiv preprint arXiv:2502.13923*, 2025.
- [6]
Brandon Castellano.
PySceneDetect.
[https://github.com/Breakthrough/PySceneDetect](https://github.com/Breakthrough/PySceneDetect).
- Caba Heilbron et al. [2015]
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles.
Activitynet: A large-scale video benchmark for human activity understanding.
In *Proceedings of the ieee conference on computer vision and pattern recognition*, pages 961–970, 2015.
- Chang et al. [2023]
Di Chang, Yichun Shi, Quankai Gao, Jessica Fu, Hongyi Xu, Guoxian Song, Qing Yan, Xiao Yang, and Mohammad Soleymani.
Magicdance: Realistic human dance video generation with motions & facial expressions transfer.
*CoRR*, 2023.
- Chen et al. [2024]
Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, and Xie Chen.
F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching.
*arXiv preprint arXiv:2410.06885*, 2024.
- Chen et al. [2025a]
Yi Chen, Sen Liang, Zixiang Zhou, Ziyao Huang, Yifeng Ma, Junshu Tang, Qin Lin, Yuan Zhou, and Qinglin Lu.
Hunyuanvideo-avatar: High-fidelity audio-driven human animation for multiple characters.
*arXiv preprint arXiv:2505.20156*, 2025a.
- Chen et al. [2025b]
Yafeng Chen, Siqi Zheng, Hui Wang, Luyao Cheng, et al.
3d-speaker-toolkit: An open source toolkit for multi-modal speaker verification and diarization.
2025b.
- Chung and Zisserman [2017]
Joon Son Chung and Andrew Zisserman.
Lip reading in the wild.
In *Computer Vision–ACCV 2016: 13th Asian Conference on Computer Vision, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13*, pages 87–103. Springer, 2017.
- Chung et al. [2018]
Joon Son Chung, Arsha Nagrani, and Andrew Zisserman.
Voxceleb2: Deep speaker recognition.
*arXiv preprint arXiv:1806.05622*, 2018.
- Deng et al. [2024]
Haoge Deng, Ting Pan, Haiwen Diao, Zhengxiong Luo, Yufeng Cui, Huchuan Lu, Shiguang Shan, Yonggang Qi, and Xinlong Wang.
Autoregressive video generation without vector quantization.
*arXiv preprint arXiv:2412.14169*, 2024.
- Deng et al. [2019a]
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.
Arcface: Additive angular margin loss for deep face recognition.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 4690–4699, 2019a.
- Deng et al. [2019b]
Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong.
Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops*, pages 0–0, 2019b.
- Du et al. [2024]
Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, et al.
Cosyvoice 2: Scalable streaming speech synthesis with large language models.
*arXiv preprint arXiv:2412.10117*, 2024.
- Jocher and the Ultralytics Team [2023]
Glenn Jocher and the Ultralytics Team.
Ultralytics YOLOv8: Cutting-edge object detection models.
[https://github.com/ultralytics/ultralytics](https://github.com/ultralytics/ultralytics), 2023.
Accessed: 2025-06-27.
- Lee et al. [2021]
Sangho Lee, Jiwan Chung, Youngjae Yu, Gunhee Kim, Thomas Breuel, Gal Chechik, and Yale Song.
Acav100m: Automatic curation of large-scale datasets for audio-visual video representation learning.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 10274–10284, 2021.
- Li et al. [2024a]
Chunyu Li, Chao Zhang, Weikai Xu, Jinghui Xie, Weiguo Feng, Bingyue Peng, and Weiwei Xing.
Latentsync: Audio conditioned latent diffusion models for lip sync.
*arXiv preprint arXiv:2412.09262*, 2024a.
- Li et al. [2025]
Hui Li, Mingwang Xu, Yun Zhan, Shan Mu, Jiaye Li, Kaihui Cheng, Yuxuan Chen, Tan Chen, Mao Ye, Jingdong Wang, et al.
Openhumanvid: A large-scale high-quality dataset for enhancing human-centric video generation.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pages 7752–7762, 2025.
- Li et al. [2024b]
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He.
Autoregressive image generation without vector quantization.
*Advances in Neural Information Processing Systems*, 37:56424–56445, 2024b.
- Lin et al. [2024]
Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang, Xianyi He, Yang Ye, Shenghai Yuan, Liuhan Chen, et al.
Open-sora plan: Open-source large video generation model.
*arXiv preprint arXiv:2412.00131*, 2024.
- Lin et al. [2025a]
Gaojie Lin, Jianwen Jiang, Chao Liang, Tianyun Zhong, Jiaqi Yang, Zerong Zheng, and Yanbo Zheng.
Cyberhost: A one-stage diffusion framework for audio-driven talking body generation.
In *The Thirteenth International Conference on Learning Representations*, 2025a.
- Lin et al. [2025b]
Gaojie Lin, Jianwen Jiang, Jiaqi Yang, Zerong Zheng, and Chao Liang.
Omnihuman-1: Rethinking the scaling-up of one-stage conditioned human animation models.
*arXiv preprint arXiv:2502.01061*, 2025b.
- Low and Wang [2025]
Chetwin Low and Weimin Wang.
Talkingmachines: Real-time audio-driven facetime-style video via autoregressive diffusion models.
*arXiv preprint arXiv:2506.03099*, 2025.
- Luo et al. [2025]
Yuxuan Luo, Zhengkun Rong, Lizhen Wang, Longhao Zhang, Tianshu Hu, and Yongming Zhu.
Dreamactor-m1: Holistic, expressive and robust human image animation with hybrid guidance.
*arXiv preprint arXiv:2504.01724*, 2025.
- Nagrani et al. [2017]
Arsha Nagrani, Joon Son Chung, and Andrew Zisserman.
Voxceleb: a large-scale speaker identification dataset.
*arXiv preprint arXiv:1706.08612*, 2017.
- Ng et al. [2022]
Evonne Ng, Hanbyul Joo, Liwen Hu, Hao Li, Trevor Darrell, Angjoo Kanazawa, and Shiry Ginosar.
Learning to listen: Modeling non-deterministic dyadic facial motion.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 20395–20405, 2022.
- Ng et al. [2023]
Evonne Ng, Sanjay Subramanian, Dan Klein, Angjoo Kanazawa, Trevor Darrell, and Shiry Ginosar.
Can language models learn to listen?
In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 10083–10093, 2023.
- Peng et al. [2025]
Ziqiao Peng, Jiwen Liu, Haoxian Zhang, Xiaoqiang Liu, Songlin Tang, Pengfei Wan, Di Zhang, Hongyan Liu, and Jun He.
Omnisync: Towards universal lip synchronization via diffusion transformers.
*arXiv preprint arXiv:2505.21448*, 2025.
- Prajwal et al. [2020]
KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar.
A lip sync expert is all you need for speech to lip generation in the wild.
In *Proceedings of the 28th ACM international conference on multimedia*, pages 484–492, 2020.
- Qi et al. [2025]
Jinwei Qi, Chaonan Ji, Sheng Xu, Peng Zhang, Bang Zhang, and Liefeng Bo.
Chatanyone: Stylized real-time portrait video generation with hierarchical motion diffusion model.
*arXiv preprint arXiv:2503.21144*, 2025.
- Radford et al. [2021]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.
Learning transferable visual models from natural language supervision.
In *International conference on machine learning*, pages 8748–8763. PmLR, 2021.
- Radford et al. [2023]
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever.
Robust speech recognition via large-scale weak supervision.
In *International conference on machine learning*, pages 28492–28518. PMLR, 2023.
- Raina and Arora [2022]
Akshay Raina and Vipul Arora.
Syncnet: Using causal convolutions and correlating objective for time delay estimation in audio signals.
*arXiv preprint arXiv:2203.14639*, 2022.
- Shahroudy et al. [2016]
Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang.
Ntu rgb+ d: A large scale dataset for 3d human activity analysis.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 1010–1019, 2016.
- Soomro et al. [2012]
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.
Ucf101: A dataset of 101 human actions classes from videos in the wild.
*arXiv preprint arXiv:1212.0402*, 2012.
- Tian et al. [2024]
Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo.
Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions.
In *European Conference on Computer Vision*, pages 244–260. Springer, 2024.
- Valevski et al. [2024]
Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter.
Diffusion models are real-time game engines.
*arXiv preprint arXiv:2408.14837*, 2024.
- Wang et al. [2023]
Duomin Wang, Yu Deng, Zixin Yin, Heung-Yeung Shum, and Baoyuan Wang.
Progressive disentangled representation learning for fine-grained controllable talking head synthesis.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 17979–17989, 2023.
- Wang et al. [2025a]
Duomin Wang, Bin Dai, Yu Deng, and Baoyuan Wang.
Disentangling planning, driving and rendering for photorealistic avatar agents.
In *Computer Vision – ECCV 2024 Workshops*, pages 137–147, Cham, 2025a. Springer Nature Switzerland.
- Wang et al. [2020]
Kaisiyuan Wang, Qianyi Wu, Linsen Song, Zhuoqian Yang, Wayne Wu, Chen Qian, Ran He, Yu Qiao, and Chen Change Loy.
Mead: A large-scale audio-visual dataset for emotional talking-face generation.
In *European conference on computer vision*, pages 700–717. Springer, 2020.
- Wang et al. [2025b]
Mengchao Wang, Qiang Wang, Fan Jiang, Yaqi Fan, Yunpeng Zhang, Yonggang Qi, Kun Zhao, and Mu Xu.
Fantasytalking: Realistic talking portrait generation via coherent motion synthesis.
*arXiv preprint arXiv:2504.04842*, 2025b.
- Wang et al. [2021]
Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu.
One-shot free-view neural talking-head synthesis for video conferencing.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 10039–10049, 2021.
- Wei et al. [2025]
Cong Wei, Bo Sun, Haoyu Ma, Ji Hou, Felix Juefei-Xu, Zecheng He, Xiaoliang Dai, Luxin Zhang, Kunpeng Li, Tingbo Hou, et al.
Mocha: Towards movie-grade talking character synthesis.
*arXiv preprint arXiv:2503.23307*, 2025.
- Wu et al. [2023]
Haoning Wu, Erli Zhang, Liang Liao, Chaofeng Chen, Jingwen Hou, Annan Wang, Wenxiu Sun, Qiong Yan, and Weisi Lin.
Exploring video quality assessment on user generated contents from aesthetic and technical perspectives.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 20144–20154, 2023.
- Xu et al. [2025]
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al.
Qwen2. 5-omni technical report.
*arXiv preprint arXiv:2503.20215*, 2025.
- Xu et al. [2024]
Sicheng Xu, Guojun Chen, Yu-Xiao Guo, Jiaolong Yang, Chong Li, Zhenyu Zang, Yizhong Zhang, Xin Tong, and Baining Guo.
Vasa-1: Lifelike audio-driven talking faces generated in real time.
*Advances in Neural Information Processing Systems*, 37:660–684, 2024.
- Yang et al. [2025]
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al.
Qwen3 technical report.
*arXiv preprint arXiv:2505.09388*, 2025.
- Yang et al. [2023]
Zhendong Yang, Ailing Zeng, Chun Yuan, and Yu Li.
Effective whole-body pose estimation with two-stages distillation.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 4210–4220, 2023.
- Yu et al. [2023]
Jianhui Yu, Hao Zhu, Liming Jiang, Chen Change Loy, Weidong Cai, and Wayne Wu.
Celebv-text: A large-scale facial text-video dataset.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 14805–14814, 2023.
- Yu et al. [2022]
Zhentao Yu, Zixin Yin, Deyu Zhou, Duomin Wang, Finn Wong, and Baoyuan Wang.
Talking head generation with probabilistic audio-to-visual diffusion priors. 2023 ieee.
In *CVF International Conference on Computer Vision (ICCV)*, pages 7611–7621, 2022.
- Zhang et al. [2021]
Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan.
Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 3661–3670, 2021.
- Zhou et al. [2021]
Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, and Ziwei Liu.
Pose-controllable talking face generation by implicitly modularized audio-visual representation.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 4176–4186, 2021.
- Zhou et al. [2022]
Mohan Zhou, Yalong Bai, Wei Zhang, Ting Yao, Tiejun Zhao, and Tao Mei.
Responsive listening head generation: a benchmark dataset and baseline.
In *European conference on computer vision*, pages 124–142. Springer, 2022.
- Zhu et al. [2022]
Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang, Siwei Tang, Li Zhang, Ziwei Liu, and Chen Change Loy.
Celebv-hq: A large-scale video facial attributes dataset.
In *European conference on computer vision*, pages 650–667. Springer, 2022.
- Zhu et al. [2025]
Yongming Zhu, Longhao Zhang, Zhengkun Rong, Tianshu Hu, Shuang Liang, and Zhipeng Ge.
Infp: Audio-driven interactive head generation in dyadic conversations.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pages 10667–10677, 2025.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Are the person ids per video or are they consistent across multiple videos, i.e. same speaker, different day?
soundness: 3
strengths: Dyadic human videos are an important mode of human-centric video generation. Having a large dataset with paired ASR, Audio and video is highly valuable. The paper is easy to follow and the authors provide an extensive ethics statement. VidChatBench is a reasonable benchmark suite for the proposed dataset.
summary: SpeakerVid-5M îs a large-scale dyadic talking humans dataset. The dataset contains automatically extracted annotations of 2D kpts, audio, audio-to-text, and scene descriptions. The dataset is accompanied by an extensive video benchmark.
weaknesses: It seems that the dataset only contains pairs of videos - initiator -> respond. However, to future-prove this, I wonder if the authors could make available extended back-and-forth sequences of their data as well, i.e. where the initiator and responder engage in a back and forth way.

The authors claim that the video resolution is 1080P - however, their sample videos in the supplementary material are crops which are significantly smaller than 1080P. Could the authors clarify if they will release the full resolution? 

I find Figure 1 a bit unclear: what are the multi-modal annotations? I think the figure should make more clear in what the model inputs and outputs are for the generation process. 

It seems some of the example videos contain jump cuts at the end (Body Composition/full_body/3.mp4 ) - I wonder how many videos contain those “”incorrect cuts - could the authors comment if they are planning to post-process the dataset to identify those instances or if they believe those to not be a problem?


Suggested additional citations: TalkCuts [1].


[1] A Large-Scale Dataset for Multi-Shot Human Speech Video Generation; NeurIPS DBT 2025

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Modular Baseline Comparison: Could the authors provide results for a strong modular baseline on VidChatBench?  See W3
soundness: 3
strengths: 1. The paper addresses a critical and timely problem. The research community's focus is clearly shifting from passive, audio-driven "talking heads" to proactive, interactive digital humans. The authors correctly identify that the single greatest barrier to open-source academic research in this area is the lack of a large-scale, high-quality dataset specifically capturing dyadic audio-visual interactions. This contribution directly unblocks this important future direction.
2. The dataset is a significant contribution, and its value is strongly reinforced by the rigorous and transparent data curation pipeline. The authors provide excellent clarity (particularly in Figure 2) on each methodical step, which builds trust in the data's fidelity. The use of a modern, multi-model suite for filtering and annotation is a major strength. For example, employing DOVER for video quality assessment, SyncNet to ensure tight audio-visual synchronization, and a powerful VLM like Qwen2.5-VL to generate rich, structured annotations (like motion, expression, and pose) goes far beyond simple ASR transcripts and ensures the data is high-quality and multi-faceted.
3. The proposed model architecture makes sense to me. The baseline is built on Qwen2.5-Omni. The 'thinker' to understand the initiator's multimodal input, and a joint Audio-Visual Generator that acts as a 'renderer' to produce the audio and video response. This end-to-end approach is a modern paradigm, and providing this baseline (along with the VidChatBench) gives the community a strong starting point and clearly demonstrates the dataset's immediate utility.
summary: This paper introduces a new dataset and a new benchmark, accompanied by a baseline model.
The dataset, SpeakerVid-5M, is a large-scale, high-quality collection specifically designed for audio-visual dyadic interactive virtual human generation. It contains over 8K hours of human portrait video clips.
The dataset is richly annotated with multi-modal information, including structured textual captions, human skeletal sequences (DWpose), ASR transcripts, and blur scores. It is further organized into four interaction types (dialogue, single, listening, and multi-turn branches) and divided into two tiers: a large-scale pre-training subset and a high-quality Supervised Fine-Tuning (SFT) subset.
The authors also propose an autoregressive (AR)-based video chat baseline model trained on this dataset and introduce VidChatBench, a dedicated benchmark with tailored metrics for evaluating performance on the complex task of audio-visual dyadic generation, which requires both multi-modal understanding and generation capabilities.
The dataset and processing code are planned to be publicly released.
weaknesses: 1. Limited Data Domain and Generality: The data sources, while high-quality, are heavily skewed towards formal or semi-formal scenarios (interviews, news, seminars, debates). The dataset appears to lack more casual, "in-the-wild" interaction styles, such as personal vlogs, movie/TV drama scenes, or general user-generated content. This "domain bias" might limit the ability of models trained on this data to generalize to the full spectrum of human interaction needed for a truly "general purpose" digital human.

2. Ambiguous Baseline Model Architecture: The description of the autoregressive baseline in Section 5 is too high-level, leaving several critical implementation details unclear. This makes the model difficult to reproduce or fairly assess.

"Diffusion MLP": This component is a black box. What is its architecture? Is it a simple MLP, or a more complex structure like a DiT? How are the time-step and visual token conditions precisely integrated?

"AR Generation Loop": This is the most confusing part. The model is autoregressive, but it also uses a diffusion process to "refine" latents. For the next generation step, what exactly is fed back into the AR model? Is it the initial, raw tokens from the AR generator, or the refined latents after the diffusion/denoising step? .

"Next-chunk prediction": What is the temporal size of a "chunk"? Are the visual tokens continuous or discrete token?

3. Missing SOTA Modular Baseline Comparison: The paper rightly claims no single end-to-end model exists for this task, but it fails to compare against the most obvious and powerful SOTA alternative: a modular pipeline. A proper evaluation would benchmark the proposed model against a baseline that 1) uses an LLM/VLM to generate a text response, 2) uses a TTS model for speech, and 3) uses a state-of-the-art speech-to-video model (like Hallo3, etc.) to synthesize the video. Without this comparison, it's impossible to know if the proposed end-to-end method is actually superior to (or even competitive with) what's already possible by combining existing SOTA components.

---

confidence: 2
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. Beyond scale, what distinguishes SpeakerVid-5M from VoxCeleb2 + LRS3? Are there new annotation dimensions (e.g., speaker emotion, interaction type, recording context) that justify the “5M” contribution?

2. Have authors quantified the rate of misaligned or noisy samples? What proportion of clips have accurate lip audio synchronization?

3. Could you report statistics on gender, age, ethnicity, and geographic diversity? How do you mitigate bias from the over-representation of Western/English speakers?

4. Are the improvements in Table 4 solely due to dataset size, or did you modify architecture or hyperparameters? A same-architecture ablation on VoxCeleb vs. SpeakerVid would help isolate the effect.
soundness: 3
strengths: 1. SpeakerVid-5M is one of the largest open-source datasets in the speaker understanding domain, with diverse speakers, recording conditions, and languages, making it valuable for pretraining multimodal encoders.

2. By integrating speaker verification, lip-reading, and voice–face retrieval within one dataset, it provides a practical testbed for representation transfer and multi-task learning.

3. The use of automatic speech-face alignment and quality filtering (lip-sync thresholding, blur detection) demonstrates thoughtful engineering and scalability.
summary: This paper presents SpeakerVid-5M, a large-scale audio-visual speaker understanding dataset containing 5 million labeled video clips paired with aligned face and speech data. The authors aim to enable multi-modal representation learning for speaker identification, speech-to-face generation, and lip-synchronization tasks. The dataset is collected from online sources (e.g., YouTube) using an automated pipeline that detects talking faces, aligns speech segments, and filters content for language and quality.
weaknesses: 1. The core contribution is the dataset’s size, not a new method, model, or theoretical insight. It extends existing pipelines rather than rethinking them.

2. There is little quantitative characterization of dataset diversity, no breakdown by gender, age, language, recording environment, or cultural representation.

3. Large-scale scraping of online videos raises questions about consent, copyright, and potential bias amplification. The ethical section is superficial and lacks compliance discussion (e.g., GDPR, US fair-use).


Decision: Conditional Accept (Poster)