AV-STE: Audio-Visual Speech Token Enhancement

EMNLP 2026 — Noise Adaptive Streaming Audio-Visual Speech Token Enhancement for Robust Full-Duplex Spoken Dialogue Models

Code, demo, and full documentation: github.com/bellagodiva/av-ste

AV-STE recovers clean Mimi semantic tokens from noisy speech by fusing audio features with lip-ROI video through an entropy-gated cross-attention module built on top of AV-HuBERT. Enhanced tokens can be fed directly into any Mimi-based TTS or codec LM (e.g. Moshi) in place of the noisy ones.

Files

Checkpoint	Use for
avste.pt	Clean speech, non-speech background noise, cross-dataset speaker interference
avste_lrs3_interference.pt	Same-corpus-style competing speakers, out-of-domain video

Together they reproduce every semantic-token-accuracy row in the paper's main table. large_vox_iter5.pt (the public AV-HuBERT-Large VoxCeleb2 backbone both were fine-tuned from) is not hosted here — the GitHub repo's download script fetches it directly from its original source.

Full checkpoint-selection guidance: checkpoints/README.md in the code repo.

Usage

from huggingface_hub import hf_hub_download

path = hf_hub_download(repo_id="bgdv99/av-ste", filename="avste.pt")
# or filename="avste_lrs3_interference.pt"

Both are fairseq model ensemble files:

from fairseq import checkpoint_utils

models, cfg, task = checkpoint_utils.load_model_ensemble_and_task(
    ["avste.pt"],
    arg_overrides={"w2v_path": "large_vox_iter5.pt"},
)

See the GitHub repo for the full inference pipeline (scripts/infer_avste.py), a runnable demo, and training instructions.

Data

These weights were fine-tuned on LRS3 (license-gated) with AudioSet noise/interference augmentation. No training or evaluation data is redistributed here — only the resulting model parameters.

Citation

@inproceedings{avste2026,
  title     = {Noise Adaptive Streaming Audio-Visual Speech Token Enhancement for Robust Full-Duplex Spoken Dialogue Models},
  author    = {TODO},
  booktitle = {Proceedings of EMNLP},
  year      = {2026},
}