RogerTzeng/Sparse-MERIT

Sparse MERIT

Official pretrained checkpoints for Sparse Mixture-of-Experts Representation Integration Technique (Sparse MERIT), introduced in:

Joint Learning Using Mixture-of-Expert-Based Representation for Speech Enhancement and Robust Emotion Recognition
Jing-Tong Tzeng, Carlos Busso, and Chi-Chun Lee
IEEE Transactions on Audio, Speech and Language Processing, vol. 34, pp. 3026–3038, 2026.

Paper · arXiv · Code and evaluation scripts

Model overview

Sparse MERIT jointly learns speech enhancement (SE) and speech emotion recognition (SER) from noisy speech. A shared WavLM-Large backbone supplies frame-level representations to a mixture of experts, with separate task-specific routers selecting expert features for enhancement and emotion classification. This design supports task specialization within a shared representation framework.

Component	Description
Speech backbone	WavLM-Large; 24 Transformer layers plus the pre-Transformer representation
Representation integration	Concatenation of 25 representations, each with 1,024 dimensions
Experts	Three shared feed-forward experts; 25,600 → 4,096 → 1,024 dimensions
Routing	Task-specific, frame-wise Top-1 expert selection
SER head	Attentive statistics pooling followed by a classification head
SE head	BSSE-SE-based decoder combining expert features with the noisy log-magnitude spectrogram
Audio input	Mono speech at 16 kHz, using training-set waveform normalisation
SER output order	0: Angry, 1: Sad, 2: Happy, 3: Neutral

These are component checkpoints for the custom PyTorch implementation in the linked GitHub repository. Use its model definitions and loading code to assemble the system.

Released files

The root directory contains WavLM-Large.pt and the first-stage head checkpoints pretrained_pool.pt, pretrained_ser.pt, and pretrained_se.pth.tar.

The final checkpoint components are stored under:

bsse_noisy_weight_cat_ser_4_classes_git_ws_1_1_5e-5_SSL_0_5_Original_weight_switch_mmoe_3experts/wavLM_adamW/7/

File	Contents
final_ssl.pt	Fine-tuned speech backbone
final_mmoe.pt	Experts and task-specific routing modules
final_pool.pt	SER pooling module
final_ser.pt	Emotion classification head
final_se.pt	Speech enhancement head
final_loss.pt	Saved training loss-module state
train_norm_stat.pkl	Noisy-training waveform mean and standard deviation, used for evaluation input
clean_train_norm_stat.pkl	Clean-training waveform mean and standard deviation

Download and use

Clone the implementation and install its dependencies. The repository documents Python 3.9 as its development environment.

git clone https://github.com/RogerTzeng/Sparse-MERIT.git
cd Sparse-MERIT
pip install -r requirements.txt
pip install huggingface_hub

Run the following Python code from the cloned repository to download weights and normalisation files into the expected directory layout:

from huggingface_hub import HfApi, hf_hub_download

repo_id = "RogerTzeng/Sparse-MERIT"
extensions = (".pt", ".pth", ".pth.tar", ".safetensors", ".ckpt", ".pkl")

for filename in HfApi().list_repo_files(repo_id):
    if filename.endswith(extensions):
        hf_hub_download(
            repo_id=repo_id,
            filename=filename,
            local_dir="model" if "/" in filename else "pretrained_models",
        )

For dataset evaluation, configure the audio and label paths in config_cat.json and eval.sh, and set --model_path to the downloaded checkpoint directory under model/. Select the intended dataset split and noise condition. Align the WavLM checkpoint path in eval.py with pretrained_models/WavLM-Large.pt and use the WavLM implementation supplied by the repository. See eval.py for component loading and the SER forward pass, and train.py for joint SE/SER processing.

Waveform normalisation

Each .pkl contains a (mean, std) tuple:

Statistics file	Mean	Standard deviation
train_norm_stat.pkl	-0.00016752422864340985	0.09842836134288799
clean_train_norm_stat.pkl	-0.00008088798101574731	0.08598362556210874

The repository computes sample-weighted global statistics across the loaded training waveforms, before duration truncation or batch padding:

N = total number of waveform samples
mean = sum(x) / N
std = sqrt(sum(x²) / N - mean²)
x_normalised = (x - mean) / (std + 1e-8)

Use train_norm_stat.pkl for the evaluation waveform input, as in eval.py. Retain these training statistics when evaluating other conditions; do not recompute them on the test set. The repository's waveform dataset uses a default maximum duration of 12 seconds.

Training and evaluation in the paper

The experiments use MSP-Podcast v1.11 with four emotion categories: anger, sadness, happiness, and neutral. The paper describes excluding background music and overlapping speech and retaining source recordings with predicted SNR above 20 dB. The Train partition is used for training, Development for model selection and early stopping, and Test1 for evaluation.

Training and development speech is mixed with CRSS-4ENGLISH-14 babble noise at 5 dB SNR. Evaluation covers −5, 0, 5, and 10 dB SNR, using CRSS noise and unseen Freesound and ICASSP 2023 DNS Challenge noise. Overlapping DNS/Freesound segments are removed, and room impulse responses are excluded.

The paper uses two training phases:

1. Freeze WavLM and train the SE and SER heads independently with AdamW at 5e-5: 130 epochs with batch size 16 for SE, and 20 epochs with batch size 32 for SER.
2. Jointly fine-tune for 20 epochs with batch size 32, using 5e-5 for experts, routers, and task heads, and 2.5e-5 for WavLM Transformer layers. The convolutional feature extractor remains frozen.

The joint objective combines class-weighted cross-entropy for SER and L1 spectral reconstruction loss for SE.

SER is evaluated with macro- and micro-F1 across four training seeds and five non-overlapping Test1 subsets per condition. SE is evaluated using a fixed training seed with PESQ, CSIG, CBAK, COVL, segmental SNR, and STOI. Full condition-specific results and statistical comparisons appear in Tables II and III of the paper.

The paper reports improved SER robustness under low-SNR, unseen-noise conditions and competitive enhancement performance. Its aggregate SER results cover multiple training runs; this Hub repository currently provides the final checkpoint set in the 7/ directory.

Scope and data access

The model is intended for research on noisy-speech emotion classification, speech enhancement, and multi-task representation learning. Its evaluation covers English podcast speech and the four listed emotion classes. Performance on other languages, recording domains, emotion taxonomies, or noise distributions requires separate evaluation. Emotion predictions reflect learned annotation categories rather than direct measurements of a speaker's internal state.

MSP-Podcast audio and noise datasets are not distributed here. Obtain MSP-Podcast through its official access process and follow the terms of each dataset. Results using a different corpus release or noise construction should be identified with that experimental configuration.

The source code is distributed under the GitHub repository's MIT license. Refer to the original providers for terms applicable to third-party pretrained resources and datasets.

Citation

@article{tzeng_2026_taslp,
  author  = {Tzeng, Jing-Tong and Busso, Carlos and Lee, Chi-Chun},
  title   = {Joint Learning Using Mixture-of-Expert-Based Representation for Speech Enhancement and Robust Emotion Recognition},
  journal = {IEEE Transactions on Audio, Speech and Language Processing},
  year    = {2026},
  volume  = {34},
  pages   = {3026--3038},
  doi     = {10.1109/TASLPRO.2026.3688928}
}

Acknowledgements

The framework builds on WavLM and adapts speech enhancement components from BSSE-SE. We thank the authors for sharing their implementations and pretrained resources.