🎭 TACME

Task-Aware Cross-Modal Experts for Disjoint-Corpus Multimodal Behavioral Understanding

One shared fusion model · Four feature streams · Three behavioral tasks · Disjoint-corpus supervision

Try the Space · Read the paper · Browse the code · Download the checkpoint

---

At a glance

Research setting	Released artifact	Prediction targets	Evaluation
Disjoint-corpus multimodal MTL	3.99M-parameter fusion model (approximately 16.0 MB Safetensors file)	7 emotions · 5 apparent personality scores · 2 ambivalence classes	CMU-MOSEI · FIv2 · BAH · zero-shot MELD

Release scope. This repository contains the trained TACME fusion checkpoint, not a standalone raw-video foundation model. To analyze a video or audio recording, use the TACME application and its upstream feature extractors. Those pretrained backbone weights are not bundled in tacme.safetensors.

Why TACME?

Emotion recognition, apparent personality assessment, and ambivalence recognition are usually trained on different datasets with different annotations. TACME addresses this fragmented-supervision problem by sharing cross-modal interactions while learning task-specific expert mixtures.

The approach combines:

1. Directed cross-modal Transformer experts, which model ordered interactions between modality pairs.
2. Task-aware expert gating, which selects different mixtures for emotion, personality, and ambivalence.
3. Two-token statistical aggregation, retaining the mean and standard deviation of each feature sequence.
4. Feature-level and prediction-level flow matching (FM), investigated as training regularization.
5. Progress-Aware Gradient Balancing (PAGB), a multi-task training strategy informed by validation progress and gradient dynamics.

Important: FM and PAGB are part of the training methodology; they are not extra labels predicted by the published checkpoint.

Model architecture and inputs

The released MultiModalFusionModel_v2 consumes four precomputed feature streams:

Fusion input key	Source in the end-to-end application	Feature size
scene	CLIP vision encoder	512
audio	Emotion-oriented Wav2Vec2	1,024
text	Whisper transcription → emotion-oriented RoBERTa	768
behavior	Qwen3-VL-4B-Instruct video description → emotion-oriented RoBERTa	768

                          VIDEO + AUDIO
                                │
           ┌────────────────────┼─────────────────────┐
           │                    │                     │
     Video frames          Speech audio           Video frames
           │                    │                     │
          CLIP             ┌─────┴──────┐           Qwen3-VL
           │            Wav2Vec2      Whisper           │
           │               │             │          RoBERTa
           │               │          RoBERTa           │
           ▼               ▼             ▼              ▼
         Scene            Audio         Text         Behavior
         512 D           1024 D        768 D          768 D
           └───────────────┴─────────────┴──────────────┘
                           │
                  Mean + STD (2 tokens)
                           │
                  Modality encoders
                           │
                Shared directed experts
                           │
                  Task-specific gates
                     ┌─────┼─────┐
                     ▼     ▼     ▼
                  Emotion Personality Ambivalence

Fusion configuration: 128-dimensional hidden representations, 8 attention heads, 5 Transformer layers, 12 ordered pairwise modality interactions (4 × 3), and task-specific expert gates. The deployed model expects a batch structured as features and optional feature_masks mappings. Aggregated modality tensors have shape [batch, 2, dimension]; masks have shape [batch, 2] and boolean dtype. Consult the reference smoke test for the exact input contract.

Prediction heads

Task	Output tensor	Classes / scores
Emotion recognition	emotion_logits — [B, 7]	Neutral, anger, disgust, fear, happiness, sadness, surprise
Apparent personality	personality_scores — [B, 5]	Openness, conscientiousness, extraversion, agreeableness, emotional stability (NNEU / non-neuroticism)
Ambivalence	ah_logits — [B, 2]	Absence, presence

The personality values are regression scores, not class probabilities or validated psychometric measures. Emotion and ambivalence are model predictions, not independently verified psychological states.

---

Experimental results

All numbers below are from Ryumina et al., Findings of EMNLP 2026. They report the paper's evaluation experiments, not a newly executed benchmark of this uploaded file. The study uses the original dataset splits and a common feature-extraction protocol for controlled comparisons.

1. Comparison with multimodal fusion models

Paper Table 1. The table shows the best-ranked reported variant of each compared architecture within the paper's disjoint-corpus multi-task evaluation. Larger task scores are better; smaller average rank is better. Different models may use different aggregation or regularization variants, as specified in the paper.

Fusion model	CMU-MOSEI mMF1 ↑	CMU-MOSEI mWAcc ↑	FIv2 mACC ↑	FIv2 CCC ↑	BAH MF1 ↑	BAH UAR ↑	Avg. rank ↓
UniGraph2	61.31	61.61	91.25	66.54	68.70	69.36	9.50
DPDF-LQ	62.38	62.73	91.04	62.77	69.44	70.03	5.33
HME	62.28	62.30	91.62	64.89	68.16	67.99	8.17
ComP	60.71	61.93	90.49	51.95	69.38	68.87	11.50
TACME (task-aware + FM + PAGB)	62.68	63.29	91.71	69.89	71.15	72.07	1.00

Interpretation. TACME achieves the lowest average rank in the controlled fusion-model comparison across six task-specific metrics. This is a within-study comparison, not a claim that one checkpoint sets the best score on every external benchmark or every metric. See Table 1 in the paper.

2. Single-task performance versus multi-task generalization

Paper Table 4, mean ± standard deviation over five random seeds:

Training setting	MOSEI mMF1 ↑	MOSEI mWAcc ↑	FIv2 mACC ↑	FIv2 CCC ↑	BAH MF1 ↑	BAH UAR ↑
Single-task TACME	67.82 ± 0.37	68.17 ± 0.42	92.59 ± 0.18	77.47 ± 1.17	78.05 ± 0.62	78.63 ± 0.78
Multi-task TACME	62.54 ± 0.26	63.32 ± 0.24	91.76 ± 0.05	70.14 ± 0.78	70.29 ± 0.90	71.71 ± 0.53

Trade-off: separately trained single-task TACME models have higher scores on all six source-corpus metrics; joint training is more valuable for the cross-task / out-of-domain generalization question investigated in the paper. These single-task rows refer to separately trained experimental models, not additional checkpoints included in this repository. See Table 4.

3. Zero-shot transfer to unseen MELD

Paper Table 6. MELD was not used for training or model selection; the evaluation includes 2,610 test utterances over seven emotions.

Model	UAR ↑
Single-task TACME	28.9%
Multi-task TACME	35.9%

+7.0 percentage points UAR for the multi-task model on this unseen test set. The gain is not uniform across emotions, as the class-wise results below show.

View class-wise zero-shot recall on MELD (paper Table 6)

Model	Neutral	Anger	Disgust	Fear	Happiness	Sadness	Surprise	UAR
Single-task	33.1	6.4	16.2	0.0	91.3	24.5	31.0	28.9
Multi-task	36.1	7.5	50.0	6.0	65.7	36.5	49.8	35.9

All figures are recall (%). Multi-task training improves most classes but reduces happiness recall from 91.3% to 65.7%. See Table 6.

4. What do the ablations indicate?

The paper also investigates modality contributions (Table 3) and multi-task optimization (Table 2):

• Removing the speech transcript degrades BAH UAR from 72.07 to 66.58 in the modality ablation.
• Removing the visual signal degrades FIv2 CCC from 69.89 to 52.17.
• PAGB achieves an average rank of 2.33 among tested MTL strategies, compared with 2.67 for CAGrad under that separate experiment.

These are ablations and controlled strategy comparisons; their rank values should not be conflated with Table 1's overall ranking. Full results, uncertainty, and experimental settings appear in the paper.

---

Installation and verification

Option A — Validate the published fusion checkpoint (CPU)

This is the fastest reproducible check of the actual release. It does not download CLIP, Wav2Vec2, Whisper, RoBERTa or Qwen3-VL.

Requirements: uv and Python 3.14.7, plus a platform with compatible PyTorch wheels.

# Clone the official implementation.
git clone https://github.com/DmitryRyumin/TACME.git
cd TACME

# Install the project's locked runtime dependencies.
uv sync --locked --no-dev

# Download the exact published file to the loader's expected location.
uv run --no-dev hf download DmitryRyumin/TACME \
  tacme.safetensors --local-dir models

# Check metadata, architecture, output shapes and deterministic inference.
uv run --no-dev python -m scripts.check_model --device cpu

Expected confirmation from the repository's smoke-test script:

TACME runtime model check passed.
Model: MultiModalFusionModel_v2
Device: cpu
Parameters: 3,990,047
emotion_logits: shape=(2, 7), dtype=torch.float32, finite=True
personality_scores: shape=(2, 5), dtype=torch.float32, finite=True
ah_logits: shape=(2, 2), dtype=torch.float32, finite=True
Repeated inference: stable

These are expected success messages, not a claim that the verification was executed on your specific machine. The script uses deterministic synthetic features, so passing it establishes runtime integrity and output compatibility, not benchmark accuracy.

Option B — Run complete media inference (CPU)

After completing Option A, the Python application can load the four upstream streams and analyze a compatible recording:

# From the root of the cloned TACME repository:
uv run --no-dev python app.py

The interface includes English / Russian localization, media information, prediction results, processing-stage details and feature-attribution visualizations. Opening the application also downloads upstream backbone weights; the full pipeline needs much more RAM and disk space than the 16 MB fusion file. Ensure FFmpeg and the project-required browser components are installed; see the repository and the Dockerfile for runtime dependencies.

Local network note: The current app.py uses Gradio sharing outside Hugging Face Spaces. A local run may create a public sharing link. Review the launch configuration if you intend to keep uploaded recordings private.

For a ready-to-use hosted interface, open TACME on Hugging Face Spaces. The Space uses a CPU Docker deployment. Availability, memory usage, and inference latency depend on its hardware and upstream model downloads.

Option C — Call the Python inference engine

For application-level inference from a local media file, the repository exposes:

from tacme.inference import TacmeInferenceEngine

engine = TacmeInferenceEngine(device="cpu")
result = engine.predict("path/to/recording.mp4")
print(result.emotion)
print(result.personality)
print(result.ambivalence)

Run this after installing the environment and downloading the fusion checkpoint. This initializes the full pretrained feature stack, and the recording must pass TACME's media validation. It is not a direct transformers.pipeline() or AutoModel.from_pretrained() model.

---

Repository files and checkpoint compatibility

File	Description
tacme.safetensors	Trained multi-task fusion-model weights (16,023,444 bytes)
README.md	Model Card, methodology, performance, installation and limitations
CITATION.bib	Paper reference
LICENSE	MIT license for this release

The loader tacme/models/loader.py checks Safetensors metadata, model architecture, tensor count, keys, shapes and dtypes, and loads the checkpoint with strict state-dictionary matching. It expects 496 tensors and 3,990,047 parameters in the released architecture. Preserve the original checkpoint filename and use a compatible revision of the source code.

Reproducibility boundary: the paper's aggregate and per-seed results do not follow automatically from a single checkpoint and a smoke test. Replicating the experiments also requires the exact corpora, splits, feature-extraction settings, training protocol and evaluation procedures detailed in the paper.

Limitations and responsible use

• Not a diagnostic instrument. Do not interpret emotion, apparent personality or ambivalence predictions as psychological diagnoses, clinical findings, or verified personal attributes.
• No high-stakes individual decisions. Not intended for employment screening, educational eligibility, legal assessment or comparable consequential decisions.
• Disjoint labels and corpus confounding. In the study, tasks are coupled to datasets; high corpus separability remains a methodological limitation.
• Domain shift. Results can change with language, demographic composition, culture, recording equipment, background noise and video context. Zero-shot MELD results are not evidence of universal robustness.
• Generated behavior descriptions. Upstream vision-language models can produce interpretive or stereotyped descriptions. Such output should be inspected rather than accepted as an objective behavioral record.
• Temporal detail. Mean/standard-deviation aggregation can discard fine-grained temporal information.
• Compute costs. The fusion checkpoint is compact; the full pipeline includes much larger upstream models and can be slow or memory-intensive on CPU.
• Consent and privacy. Obtain appropriate consent before processing recordings of people, and handle uploaded media and inferences in accordance with applicable data-use requirements.

See the paper's Limitations, Ethical Considerations and subgroup analyses for fuller discussion.

Authors and citation

Elena Ryumina · Alexandr Axyonov · Dmitry Ryumin
Findings of the Association for Computational Linguistics: EMNLP 2026, pp. 20210–20234.

If TACME, this checkpoint or the corresponding code is useful in your research, please cite:

@inproceedings{ryumina-etal-2026-tacme,
    title = "{TACME}: Task-Aware Cross-Modal Experts for Disjoint-Corpus Multimodal Behavioral Understanding",
    author = "Ryumina, Elena and Axyonov, Alexandr and Ryumin, Dmitry",
    editor = "Lee, Hung-Yi and Dev, Sunipa and Li, Jessy and Elliott, Desmond",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {EMNLP} 2026",
    month = oct,
    year = "2026",
    address = "Budapest, Hungary",
    publisher = "Association for Computational Linguistics",
    pages = "20210--20234",
    url = "https://aclanthology.org/2026.findings-emnlp.957/",
    doi = "10.18653/2026.findings-emnlp.957",
    ISBN = "979-8-89176-462-0"
}

License: MIT for the TACME code and released fusion checkpoint, as distributed in this repository. Third-party pretrained models and datasets are not redistributed by this model repository and retain their own licenses and usage terms.