OpenMOSS-Team

/

MOSS-Transcribe-preview-2B

like 37

Follow

OpenMOSS 1.02k

I noticed a suspicious transcript-matching pattern in the MOSS-Transcribe-preview-2B evaluation results.

In one evaluation sample, the model prediction reproduces the same unusual capitalization and abbreviation formatting as the reference transcript.

Reference:

And uh because, obviously, all T. V. s use this, the same infrared…

Prediction:

And uh because obviously all T. V. s use this same infrared…

Another part of the same sample contains:

Reference:

… an L. C. D. screen in the remote control …

Prediction:

… an L. C. D. screen in the remote control …

The suspicious part is not the recognition of the words themselves, but the exact reproduction of the reference transcript’s annotation conventions:

• Acronym letters are represented as separated uppercase letters (T. V. and L. C. D.).
• The periods and spaces inside the abbreviations are reproduced exactly.
• The plural suffix in T. V. s remains lowercase.
• The same unusual abbreviation, capitalization, spacing, and normalization conventions are preserved.

The audio may indicate that the speaker said “TVs” and “LCD,” but it does not determine whether these expressions should be written as TVs, T.V.s, T. V. s, LCD, L.C.D., or L. C. D.. In
particular, the lowercase plural suffix in T. V. s, as well as the exact placement of periods and spaces, reflects transcript annotation choices rather than acoustic information.

These formatting artifacts are therefore difficult to explain through acoustic recognition alone. Their exact reproduction may indicate exposure to the same transcript examples or the use of the
same annotation or text-normalization pipeline.

This raises concerns about possible overlap between the training and evaluation data.

In addition, on the Hugging Face Open ASR Leaderboard (https://huggingface.co/spaces/hf-audio/open_asr_leaderboard), MOSS-Transcribe-preview-2B performs substantially worse on the private-data evaluation, ranking outside the top 40, while achieving much stronger results on public test sets. Although this discrepancy alone does not prove data contamination, the unusually large gap between public and private evaluations—combined with the exact reproduction of reference-specific transcript formatting—raises further concerns about whether the model may have been exposed or overfitted to the public evaluation data.

Could you please clarify:

1. Was this evaluation set completely excluded from all training stages?
2. Was deduplication performed between the training and evaluation datasets?
3. Did the deduplication procedure compare audio, transcript text, or both?
4. Are these evaluation transcripts from a source corpus that was also used for training?
5. Are the displayed predictions raw model outputs, or were they processed by a text-normalization or punctuation-restoration pipeline?
6. What procedures were used to prevent evaluation-data contamination?

If evaluation samples, their transcripts, or duplicate versions of the same recordings were included in the training data, the reported benchmark results may not accurately represent the
generalization ability of MOSS-Transcribe-preview-2B.

Please investigate this sample and clarify the dataset construction, deduplication, and contamination-prevention procedures.