Jesujuwon/yoruba-tone-mark-evaluation

Yoruba Tone-Mark Sensitivity in Qwen3-4B

1. Research motivation: a blind spot in African NLP

My interest in this problem comes from my work on African language NLP and low-resource language evaluation. Standard multilingual benchmarks can report whether a model translates a sentence correctly, but may not reveal how robust its interpretation is to linguistic information that is easily lost during text processing.

Yoruba uses tone marks, and removing them can make written text ambiguous. I wanted to investigate whether a small open-weight language model changes its interpretation when those marks are removed.

Research question: How does removing tone marks from Yoruba input affect the English translations produced by Qwen3-4B?

2. Model and experimental setup

• Model: Qwen3-4B (https://huggingface.co/Qwen/Qwen3-4B)
• Task: Yoruba-to-English translation
• Pilot dataset: 15 Yoruba sentences with speaker-verified English references
• Conditions: Original tone-marked sentences versus versions with tone marks removed
• Outputs: 30 translations, with each sentence evaluated in both conditions
• Scoring: Manual comparison of translations against the English reference meanings

The model received the same translation instruction in both conditions. Decoding was deterministic ("do_sample=False"), with a maximum of 100 new tokens.

3. Pilot results

Metric| Tone marks present| Tone marks removed Correct translations| 2/15| 0/15 Accuracy| 13.3%| 0%

Both correct translations occurred in the tone-marked condition. The exact paired test yielded a two-sided p-value of 0.50.

These results are preliminary.

4. Exploratory corpus experiment

I also evaluated 200 sentence pairs sampled from an external English–Yoruba parallel corpus. In 190 of the 200 pairs, the model produced different outputs when tone marks were removed.

5. Limitations and next steps

The current pilot has three major limitations:

1. Fifteen sentences are insufficient to support broad conclusions.
2. The examples do not all isolate tone as the only difference in meaning.
3. Low accuracy in both conditions makes it difficult to distinguish tone-mark sensitivity from broader Yoruba translation weaknesses.

My next step is to construct a larger benchmark of controlled tone contrasts 6. Research artifacts

This repository contains the pilot inputs, model outputs, audited scores, summary statistics, and exploratory corpus experiment files.

• "yoruba_verified_15.csv" — speaker-verified pilot sentences and references
• "yoruba_verified_15_evaluation.csv" — marked and unmarked model outputs
• "yoruba_verified_15_audit.csv" — paired output audit
• "yoruba_verified_15_scoring_final.csv" — audited correctness scores
• "yoruba_pilot_results_summary_final.json" — pilot metrics
• "yoruba_200_paired_evaluation.csv" — exploratory corpus experiment outputs
• "yoruba_sample_200_test_set_cleaned.csv" — sampled external-corpus inputs

7. Proposed path forward

A practical intervention would combine speaker-validated tone-contrast data, contrastive training, and targeted evaluation across marked and unmarked text. The dataset should preserve the original orthography, document ambiguity, and include independent linguistic review. This could help determine whether models are learning meaningful distinctions rather than relying on incomplete surface patterns.

The broader goal is to make African language model evaluations more representative of the linguistic distinctions that matter to their speakers.