# A corpus of spontaneous L2 English speech for real-situation speaking assessment

Sylvain Coulange, Marie-Hélène Fries, Monica Masperi, Solange Rossato

► To cite this version:

Sylvain Coulange, Marie-Hélène Fries, Monica Masperi, Solange Rossato. A corpus of spontaneous L2 English speech for real-situation speaking assessment. the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), May 2024, Torino, Italy., A corpus of spontaneous L2 English speech for real-situation speaking assessment, pp.293-297, 2024. [hal-04595945](https://hal.science/hal-04595945)

HAL Id: hal-04595945

https://hal.science/hal-04595945

Submitted on 31 May 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_1.png" /></figure>

HAL
open science

# A corpus of spontaneous L2 English speech for real-situation speaking assessment

Sylvain Coulange, Marie-Hélène Fries, Monica Masperi, Solange Rossato

<figure><img src="image_2.png" /></figure>

## ► To cite this version:

Sylvain Coulange, Marie-Hélène Fries, Monica Masperi, Solange Rossato. A corpus of spontaneous L2 English speech for real-situation speaking assessment. the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), May 2024, Torino, Italy., A corpus of spontaneous L2 English speech for real-situation speaking assessment, pp.293-297, 2024. hal-04595945

HAL Id: hal-04595945

https://hal.science/hal-04595945v1

Submitted on 31 May 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

<figure><img src="image_3.png" /></figure>

# A corpus of spontaneous L2 English speech for real-situation speaking assessment

Sylvain COULANGE¹,², Marie-Hélène FRIES³, Monica MASPERI¹, Solange ROSSATO²

1. *Univ. Grenoble Alpes*, Laboratory of Linguistics and
   Didactics of Foreign and Mother Tongues (LIDILEM)
   38000 Grenoble, France

2. *Univ. Grenoble Alpes*, CNRS, Institute of Engineering,
   Grenoble Computer Science Laboratory (LIG)
   38000 Grenoble, France

3. National Coordination for the Certificate of language
   skills in French higher education (CLES)

{ sylvain.coulange, monica.masperi, solange.rossato}@univ-grenoble-alpes.fr, coordination-nationale@certification-cles.fr

## Context:

* Computer Assisted Pronunciation Training tasks rarely deal with **spontaneous speech**

* Lack of L2 **spontaneous** speech corpus.

* Lack of speech in **peer dialogue** situations.

## Creation of a speech corpus:

* We started gathering **L2 spontaneous speech** data recorded in exam situations.

* Our first aim is to **train score prediction models** based on near-real-situation L2 speech, but this corpus can also serve other purposes in L2 acquisition, teaching, testing, or L2 speech processing.

## Automated file processing:

* We made a dedicated **speech processing pipeline** to annotate this challenging type of speech [2].

* In this study, we focused on **speech rhythm measurement** through syllabic prominence of polysyllabic words

## Corpus

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

* ✓ The CLES is a state certificate established by the French Ministry of Higher Education and Research, designed for university-level language proficiency assessment. [1]

* ✓ The collection of spontaneous L2 speech recordings comprises 2 types of role-play:

— CLES B2 —
Argumentative discussions
(2 or 3 candidates)
Mean dur.: 9'35''

— CLES B1 —
Vocal messages
(monologues)
Mean dur.: 3'20''

* ✓ Each recording is provided with high-quality certification-level proficiency ratings.

<table><thead><tr><th>Type</th><th>Nb. of speakers</th><th>Duration</th></tr></thead><tbody><tr><td>3-speaker</td><td>15</td><td>1h03'44''</td></tr><tr><td>2-speaker</td><td>232</td><td>18h16'50''</td></tr><tr><td>1-speaker</td><td>13</td><td>39'28''</td></tr><tr><td>Total</td><td>260</td><td>20h00'02''</td></tr></tbody></table>

* ✓ Each candidate assumed a specific given role, either advocating for or against the subject.

* ✓ The objective is to negotiate and work towards a compromise.

<table><thead><tr><th>Proficiency</th><th>Nb. of speakers</th><th>Proportion</th></tr></thead><tbody><tr><td>B2</td><td>151</td><td>58 %</td></tr><tr><td>B1</td><td>75</td><td>29 %</td></tr><tr><td>Non-validated</td><td>34</td><td>13 %</td></tr><tr><td>Total</td><td>260</td><td></td></tr></tbody></table>

Note: a similar corpus is being made involving native speakers and Japanese-L1 speakers of English [3].

## Processing Pipeline

<figure><img src="image_6.png" /></figure>

Neural speaker diarization
(Pyannote)

<figure><img src="image_7.png" /></figure>

Automatic speech recognition
And Word-level forced alignment
(WhisperX, Wav2Vec2.0)

<figure><img src="image_8.png" /></figure>

Syllable nuclei detection [4]

<figure><img src="image_9.png" /></figure>

Syntactic analysis
(Spacy, Benepar)

<figure><img src="image_10.png" /></figure>

Lexical stress analysis

<figure><img src="image_11.png" /></figure>

Pause position analysis

<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /></figure>

<figure><img src="image_14.png" /></figure>

Speaker diarization
output in TextGrid format

Lexical stress analysis ►
annotation example

Only polysyllabic words with adequat number of syllable nuclei detected are annotated, in order to filter bad word alignments.

<figure><img src="image_15.png" /></figure>

<figure><img src="image_16.png" /></figure>

## Lexical stress analysis:

* Lexical stress is estimated from prosodic measures on syllable nuclei, based on F0, intensity and syllable duration.

* Each prosodic measure is converted in speaker percentile, (50 means median prominence level for any speaker, 0 is minimum and 100 is maximum)

* Comparison with the prescripted stress pattern (CMU dictionary).

<figure><img src="image_17.png" /></figure>

## Pipeline Evaluation & Limitations:

* As the pipeline combines several modules, errors can occur at different levels, often leading to incorrect annotations.

* ▲ Syllable detection and word alignment often mismatches, leading to a limited nb. of target words (only **41%** of polysyllabic words in the study below were **target words**).

* ▲ Manual evaluation of random 100 target words showed that **17%** were miss-recognized or miss-aligned, potentially leading to wrong judgments that can be problematic in a real assessment context.

* ▲ Intrinsic vowel length and word ending lengthening need to be considered in order to improve stress estimation.

* ▲ Some cases of **vowel devoicing** also impacted F0 measures (tackled with linear interpolation for now)

## Lexical stress position accuracy and degree of prosodic contrast

### Sub-corpus:

* French-L1 speakers with either B1 or B2 speaking proficiency (n=176, 11 hours)

* Speaking B1 level: 34%, B2 level: 66%

* 6350 target words.

### Hypothesis:

* Position accuracy B2>B1.

* Shift to last syllable.

* Stress mainly by duration change.

* F0 and intensity used mainly by high proficiency speakers.

### Main observations:

* Mean stress position accuracy varies greatly among speakers (0–68.4%, mean: 35.4%).

* B2 speakers perform better than B1 in terms of stress position accuracy (36% vs. 29.6%) and prosodic contrast.

* Syllabic prominence is often detected on the last syllable of words, which might be caused by L1 influence.

* The better the speaker mean stress position accuracy, the higher pitch and intensity of expected stressed syllable.

* Duration parameter is the most discriminant.

<caption>Proportion of target words with
expected stress position per speaker</caption>

<figure><img src="image_18.png" /><figcaption>Number of observed stress patterns for each expected pattern</figcaption></figure>

<figure><img src="image_19.png" /><figcaption>Prosodic contrast</figcaption></figure>

<figure><img src="image_20.png" /><figcaption>Prosodic contrast per speaker</figcaption></figure>

<figure><img src="image_21.png" /><figcaption>SpeakerA</figcaption></figure>

<figure><img src="image_22.png" /><figcaption>SpeakerB</figcaption></figure>

<figure><img src="image_23.png" /></figure>

### References:

[1] CLES official website: https://www.certification-cles.fr/english/

[2] Coulange S, Kato T, Rossato S, Masperi M. (2024). Enhancing Language Learners' Comprehensibility through Automated Analysis of Pause Positions and Syllable Prominence. Languages 9(3):78

[3] Coulange S, Konishi T, Kato T, Sugahara M, Rossato R, Masperi M. (2024). A corpus of spontaneous dialogues in L2 English by French and Japanese L1 speakers for automated assessment of fluency. 6th International Symposium on Learner Corpus Studies in Asia and the World (LCSAW6), Feb. 2024, Kobe, Japan.

[4] De Jong, N. H., Pacilly, J., Heeren, W. (2021) "Praat scripts to measure speed fluency and breakdown fluency in speech automatically." Assessment in Education: Principles, Policy & Practice, 28, 456-476.
<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

# A corpus of spontaneous L2 English speech for real-situation speaking assessment

Sylvain COULANGE¹,², Marie-Hélène FRIES³, Monica MASPERI¹, Solange ROSSATO²

1. *Univ. Grenoble Alpes*, Laboratory of Linguistics and
   Didactics of Foreign and Mother Tongues (LIDILEM)
   38000 Grenoble, France

2. *Univ. Grenoble Alpes*, CNRS, Institute of Engineering,
   Grenoble Computer Science Laboratory (LIG)
   38000 Grenoble, France

3. National Coordination for the Certificate of language
   skills in French higher education (CLES)

{ sylvain.coulange, monica.masperi, solange.rossato}@univ-grenoble-alpes.fr, coordination-nationale@certification-cles.fr

## Context:

* Computer Assisted Pronunciation Training tasks rarely deal with **spontaneous speech**

* Lack of L2 **spontaneous** speech corpus.

* Lack of speech in **peer dialogue** situations.

## Creation of a speech corpus:

* We started gathering L2 **spontaneous speech** data recorded in exam situations.

* Our first aim is to **train score prediction models** based on near-real-situation L2 speech, but this corpus can also serve other purposes in L2 acquisition, teaching, testing, or L2 speech processing.

## Automated file processing:

* We made a dedicated **speech processing pipeline** to annotate this challenging type of speech [2].

* In this study, we focused on **speech rhythm measurement** through syllabic prominence of polysyllabic words

## Corpus

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /></figure>

* ✓ The CLES is a state certificate established by the French Ministry of Higher Education and Research, designed for university-level language proficiency assessment. [1]

* ✓ The collection of spontaneous L2 speech recordings comprises 2 types of role-play:

— CLES B2 —
Argumentative discussions
(2 or 3 candidates)
Mean dur.: 9'35''

— CLES B1 —
Vocal messages
(monologues)
Mean dur.: 3'20''

* ✓ Each recording is provided with high-quality certification-level proficiency ratings.

<table><thead><tr><th>Type</th><th>Nb. of speakers</th><th>Duration</th></tr></thead><tbody><tr><td>3-speaker</td><td>15</td><td>1h03'44''</td></tr><tr><td>2-speaker</td><td>232</td><td>18h16'50''</td></tr><tr><td>1-speaker</td><td>13</td><td>39'28''</td></tr><tr><td>Total</td><td>260</td><td>20h00'02''</td></tr></tbody></table>

* ✓ Each candidate assumed a specific given role, either advocating for or against the subject.

* ✓ The objective is to negotiate and work towards a compromise.

<table><thead><tr><th>Proficiency</th><th>Nb. of speakers</th><th>Proportion</th></tr></thead><tbody><tr><td>B2</td><td>151</td><td>58 %</td></tr><tr><td>B1</td><td>75</td><td>29 %</td></tr><tr><td>Non-validated</td><td>34</td><td>13 %</td></tr><tr><td>Total</td><td>260</td><td></td></tr></tbody></table>

Note: a similar corpus is being made involving native speakers and Japanese-L1 speakers of English [3].

## Processing Pipeline

<figure><img src="image_8.png" /></figure>

Neural speaker diarization
(Pyannote)

<figure><img src="image_9.png" /></figure>

Automatic speech recognition
And Word-level forced alignment
(WhisperX, Wav2Vec2.0)

<figure><img src="image_10.png" /></figure>

Syllable nuclei detection [4]

<figure><img src="image_11.png" /></figure>

Syntactic analysis
(Spacy, Benepar)

<figure><img src="image_12.png" /></figure>

Lexical stress analysis

<figure><img src="image_13.png" /></figure>

Pause position analysis

<figure><img src="image_14.png" /></figure>

<figure><img src="image_15.png" /></figure>

<figure><img src="image_16.png" /></figure>

Speaker diarization
output in TextGrid format

Lexical stress analysis ▶
annotation example

Only polysyllabic words with adequat number of syllable nuclei detected are annotated, in order to filter bad word alignments.

<figure><img src="image_17.png" /></figure>

<figure><img src="image_18.png" /></figure>

## Lexical stress analysis:

* Lexical stress is estimated from prosodic measures on syllable nuclei, based on F0, intensity and syllable duration.

* Each prosodic measure is converted in speaker percentile, (50 means median prominence level for any speaker, 0 is minimum and 100 is maximum)

* Comparison with the prescripted stress pattern (CMU dictionary).

<figure><img src="image_19.png" /></figure>

## Pipeline Evaluation & Limitations:

* As the pipeline combines several modules, errors can occur at different levels, often leading to incorrect annotations.

▲ Syllable detection and word alignment often mismatches, leading to a limited nb. of target words (only **41%** of polysyllabic words in the study below were **target words**).

▲ Manual evaluation of random 100 target words showed that **17%** were miss-recognized or miss-aligned, potentially leading to wrong judgments that can be problematic in a real assessment context.

▲ Intrinsic vowel length and
word ending lengthening
need to be considered in order
to improve stress estimation.

▲ Some cases of **vowel devoicing** also impacted F0
measures (tackled with linear
interpolation for now)

## Lexical stress position accuracy and degree of prosodic contrast

### Sub-corpus:

* French-L1 speakers with either B1 or B2 speaking proficiency (n=176, 11 hours)

* Speaking B1 level: 34%, B2 level: 66%

* 6350 target words.

### Hypothesis:

* Position accuracy B2>B1.

* Shift to last syllable.

* Stress mainly by duration change.

* F0 and intensity used mainly by high proficiency speakers.

### Main observations:

* Mean stress position accuracy varies greatly among speakers (0–68.4%, mean: 35.4%).

* B2 speakers perform better than B1 in terms of stress position accuracy (36% vs. 29.6%) and prosodic contrast.

* Syllabic prominence is often detected on the last syllable of words, which might be caused by L1 influence.

* The better the speaker mean stress position accuracy, the higher pitch and intensity of expected stressed syllable.

* Duration parameter is the most discriminant.

<caption>Proportion of target words with
expected stress position per speaker</caption>

<figure><img src="image_20.png" /><figcaption>Number of observed stress patterns for each expected pattern</figcaption></figure>

<figure><img src="image_21.png" /><figcaption>Prosodic contrast</figcaption></figure>

<figure><img src="image_22.png" /><figcaption>Prosodic contrast per speaker</figcaption></figure>

<figure><img src="image_23.png" /><figcaption>SpeakerA</figcaption></figure>

<figure><img src="image_24.png" /><figcaption>SpeakerB</figcaption></figure>

<figure><img src="image_25.png" /></figure>

### References:

[1] CLES official website: https://www.certification-cles.fr/english/

[2] Coulange S, Kato T, Rossato S, Masperi M. (2024). Enhancing Language Learners' Comprehensibility through Automated Analysis of Pause Positions and Syllable Prominence. Languages 9(3):78

[3] Coulange, S., Konishi, T., Kato, T., Sugahara, M., Rossato, R., Masperi, M. (2024). A corpus of spontaneous dialogues in L2 English by French and Japanese L1 speakers for automated assessment of fluency. 6th International Symposium on Learner Corpus Studies in Asia and the World (LCSAW6), Feb. 2024, Kobe, Japan.

[4] De Jong, N. H., Pacilly, J., Heeren, W. (2021) "Praat scripts to measure speed fluency and breakdown fluency in speech automatically." Assessment in Education: Principles, Policy & Practice, 28, 456-476.