Evaluation of Consensus on the Annotation of Prosodic Breaks in the Romance Corpus of Spontaneous Speech “C-ORAL-ROM”

C-ORAL-ROM, Integrated Reference Corpora For Spoken Romance Languages, is a multilingual corpus of spontaneous speech delivered within the IST Program. Corpora are tagged with respect to terminal and non terminal prosodic breaks. Terminal breaks are considered the most perceptively relevant cues to determine the utterance boundaries in spontaneous speech resources. The paper presents the evaluation of the inter-annotator agreement accomplished by an institution external to the consortium and shows the level of reliability of the tagging delivered and the annotation scheme adopted. The data show, at cross-linguistic level, a very high K coefficient (between 7.7 and 9.2, according to the language resource). A strong level of agreement specifically for terminal breaks has also been recorded. The data thus show that the annotation of the utterances identified in terms of their prosodic breaks is able to capture relevant perceptual facts, and it appears that the proposed coding scheme can be applied in a highly replicable way. 1. C-ORAL-ROM Prosodic Tagging The C-ORAL-ROM Project (IST2000-26228) provides four multimedia corpora of spontaneous speech for French, Italian, Portuguese and Spanish, (Cresti and Moneglia, forthcoming). They consist of different speech materials, sampled following an explicit set of parameters (formal versus informal, dialogue versus monologue; media versus natural context), applied in such a way that the proportional representation of the different speech styles is equal in the four languages. In C-ORAL-ROM the textual string is presented as short consecutive chunks of words in orthographic transcription, which are separated by prosodically motivated tags: these are based on the occurrence of terminal and non terminal breaks in the speech waveform. The presence of a terminal break is considered the main cue for the detection of utterances (Austin, 1962); that is, the linguistically relevant information units in the domain of spontaneous speech (Biber et al., 1999; Cresti, 2000). The motivation for this is that terminal breaks are assumed to mark the utterance limit (Karcevsky, 1931; Crystal, 1975). The rough equivalence between utterance units and sequences separated by terminal breaks is based on the idea that competent speakers are extremely sensible to intentional prosodic variation (’t Hart et al., 1990) and that the voluntary accomplishment of a speech act is always accompanied by such variations. The notion of Prosodic break is specified as follows: Concept: Prosodic break Definition: Perceptively relevant prosodic variation in the speech continuum such as to cause the parsing of the continuum into discrete prosodic units. Concept: Terminal prosodic breaks Definition: Given one or more prosodic units, a prosodic break is said terminal if a competent speaker assigns to it the quality of concluding the sequence. Concept: Non-terminal prosodic breaks Definition: Given a sequence of one or more prosodic units, a prosodic break is said non-terminal if a competent speaker assigns to it the quality of being non conclusive. In C-ORAL-ROM each word boundary (W) is considered a possible position for a break (within-word breaks are not considered) and necessarily has one of the following values: 1) no break (O); 2) terminal break (T); non-terminal (N). Tagging is based only on perceptual judgments and does not require any specific linguistic knowledge, although the notion of speech act is always familiar to the expert transcribers (PhD and PhD students) who annotated the corpus according to the following procedure: 1) Tagging of prosodic breaks simultaneous to the transcription by a first labeler; 2) Revision of tagging by a different labeler in connection to the revision of transcripts; 3) Revision of tagging and specific check of terminal breaks by a third labeler during the alignment. This process already ensures control on the interannotator relevance of tags and a maximum accuracy in the detection of terminal breaks. The accuracy with respect to non-terminal breaks is by definition lower. The current paper will report on additional evaluations of the prosodic tags by another group of independent annotators. 1.1. Evaluation background In the recent literature the evaluation of inter-annotator agreement with respect to various kinds of prosodic boundaries, mainly regards ToBI annotation (by trained labelers) of mostly non-spontaneous speech resources (Pitrelli, Beckman, Hirschberg, 1994; Grice et al., 1996; Syrdal & Mc Gorg 2000). The prosodic annotation in the Dutch corpus of spontaneous speech has also been recently verified by non-experts, who had received a minimal amount of training (Buhmann et al., 2002). Roughly speaking such literature testifies a high degree of agreement on “boundary tones” (from 85 to 92%; for