Question: How do they perform the tests?
Evidence: In this first exploratory work, only documents in French were considered, but the system can be adapted to other languages. The evaluation is based on the correspondence of word pairs representing a border. In this way we compare the Annodis segmentation with the automatically produced segmentation. For each pair of reference segments, a $L_r$ list of word pairs is provided: the last word of the first segment and the first word of the second.
For example, considering the reference text wik1_01_02-04-2006.seg, from Annodis corpus:
[Le Ban Amendment]$_1$ [Après avoir adopté la Convention,]_2 [un certain nombre de PED et d'associations de défense de l'environnement soutinrent]_3 [que le document n'allait pas assez loin.]_4 [De nombreux pays et ONG militèrent]_5 [en faveur d'une interdiction totale de l'expédition de déchets dangereux à destinations des PED.]_6 [Plus exactement,]_7 [la Convention originale n'interdisait pas l'exportation de déchets,]_8 [excepté vers l'Antarctique.]_9 [Elle n'exigeait]_10 [qu'une procédure de consentement préalable en connaissance de cause]_11 [(PIC, Prior Informed Consent).]_12
Here are the word pairs of the created reference list (punctuation marks are disregarded):
$L_r$={[Convention – un], [soutinrent – que], [loin – de], [militèrent – en], [exactement – la], [PED – plus], [exactement – la], [déchets – excepté], [Antartique – Elle], [exigeait – qu'une], [cause – PIC] }
We decided to count the word pairs instead of the segments, as this is a first version of the evaluation protocol. In fact, the segments may be nested, which complicates the evaluation process. Although there are some errors, word boundaries allow us to detect segments more easily.
We have built a second $L_c$ list for the automatically identified segments, following the same criteria of $L_r$. The $L_r$ and $L_c$ lists regroup, pair by pair, the segment border. We then count the common pair intersection of the two lists. Each pair in the $L_c$ list is also present in the $L_r$ reference list and is a correctly assigned to the class pair. A word pair belonging to the $L_c$ list but not belonging to the $L_r$ reference list, will be a pair assigned to the class.
For that same text, the $L_c$ list of candidate pairs obtained with the Segmentator$_{\mu }$ is:
$L_r$={[loin–De], [pays–et], [militèrent–en], [dangereux–à], [PED–Plus], [Antarctique–Elle], [préalable–en], [cause–PIC] }
We calculate the precision $P$, the recall $R$ and the $F$-score on the text corpus used in our tests, as follow:
The precision, the recall and the $F$-score for this example is: $P$ = 5 / 11 = 0.45; $R$ = 5 / 8 = 0.625; F-score = 2 $\times \frac{ 0.45 \times 0.625}{ 0.45 + 0.625} = 0.523$. We used the documents in the Annodis corpus without segmentation, because they had been segmented with the Segmenter$_{\mu }$ and with the grammar segmenters.
Two batch of tests were performed. The first on the $D$ set of documents common to the two subcorpus “specialist” $E$ and “naive” $N$ from Annodis. $D$ contains 38 documents with 13 364 words. This first test allowed to measure the distance between the human markers. In fact, in order to get an idea of the quality of the human segmentations, the cuts in the texts made by the specialists were measured it versus the so-called “naifs” note takers and vice versa. The second series of tests consisted of using all the documents of the subcorpus “specialist” $E$, because the documents of the subcorpus of Annodis are not identical. Then we benchmarked the performance of the three systems automatically.
Two batch of tests were performed. The first on the D set of documents common to the two subcorpus “specialist” E and “naive” N from ANNODIS. D contains 38 documents with 13 364 words. This first test allowed to measure the distance between the human markers. In fact, in order to get an idea of the quality of the human segmentations, the cuts in the texts made by the specialists were measured it versus the so-called “naifs” note takers and vice versa. The second series of tests consisted of using all the documents of the subcorpus “specialist” E, because the documents of the subcorpus of ANNODIS are not identical. Then we benchmarked the performance of the three systems automatically.
Answer: They perform two batches of tests. The first on the D set of documents common to the two subcorpus “specialist” E and “naive” N from ANNODIS. The second consisted of using all the documents of the subcorpus “specialist” E. Then they benchmarked the performance of the three systems automatically.