Question: How is the quality of the professional translations in the corpus controlled?
Evidence: Common Voice BIBREF10 is a crowdsourcing speech recognition corpus with an open CC0 license. Contributors record voice clips by reading from a bank of donated sentences. Each voice clip was validated by at least two other users. Most of the sentences are covered by multiple speakers, with potentially different genders, age groups or accents.
Raw CoVo data contains samples that passed validation as well as those that did not. To build CoVoST, we only use the former one and reuse the official train-development-test partition of the validated data. As of January 2020, the latest CoVo 2019-06-12 release includes 29 languages. CoVoST is currently built on that release and covers the following 11 languages: French, German, Dutch, Russian, Spanish, Italian, Turkish, Persian, Swedish, Mongolian and Chinese.
Validated transcripts were sent to professional translators. Note that the translators had access to the transcripts but not the corresponding voice clips since clips would not carry additional information. Since transcripts were duplicated due to multiple speakers, we deduplicated the transcripts before sending them to translators. As a result, different voice clips of the same content (transcript) will have identical translations in CoVoST for train, development and test splits.
In order to control the quality of the professional translations, we applied various sanity checks to the translations BIBREF11. 1) For German-English, French-English and Russian-English translations, we computed sentence-level BLEU BIBREF12 with the NLTK BIBREF13 implementation between the human translations and the automatic translations produced by a state-of-the-art system BIBREF14 (the French-English system was a Transformer big BIBREF15 separately trained on WMT14). We applied this method to these three language pairs only as we are confident about the quality of the corresponding systems. Translations with a score that was too low were manually inspected and sent back to the translators when needed. 2) We manually inspected examples where the source transcript was identical to the translation. 3) We measured the perplexity of the translations using a language model trained on a large amount of clean monolingual data BIBREF14. We manually inspected examples where the translation had a high perplexity and sent them back to translators accordingly. 4) We computed the ratio of English characters in the translations. We manually inspected examples with a low ratio and sent them back to translators accordingly. 5) Finally, we used VizSeq BIBREF16 to calculate similarity scores between transcripts and translations based on LASER cross-lingual sentence embeddings BIBREF17. Samples with low scores were manually inspected and sent back for translation when needed.
We also sanity check the overlaps of train, development and test sets in terms of transcripts and voice clips (via MD5 file hashing), and confirm they are totally disjoint.
In order to control the quality of the professional translations, we applied various sanity checks to the translations (Guzman et al., 2019). 1) For German-English, French-English and Russian-English translations, we computed sentence-level BLEU (Chen and Cherry, 2014) with the NLTK (Bird et al., 2009) implementation between the human translations and the automatic translations produced by a state-of-the-art system (Ng et al., 2019) (the French- English system was a Transformer big (Vaswani et al., 2017) separately trained on WMT14). We applied this method to these three language pairs only as we are confident about the quality of the corresponding systems. Translations with a score that was too low were manually in_x001f_spected and sent back to the translators when needed. 2) We manually inspected examples where the source transcript was identical to the translation. 3) We measured the perplexity of the translations using a language model trained on a large amount of clean monolingual data (Ng et al., 2019). We manually inspected examples where the translation had a high perplexity and sent them back to translators accordingly. 4) We computed the ratio of English characters in the translations. We manually inspected examples with a low ratio and sent them back to translators accordingly. 5) Finally, we used VizSeq (Wang et al., 2019) to calculate similarity scores between transcripts and translations based on LASER cross-lingual sentence embeddings (Artetxe and Schwenk, 2019). Samples with low scores were manually inspected and sent back for translation when needed. We also checked the overlap between train, development and test sets in terms of transcripts and voice clips (via MD5 file hashing), and confirmed they are disjoint.
Answer: They applied various sanity checks to the translations (Guzman et al., 2019). 1) For German-English, French-English and Russian-English translations, they computed sentence-level BLEU (Chen and Cherry, 2014) with the NLTK (Bird et al., 2009) implementation between the human translations and the automatic translations produced by a state-of-the-art system (Ng et al., 2019). Translations with a score that was too low were manually inspected and sent back to the translators when needed. 2) They manually inspected examples where the source transcript was identical to the translation. 3) They measured the perplexity of the translations using a language model trained on a large amount of clean monolingual data (Ng et al., 2019). They manually inspected examples where the translation had a high perplexity and sent them back to translators accordingly. 4) They computed the ratio of English characters in the translations. They manually inspected examples with a low ratio and sent them back to translators accordingly. 5) Finally, they used VizSeq (Wang et al., 2019) to calculate similarity scores between transcripts and translations based on LASER cross-lingual sentence em_x001f_beddings (Artetxe and Schwenk, 2019). Samples with low scores were manually inspected and sent back for translation when needed. They also checked the overlap between train, development and test sets in terms of transcripts and voice clips (via MD5 file hashing), and confirmed they are disjoint.