SlowGuess/ACL_2020_OCR huggingface dataset:
Dioxygen is used in cellular respiration and many major classes of organic molecules in living organisms contain oxygen, such as proteins, nucleic acids, carbohydrates, and fats, as do the major constituent inorganic compounds of animal shells, teeth, and bone. Most of the mass of living organisms is oxygen as a component of water, the major constituent of lifeforms. Oxygen is continuously replenished in Earth's atmosphere by photosynthesis, which uses the energy of sunlight to produce oxygen from water and carbon dioxide. Oxygen is too chemically reactive to remain a free element in air without being continuously replenished by the photosynthetic action of living organisms. Another form (allotrope) of oxygen, ozone (O3), strongly absorbs ultraviolet UVB radiation and the high-altitude ozone layer helps protect the biosphere from ultraviolet radiation.
Your Question:
Ask a question here.
Try using your own words
No question possible?
Select Answer
Figure 5: English QA annotation interface screenshot
en	de	es	ar	zh*	vi	hi
Context	157.5	102.2	103.4	116.8	222.9	195.1	141.5
Question	8.4	7.7	8.6	7.6	14.3	10.6	9.3
Answer	3.1	3.2	4.1	3.4	8.2	4.5	3.6
Table 8: Mean Sequence lengths (tokens) in MLQA. *calculated with mixed segmentation (section 4.1)
A.1 Annotation Interface
Figure 5 shows a screenshot of the annotation interface. Workers are asked to write a question in the box, and highlight an answer using the mouse in the sentence that is in bold. There are a number of data input validation features to assist workers, as well as detailed instructions in a drop-down window, which are shown in Figure 6
A.2 Additional MLQA Statistics
Figure 7 shows the distribution of wh words in questions in both MLQA-en and SQuAD v.1.1. The distributions are very similar, suggesting training on SQuAD data is an appropriate training dataset choice.
Table 4 shows the number of Wikipedia articles that feature at least one of their paragraphs as a context paragraph in MLQA, along with the number of unique context paragraphs in MLQA. There are 1.9 context paragraphs from each article on average. This is in contrast to SQuAD, which instead features a small number of curated articles, but more densely annotated, with 43 context paragraphs per article on average. Thus, MLQA covers a much broader range of topics than SQuAD.
Table 8 shows statistics about the lengths of con
HIT Instructions (Click to collapse)
Spend around 6 minutes in total to ask one question on each of these five paragraphs.
Select the answer from the paragraph by clicking on 'select answer' and then highlight the smallest segment of the paragraph that answers the question.
The answer must come from the sentence in bold (the interface will not allow answers to be selected from other sentences)
When asking questions:
- Avoid using the same words/phrases as in the paragraph.
You are encouraged to pose hard questions
- Be specific, and avoid pronouns
- Do not ask 'fill in the blank style' questions
- Select the shortest answer to your question (NOT the entire bold sentence)
If you cant think of a question for a particular paragraph, click the "no question possible" button. You should only do this if there is no question you could possibly ask, so please try hard to think of questions.
An example is provided below:
Most of the mass of living organisms is oxygen as a component of water, the major constituent of lifeforms. Oxygen is continuously replenished in Earth's atmosphere by photosynthesis, which uses the energy of sunlight to produce oxygen from water and carbon dioxide. Oxygen is too chemically reactive to remain a free element in air without being continuously replenished by the photosynthetic action of living organisms.
What is the name of the biol
Answer Saved. Click To Change
Your Answer:
photosynthesis
Detailed instructions for using the interface:
First, click on the "Your Question" field, and write your question. Then, click the "Select Answer" button, and highlight your answer from the sentence in bold in the paragraph using the mouse/cursor. The "Your Answer" field will automatically populate with your highlighted answer. If you made a mistake highlighting your answer, click "Answer Saved. Click to change" and highlight a new answer using the mouse/cursor. Once you have provided a question and answer for the first paragraph, the second will appear. Once you have provided questions and answers for five paragraphs, click the submit button to end the HIT.
If you encounter problems when performing this HIT, you can contact us Here.
Figure 6: English annotation instructions screenshot
texts, questions and answers in MLQA. Vietnamese has the longest contexts on average and German are shortest, but all languages have a substantial tail of long contexts. Other than Chinese, answers are on average 3 to 4 tokens.
A.3 QA Performance stratified by question and answer types
To examine how performance varies across languages for different types of questions, we stratify MLQA with three criteria — By English Wh-word, by answer Named-Entity type and by English Question Difficulty
7327
Figure 7: Question type distribution (by "wh" word) in MLQA-en and SQuAD V1.1. The distributions are strikingly similar
Figure 8: F1 score stratified by named entity types in answer spans, relative to overall F1 score for XLM
By wh-word: First, we split by the English Wh* word in the question. This resulting change in F1 score compared to the overall F1 score is shown in Figure 3, and discussed briefly in the main text. The English wh* word provides a clue as to the type of answer the questioner is expecting, and thus acts as a way of classifying QA instances into types. We chose the 5 most common wh* words in the dataset for this analysis. We see that "when" questions are consistently easier than average across the languages, but the pattern is less clear for other question types. "Who" questions also seem easier than average, except for Hindi, where the performance is quite low for these questions. "How"-type questions (such as "how much", "how many" or "how long") are also more challenging to answer than average in English compared to the other languages. "Where" questions also seem challenging
for Spanish, German, Chinese and Hindi, but this is not true for Arabic or Vietnamese.
By Named-Entity type We create subsets of MLQA by detecting which English named entities are contained in the answer span. To achieve this, we run Named Entity Recognition using SPaCy (Honnibal and Montani, 2017), and detect where named entity spans overlap with answer spans. The F1 scores for different answer types relative to overall F1 score are shown for various Named Entity types in Figure 8. There are some clear trends: Answer spans that contain named entities are easier to answer than those that do not (the first two rows) for all the languages, but the difference is most pronounced for German. Secondly, "Temporal" answer types (DATE and TIME entity labels) are consistently easier than average for all languages, consistent with the high scores for "when" questions in the previous section. Again, this result is most pronounced in German, but is also very strong for Spanish, Hindi, and Vietnamese. Arabic also performs well for ORG, GPE and LOC answer types, unlike most of the other languages. Numeric questions (CARDINAL, ORDINAL, PERCENT, QUANTITY and MONEY entity labels) also seem relatively easy for the model in most languages.
By English Question Difficulty Here, we split MLQA into two subsets, according to whether the XLM model got the question completely wrong (no word overlap with the correct answer). We then evaluated the mean F1 score for each language on the two subsets, with the results shown in Figure 4. We see that questions that are "easy" in English also seem to be easier in the target languages, but the drop in performance for the "hard" subset is not as dramatic as one might expect. This suggests that not all questions that are hard in English in MLQA are hard in the target languages. This could be due to the grammar and morphology of different languages leading to questions being easier or more difficult to answer, but an another factor is that context documents can be shorter in target languages for questions the model struggled to answer correctly in English, effectively making them easier. Manual inspection suggests that whilst context documents are often shorter for when the model is correct in the target language, this effect is not sufficient to explain the difference in performance.
7328
A.4 Additional G-XLT results
Table 6 in the main text shows for XLM on the G-XLT task, and Table 9 for Multilingual-BERT respectively. XLM outperforms M-BERT for most language pairs, with a mean G-XLT performance of 53.4 F1 compared to 47.2 F1 (mean of off-diagonal elements of Tables 6 and 9). Multilingual BERT exhibits more of a preference for English than XLM for G-XLT, and exhibits a bigger performance drop going from XLT to G-XLT (10.5 mean drop in F1 compared to 8.2).
c/q	en	es	de	ar	hi	vi	zh
en	77.7	64.4	62.7	45.7	40.1	52.2	54.2
es	67.4	64.3	58.5	44.1	38.1	48.2	51.1
de	62.8	57.4	57.9	38.8	35.5	44.7	46.3
ar	51.2	45.3	46.4	45.6	32.1	37.3	40.0
hi	51.8	43.2	46.2	36.9	43.8	38.4	40.5
vi	61.4	52.1	51.4	34.4	35.1	57.1	47.1
zh	58.0	49.1	49.6	40.5	36.0	44.6	57.5
Table 9: F1 Score for M-BERT for G-XLT. Columns show question language, rows show context language.
A.5 Additional preprocessing Details
OpenCC (https://github.com/BYVoid/OpenCC) is used to convert all Chinese contexts to Simplified Chinese, as wikipedia dumps generally consist of a mixture of simplified and traditional Chinese text.
A.6 Further details on Parallel Sentence mining
Table 10 shows the number of mined parallel sentences found in each language, as function of how many languages the sentences are parallel between. As the number of languages that a parallel sentence is shared between increases, the number of such sentences decreases. When we look for 7-way aligned examples, we only find 1340 sentences from the entirety of the 7 Wikipedia. Additionally, most of these sentences are the first sentence of the article, or are uninteresting. However, if we choose 4-way parallel sentences, there are plenty of sentences to choose from. We sample evenly from each combination of English and 3 of the 6 target languages. This ensures that we have an even distribution over all the target languages, as well as ensuring we have even numbers of instances that will be parallel between target language combinations.
7329
N-way	en	de	es	ar	zh	vi	hi
2	12219436	3925542	4957438	1047977	1174359	904037	210083
3	2143675	1157009	1532811	427609	603938	482488	83495
4	385396	249022	319902	148348	223513	181353	34050
5	73918	56756	67383	44684	58814	54884	13151
6	12333	11171	11935	11081	11485	11507	4486
7	1340	1340	1340	1340	1340	1340	1340
Table 10: Number of mined parallel sentences as a function of how many languages the sentences are parallel between
7330