SlowGuess/NAACL_2022_OCR huggingface dataset:
Different variants of masked language models have been investigated to improve performance in downstream tasks. Joshi et al. (2020) leveraged a masked span generation task instead of word prediction. Fei et al. (2020) and Shen et al. (2020) proposed models that learns better syntax knowledge with syntactic distances (Shen et al., 2018) and heights (Luo et al., 2019). Henderson et al. (2019) and Humeau et al. (2019) showed that pretraining language models on dialog corpora perform better on dialog-related downstream tasks, as compared to pretraining on Wikipedia. A span selection pretraining objective is proposed in Glass et al. (2019) to reduce the gap between the pretraining and downstream finetuning stages and to improve the performance on the QA task. Some applications of generated questions are shown in (Lewis et al., 2021; Jia et al., 2021).
In contrast to self-training methods that usually adopt a teacher-student learning strategy, cooperative learning pipelines contain several agents working together to learn as much knowledge as possible. A typical cooperative learning framework is generative adversarial networks (GAN) (Goodfellow, 2016; Goodfellow et al., 2014), where a generator is optimized to confuse a discriminator, and a discriminator is trained to distinguish real examples from generated ones. Sequence GAN is further designed for learning diverse text generation (Yu et al., 2017). Unlike the adversarial learning method where two networks work for opposite goals, other studies proposed learning environments in which different agents learn the same objective functions for language emergence (Lazaridou et al., 2016; Mordatch and Abbeel, 2018; Havrylov and Titov, 2017), including simple natural language, compositional language, and
Dataset	Num. Synthetic QA
BioASQ	123121
TextbookQA	133773
RACE	115847
RelExt.	52142
DuoRC	250698
DROP	100394
Table 9: Number of synthetic QA of each MRQA domain.
symbolic language.
The SQuAD v1.1 is the easiest QA corpus used in this paper. The dataset contains 107,785 question-answer pairs on 536 articles, which are split into passages. Each question is labeled with an answer that can be extracted from the given passage.
The Natural Questions dataset is a large-scale corpus designed for open-domain question answering. The dataset is more challenging than SQuAD. All questions are collected from human search queries and are annotated with long and abstractive answers. Some of the questions are also labeled with a short answer for learning answer-span extraction or reading comprehension. Focusing on the machine reading comprehension task, we select 106,926 questions labeled with both long and short answers from the dataset for experiments.
For each target domain in MRQA, we collect the corresponding training data and sample 3000 passages for QA synthesis. The number of synthetic QAs varies based on the length of input passages, and is shown in Table 9.
C Answer Entity Recognition Details
In this section, we describe details of the AER methods, which are not covered in detail in previous sections. All AER models are pretrained on the Natural Questions corpus. To solve the sparsity problem, in other words, the passages are long but not all potential question-answer pairs are annotated, we train all following AER models by using the sentence containing the annotated answer entities as inputs, instead of the whole passage. If a sentence in the passage does not contain an annotated answer entity, we do not use it for training.
In this work, we introduce two types of AER methods, tagging based AER (AER-tag) and extrac
tion based AER (AER-Search and AER-Coop). We describe their training and how we use the trained model to recognize answer entities in our experiments.
C.1 AER-Tag
C.1.1 Training
We apply a BIO tagging model for answer entity recognition in the AER-Tab method. We train the model to classify all tokens in the input sentence into three classes,
- B(egin) - the first token of the annotated answer entity
- I(nsize) - other tokens of the annotated answer entity
- O(outside) - tokens that are not a part of the annotated answer entity
C.1.2 Evaluation
Given an input passage, we run the trained BIO tagging model on each of its sentences and greedily predict answer entities. There might be more than one answer entities predicted in each sentence, and we only use the answer entities start with a predicted B tag.
C.2 AER-LM
C.2.1 Training
For AER-LM method, we need to pretrain an extraction-based AER model. We also take a sentence of  $L$  tokens containing an annotated answer entity as an example. Using an extraction model, which is similar as our question answering model, we train the model to predict the start and end location of the annotated answer entity. The model outputs a start score and an end score for each token, and predicts the start/end locations by selecting the tokens that are assigned with highest scores. The model is trained with cross-entropy loss, by regarding the extraction task as two  $L$ -class classification tasks.
C.2.2 Evaluation
In evaluation, we first run the model on each sentence of the input passages and calculate the start and end scores for each token. For each span  $(x_{i}, x_{i+1}, \ldots, x_{j})$  that is not longer than  $L_{\text{span}}$  tokens, we calculate the span score with
$$
s _ {i j} = s _ {s t} ^ {i} + s _ {e d} ^ {j} \tag {2}
$$
where  $s_{st}^i$  is the start score of the first token of span  $(i,j)$ , and  $s_{ed}^j$  is the end score of the last token of the span. In practice, we set  $L_{span} = 10$ .
To re-rank all possible answer entities, we select top  $N_0 = 40$  spans according to  $s_{ij}$  for each passage. For all selected answer entities, we generated questions with a pretrained question generator and collect the generation perplexity of the questions. We select  $N_{search} = 5$  question-answer pairs with lowest perplexities for the final question-answering finetuning.
C.3 AER-Coop
In AER-Coop, we use the same extraction training method applied in AER-Search, and we also use the  $s_{ij}$  scores to select the top  $N_0 = 40$  preliminary answer entities for further search. The difference is that we search for final answer entities cooperatively with the pretrained question generator and question answering extractor.
With the question generator and question answering extractor, we re-rank the recognized answer entities with the following score
$$
s _ {i j} ^ {c} = \gamma \cdot I _ {c} - p \tag {3}
$$
where  $\gamma$  is a large, positive coefficient,  $p$  is the perplexity of generated question based on span  $(i,j)$ , and  $I_{c} = 1$  if the generated question is correctly answered, and otherwise  $I_{c} = 0$ .
C.4 Answer Entity Overlapping
We found the extraction-based AER model leads to overlapping problems, since a large start or end score assigned to a token leads to many candidate answer entities start or end at the token. In practice, if an answer entity is selected by the AER-Search and AER-Coop method, we no longer consider any other answer entities that overlap with the selected ones.
D RGX Examples
In this section, we show some examples of our full model. The examples are contained in Table 10.
The National History Museum of Montevideo is located in the historical residence of General Fructuoso Rivera. It exhibits artifacts related to the history of Uruguay. In a process begun in 1998, the National Museum of Natural History (1837) and the National Museum of Anthropology (1981), merged in 2001, becoming the National Museum of Natural History and Anthropology. In July 2009, the two institutions again became independent. The Historical Museum has annexed eight historical houses in the city, five of which are located in the Ciudad Vieja. One of them, on the same block with the main building, is the historic residence of Antonio Montero, which houses the Museo Romanticico.
When was the national history museum of montevideo founded?
Monarism focuses on the relationship between the?
Starting in 2006, Apple's industrial design shifted to favor aluminum, which was used in the construction of the first MacBook Pro. Glass was added in 2008 with the introduction of the unibody MacBook Pro. These materials are billed as environmentally friendly. The iMac, MacBook Pro, MacBook Air, and Mac Mini lines currently all use aluminum enclosures, and are now made of a single unibody. Chief designer Jonathan Ive continues to guide products towards a minimalist and simple feel, including eliminating of replaceable batteries in notebooks. Multi-touch gestures from the iPhone's interface have been applied to the Mac line in the form of touch pads on notebooks and the Magic Mouse and Magic Trackpad for desktops.
Who is the designer of the macbook pro?
The city's total area is 468.9 square miles (1,214 km²). 164.1 sq mi (425 km²) of this is water and 304.8 sq mi (789 km²) is land. The highest point in the city is Todt Hill on Staten Island, which, at 409.8 feet (124.9 m) above sea level, is the highest point on the Eastern Seaboard south of Maine. The summit of the ridge is mostly covered in woodlands as part of the Staten Island Greenbelt.
Where is the highest point in new york city?
In 1922, the number of supporters had surpassed 20,000 and by lending money to the club, Barça was able to build the larger Camp de Les Corts, which had an initial capacity of 20,000 spectators. After the Spanish Civil War the club started attracting more members and a larger number of spectators at matches. This led to several expansion projects: the grandstand in 1944, the southern stand in 1946, and finally the northern stand in 1950. After the last expansion, Les Corts could hold 60,000 spectators.
What is the capacity of barcelona's stadium?
On 1 November 2013, international postal services for Somalia officially resumed. The Universal Postal Union is now assisting the Somali Postal Service to develop its capacity, including providing technical assistance and basic mail processing equipment.
Who is responsible for supporting the somali postal service?
In addition to membership, as of 2010[update] there are 1,335 officially registered fan clubs, called penyes, around the world. The fan clubs promote Barcelona in their locality and receive beneficial offers when visiting Barcelona. Among the best supported teams globally, Barcelona has the highest social media following in the world among sports teams, with over 90 million Facebook fans as of February 2016. The club has had many prominent people among its supporters, including Pope John Paul II, who was an honorary member, and former prime minister of Spain José Luis Rodríguez Zapatero. FC Barcelona has the second highest average attendance of European football clubs only behind Borussia Dortmund.
Who was an honorary member of barcelona football club?
In April 1758, the British concluded the Anglo-Prussian Convention with Frederick in which they committed to pay him an annual subsidy of £670,000. Britain also dispatched 9,000 troops to reinforce Ferdinand's Hanoverian army, the first British troop commitment on the continent and a reversal in the policy of Pitt. Ferdinand had succeeded in driving the French from Hanover and Westphalia and re-captured the port of Emden in March 1758 before crossing the Rhine with his own forces, which caused alarm in France. Despite Ferdinand's victory over the French at the Battle of Krefeld and the brief occupation of Düsseldorf, he was compelled by the successful manoeuvring of larger French forces to withdraw across the Rhine.
What did france pay to the prussian monarchy?
Executives at Trump Entertainment Resorts, whose sole remaining property will be the Trump Taj Mahal, said in 2013 that they were considering the option of selling the Taj and winding down and exiting the gaming and hotel business.
What is the future of the trump taj mahal?
Vehicles typically include headlamps and tail lights. Headlamps are white or selective yellow lights placed in the front of the vehicle, designed to illuminate the upcoming road and to make the vehicle more visible. Many manufactures are turning to LED headlights as an energy-efficient alternative to traditional headlights. Tail and brake lights are red and emit light to the rear so as to reveal the vehicle's direction of travel to following drivers. White rear-facing reversing lamps indicate that the vehicle's transmission has been placed in the reverse gear, warning anyone behind the vehicle that it is moving backwards, or about to do so. Flashing turn signals on the front, side, and rear of the vehicle indicate an intended change of position or direction. In the late 1950s, some automakers began to use electroluminescent technology to backlight their cars' speedometers and other gauges or to draw attention to logos or other decorative elements.
When did they start putting back up lights in cars?
Table 10: Examples of recognized answer entities and generated questions with the full RGX model