dataesr/scientific-paragraphs-categorization huggingface dataset:
The marginal effect of the first early offer is calculated in a similar manner. b The marginal effect of a non-first early offer measured in distance is calculated as the reduction in distance from 126 km that is needed to equalize the effect on utility of switching EarlyOffer from one to zero. A similar calculation is performed for the marginal effect of the first early offer. consistent with the finding that early offers are not from more attractive programs. Columns 4-6 of Panel A reveal that receiving an early offer induces a student to rank that program higher in her final ROL. Similarly, the first early offer enjoys a premium. Using the estimation results, we further quantify the effect of receiving an early offer and find results that are almost identical to those for acceptance probability (Table  3 ).
A. 1. 3 Sample Restrictions The DoSV data contain 183,028 students applying to university programs for the winter term of 2015-16. We exclude 31,066 students for whom the Abitur grade is missing and cannot be inferred using the procedure described above, as well as 2,252 students with missing socio-demographic or postal code information. We further remove from the A-3 sample 4,097 students who registered to the clearinghouse after the start of Phase 1. Finally, we exclude 34,832 students who applied to specific programs with complex ranking rules, these students being mostly those wanting to become teachers and who have to choose multiple subjects (e. g. , math and English). This leaves us with a sample of 110,781 students. Table  1  in the main text provides summary statistics for this sample, as well as for the subsample of students who applied to at least two programs (64,876 students).
The sample only includes students who applied to at least two feasible programs and who either actively accepted an early offer during Phase 1 or were automatically assigned to their best offer in Phase 2. Each student's choice set is restricted to the feasible programs that she included in her initial ROL, i. e. , to the programs from which she could have received an offer by the end of Phase 2. EarlyOffer is a dummy variable, equal to one if the program became feasible to the student during Phase 1 and zero if it became feasible in Phase 2. FirstEarlyOffer is a dummy variable, equal to one if the program is the first to have become feasible to the student during Phase 1. A program's ranking of the student is computed as the student's percentile (between 0 and 1) among all applicants to the program under the Abitur quota.
A. Estimates EarlyOffer Notes: See notes of Table  3  in the main text. This table shows the results based on the specification in column 4 of Table  3  when we artificially contract students' sets of feasible programs. We proceed by relabelling as "infeasible" from the student's perspective any program that a student included in her initial ROL and whose cutoff was barely cleared by the student in Phase 2. Starting from the main analysis sample, i. e. , students who applied to at least two feasible programs and who either accepted an early offer during Phase 1 or were automatically assigned to their best offer in Phase 2, we modify students' feasible sets and acceptance decisions as follows: (i) We relabel as "infeasible" any program that became feasible to the student in Phase 2 and for which the ratio r between the student's rank under the most favorable quota and the rank of the last student who received an offer from the program under that quota is between r and 1, with r ă 1 (the most favorable quota is the quota under which the program first became feasible to the student); (ii) we restrict the sample to students who applied to at least two feasible programs under the new definition of program feasibility; (iii) if the student accepted an offer from a program that became feasible in Phase 2 but is no longer feasible under the new definition, we modify the student's acceptance decision by considering that the student accepted the highest-ranked offer among the programs that she ranked upon entering Phase 2 and that remain feasible under the new definition of feasibility.
e. , to the programs from which she could have received an offer by the end of Phase 2. EarlyOffer is a dummy variable, equal to one if the program became feasible to the student during Phase 1 and zero if it became feasible in Phase 2. FirstEarlyOffer is a dummy variable, equal to one if the program is the first to have become feasible to the student during Phase 1. A program's ranking of the student is computed as the student's percentile (between 0 and 1) among all applicants to the program under the Abitur quota. The chances of not receiving an offer from a program in Phase 2 are proxied by the ratio between the student's rank and the rank of the last student who received an offer from the program in Phase 1; the variable is zero if the student has an early from the program. *: p ă0. 1; **: p ă0.
EarlyOffer is a dummy variable, equal to one if the program became feasible to the student during Phase 1 and zero if it became feasible in Phase 2. FirstEarlyOffer is a dummy variable, equal to one if the program is the first to have become feasible to the student during Phase 1. A program's ranking of the student is computed as the student's percentile (between 0 and 1) among all applicants to the program under the Abitur quota. The chances of not receiving an offer from a program in Phase 2 are proxied by the ratio between the student's rank and the rank of the last student who received an offer from the program in Phase 1; the variable is zero if the student has an early from the program. *: p ă0. 1; **: p ă0. 05: ***: p ă0. 01. r " 1. 0 r " 1. 1 r " 1. 2 r " 1. 3 r " 1. 4 r " 1. 5 (1) (2) (3) (4) (5) (6).
The final ROL is constructed as follows: (i) when a student actively accepted an early offer during Phase 1, we only assume that she prefers the accepted offer to all other feasible programs in her ROL; (ii) when a student was assigned to a program in Phase 2, we use the rank order among the feasible programs in her final ROL up to the first program that made her an early offer in Phase 1-programs ranked below this highest-ranked early offer are only assumed to be less preferred than those ranked above (their relative rank order is ignored). EarlyOffer is a dummy variable, equal to one if the program became feasible to the student during Phase 1 and zero if it became feasible in Phase 2. FirstEarlyOffer is a dummy variable, equal to one if the program is the first to have become feasible to the student during Phase 1.
A-30 Empirical specification. Our RD estimates of the early-offer effect are based on the following empirical specification: Accept i,k,q " δ EarlyOffer i,k,q `f p R1 i,k,q q `ǫi,k,q , (A. 4) EarlyOffer i,k,q " π1t R1 i,k,q ď 0ugp R1 i,k,q q `νi,k,q , (A. 5) where Accept i,k,q is an indicator that equals one if student i, ranked under program quota pk, qq, accepted an offer (in Phase 1 or Phase 2) from program k under any quota; EarlyOffer i,k,q is an indicator that takes the value one if the student had a potential early offer (i. e. , before the end of Phase 1) from program k under any quota; the forcing variable R1 i,k,q " R i,k,q ´R1 k,q is the distance between the student's rank under the program quota pk, qq and the rank of the last student who received an offer in Phase 1 under that quota; 1t R1 i,k,q ď 0u is an indicator that is equal to one if the student cleared the Phasecutoff for program k under quota q, and hence was eligible to receive an early offer from the program under that quota; f p¨q and gp¨q are polynomial functions of the forcing variable R1 i,k,q.
The RD instrumental variable estimator using 1t R1 i,k,q ď 0u as an instrument for EarlyOffer i,k,q identifies the local average treatment effect of early offers on the acceptance probability under two main assumptions: (i) Epǫ i,k,q | R1 i,k,q q and Epν i,k,q | R1 i,k,q q are continuous, i. e. , the assignment of students on either side of Phase-1 cutoff is as good as random; (ii) crossing the cutoff affects students' acceptance probability only through the increased probability of receiving an early offer. We implemented this fuzzy RD design using the statistical package rdrobust described in Calonico et al. (2017). We present estimates from data-driven bandwidths that are mean square error (MSE)-optimal as proposed by Imbens and Kalyanaraman (2012) as well as the bias-corrected estimates and robust standard errors proposed by Calonico et al.
The sample pools together all students who applied to at least two feasible programs and whose rank under one of the considered program quotas was above the quota's Phase-2 cutoff. Following de Chaisemartin and Behaghel (2020), the last student to receive an early offer under any program quota is removed. The plots are obtained using the Stata command rdplot described in Calonico et al. (2017). The dots represent early-offer probabilities (Panel A) and acceptance probabilities (Panel B) averaged within bins. The bins are selected to balance squared-bias and variance so that the integrated mean squared error is approximately minimized. The solid lines represent kernel-weighted fourth-order polynomial fits using a uniform kernel. The shaded area represents the 95 percent confidence intervals for local means within each bin.
This table reports estimates from a conditional logit model for the probability of accepting a program among feasible programs. The sample is the same as that used in Table3in the main text but is restricted to students who only applied to programs located in their municipality of residence. Each student's choice set is restricted to the feasible programs that she included in her initial ROL, i. e. , those from which she could have received an offer by the end of Phase 2. EarlyOffer is a dummy variable, equal to one if the program became feasible to the student during Phase 1 and zero if it became feasible in Phase 2. FirstEarlyOffer is a dummy variable, equal to one if the program is the first to have become feasible to the student during Phase 1. A program's ranking of the student is computed as the student's percentile (between 0 and 1) among all applicants to the program under the Abitur quota.
This table reports estimates from a conditional logit model for the probability of accepting a program among feasible programs. The sample is the same as that used in Table3in the main text but is further restricted to students who did not accept an early offer until at least halfway through Phase 1, i. e. , until August 2 (Phase 1 lasted 34 days, from July 16 to August 18). Each student's choice set is restricted to the feasible programs that she included in her initial ROL, i. e. , to the programs from which she could have received an offer by the end of Phase 2. EarlyOffer is a dummy variable, equal to one if the program became feasible to the student during Phase 1 and zero if it became feasible in Phase 2. FirstEarlyOffer is a dummy variable, equal to one if the program is the first to have become feasible to the student during Phase 1.
Jennifer Rontganger  provided excellent copy editing service. The authors are grateful to the  Stiftung für Hochschulzulassung , in particular  Matthias Bode , for invaluable assistance. Financial support from the  National Science Foundation  through grant no. SES-1730636 , the  French National Research Agency (Agence Nationale de la Recherche)  through project  ANR-14-FRAL-0005 FDA  and  EUR  grant  ANR-17-EURE-0001 , and the  German Research Foundation (Deutsche Forschungsgemeinschaft)  through projects  KU 1971/3-1  and  CRC   TRR 190  is gratefully acknowledged.
Marginal effects on acceptance probability of feasible programs a This table reports estimates from a conditional logit model for the probability of accepting a program among feasible programs. The sample is the same as that used in Table3in the main text but is restricted to students who only applied to programs located in their municipality of residence. Each student's choice set is restricted to the feasible programs that she included in her initial ROL, i. e. , those from which she could have received an offer by the end of Phase 2. EarlyOffer is a dummy variable, equal to one if the program became feasible to the student during Phase 1 and zero if it became feasible in Phase 2. FirstEarlyOffer is a dummy variable, equal to one if the program is the first to have become feasible to the student during Phase 1.
Marginal effects on acceptance probability of feasible programs a This table reports estimates from a conditional logit model for the probability of accepting a program among feasible programs. The sample is the same as that used in Table3in the main text but is further restricted to students who did not accept an early offer until at least halfway through Phase 1, i. e. , until August 2 (Phase 1 lasted 34 days, from July 16 to August 18). Each student's choice set is restricted to the feasible programs that she included in her initial ROL, i. e. , to the programs from which she could have received an offer by the end of Phase 2. EarlyOffer is a dummy variable, equal to one if the program became feasible to the student during Phase 1 and zero if it became feasible in Phase 2. FirstEarlyOffer is a dummy variable, equal to one if the program is the first to have become feasible to the student during Phase 1.
These observations allowed us to give several suggestions on how to implement effective OCR post-processing approaches when intending to perform named entity recognition and linking. The rest of this paper is organized as follows: in Section 2, we describe typical NER and NEL approaches. Section 2. 3 studies the impact of OCR on many NER and NEL systems processing its outputs. Section 3 consists of two parts. The first one concerns the datasets, with an overview of the NER and NEL datasets used. The second part outlines the impact of OCR errors on NER and NEL systems from a global point of view, using results on clean and noisy OCRed texts. Based on the resulting observations, we propose an in-depth analysis of the types of OCR errors and their impact on NER and NEL in Section 4. Finally, Section 5 concludes the paper.
2014). Many other works have been proposed to enrich word representations with sub-word and contextual information, such as ELMo  (Peters et al. 2018) , Flair  (Akbik et al. 2019 ) and BERT  (Devlin et al. 2019). The effectiveness of NER systems using neural networks is due to their ability to be updated and generalized. These systems can jointly learn effective features with model parameters directly from the training dataset, instead of relying on hand-crafted features developed for a specific dataset.
Natural language processing of OCRed documents Despite decades of research on OCR, outputs of such systems often contain errors especially when the OCR input document is damaged, old or badly digitized. OCR systems are always located at the beginning of pipeline processes and their errors might have a negative impact and can be f https://github. com/dalab/end2end_neural_el sometimes harmful for further tasks. For this reason, many researchers have studied the problems relating to processing text data from noisy sources. Those studies where led in order to understand the effects of optical recognition errors on text analysis routines and eventually estimate which zones of digitized documents may require a prior error correction process. Many works have been done in the field of NLP to process noisy data  (Lopresti 2005).
2003)  and topic modelling  (Mutuvi et al. 2018). Several works focused on information retrieval from noisy data  (Croft et al. 1994). Chiron et al. (2017)  proposed a method to estimate the impact of OCR errors on the use of digital libraries. They built an OCR error model using a large corpus of OCRed documents aligned with their corresponding gold standard. Their model estimated the risk that a user's query might fail to match with the targeted documents. Taghva et al. (1996)  showed that moderate OCR errors have no desperate impact on the effectiveness of classical information retrieval measures. Other studies focused on the impact of OCR errors on the classification of pathology reports for cancer notification  (Zuccon et al. 2012). They concluded that OCR errors even with modest rates are not perceptible for extracting cancer notification items.
(  2020 ) conducted a large-scale analysis of the impact of OCR errors on several NLP tasks. They found that the impact on NER is less significant than on other tasks such as dependency parsing and sentence segmentation. Interestingly, the damaging effect seems greater on geo-political entities than person names or dates. Hamdi et al. (2019)  focused on NER. They processed five noisy datasets using a BiLSTM NER system and reported that NER F1-score drop about 30 percentage points when the error rate is around 20%. Huynh et al. (  2020 ) then applied a post-OCR correction method on these datasets and showed that the OCR impact can be considerably attenuated by only correcting OCR words with up to two erroneous characters. Additional related work proposed to create collections for NER on digitised Chinese documents  (Lawrie et al.
OCR errors and Named-Entity Linking Concerning NEL, we previously evaluated the performance of state-of-the-art named entity linking approaches over digitized documents with different levels of OCR quality  (Linhares Pontes et al. 2019). We simulated OCR mistakes on contemporary datasets and analyzed the performance of  Ganea and Hofmann (2017)  and  Le and Titov (2018)  systems on these data. Ganea and Hofmann embed entities and words in a common vector space and use a neural attention mechanism over local context windows to select words that are informative for the disambiguation decision. Le and Titov relied on representation learning and learn embeddings of mentions, contexts, and relations to reduce the amount of human expertise required to construct the system and make the analysis more portable across languages and domains.
2020a). They represented the entities in a continuous space and combined them with a neural attention mechanism to analyze context words and candidate entity embeddings to disambiguate mentions in historical documents. In addition, they developed several modules to handle the multilingualism and errors related to OCR engines. Similarly to van Strien et al. (  2020 ), we propose in the next sections to study the impact of OCR quality on NLP tasks, specifically NER and NEL, however, running detailed analysis over noisy data. Unlike previous work, we use larger corpora for evaluation without relying on postcorrection. The datasets used in this work cover several languages and the noisy version contains different types of degradation that might be related to storage or digitization processes. Finally we present a deep analysis of the OCR quality and provide different types and levels of OCR errors required to perform reasonably reliable NER and NEL approaches.
Resources Processing named entities in a noisy context is very common with historical content, since the text to be analysed is almost always resulting from a digitization and an OCR process. Few annotated datasets (with named entities and their links) in a noisy context aligned with their ground truth are publicly available to assess the impact of OCR errors on those tasks. As the main objective of this paper is to have a deep analysis of OCR error categories for NER/NEL tasks, we first review experiments on the global impact of OCR errors. To this end, we start with an overview of the two publicly available datasets we used. We then present how we simulated degradations of the digitization process and their OCR errors. Finally, we present the obtained results from the point of view of OCR quality, using the classical measures that are character error rate (CER) and word error rate (WER).
Named entity recognition datasets First, we focused on NER on a publicly available dataset g presented by  Hamdi et al. (2020). These corpora are based on dataset presented in the conference on natural language learning in  2002 and 2003 (CoNLL-02 and CoNLL-03). Then, we performed some operations to synthesize real OCR errors. These resources consist of 3 datasets covering 3 languages : English, Spanish and Dutch. Each dataset is cut into the 3 subsets that are commonly used in machine learning, i. e. training set, development set and test set. For each corpus (English, Dutch and Spanish), degraded images and noisy texts extracted by the OCR as well as the aligned version with clean data at the word and the character levels are provided. Named entities are classified into 4 pre-defined categories: PER for persons, LOC for locations, ORG for organisations and MISC for miscellaneous, which is used to annotate all NEs not belonging to any of the other three classes.
Simulation of noisy data Both datasets follow the same process of construction. Due to the lack of real noisy annotated data, they were built by synthesizing the process of text extraction from digitized documents. First, this was done by generating clean images from raw text coming from NER-NEL datasets. To simulate the noise induced by digitization, they then used the DocCreator tool i developed by  Journet et al. (2017). This tool provides many filters to apply various degradations to document images such as blurring, bleeding-through, ink degradation, holes and more. In order to simulate OCRed versions, the raw texts are extracted from the NER and NEL annotated corpora. They are then converted into images which have been contaminated by injecting common OCR degradation when using a scanner.
Using the tesseract OCR engine, the noisy texts are then extracted from the degraded document images. Finally, the original and noisy texts are aligned using the RETAS j tool, to match the annotations from the original corpus to the noisy version. Four types of degradation are applied to both datasets: the character degradation adds small ink spots on characters due to the age of documents. The phantom degradation simulates eroded characters that can occur after successive uses of documents. The bleed-through simulates the ink from the back side of a page, appearing on its front side. The blurring adds a blurring effect. Each of these types of degradation are performed at two levels in the NER datasets: LEV-1 level where noises are applied rarely and LEV-2 where they are applied more frequently.
On the NEL dataset, only the LEV-1 has been applied. Figure  1  shows an original image and its degraded version. Two additional degradations have been defined: LEV-0 and LEV-MIX. LEV-0 is the re-OCRred version of original images with no degradation added. It aims to provide a baseline of the OCR engine with a clean image. LEV-MIX is more of a real-world example, representing the result of simultaneously applying the four types of degradation at LEV-1 to the original texts.
Global impact of OCR errors on NER/NEL Neural networks as well as the training process have several hyper-parameters such as character embedding dimension, character-based token embedding, LSTM dimension, token embedding dimension, etc. The same parameters for training and testing have been used for both OCRed and clean datasets. In order to quantitatively estimate the impact of OCR errors on the NER and NEL tasks, we highlighted our previous works  (Hamdi et al. 2020; Linhares Pontes et al. 2019 ) over the clean corpora and the noisy simulated ones.