dataesr/scientific-paragraphs-categorization huggingface dataset:
Data We provided the participants with training data of parallel documents for the three language pairs as well as monolingual documents for each of the four languages, as summarized in Table  14. We did not provide any development data and the participants were free to split the training data into a training and a development datasets. The training data consisted mainly of the Scielo corpus  (Neves et al. , 2016) , a parallel collection of scientific publications composed of either titles, abstracts or title and abstracts which were retrieved from the Scielo database. For the Scielo corpus, we compiled parallel documents for all language pairs in the two sub-domains, except for the EN/FR, where only health was considered, as there were inadequate parallel documents available for biology in that pair.
LIMSI-TLP The LIMSI-TLP system is a MOSES-based statistical machine translation system, rescored with Structured Output Layer neural network models. It relied on additional indomain data, including data from the WMT'14 medical translation task (English-French) and a set of English-French Cochrane systematic review abstracts. They also experiment with a confusion network system combination which combines the outputs of Phrase Based SMT systems trained either to translate entire source sentences or specific syntactic constructs extracted from those sentences. The approach is implemented using Confusion Network decoding. uedin The University of Edinburgh team used the phrase-based statistical model from Moses including hierarchical lexicalized reordering model with four orientations in both directions.
We rank the systems as follows according to their BLEU scores, with B=biology and H=health, and bl=baseline: • en-pt(B): Istrionbox>bl; • en-pt(H): Istrionbox>bl; • pt-en(B): Istrionbox>bl; • pt-en(H): Istrionbox>bl; • en-es(B): TALP>IXA>bl; • en-es(H): TALP>IXA>bl; • es-en(B): uedin>IXA>TALP>bl; • es-en(H): uedin>IXA>TALP>bl; • en-fr(H): LIMSI>bl;  For the pairwise manual validation of sentences, and given the high number of runs for some language pairs, e. g. , Spanish-English and English-Spanish, we did not perform a pairwise evaluation for every pair of two systems. Instead, we considered only one run from each participant for each language pair and dataset: the one that achieved the best BLEU score in the automatic evaluation. An exception was made for the English-French and English-Portuguese tasks for which we had only one participating team: we considered all combinations of runs and reference translations for English-French and combinations of the reference translation and both the run with best BLEU score and the one that the participant (Istrionbox) reported as their best run.
The results of the manual validation are presented in Table  17. Only one run (IXA run 3, English-Spanish, health dataset) was comparable to the reference translation: 30 vs. 26 for A>B and A<B, respec-tively. For all other cases, the reference translation was assigned to be better than the other translation at least twice as many times. Regarding comparison between teams and runs, i. e. , ES2PT (biological and health) and English-French, we did not observe much difference when comparing distinct runs of the same team. When comparing runs from distinct teams, IXA clearly outperformed TALP in two comparisons: Spanish-English biological (57 vs. 24) and Spanish-English health (48 vs. 22). On the other hand, TALP slightly outperformed IXA in one dataset: English-Spanish biological (16 vs. 7). Finally, the uedin system was clearly superior to TALP in the Spanish-English biological dataset (60 vs.
20) and to both TALP and IXA in the Spanish-English health dataset (54 vs. 19 and 41 vs. 15, respectively). We rank the systems as follows according to our manual validation (ref=reference): • en-pt (B): ref>Istrionbox; • en-pt (H): ref>Istrionbox; • pt-en (B): ref>Istrionbox; • pt-en (H): ref>Istrionbox; • en-es (B): ref>TALP> IXA; • en-es (H): {IXA,ref}>TALP; • es-en (B): ref>uedin>IXA>TALP; • es-en (H): ref>uedin> IXA>TALP; • en-fr (H): ref>LIMSI;.
In this first edition of the task, the training and test data was obtained from the parallel publications available in Scielo. We did not perform manual translation of the documents for either the training or the test data, but rather used the original text available in Scielo for all languages under consideration here. In practice, this means that the reference translations were produced by the article authors independently of the WMT challenge goals. These authors are not professional writers or professional translators, and some of them may have limited proficiency in the languages they are required to use for publication. This situation has an impact on the quality of the reference translations, compared to other WMT tasks. It is reflected in the manual evaluation which indicates that for some language pairs (notably English-Spanish health), participant runs were rated overall as better or equal to the reference translation.
Our experience with this first edition of the task indicates that the Scielo corpus is a valuable resource for biomedical WMT, however more work is needed in terms of quality assurance to ensure that meaningful evaluation results can be obtained. Plan for future editions. In next editions, we plan to build on the established pipeline to collect and pre-process Scielo data to prepare a new test dataset. More importantly, we plan to work towards improved data and evaluation quality. While we initially focused on characterizing the quality of the alignment in the parallel Scielo corpus, we are planning to craft a higher quality dataset by removing any sentence pairs with alignment issues. Furthermore, the data set will also be pruned for sentences exhibiting lexical, grammatical or fluency issues.
Quality Estimation The fifth edition of the WMT shared task on quality estimation (QE) of machine translation (MT) builds on the previous editions of the task  (Callison-Burch et al. , 2012; Bojar et al. , 2013 Bojar et al. , , 2014 Bojar et al. , , 2015)) , with "traditional" tasks at sentence and word levels, a new task for entire documents quality prediction, and a variant of the word-level task: phrase-level estimation. The goals of this year's shared task were: • To advance work on sentence and wordlevel quality estimation by providing domainspecific, larger and professionally annotated datasets. • To analyse the effectiveness of different types of quality labels provided by humans for longer texts in document-level prediction. • To investigate quality estimation at a new level of granularity: phrases.
These goals are addressed through three groups of tasks: Task 1 at sentence level (Section 6. 3), Task 2 at word and phrase levels (Section 6. 4), and Task 3 at document level (Section 6. 6). Tasks 1 and 2 provide the same dataset with English-German translations generated by a statistical machine translation (SMT) system, while Task 3 provides an English-Spanish dataset of translations taken from all participating systems in WMT08-WMT13. These datasets were annotated with different labels for quality: for Tasks 1 and 2, the labels were automatically derived from the post-editing of the machine translation output, while for Task 3, scores were computed based on a two-stage post-editing process. Any external resource, including additional quality estimation training data, could be used by participants (no distinction between constrained and unconstrained tracks was made).
They also tried bilingual models with RNN-LM and found that they perform better than monolingual models. In the training data, the distribution of labels (OK/BAD) is skewed, with significantly more OK labels. To handle this issue, they use strategies to replace the OK label with sublabels to balance the distribution. The sub-labels are OK B, OK I, OK E, depending on the location of the token in the sentence. POSTECH's RNN-based systems are entirely neural approaches for QE. Due to the small amount of data to train the prediction models, each component of the systems is trained separately by using different training data. To train the first component of the systems, the Europarl v7 English-German parallel corpus was used. To train the second component of the systems, WMT16 QE task English-German datasets were used.
To calculate the quality of an entire sentence translation, they split its source/target side in ngrams, estimate their quality individually, then average their quality scores. They use word embedding models trained over 7 billion words as external resource (English and German) using word2vec. ding features along with the cross entropy and NMT likelihood led to large improvements in prediction, and achieved third place in the scoring and second place in the ranking task variants according to the official evaluation metrics. Neural network features alone also performed very well. This is a very encouraging finding since for many language pairs it is sometime hard to find appropriate resources to build hand-crafted features, while the neural network features used only require (sufficient) monolingual data to train models, which is available in abundance for many languages.
17 Two systems were submitted, both for word-level and phrase-level QE tasks: one using only features based on external sources of bilingual information, and another combining them with the baseline features provided by the task organisers. UFAL (Task 1): The submission is based on word alignments and bilingual distributed representations to introduce a new set of features for the sentence-Level QE task. The features extracted include three alignment-based features, three bilingual embedding-based features, two embeddingbased features constrained on alignment links, as well as a set of 74 bigrams used as boolean features. The set of bigrams represents the most frequent bigrams in translations that have changed after the post-edition, and they are compiled by aligning translations to their post-editions provided in the WMT QE datasets.
To produce these features, GIZA++  (Och and Ney, 2003)  was used for word alignment and Multivec  (Berard et al. , 2016)  was used for the bilingual model, which jointly learns distributed representations for source and target languages using a parallel corpus. To build the bilingual model, domain-specific data compiled from the resources made available for the WMT 16 IT-Domain shared task was used. As prediction model, a Linear Regression model using scikit-learn was built using a combination of QuEst++ baseline features and the new features proposed. UGENT-LT3 (Task 1, Task 2): The submissions for the word-level task use 41 features in combination with the baseline feature set to train binary classifiers. The 41 additional features attempt to capture accuracy errors (concerned with the meaning transfer from the source to target sentences) using word and phrase alignment probabilities, fluency errors (concerned with the wellformedness of target sentence) using language models trained on word surface forms and on part-of-speech tags, and terminology errors (concerned with the domain-specific terminology) using a bilingual terminology list.
Data The data is the same as that used for the WMT16 Automatic Post-editing task, collected By the QT21 Project 19  in the Information Technology (IT) domain. 20 Source segments are English sentences and target segments are German translations produced by a strong SMT system built within the QT21 Project. The human post-editions are a manual revision of the target, done by professional translators using the PET post-editing tool  (Aziz et al. , 2012). HTER labels were computed using the TERCOM tool 21 with default settings (tokenised, case insensitive, exact matching only), and scores capped to 1. As training and development data, we provided English-German datasets with 12,000 and 1,000 source sentences, their machine translations, posteditions and HTER scores. As test data, we provided an additional set of 2,000 English-German source-translations pairs produced by the same SMT system used for the training data.
Task 2: Predicting word-level quality The goal of this task is to evaluate the extent to which we can detect word-level errors in MT output. Various classes of errors can be found in translations, but for this task we consider all error types together, aiming at making a binary distinction between OK and BAD tokens. The decision to bucket all error types together was made because of the lack of sufficient training data that could allow consideration of more fine-grained error tags. Data This year's word-level task uses the same dataset as Task 1, for a single language pair: English-German. Each instance of the training, 21 http://www. cs. umd. edu/ ˜snover/tercom/ development and test sets consists of the following elements: • Source sentence (English). • Automatic translation (German). • Manual post-edition of the automatic translation.
• Word-level binary (OK/BAD) labelling of the automatic translation. The binary labels for the datasets were acquired automatically with the TERCOM tool. The tool identifies four types of errors: substitution of a word with another word, deletion of a word (word was omitted by the translation system), insertion of a word (a spurious word was added by the translation system), and word or sequence of words shift (word order error). Every word in the machinetranslated sentence is tagged with one of these error types or not tagged if it matches a word from the reference. All the untagged (correct) words were tagged with OK, while the words tagged with substitution and insertion errors were assigned the tag BAD. The deletion errors are not associated with any word in the automatic translation, so we could not consider them.
We also disabled the shift errors by running TERCOM with the option '-d 0'. The reason for that is the fact that searching for shifts introduces significant noise in the annotation. The tool cannot discriminate between cases where a word was really shifted and where a word (especially common words such as prepositions, articles and pronouns) was deleted in one part of the sentence and then independently inserted in another part of this sentence, i. e. to correct an unrelated error. The statistics of the datasets are outlined in Table  20. Evaluation This year's evaluation procedure is different from the one used in previous QE tasks. Previously, the submissions were evaluated in terms of F 1 -score for the BAD class. However, this metric was criticised for being biased towards "pessimistic" labellings.
The phrase segmentation used in this dataset is the original segmentation of sentences produced by the SMT decoder during translation. The dataset statistics are outlined in Table  23  (this is similar to Table  20 , but shows the percentage of incorrect phrases instead of words). Evaluation Although the QE was produced at the level of phrases, we used word-level metrics to evaluate the performance of participating systems. This choice was motivated by the fact that the length of phrases can vary significantly, and an incorrectly labelled phrase can actually mean 1 to 5 incorrectly labelled words, while phrase-level metrics do not weigh incorrect labels by the length of the phrases. We decided to use word-level evaluation to make the results of this task more intuitive. We used the same metric as the one used in task 2: multiplication of word-level F 1 -OK and word-level F 1 -BAD (F 1 -mult).
Domain specific, professionally done post-editions Last year we used the largest dataset of all editions of the shared task to date (for sentence and phraselevel QE): ∼14K segment pairs altogether. However, the findings were somewhat inconclusive as the quality of the dataset was dubious (crowdsourced post-editions). This year we were able to collect a dataset of comparable size (15K) but in a completely controlled way, and with professional (paid) translators to ensure the quality of the data. Another critical difference in this year's main dataset is its domain: IT, as opposed to the rather general, "news" domain that had been used so far. Finally, we had access to the SMT system that produced the translations, which was very important for the new task introduced this yearphrase-level QE.
For phrase-level QE, the segmentation of the sentences in phrases was necessary. Having a more repetitive text domain was deemed particularly relevant for the word and phrase-level tasks, where data sparsity is a major issue. In practice, we found that this year's main dataset is similar to last year's in terms of error distribution at the word-level: about 20% of the words are labelled as BAD. One thing to notice, however, is that with the new data systems did not seem to benefit from filtering data out. Last year, various systems reported improvements from filtering out significant portions of the "all/mostly GOOD" sentences, which could have meant that these sentences may not have been correct, but did not get post-edited by the crowdworkers. In terms of progress with respect to last year for comparable tasks, although direct comparisons are not possible, we observed that: • For sentence-level, the Pearson correlation of the winning submission last year was 0.
Overall, there is no single answer to what should be considered as a "phrase" in a phraselevel QE system. A fully-fledged phrase-level QE system should be able to handle both the segmentation of a sentence into phrases and the labelling of each phrase for quality. However, each of these two steps is a complex problem on itself. Therefore, for the first edition of the task we decided to simplify it and provide the phrase segmentation. Following  Logacheva et al. (2015) , we considered a "phrase" the final segmentation produced by the SMT decoder by an MT decoder that generated the automatic translations in the dataset. This segmentation is useful for decoding-time QE. The baseline phrase-level QE system uses a set of features which were originally designed for sentences and later adapted for smaller sequences.
Also this year, the general framework consisted in a "black box" scenario in which the MT system that produced the translations is unknown to the participants and cannot be modified. However, building on the lessons learned in the first pilot round  (Bojar et al. , 2015) , some changes have been made. The major differences concern the domain and the origin of the data. First, we moved from the general news domain to the more specific information technology (IT) domain. This novelty is motivated by the difficulties observed in the pilot round, in which the baseline (the simple do-nothing APE system that leaves all the test sentences unmodified) remained unbeaten. Indeed, the scarce repetitiveness of the news domain prevented participants to learn from the training data effective correction patterns that are also applicable to the test set.
The training and development sets respectively consist of 12, 000 and 1, 000 instances. In each instance: • The source (SRC) is a tokenized English sentence whose length ranges between 3 and 30 tokens; • The target (TGT) is a tokenized German translation of the source. Translations were obtained with a statistical MT system. 28 This information, however, was unknown to participants, for which the MT system was a black-box. • The human post-edit (PE) is a manuallyrevised version of the target, done by professional translators. 29   Test data (2, 000 instances) consists of (source, target) pairs having similar characteristics of those in the training set. Human post-edits of the test target instances were left apart to measure system performance. Table  31  provides some basic statistics about the data.
As discussed in Section 7. 3, the differences in the domain and the origin of this year's data can contribute to explain the large improvements over the baseline, which in the first pilot round unfortunately remained unbeaten. These differences are highlighted by the Repetition Rate 27 The source sentences (together with their reference translations which were not used for the task) were provided by TAUS (https://www. taus. net/) and originally come from a unique IT vendor. 28 It consists of a phrase-based machine translation system leveraging generic and in-domain parallel training data and using a pre-reordering technique  (Herrmann et al. , 2013). It takes also advantages of POS and word class-based language models. 29 German native speakers working at Text&Form https: //www. textform. com/.
Participants This year, six teams (two more than in the pilot round) participated in the APE task by submitting a total of eleven runs. Participants are listed in Table  33 ; a short description of their systems is provided in the following. Adam Mickiewicz University. This system is among the very first ones exploring the application of neural translation models to the APE task. In particular, it investigates the following aspects: i) the use of artificially-created postedited data to train the neural models, ii) the loglinear combination of monolingual and bilingual models in an ensemble-like manner, iii) the addition of task-specific features in the log-linear model to control the final output quality. Concerning the data, in addition to the official training and development material, the system exploits the English-German bilingual training material released for the IT-domain and news translation shared tasks.
We carried out six annotation sessions in a controlled environment of approximately 45 to 60 minutes each, divided in two blocks of equal duration with a small break in between. Prior to the human evaluation task, we provided annotators with a pilot study in order to be introduced to the ranking task and be familiarized with the annotation interface. For each source sentence, five systems' outputs were randomly selected among the partic-37 https://github. com/cfedermann/Appraise ipants and the non-post-edited MT output. For the second evaluation run, the human post-edit was included in the random selection of target sentences to annotate. The human annotators then ranked the outputs from 1 to 5 (1 being the best) with ties allowed. All source segments were evaluated by at least 3 annotators.
Acknowledgments This work was supported in parts by the MosesCore, QT21, QTLeap, EXPERT and CRACKER projects funded by the  European Commission  ( 7th Framework Programme  and  H2020 ). The APE task organizers would also like to thank  Jan Niehues  for training the KIT system used to produce the MT output, Text&Form for producing the manual post-edits, and the annotators involved in the manual evaluation.
Biopsy harvesting procedure Following a protocol described in the literature  [10] , biopsies of full-depth mucosa (down to the bone level) from pristine keratinized gingival areas and newly formed keratinized gingiva were harvested under local anesthesia using a 3-mm biopsy punch, prior to surgery, after 7 and after 14 days, for a different histological study (data to be published). A part of each sample was used for cell cultures in the present study, the rest was used for further detailed histological analysis. All biopsies were performed from the central zone of the grafted area under the dental operating microscope with the aid of microsurgical instruments to avoid any disturbance of the healing process. To determine the exact region of harvesting and to avoid harvesting twice from the same site, preoperative and postoperative photographs were taken and surgical sketches were drawn.
Funding D. R and S. I. S. acknowledge that the study has been supported by an internal research grant (KERATOGRAFT, code  PII-C1-TC-2013-08 , contract No. 15250/ 19. 12. 2012 ) of the  Victor Babes University Medicine and Pharmacy, Timisoara ,  Romania. B. C. aknowledges that this work was supported by a grant of the  Romanian National Authority for Scientific Research and Innovation ,  CNCS-UEFISCDI , project number  PN-II-RU-TE-2014-4-1879. B. C would also like to thank for the Senior Fulbright Grant 2015-2016 received from the  Romanian-US Fulbright Commission.