prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Did they experiment with this new dataset?
Context 1: Introduction
How humans process language has become increasingly relevant in natural language processing since physiological data during language understanding is more accessible and recorded with less effort  In this work, we focus on eye-tracking and electroencephalography (EEG) recordings to capture the reading process  On one hand, eye movement data provides millisecond-accurate records about where humans look when they are reading, and is highly correlated with the cognitive load associated with different stages of text processing  On the other hand, EEG records electrical brain activity across the scalp and is a direct measure of physiological processes, including language processing  The combination of both measurement methods enables us to study the language understanding process in a more natural setting, where participants read full sentences at a time, in their own speed  Eye-tracking then permits us to define exact word boundaries in the timeline of a subject reading a sentence, allowing the extraction of brain activity signals for each word 
Human cognitive language processing data is immensely useful for NLP: Not only can it be leveraged to improve NLP applications (e g  barrett2016weakly for part-of-speech tagging or klerke2016improving for sentence compression), but also to evaluate state-of-the-art machine learning systems  For example, hollenstein2019cognival evaluate word embeddings, or schwartz2019inducing fine-tune language models with brain-relevant bias 
Additionally, the availability of labelled data plays a crucial role in all supervised machine learning applications  Physiological data can be used to understand and improve the labelling process (e g  tokunaga2017eye), and, for instance, to build cost models for active learning scenarios BIBREF0  Is it possible to replace this expensive manual work with models trained on physiological activity data recorded from humans while reading? That is to say, can we find and extract relevant aspects of text understanding and annotation directly from the source, i e  eye-tracking and brain activity signals during reading?
Motivated by these questions and our previously released dataset, ZuCo 1 0 BIBREF1, we developed this new corpus, where we specifically aim to collect recordings during natural reading as well as during annotation 
We provide the first dataset of simultaneous eye movement and brain activity recordings to analyze and compare normal reading to task-specific reading during annotation  The Zurich Cognitive Language Processing Corpus (ZuCo) 2 0, including raw and preprocessed eye-tracking and electroencephalography (EEG) data of 18 subjects, as well as the recording and preprocessing scripts, is publicly available at https://osf io/2urht/  It contains physiological data of each subject reading 739 English sentences from Wikipedia (see example in Figure FIGREF1)  We want to highlight the re-use potential of this data  In addition to the psycholinguistic motivation, this corpus is especially tailored for training and evaluating machine learning algorithms for NLP purposes  We conduct a detailed technical validation of the data as proof of the quality of the recordings 
Related Work
Some eye-tracking corpora of natural reading (e g  the Dundee BIBREF2, Provo BIBREF3 and GECO corpus BIBREF4), and a few EEG corpora (for example, the UCL corpus BIBREF5) are available  It has been shown that this type of cognitive processing data is useful for improving and evaluating NLP methods (e g  barrett2018sequence,hollenstein2019cognival, hale2018finding)  However, before the Zurich Cognitive Language Processing Corpus (ZuCo 1 0), there was no available data for simultaneous eye-tracking and EEG recordings of natural reading  dimigen2011coregistration studied the linguistic effects of eye movements and EEG co-registration in natural reading and showed that they accurately represent lexical processing  Moreover, the simultaneous recordings are crucial to extract word-level brain activity signals 
While the above mentioned studies analyze and leverage natural reading, some NLP work has used eye-tracking during annotation (but, as of yet, not EEG data)  mishra2016predicting and joshi2014measuring recorded eye-tracking during binary sentiment annotation (positive/negative)  This data was used to determine the annotation complexity of the text passages based on eye movement metrics and for sarcasm detection BIBREF6  Moreover, eye-tracking has been used to analyze the word sense annotation process in Hindi BIBREF7, named entity annotation in Japanese BIBREF8, and to leverage annotator gaze behaviour for coreference resolution BIBREF9  Finally, tomanek2010cognitive used eye-tracking data during entity annotation to build a cost model for active learning  However, until now there is no available data or research that analyzes the differences in the human processing of normal reading versus annotation 
Related Work ::: ZuCo1 0
In previous work, we recorded a first dataset of simultaneous eye-tracking and EEG during natural reading BIBREF1  ZuCo 1 0 consists of three reading tasks, two of which contain very similar reading material and experiments as presented in the current work  However, the main difference and reason for recording ZuCo 2 0, consists in the experiment procedure  For ZuCo 1 0 the normal reading and task-specific reading paradigms were recorded in different sessions on different days  Therefore, the recorded data is not appropriate as a means of comparison between natural reading and annotation, since the differences in the brain activity data might result mostly from the different sessions due to the sensitivity of EEG  This, and extending the dataset with more sentences and more subjects, were the main factors for recording the current corpus  We purposefully maintained an overlap of some sentences between both datasets to allow additional analyses (details are described in Section SECREF7) 
Corpus Construction
In this section we describe the contents and experimental design of the ZuCo 2 0 corpus 
Corpus Construction ::: Participants
We recorded data from 19 participants and discarded the data of one of them due to technical difficulties with the eye-tracking calibration  Hence, we share the data of 18 participants  All participants are healthy adults (mean age = 34 (SD=8 3), 10 females)  Their native language is English, originating from Australia, Canada, UK, USA or South Africa  Two participants are left-handed and three participants wear glasses for reading  Details on subject demographics can be found in Table TABREF4  All participants gave written consent for their participation and the re-use of the data prior to the start of the experiments  The study was approved by the Ethics Commission of the University of Zurich 
Corpus Construction ::: Reading materials
During the recording session, the participants read 739 sentences that were selected from the Wikipedia corpus provided by culotta2006integrating  This corpus was chosen because it provides annotations of semantic relations  We included seven of the originally defined relation types: political_affiliation, education, founder, wife/husband, job_title, nationality, and employer  The sentences were chosen in the same length range as ZuCo 1 0, and with similar Flesch reading ease scores  The dataset statistics are shown in Table TABREF2 
Of the 739 sentences, the participants read 349 sentences in a normal reading paradigm, and 390 sentences in a task-specific reading paradigm, in which they had to determine whether a certain relation type occurred in the sentence or not  Table TABREF3 shows the distribution of the different relation types in the sentences of the task-specific annotation paradigm 
Purposefully, there are 63 duplicates between the normal reading and the task-specific sentences (8% of all sentences)  The intention of these duplicate sentences is to provide a set of sentences read twice by all participants with a different task in mind  Hence, this enables the comparison of eye-tracking and brain activity data when reading normally and when annotating specific relations (see examples in Section SECREF4) 
Furthermore, there is also an overlap in the sentences between ZuCo 1 0 and ZuCo 2 0  100 normal reading and 85 task-specific sentences recorded for this dataset were already recorded in ZuCo 1 0  This allows for comparisons between the different recording procedures (i e  session-specific effects) and between more participants (subject-specific effects) 
Corpus Construction ::: Experimental design
As mentioned above, we recorded two different reading tasks for the ZuCo 2 0 dataset  During both tasks the participants were able to read in their own speed, using a control pad to move to the next sentence and to answer the control questions, which allowed for natural reading  Since each subject reads at their own personal pace, the reading speed between varies between subjects  Table TABREF4 shows the average reading speed for each task, i e  the average number of seconds a subject spends per sentence before switching to the next one 
All 739 sentences were recorded in a single session for each participant  The duration of the recording sessions was between 100 and 180 minutes, depending on the time required to set up and calibrate the devices, and the personal reading speed of the participants 
We recorded 14 blocks of approx  50 sentences, alternating between tasks: 50 sentences of normal reading, followed by 50 sentences of task-specific reading  The order of blocks and sentences within blocks was identical for all subjects  Each sentence block was preceded by a practice round of three sentences 
Corpus Construction ::: Experimental design ::: Normal reading (NR)
In the first task, participants were instructed to read the sentences naturally, without any specific task other than comprehension  Participants were told to read the sentences normally without any special instructions  Figure FIGREF8 (left) shows an example sentence as it was depicted on the screen during recording  As shown in Figure FIGREF8 (middle), the control condition for this task consisted of single-choice questions about the content of the previous sentence  12% of randomly selected sentences were followed by such a comprehension question with three answer options on a separate screen 
Corpus Construction ::: Experimental design ::: Task-specific reading (TSR)
In the second task, the participants were instructed to search for a specific relation in each sentence they read  Instead of comprehension questions, the participants had to decide for each sentence whether it contains the relation or not, i e  they were actively annotating each sentence  Figure FIGREF8 (right) shows an example screen for this task  17% of the sentences did not include the relation type and were used as control conditions  All sentences within one block involved the same relation type  The blocks started with a practice round, which described the relation and was followed by three sample sentences, so that the participants would be familiar with the respective relation type 
Corpus Construction ::: Linguistic assessment
As a linguistic assessment, the vocabulary and language proficiency of the participants was tested with the LexTALE test (Lexical Test for Advanced Learners of English, lemhofer2012introducing)  This is an unspeeded lexical decision task designed for intermediate to highly proficient language users  The average LexTALE score over all participants was 88 54%  Moreover, we also report the scores the participants achieved with their answers to the reading comprehension control questions and their relation annotations  The detailed scores for all participants are also presented in Table TABREF4 
Corpus Construction ::: Data acquisition
Data acquisition took place in a sound-attenuated and dark experiment room  Participants were seated at a distance of 68cm from a 24-inch monitor with a resolution of 800x600 pixels  A stable head position was ensured via a chin rest  Participants were instructed to stay as still as possible during the tasks to avoid motor EEG artifacts  Participants were also offered snacks and water during the breaks and were encouraged to rest  All sentences were presented at the same position on the screen and could span multiple lines  The sentences were presented in black on a light grey background with font size 20-point Arial, resulting in a letter height of 0 8 mm  The experiment was programmed in MATLAB 2016b BIBREF10, using PsychToolbox BIBREF11  Participants completed the tasks sitting alone in the room, while two research assistants were monitoring their progress in the adjoining room  All recording scripts including detailed participant instructions are available alongside the data 
Corpus Construction ::: Data acquisition ::: Eye-tracking acquisition
Eye position and pupil size were recorded with an infrared video-based eye tracker (EyeLink 1000 Plus, SR Research) at a sampling rate of 500 Hz  The eye tracker was calibrated with a 9-point grid at the beginning of the session and re-validated before each block of sentences 
Corpus Construction ::: Data acquisition ::: EEG acquisition
High-density EEG data were recorded at a sampling rate of 500 Hz with a bandpass of 0 1 to 100 Hz, using a 128-channel EEG Geodesic Hydrocel system (Electrical Geodesics)  The recording reference was set at electrode Cz  The head circumference of each participant was measured to select an appropriately sized EEG net  To ensure good contact, the impedance of each electrode was checked prior to recording, and was kept below 40 kOhm  Electrode impedance levels were checked after every third block of 50 sentences (approx  every 30 mins) and reduced if necessary 
Corpus Construction ::: Preprocessing and feature extraction ::: Eye-tracking
The eye-tracking data consists of (x,y) gaze location entries for all individual fixations (Figure FIGREF1b)  Coordinates were given in pixels with respect to the monitor coordinates (the upper left corner of the screen was (0,0) and down/right was positive)  We provide this raw data as well as various engineered eye-tracking features  For this feature extraction only fixations within the boundaries of each displayed word were extracted  Data points distinctly not associated with reading (minimum distance of 50 pixels to the text) were excluded  Additionally, fixations shorter than 100 ms were excluded from the analyses, because these are unlikely to reflect fixations relevant for reading BIBREF12  On the basis of the GECO and ZuCo 1 0 corpora, we extracted the following features: (i) gaze duration (GD), the sum of all fixations on the current word in the first-pass reading before the eye moves out of the word; (ii) total reading time (TRT), the sum of all fixation durations on the current word, including regressions; (iii) first fixation duration (FFD), the duration of the first fixation on the prevailing word; (iv) single fixation duration (SFD), the duration of the first and only fixation on the current word; and (v) go-past time (GPT), the sum of all fixations prior to progressing to the right of the current word, including regressions to previous words that originated from the current word  For each of these eye-tracking features we additionally computed the pupil size  Furthermore, we extracted the number of fixations and mean pupil size for each word and sentence 
Corpus Construction ::: Preprocessing and feature extraction ::: EEG
The EEG data shared in this project are available as raw data, but also preprocessed with Automagic (version 1 4 6, pedroni2019automagic), a tool for automatic EEG data cleaning and validation  105 EEG channels (i e  electrodes) were used from the scalp recordings  9 EOG channels were used for artifact removal and additional 14 channels lying mainly on the neck and face were discarded before data analysis  Bad channels were identified and interpolated  We used the Multiple Artifact Rejection Algorithm (MARA), a supervised machine learning algorithm that evaluates ICA components, for automatic artifact rejection  MARA has been trained on manual component classifications, and thus captures a wide range of artifacts  MARA is especially effective at detecting and removing eye and muscle artifact components  The effect of this preprocessing can be seen in Figure FIGREF1d 
After preprocessing, we synchronized the EEG and eye-tracking data to enable EEG analyses time-locked to the onsets of fixations  To compute oscillatory power measures, we band-pass filtered the continuous EEG signals across an entire reading task for five different frequency bands resulting in a time-series for each frequency band  The independent frequency bands were determined as follows: theta$_1$ (4–6 Hz), theta$_2$ (6 5–8 Hz), alpha$_1$ (8 5–10 Hz), alpha$_2$ (10 5–13 Hz), beta$_1$ (13 5–18 Hz), beta$_2$ (18 5–30 Hz), gamma$_1$ (30 5–40 Hz), and gamma$_2$ (40–49 5 Hz)  We then applied a Hilbert transformation to each of these time-series  We specifically chose the Hilbert transformation to maintain the temporal information of the amplitude of the frequency bands, to enable the power of the different frequencies for time segments defined through the fixations from the eye-tracking recording  Thus, for each eye-tracking feature we computed the corresponding EEG feature in each frequency band  Furthermore, we extracted sentence-level EEG features by calculating the power in each frequency band, and additionally, the difference of the power spectra between frontal left and right homologue electrodes pairs  For each eye-tracking based EEG feature, all channels were subject to an artifact rejection criterion of $90\mu V$ to exclude trials with transient noise 
Data Validation
The aim of the technical validation of the data is to guarantee good recording quality and to replicate findings of previous studies investigating co-registration of EEG and eye movement data during natural reading tasks (e g  dimigen2011coregistration)  We also compare the results to ZuCo 1 0 BIBREF1, which allows a more direct comparison due to the analogous recording procedure 
Data Validation ::: Eye-tracking
We validated the recorded eye-tracking data by analyzing the fixations made by all subjects through their reading speed and omission rate on sentence level  The omission rate is defined as the percentage of words that is not fixated in a sentence  Figure FIGREF10 (middle) shows the mean reading speed over all subjects, measured in seconds per sentence and Figure FIGREF10 (right) shows the mean omission rates aggregated over all subjects for each task  Clearly, the participants made less fixations during the task-specific reading, which lead to faster reading speed 
Moreover, we corroborated these sentence-level metrics by visualizing the skipping proportion on word level (Figure FIGREF13)  The skipping proportion is the average rate of words being skipped (i e  not being fixated) in a sentence  As expected, this also increases in the task-specific reading 
Although the reading material is from the same source and of the same length range (see Figure FIGREF10 (left)), in the first task (NR) passive reading was recorded, while in the second task (TSR) the subjects had to annotate a specific relation type in each sentence  Thus, the task-specific annotation reading lead to shorter passes because the goal was merely to recognize a relation in the text, but not necessarily to process the every word in each sentence  This distinct reading behavior is shown in Figure FIGREF15, where fixations occur until the end of the sentence during normal reading, while during task-specific reading the fixations stop after the decisive words to detect a given relation type  Finally, we also analyzed the average reading times for each of the extracted eye-tracking features  The means and distributions for both tasks are shown in Figure FIGREF21  These results are in line with the recorded data in ZuCo 1 0, as well as with the features extracted in the GECO corpus BIBREF4 
Data Validation ::: EEG
As a first validation step, we extracted fixation-related potentials (FRPs), where the EEG signal during all fixations of one task are averaged  Figure FIGREF24 shows the time-series of the resulting FRPs for two electrodes (PO8 and Cz), as well as topographies of the voltage distributions across the scalp at selected points in time  The five components (for which the scalp topographies are plotted) are highly similar in the time-course of the chosen electrodes to dimigen2011coregistration as well as to ZuCo 1 0 
Moreover, these previous studies were able to show an effect of fixation duration on the resulting FRPs  To show this dependency we followed two approaches  First, for each reading task, all single-trial FRPs were ordered by fixation duration and a vertical sliding time-window was used to smooth the data BIBREF13  Figure FIGREF25 (bottom) shows the resulting plots  In line with this previous work, a first positivation can be identified at 100 ms post-fixation onset  A second positive peak is located dependent on the duration of the fixation, which can be explained by the time-jittered succeeding fixation  The second approach is based on henderson2013co in which single trial EEG segments are clustered by the duration of the current fixation  As shown in Figure FIGREF25 (top), we chose four clusters and averaged the data within each cluster to four distinct FRPs, depending on the fixation duration  Again, the same positivation peaks become apparent  Both findings are consistent with the previous work mentioned and with our findings from ZuCo 1 0 
Conclusion
We presented a new, freely available corpus of eye movement and electrical brain activity recordings during natural reading as well as during annotation  This is the first dataset that allows for the comparison between these two reading paradigms  We described the materials and experiment design in detail and conducted an extensive validation to ensure the quality of the recorded data  Since this corpus is tailored to cognitively-inspired NLP, the applications and re-use potentials of this data are extensive  The provided word-level and sentence-level eye-tracking and EEG features can be used to improve and evaluate NLP and machine learning methods, for instance, to evaluate linguistic phenomena in neural models via psycholinguistic data  In addition, because the sentences contains semantic relation labels and the annotations of all participants, it can also be widely used for relation extraction and classification  Finally, the two carefully constructed reading paradigms allow for the comparison between normal reading and reading during annotation, which can be relevant to improve the manual labelling process as well as the quality of the annotations for supervised machine learning 
Context 2: Introduction
Text summarization generates summaries from input documents while keeping salient information  It is an important task and can be applied to several real-world applications  Many methods have been proposed to solve the text summarization problem BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3   There are two main text summarization techniques: extractive and abstractive  Extractive summarization generates summary by selecting salient sentences or phrases from the source text, while abstractive methods paraphrase and restructure sentences to compose the summary  We focus on abstractive summarization in this work as it is more flexible and thus can generate more diverse summaries 
Recently, many abstractive approaches are introduced based on neural sequence-to-sequence framework BIBREF4 , BIBREF0 , BIBREF3 , BIBREF5   Based on the sequence-to-sequence model with copy mechanism BIBREF6 , BIBREF0 incorporates a coverage vector to track and control attention scores on source text  BIBREF4 introduce intra-temporal attention processes in the encoder and decoder to address the repetition and incoherent problem 
There are two issues in previous abstractive methods: 1) these methods use left-context-only decoder, thus do not have complete context when predicting each word  2) they do not utilize the pre-trained contextualized language models on the decoder side, so it is more difficult for the decoder to learn summary representations, context interactions and language modeling together 
Recently, BERT has been successfully used in various natural language processing tasks, such as textual entailment, name entity recognition and machine reading comprehensions  In this paper, we present a novel natural language generation model based on pre-trained language models (we use BERT in this work)  As far as we know, this is the first work to extend BERT to the sequence generation task  To address the above issues of previous abstractive methods, in our model, we design a two-stage decoding process to make good use of BERT's context modeling ability  On the first stage, we generate the summary using a left-context-only-decoder  On the second stage, we mask each word of the summary and predict the refined word one-by-one using a refine decoder  To further improve the naturalness of the generated sequence, we cooperate reinforcement objective with the refine decoder 
The main contributions of this work are:
1  We propose a natural language generation model based on BERT, making good use of the pre-trained language model in the encoder and decoder process, and the model can be trained end-to-end without handcrafted features 
2  We design a two-stage decoder process  In this architecture, our model can generate each word of the summary considering both sides' context information 
3  We conduct experiments on the benchmark datasets CNN/Daily Mail and New York Times  Our model achieves a 33 33 average of ROUGE-1, ROUGE-2 and ROUGE-L on the CNN/Daily Mail, which is state-of-the-art  On the New York Times dataset, our model achieves about 5 6% relative improvement over ROUGE-1 
Text Summarization
In this paper, we focus on single-document multi-sentence summarization and propose a supervised abstractive model based on the neural attentive sequence-to-sequence framework which consists of two parts: a neural network for the encoder and another network for the decoder  The encoder encodes the input sequence to intermediate representation and the decoder predicts one word at a time step given the input sequence representation vector and previous decoded output  The goal of the model is to maximize the probability of generating the correct target sequences  In the encoding and generation process, the attention mechanism is used to concentrate on the most important positions of text  The learning objective of most sequence-to-sequence models is to minimize the negative log likelihood of the generated sequence as following equation shows, where $y^*_i$ is the i-th ground-truth summary token 
$$Loss = - \log \sum _{t=1}^N P(y_t^*|y_{<t}^*, X)$$   (Eq  3)
However, with this objective, traditional sequence generation models consider only one direction context in the decoding process, which could cause performance degradation since complete context of one token contains preceding and following tokens, thus feeding only preceded decoded words to the decoder so that the model may generate unnatural sequences  For example, attentive sequence-to-sequence models often generate sequences with repeated phrases which harm the naturalness  Some previous works mitigate this problem by improving the attention calculation process, but in this paper we show that feeding bi-directional context instead of left-only-context can better alleviate this problem 
Text summarization models are usually classified to abstractive and extractive ones  Recently, extractive models like DeepChannel BIBREF8 , rnn-ext+RL BIBREF9 and NeuSUM BIBREF2 achieve higher performances using well-designed structures  For example, DeepChannel propose a salience estimation network and iteratively extract salient sentences  BIBREF16 train a sentence compression model to teach another latent variable extractive model 
Also, several recent works focus on improving abstractive methods  BIBREF3 design a content selector to over-determine phrases in a source document that should be part of the summary  BIBREF11 introduce inconsistency loss to force words in less attended sentences(which determined by extractive model) to have lower generation probabilities  BIBREF5 extend seq2seq model with an information selection network to generate more informative summaries 
Bi-Directional Pre-Trained Context Encoders
Recently, context encoders such as ELMo, GPT, and BERT have been widely used in many NLP tasks  These models are pre-trained on a huge unlabeled corpus and can generate better contextualized token embeddings, thus the approaches built on top of them can achieve better performance 
Since our method is based on BERT, we illustrate the process briefly here  BERT consists of several layers  In each layer there is first a multi-head self-attention sub-layer and then a linear affine sub-layer with the residual connection  In each self-attention sub-layer the attention scores $e_{ij}$ are first calculated as Eq  ( 5 ) () shows, in which $d_e$ is output dimension, and $W^Q, W^K, W^V$ are parameter matrices 
$$&a_{ij} = \cfrac{(h_iW_Q)(h_jW_K)^T}{\sqrt{d_e}}  \\ &e_{ij} = \cfrac{\exp {e_{ij}}}{\sum _{k=1}^N\exp {e_{ik}}} $$   (Eq  5)
Then the output is calculated as Eq  ( 6 ) shows, which is the weighted sum of previous outputs $h$ added by previous output $h_i$   The last layer outputs is context encoding of input sequence 
$$o_i = h_i + \sum _{j=1}^{N} e_{ij}(h_j W_V) $$   (Eq  6)
Despite the wide usage and huge success, there is also a mismatch problem between these pre-trained context encoders and sequence-to-sequence models  The issue is that while using a pre-trained context encoder like GPT or BERT, they model token-level representations by conditioning on both direction context  During pre-training, they are fed with complete sequences  However, with a left-context-only decoder, these pre-trained language models will suffer from incomplete and inconsistent context and thus cannot generate good enough context-aware word representations, especially during the inference process 
Model
In this section, we describe the structure of our model, which learns to generate an abstractive multi-sentence summary from a given source document 
Based on the sequence-to-sequence framework built on top of BERT, we first design a refine decoder at word-level to tackle the two problems described in the above section  We also introduce a discrete objective for the refine decoders to reduce the exposure bias problem  The overall structure of our model is illustrated in Figure 1  
Problem Formulation
We denote the input document as $X = \lbrace x_1, \ldots , x_m\rbrace $ where $x_i \in \mathcal {X}$ represents one source token  The corresponding summary is denoted as $Y = \lbrace y_1, \ldots , y_L\rbrace $ , $L$ represents the summary length 
Given input document $X$ , we first predict the summary draft by a left-context-only decoder, and then using the generated summary draft we can condition on both context sides and refine the content of the summary  The draft will guide and constrain the refine process of summary 
Summary Draft Generation
The summary draft is based on the sequence-to-sequence model  On the encoder side the input document $X$ is encoded into representation vectors $H = \lbrace h_1, \ldots , h_m\rbrace $ , and then fed to the decoder to generate the summary draft $A = \lbrace a_1, \ldots , a_{|a|}\rbrace $  
We simply use BERT as the encoder  It first maps the input sequence to word embeddings and then computes document embeddings as the encoder's output, denoted by following equation 
$$H = BERT(x_1, \ldots , x_m)$$   (Eq  10)
In the draft decoder, we first introduce BERT's word embedding matrix to map the previous summary draft outputs $\lbrace y_1, \ldots , y_{t-1}\rbrace $ into embeddings vectors $\lbrace q_1, \ldots , q_{t-1}\rbrace $ at t-th time step  Note that as the input sequence of the decoder is not complete, we do not use the BERT network to predict the context vectors here 
Then we introduce an $N$ layer Transformer decoder to learn the conditional probability $P(A|H)$   Transformer's decoder-encoder multi-head attention helps the decoder learn soft alignments between summary and source document  At the t-th time step, the draft decoder predicts output probability conditioned on previous outputs and encoder hidden representations as Eq  ( 13 ) shows, in which $q_{<t} = \lbrace q_1, \ldots , q_{t-1}\rbrace $   Each generated sequence will be truncated in the first position of a special token '[PAD]' 
$$&P^{vocab}_t(w) = f_{dec}(q_{<t}, H)  \\ &L_{dec} = \sum _{i=1}^{|a|} -\log P(a_i = y_i^*|a_{< i}, H) $$   (Eq  13)
As Eq  () shows, the decoder's learning objective is to minimize negative likelihood of conditional probability, in which $y_i^*$ is the i-th ground truth word of summary 
However a decoder with this structure is not sufficient enough: if we use the BERT network in this decoder, then during training and inference, in-complete context(part of sentence) is fed into the BERT module, and although we can fine-tune BERT's parameters, the input distribution is quite different from the pre-train process, and thus harms the quality of generated context representations 
If we just use the embedding matrix here, it will be more difficult for the decoder with fresh parameters to learn to model representations as well as vocabulary probabilities, from a relative small corpus compared to BERT's huge pre-training corpus  In a word, the decoder cannot utilize BERT's ability to generate high quality context vectors, which will also harm performance 
This issue exists when using any other contextualized word representations, so we design a refine process to mitigate it in our approach which will be described in the next sub-section 
As some summary tokens are out-of-vocabulary words and occurs in input document, we incorporate copy mechanism BIBREF6 based on the Transformer decoder, we will describe it briefly 
At decoder time step $t$ , we first calculate the attention probability distribution over source document $X$ using the bi-linear dot product of the last layer decoder output of Transformer $o_t$ and the encoder output $h_j$ , as Eq  ( 15 ) () shows 
$$u_t^j =& o_t W_c h_j \\ \alpha _t^j =& \cfrac{\exp {u_t^j}}{\sum _{k=1}^N\exp {u_t^k}} $$   (Eq  15)
We then calculate copying gate $g_t\in [0, 1]$ , which makes a soft choice between selecting from source and generating from vocabulary, $W_c, W_g, b_g$ are parameters:
$$g_t = sigmoid(W_g \cdot [o_t, h] + b_g) $$   (Eq  16)
Using $g_t$ we calculate the weighted sum of copy probability and generation probability to get the final predicted probability of extended vocabulary $\mathcal {V} + \mathcal {X}$ , where $\mathcal {X}$ is the set of out of vocabulary words from the source document  The final probability is calculated as follow:
$$P_t(w) = (1-g_t)P_t^{vocab}(w) + g_t\sum _{i:w_i=w} \alpha _t^i$$   (Eq  17)
Summary Refine Process
The main reason to introduce the refine process is to enhance the decoder using BERT's contextualized representations, so we do not modify the encoder and reuse it during this process 
On the decoder side, we propose a new word-level refine decoder  The refine decoder receives a generated summary draft as input, and outputs a refined summary

ground_truth: No

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Why masking words in the decoder is helpful?
Context 2:  It first masks each word in the summary draft one by one, then feeds the draft to BERT to generate context vectors  Finally it predicts a refined summary word using an $N$ layer Transformer decoder which is the same as the draft decoder  At t-th time step the n-th word of input summary is masked, and the decoder predicts the n-th refined word given other words of the summary 
The learning objective of this process is shown in Eq  ( 19 ), $y_i$ is the i-th summary word and $y_{i}^*$ for the ground-truth summary word, and $a_{\ne i} = \lbrace a_1, \ldots , a_{i-1}, a_{i+1}, \ldots , a_{|y|}\rbrace $  
$$L_{refine} = \sum _{i=1}^{|y|} -\log P(y_i = y_i^*|a_{\ne i}, H) $$   (Eq  19)
From the view of BERT or other contextualized embeddings, the refine decoding process provides a more complete input sequence which is consistent with their pre-training processes  Intuitively, this process works as follows: first the draft decoder writes a summary draft based on a document, and then the refine decoder edits the draft  It concentrates on one word at a time, based on the source document as well as other words 
We design the word-level refine decoder because this process is similar to the cloze task in BERT's pre-train process, therefore by using the ability of the contextual language model the decoder can generate more fluent and natural sequences 
The parameters are shared between the draft decoder and refine decoder, as we find that using individual parameters the model's performance degrades a lot  The reason may be that we use teach-forcing during training, and thus the word-level refine decoder learns to predict words given all the other ground-truth words of summary  This objective is similar to the language model's pre-train objective, and is probably not enough for the decoder to learn to generate refined summaries  So in our model all decoders share the same parameters 
Researchers usually use ROUGE as the evaluation metric for summarization, however during sequence-to-sequence model training, the objective is to maximize the log likelihood of generated sequences  This mis-match harms the model's performance, so we add a discrete objective to the model, and optimize it by introducing the policy gradient method  For example, the discrete objective for the summary draft process is as Eq  ( 21 ) shows, where $a^s$ is the draft summary sampled from predicted distribution, and $R(a^s)$ is the reward score compared with the ground-truth summary, we use ROUGE-L in our experiment  To balance between optimizing the discrete objective and generating readable sequences, we mix the discrete objective with maximum-likelihood objective  As Eq  () shows, minimizing $\hat{L}_{dec}$ is the final objective for the draft process, note here $L_{dec}$ is $-logP(a|x)$   In the refine process we introduce similar objectives 
$$L^{rl}_{dec} = R(a^s)\cdot [-\log (P(a^s|x))]  \\ \hat{L}_{dec} = \gamma * L^{rl}_{dec} + (1 - \gamma ) * L_{dec} $$   (Eq  21)
Learning and Inference
During model training, the objective of our model is sum of the two processes, jointly trained using "teacher-forcing" algorithm  During training we feed the ground-truth summary to each decoder and minimize the objective 
$$L_{model} = \hat{L}_{dec} + \hat{L}_{refine}$$   (Eq  23)
At test time, each time step we choose the predicted word by $\hat{y} = argmax_{y^{\prime }} P(y^{\prime }|x)$ , use beam search to generate the draft summaries, and use greedy search to generate the refined summaries 
Settings
In this work, all of our models are built on $BERT_{BASE}$ , although another larger pre-trained model with better performance ( $BERT_{LARGE}$ ) has published but it costs too much time and GPU memory  We use WordPiece embeddings with a 30,000 vocabulary which is the same as BERT  We set the layer of transformer decoders to 12(8 on NYT50), and set the attention heads number to 12(8 on NYT50), set fully-connected sub-layer hidden size to 3072  We train the model using an Adam optimizer with learning rate of $3e-4$ , $\beta _1=0 9$ , $\beta _2=0 999$ and $\epsilon =10^{-9}$ and use a dynamic learning rate during the training process  For regularization, we use dropout BIBREF13 and label smoothing BIBREF14 in our models and set the dropout rate to 0 15, and the label smoothing value to 0 1  We set the RL objective factor $\gamma $ to 0 99 
During training, we set the batch size to 36, and train for 4 epochs(8 epochs for NYT50 since it has many fewer training samples), after training the best model are selected from last 10 models based on development set performance  Due to GPU memory limit, we use gradient accumulation, set accumulate step to 12 and feed 3 samples at each step  We use beam size 4 and length penalty of 1 0 to generate logical form sequences 
We filter repeated tri-grams in beam-search process by setting word probability to zero if it will generate an tri-gram which exists in the existing summary  It is a nice method to avoid phrase repetition since the two datasets seldom contains repeated tri-grams in one summary  We also fine tune the generated sequences using another two simple rules  When there are multi summary sentences with exactly the same content, we keep the first one and remove the other sentences; we also remove sentences with less than 3 words from the result 
To evaluate the performance of our model, we conduct experiments on CNN/Daily Mail dataset, which is a large collection of news articles and modified for summarization  Following BIBREF0 we choose the non-anonymized version of the dataset, which consists of more than 280,000 training samples and 11490 test set samples 
We also conduct experiments on the New York Times(NYT) dataset which also consists of many news articles  The original dataset can be applied here  In our experiment, we follow the dataset splits and other pre-process settings of BIBREF15   We first filter all samples without a full article text or abstract and then remove all samples with summaries shorter than 50 words  Then we choose the test set based on the date of publication(all examples published after January 1, 2007)  The final dataset contains 22,000 training samples and 3,452 test samples and is called NYT50 since all summaries are longer than 50 words 
We tokenize all sequences of the two datasets using the WordPiece tokenizer  After tokenizing, the average article length and summary length of CNN/Daily Mail are 691 and 51, and NYT50's average article length and summary length are 1152 and 75  We truncate the article length to 512, and the summary length to 100 in our experiment(max summary length is set to 150 on NYT50 as its average golden summary length is longer) 
On CNN/Daily Mail dataset, we report the full-length F-1 score of the ROUGE-1, ROUGE-2 and ROUGE-L metrics, calculated using PyRouge package and the Porter stemmer option  On NYT50, following BIBREF4 we evaluate limited length ROUGE recall score(limit the generated summary length to the ground truth length)  We split NYT50 summaries into sentences by semicolons to calculate the ROUGE scores 
Results and Analysis
Table 1 shows the results on CNN/Daily Mail dataset, we compare the performance of many recent approaches with our model  We classify them to two groups based on whether they are extractive or abstractive models  As the last line of the table shows, the ROUGE-1 and ROUGE-2 score of our full model is comparable with DCA, and outperforms on ROUGE-L  Also, compared to extractive models NeuSUM and MASK- $LM^{global}$ , we achieve slight higher ROUGE-1  Except the four scores, our model outperforms these models on all the other scores, and since we have 95% confidence interval of at most $\pm $ 0 20, these improvements are statistically significant 
As the last four lines of Table 1 show, we conduct an ablation study on our model variants to analyze the importance of each component  We use three ablation models for the experiments  One-Stage: A sequence-to-sequence model with copy mechanism based on BERT; Two-Stage: Adding the word-refine decoder to the One-Stage model; Two-Stage + RL: Full model with refine process cooperated with RL objective 
First, we compare the Two-Stage+RL model with Two-Stage ablation, we observe that the full model outperforms by 0 30 on average ROUGE, suggesting that the reinforcement objective helps the model effectively  Then we analyze the effect of refine process by removing word-level refine from the Two-Stage model, we observe that without the refine process the average ROUGE score drops by 1 69  The ablation study shows that each module is necessary for our full model, and the improvements are statistically significant on all metrics 
To evaluate the impact of summary length on model performance, we compare the average rouge score improvements of our model with different length of ground-truth summaries  As the above sub-figure of Figure 2 shows, compared to Pointer-Generator with Coverage, on length interval 40-80(occupies about 70% of test set) the improvements of our model are higher than shorter samples, confirms that with better context representations, in longer documents our model can achieve higher performance 
As the below sub-figure of Figure 2 shows, compared to extractive baseline: Lead-3 BIBREF0 , the advantage of our model will fall when golden summary length is greater than 80  This probably because that we truncate the long documents and golden summaries and cannot get full information, it could also because that the training data in these intervals is too few to train an abstractive model, so simple extractive method will not fall too far behind 
Additional Results on NYT50
Table 2 shows experiments on the NYT50 corpus  Since the short summary samples are filtered, NYT50 has average longer summaries than CNN/Daily Mail  So the model needs to catch long-term dependency of the sequences to generate good summaries 
The first two lines of Table 2 show results of the two baselines introduced by BIBREF15 : these baselines select first n sentences, or select the first k words from the original document  Also we compare performance of our model with two recent models, we see 2 39 ROUGE-1 improvements compared to the ML+RL with intra-attn approach(previous SOTA) carries over to this dataset, which is a large margin  On ROUGE-2, our model also get an improvement of 0 51  The experiment proves that our approach can outperform competitive methods on different data distributions 
Pre-trained language models
Pre-trained word vectors BIBREF17 , BIBREF18 , BIBREF19 have been widely used in many NLP tasks  More recently, pre-trained language models (ELMo, GPT and BERT), have also achieved great success on several NLP problems such as textual entailment, semantic similarity, reading comprehension, and question answering BIBREF20 , BIBREF21 , BIBREF22  
Some recent works also focus on leveraging pre-trained language models in summarization  BIBREF23 pretrain a language model and use it as the sentiment analyser when generating reviews of goods  BIBREF24 train a language model on golden summaries, and then use it on the decoder side to incorporate prior knowledge 
In this work, we use BERT(which is a pre-trained language model using large scale unlabeled data) on the encoder and decoder of a seq2seq model, and by designing a two stage decoding structure we build a competitive model for abstractive text summarization 
Conclusion and Future Work
In this work, we propose a two-stage model based on sequence-to-sequence paradigm  Our model utilize BERT on both encoder and decoder sides, and introduce reinforce objective in learning process  We evaluate our model on two benchmark datasets CNN/Daily Mail and New York Times, the experimental results show that compared to previous systems our approach effectively improves performance 
Although our experiments are conducted on summarization task, our model can be used in most natural language generation tasks, such as machine translation, question generation and paraphrasing  The refine decoder and mixed objective can also be applied on other sequence generation tasks, and we will investigate on them in future work 

ground_truth: ecause this process is similar to the cloze task in BERT's pre-train process, therefore by using the ability of the contextual language model the decoder can generate more fluent and natural sequences.

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which eight NER tasks did they evaluate on?
Context 3: Introduction
Pretrained Language Models (PTLMs) such as BERT BIBREF1 have spearheaded advances on many NLP tasks  Usually, PTLMs are pretrained on unlabeled general-domain and/or mixed-domain text, such as Wikipedia, digital books or the Common Crawl corpus 
When applying PTLMs to specific domains, it can be useful to domain-adapt them  Domain adaptation of PTLMs has typically been achieved by pretraining on target-domain text  One such model is BioBERT BIBREF2, which was initialized from general-domain BERT and then pretrained on biomedical scientific publications  The domain adaptation is shown to be helpful for target-domain tasks such as biomedical Named Entity Recognition (NER) or Question Answering (QA)  On the downside, the computational cost of pretraining can be considerable: BioBERTv1 0 was adapted for ten days on eight large GPUs (see Table TABREF1), which is expensive, environmentally unfriendly, prohibitive for small research labs and students, and may delay prototyping on emerging domains 
We therefore propose a fast, CPU-only domain-adaptation method for PTLMs: We train Word2Vec BIBREF3 on target-domain text and align the resulting word vectors with the wordpiece vectors of an existing general-domain PTLM  The PTLM thus gains domain-specific lexical knowledge in the form of additional word vectors, but its deeper layers remain unchanged  Since Word2Vec and the vector space alignment are efficient models, the process requires a fraction of the resources associated with pretraining the PTLM itself, and it can be done on CPU 
In Section SECREF4, we use the proposed method to domain-adapt BERT on PubMed+PMC (the data used for BioBERTv1 0) and/or CORD-19 (Covid-19 Open Research Dataset)  We improve over general-domain BERT on eight out of eight biomedical NER tasks, using a fraction of the compute cost associated with BioBERT  In Section SECREF5, we show how to quickly adapt an existing Question Answering model to text about the Covid-19 pandemic, without any target-domain Language Model pretraining or finetuning 
Related work ::: The BERT PTLM
For our purpose, a PTLM consists of three parts: A tokenizer $\mathcal {T}_\mathrm {LM} : \mathbb {L}^+ \rightarrow \mathbb {L}_\mathrm {LM}^+$, a wordpiece embedding function $\mathcal {E}_\mathrm {LM}: \mathbb {L}_\mathrm {LM} \rightarrow \mathbb {R}^{d_\mathrm {LM}}$ and an encoder function $\mathcal {F}_\mathrm {LM}$  $\mathbb {L}_\mathrm {LM}$ is a limited vocabulary of wordpieces  All words that are not in $\mathbb {L}_\mathrm {LM}$ are tokenized into sequences of shorter wordpieces, e g , tachycardia becomes ta ##chy ##card ##ia  Given a sentence $S = [w_1, \ldots , w_T]$, tokenized as $\mathcal {T}_\mathrm {LM}(S) = [\mathcal {T}_\mathrm {LM}(w_1); \ldots ; \mathcal {T}_\mathrm {LM}(w_T)]$, $\mathcal {E}_\mathrm {LM}$ embeds every wordpiece in $\mathcal {T}_\mathrm {LM}(S)$ into a real-valued, trainable wordpiece vector  The wordpiece vectors of the entire sequence are stacked and fed into $\mathcal {F}_\mathrm {LM}$  Note that we consider position and segment embeddings to be a part of $\mathcal {F}_\mathrm {LM}$ rather than $\mathcal {E}_\mathrm {LM}$ 
In the case of BERT, $\mathcal {F}_\mathrm {LM}$ is a Transformer BIBREF4, followed by a final Feed-Forward Net  During pretraining, the Feed-Forward Net predicts the identity of masked wordpieces  When finetuning on a supervised task, it is usually replaced with a randomly initialized task-specific layer 
Related work ::: Domain-adapted PTLMs
Domain adaptation of PTLMs is typically achieved by pretraining on unlabeled target-domain text  Some examples of such models are BioBERT BIBREF2, which was pretrained on the PubMed and/or PubMed Central (PMC) corpora, SciBERT BIBREF5, which was pretrained on papers from SemanticScholar, ClinicalBERT BIBREF6, BIBREF7 and ClinicalXLNet BIBREF8, which were pretrained on clinical patient notes, and AdaptaBERT BIBREF9, which was pretrained on Early Modern English text  In most cases, a domain-adapted PTLM is initialized from a general-domain PTLM (e g , standard BERT), though BIBREF5 report better results with a model that was pretrained from scratch with a custom wordpiece vocabulary  In this paper, we focus on BioBERT, as its domain adaptation corpora are publicly available 
Related work ::: Word vectors
Word vectors are distributed representations of words that are trained on unlabeled text  Contrary to PTLMs, word vectors are non-contextual, i e , a word type is always assigned the same vector, regardless of context  In this paper, we use Word2Vec BIBREF3 to train word vectors  We will denote the Word2Vec lookup function as $\mathcal {E}_\mathrm {W2V} : \mathbb {L}_\mathrm {W2V} \rightarrow \mathbb {R}^{d_\mathrm {W2V}}$ 
Related work ::: Word vector space alignment
Word vector space alignment has most frequently been explored in the context of cross-lingual word embeddings  For instance, BIBREF10 align English and Spanish Word2Vec spaces by a simple linear transformation  BIBREF11 use a related method to align cross-lingual word vectors and multilingual BERT wordpiece vectors 
Method
In the following, we assume access to a general-domain PTLM, as described in Section SECREF2, and a corpus of unlabeled target-domain text 
Method ::: Creating new input vectors
In a first step, we train Word2Vec on the target-domain corpus  In a second step, we take the intersection of $\mathbb {L}_\mathrm {LM}$ and $\mathbb {L}_\mathrm {W2V}$  In practice, the intersection mostly contains wordpieces from $\mathbb {L}_\mathrm {LM}$ that correspond to standalone words  It also contains single characters and other noise, however, we found that filtering them does not improve alignment quality  In a third step, we use the intersection to fit an unconstrained linear transformation $\mathbf {W} \in \mathbb {R}^{d_\mathrm {LM} \times d_\mathrm {W2V}}$ via least squares:
Intuitively, $\mathbf {W}$ makes Word2Vec vectors “look like” the PTLM's native wordpiece vectors, just like cross-lingual alignment makes word vectors from one language “look like” word vectors from another language  In Table TABREF7 (top), we show examples of within-space and cross-space nearest neighbors after alignment 
Method ::: Updating the wordpiece embedding layer
Next, we redefine the wordpiece embedding layer of the PTLM  The most radical strategy would be to replace the entire layer with the aligned Word2Vec vectors:
In initial experiments, this strategy led to a drop in performance, presumably because function words are not well represented by Word2Vec, and replacing them disrupts BERT's syntactic abilities  To prevent this problem, we leave existing wordpiece vectors intact and only add new ones:
Method ::: Updating the tokenizer
In a final step, we update the tokenizer to account for the added words  Let $\mathcal {T}_\mathrm {LM}$ be the standard BERT tokenizer, and let $\hat{\mathcal {T}}_\mathrm {LM}$ be the tokenizer that treats all words in $\mathbb {L}_\mathrm {LM} \cup \mathbb {L}_\mathrm {W2V}$ as one-wordpiece tokens, while tokenizing any other words as usual 
In practice, a given word may or may not benefit from being tokenized by $\hat{\mathcal {T}}_\mathrm {LM}$ instead of $\mathcal {T}_\mathrm {LM}$  To give a concrete example, 82% of the words in the BC5CDR NER dataset that end in the suffix -ia are inside a disease entity (e g , tachycardia)  $\mathcal {T}_\mathrm {LM}$ tokenizes this word as ta ##chy ##card ##ia, thereby exposing the orthographic cue to the model  As a result, $\mathcal {T}_\mathrm {LM}$ leads to higher recall on -ia diseases  But there are many cases where wordpiece tokenization is meaningless or misleading  For instance euthymia (not a disease) is tokenized by $\mathcal {T}_\mathrm {LM}$ as e ##uth ##ym ##ia, making it likely to be classified as a disease  By contrast, $\hat{\mathcal {T}}_\mathrm {LM}$ gives euthymia a one-wordpiece representation that depends only on distributional semantics  We find that using $\hat{\mathcal {T}}_\mathrm {LM}$ improves precision on -ia diseases 
To combine these complementary strengths, we use a 50/50 mixture of $\mathcal {T}_\mathrm {LM}$-tokenization and $\hat{\mathcal {T}}_\mathrm {LM}$-tokenization when finetuning the PTLM on a task  At test time, we use both tokenizers and mean-pool the outputs  Let $o(\mathcal {T}(S))$ be some output of interest (e g , a logit), given sentence $S$ tokenized by $\mathcal {T}$  We predict:
Experiment 1: Biomedical NER
In this section, we use the proposed method to create GreenBioBERT, an inexpensive and environmentally friendly alternative to BioBERT  Recall that BioBERTv1 0 (biobert_v1 0_pubmed_pmc) was initialized from general-domain BERT (bert-base-cased) and pretrained on PubMed+PMC 
Experiment 1: Biomedical NER ::: Domain adaptation
We train Word2Vec with vector size $d_\mathrm {W2V} = d_\mathrm {LM} = 768$ on PubMed+PMC (see Appendix for details)  Then, we follow the procedure described in Section SECREF3 to update the wordpiece embedding layer and tokenizer of general-domain BERT 
Experiment 1: Biomedical NER ::: Finetuning
We finetune GreenBioBERT on the eight publicly available NER tasks used in BIBREF2  We also do reproduction experiments with general-domain BERT and BioBERTv1 0, using the same setup as our model  We average results over eight random seeds  See Appendix for details on preprocessing, training and hyperparameters 
Experiment 1: Biomedical NER ::: Results and discussion
Table TABREF7 (bottom) shows entity-level precision, recall and F1  For ease of visualization, Figure FIGREF13 shows what portion of the BioBERT – BERT F1 delta is covered  We improve over general-domain BERT on all tasks with varying effect sizes  Depending on the points of reference, we cover an average 52% to 60% of the BioBERT – BERT F1 delta (54% for BioBERTv1 0, 60% for BioBERTv1 1 and 52% for our reproduction experiments)  Table TABREF17 (top) shows the importance of vector space alignment: If we replace the aligned Word2Vec vectors with their non-aligned counterparts (by setting $\mathbf {W} = \mathbf {1}$) or with randomly initialized vectors, F1 drops on all tasks 
Experiment 2: Covid-19 QA
In this section, we use the proposed method to quickly adapt an existing general-domain QA model to an emerging target domain: Covid-19  Our baseline model is SQuADBERT (bert-large-uncased-whole-word-masking-finetuned-squad), a version of BERT that was finetuned on general-domain SQuAD BIBREF19  We evaluate on Deepset-AI Covid-QA, a SQuAD-style dataset with 1380 questions (see Appendix for details on data and preprocessing)  We assume that there is no target-domain finetuning data, which is a realistic setup for a new domain 
Experiment 2: Covid-19 QA ::: Domain adaptation
We train Word2Vec with vector size $d_\mathrm {W2V} = d_\mathrm {LM} = 1024$ on CORD-19 (Covid-19 Open Research Dataset) and/or PubMed+PMC  The process takes less than an hour on CORD-19 and about one day on the combined corpus, again without the need for a GPU  Then, we update SQuADBERT's wordpiece embedding layer and tokenizer, as described in Section SECREF3  We refer to the resulting model as GreenCovidSQuADBERT 
Experiment 2: Covid-19 QA ::: Results and discussion
Table TABREF17 (bottom) shows that GreenCovidSQuADBERT outperforms general-domain SQuADBERT in all metrics  Most of the improvement can be achieved with just the small CORD-19 corpus, which is more specific to the target domain (compare “Cord-19 only” and “Cord-19+PubMed+PMC”) 
Conclusion
As a reaction to the trend towards high-resource models, we have proposed an inexpensive, CPU-only method for domain-adapting Pretrained Language Models: We train Word2Vec vectors on target-domain data and align them with the wordpiece vector space of a general-domain PTLM 
On eight biomedical NER tasks, we cover over 50% of the BioBERT – BERT F1 delta, at 5% of BioBERT's domain adaptation CO$_2$ footprint and 2% of its cloud compute cost  We have also shown how to rapidly adapt an existing BERT QA model to an emerging domain – the Covid-19 pandemic – without the need for target-domain Language Model pretraining or finetuning 
We hope that our approach will benefit practitioners with limited time or resources, and that it will encourage environmentally friendlier NLP 
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Word2Vec training
We downloaded the PubMed, PMC and CORD-19 corpora from:
https://ftp ncbi nlm nih gov/pub/pmc/oa_bulk/ [20 January 2020, 68GB raw text]
https://ftp ncbi nlm nih gov/pubmed/baseline/ [20 January 2020, 24GB raw text]
https://pages semanticscholar org/coronavirus-research [17 April 2020, 2GB raw text]
We extract all abstracts and text bodies and apply the BERT basic tokenizer (a word tokenizer that standard BERT uses before wordpiece tokenization)  Then, we train CBOW Word2Vec with negative sampling  We use default parameters except for the vector size (which we set to $d_\mathrm {W2V} = d_\mathrm {LM}$) 
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 1: Biomedical NER ::: Pretrained models
General-domain BERT and BioBERTv1 0 were downloaded from:
https://storage googleapis com/bert_models/2018_10_18/cased_L-12_H-768_A-12 zip
https://github com/naver/biobert-pretrained
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 1: Biomedical NER ::: Data
We downloaded the NER datasets by following instructions on https://github com/dmis-lab/biobert#Datasets  For detailed dataset statistics, see BIBREF2 
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 1: Biomedical NER ::: Preprocessing
We cut all sentences into chunks of 30 or fewer whitespace-tokenized words (without splitting inside labeled spans)  Then, we tokenize every chunk $S$ with $\mathcal {T} = \mathcal {T}_\mathrm {LM}$ or $\mathcal {T} = \hat{\mathcal {T}}_\mathrm {LM}$ and add special tokens:
Word-initial wordpieces in $\mathcal {T}(S)$ are labeled as B(egin), I(nside) or O(utside), while non-word-initial wordpieces are labeled as X(ignore) 
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 1: Biomedical NER ::: Modeling, training and inference
We follow BIBREF2's implementation (https://github com/dmis-lab/biobert): We add a randomly initialized softmax classifier on top of the last BERT layer to predict the labels  We finetune the entire model to minimize negative log likelihood, with the standard Adam optimizer BIBREF20 and a linear learning rate scheduler (10% warmup)  Like BIBREF2, we finetune on the concatenation of the training and development set  All finetuning runs were done on a GeForce Titan X GPU (12GB) 
Since we do not have the resources for an extensive hyperparameter search, we use defaults and recommendations from the BioBERT repository: Batch size of 32, peak learning rate of $1 \cdot 10^{-5}$, and 100 epochs 
At inference time, we gather the output logits of word-initial wordpieces only  Since the number of word-initial wordpieces is the same for $\mathcal {T}_\mathrm {LM}(S)$ and $\hat{\mathcal {T}}_\mathrm {LM}(S)$, this makes mean-pooling the logits straightforward 
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 1: Biomedical NER ::: Note on our reproduction experiments
We found it easier to reproduce or exceed BIBREF2's results for general-domain BERT, compared to their results for BioBERTv1 0 (see Figure FIGREF13, main paper)  While this may be due to hyperparameters, it suggests that BioBERTv1 0 was more strongly tuned than BERT in the original BioBERT paper  This observation does not affect our conclusions, as GreenBioBERT performs better than reproduced BERT as well 
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 2: Covid-19 QA ::: Pretrained model
We downloaded the SQuADBERT baseline from:
https://huggingface co/bert-large-uncased-whole-word-masking-finetuned-squad
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 2: Covid-19 QA ::: Data
We downloaded the Deepset-AI Covid-QA dataset from:
https://github com/deepset-ai/COVID-QA/blob/master/data/question-answering/200423_covidQA json [24 April 2020]
At the time of writing, the dataset contains 1380 questions and gold answer spans  Every question is associated with one of 98 research papers (contexts)  We treat the entire dataset as a test set 
Note that there are some important differences between the dataset and SQuAD, which make the task challenging:
The contexts are full documents rather than single paragraphs  Thus, the correct answer may appear several times, often with slightly different wordings  Only a single one of the occurrences is annotated as correct, e g :
What was the prevalence of Coronavirus OC43 in community samples in Ilorin, Nigeria?
13 3% (95% CI 6 9-23 6%) # from main text
(13 3%, 10/75)  # from abstract
SQuAD gold answers are defined as the “shortest span in the paragraph that answered the question” BIBREF19, but many Covid-QA gold answers are longer and contain non-essential context, e g :
When was the Middle East Respiratory Syndrome Coronavirus isolated first?
(MERS-CoV) was first isolated in 2012, in a 60-year-old man who died in Jeddah, KSA due to severe acute pneumonia and multiple organ failure
2012,
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 2: Covid-19 QA ::: Preprocessing
We tokenize every question-context pair $(Q, C)$ with $\mathcal {T} = \mathcal {T}_\mathrm {LM}$ or $\mathcal {T} = \hat{\mathcal {T}}_\mathrm {LM}$, which yields $(\mathcal {T}(Q), \mathcal {T}(C))$  Since $\mathcal {T}(C)$ is usually too long to be digested in a single forward pass, we define a sliding window with width and stride $N = \mathrm {floor}(\frac{509 - |\mathcal {T}(Q)|}{2})$  At step $n$, the “active” window is between $a^{(l)}_n = nN$ and $a^{(r)}_n = \mathrm {min}(|C|, nN+N)$  The input is defined as:
$p^{(l)}_n$ and $p^{(r)}_n$ are chosen such that $|X^{(n)}| = 512$, and such that the active window is in the center of the input (if possible) 
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 2: Covid-19 QA ::: Modeling and inference
Feeding $X^{(n)}$ into the pretrained QA model yields start logits $\mathbf {h^{\prime }}^{(\mathrm {start}, n)} \in \mathbb {R}^{|X^{(n)}|}$ and end logits $\mathbf {h^{\prime }}^{(\mathrm {end},n)} \in \mathbb {R}^{|X^{(n)}|}$  We extract and concatenate the slices that correspond to the active windows of all steps:
Next, we map the logits from the wordpiece level to the word level  This allows us to mean-pool the outputs of $\mathcal {T}_\mathrm {LM}$ and $\hat{\mathcal {T}}_\mathrm {LM}$ even when $|\mathcal {T}_\mathrm {LM}(C)| \ne |\hat{\mathcal {T}}_\mathrm {LM}(C)|$ 
Let $c_i$ be a whitespace-delimited word in $C$  Let $\mathcal {T}(C)_{j:j+|\mathcal {T}(c_i)|}$ be the corresponding wordpieces  The start and end logits of $c_i$ are derived as:
Finally, we return the answer span $C_{k:k^{\prime }}$ that maximizes $o^{(\mathrm {start})}_k + o^{(\mathrm {end})}_{k^{\prime }}$, subject to the constraints that $k^{\prime }$ does not precede $k$ and the answer span is not longer than 500 characters 

ground_truth: BC5CDR-disease, NCBI-disease, BC5CDR-chem, BC4CHEMD, BC2GM, JNLPBA, LINNAEUS, Species-800

BC5CDR-disease, NCBI-disease, BC5CDR-chem, BC4CHEMD, BC2GM, JNLPBA, LINNAEUS, Species-800