prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. By how much does their model outperform the state of the art results?
Context 1: Introduction
Recently, deep learning algorithms have successfully addressed problems in various fields, such as image classification, machine translation, speech recognition, text-to-speech generation and other machine learning related areas BIBREF0 , BIBREF1 , BIBREF2   Similarly, substantial improvements in performance have been obtained when deep learning algorithms have been applied to statistical speech processing BIBREF3   These fundamental improvements have led researchers to investigate additional topics related to human nature, which have long been objects of study  One such topic involves understanding human emotions and reflecting it through machine intelligence, such as emotional dialogue models BIBREF4 , BIBREF5  
In developing emotionally aware intelligence, the very first step is building robust emotion classifiers that display good performance regardless of the application; this outcome is considered to be one of the fundamental research goals in affective computing BIBREF6   In particular, the speech emotion recognition task is one of the most important problems in the field of paralinguistics  This field has recently broadened its applications, as it is a crucial factor in optimal human-computer interactions, including dialog systems  The goal of speech emotion recognition is to predict the emotional content of speech and to classify speech according to one of several labels (i e , happy, sad, neutral, and angry)  Various types of deep learning methods have been applied to increase the performance of emotion classifiers; however, this task is still considered to be challenging for several reasons  First, insufficient data for training complex neural network-based models are available, due to the costs associated with human involvement  Second, the characteristics of emotions must be learned from low-level speech signals  Feature-based models display limited skills when applied to this problem 
To overcome these limitations, we propose a model that uses high-level text transcription, as well as low-level audio signals, to utilize the information contained within low-resource datasets to a greater degree  Given recent improvements in automatic speech recognition (ASR) technology BIBREF7 , BIBREF2 , BIBREF8 , BIBREF9 , speech transcription can be carried out using audio signals with considerable skill  The emotional content of speech is clearly indicated by the emotion words contained in a sentence BIBREF10 , such as “lovely” and “awesome,” which carry strong emotions compared to generic (non-emotion) words, such as “person” and “day ” Thus, we hypothesize that the speech emotion recognition model will be benefit from the incorporation of high-level textual input 
In this paper, we propose a novel deep dual recurrent encoder model that simultaneously utilizes audio and text data in recognizing emotions from speech  Extensive experiments are conducted to investigate the efficacy and properties of the proposed model  Our proposed model outperforms previous state-of-the-art methods by 68 8% to 71 8% when applied to the IEMOCAP dataset, which is one of the most well-studied datasets  Based on an error analysis of the models, we show that our proposed model accurately identifies emotion classes  Moreover, the neutral class misclassification bias frequently exhibited by previous models, which focus on audio features, is less pronounced in our model 
Related work
Classical machine learning algorithms, such as hidden Markov models (HMMs), support vector machines (SVMs), and decision tree-based methods, have been employed in speech emotion recognition problems BIBREF11 , BIBREF12 , BIBREF13   Recently, researchers have proposed various neural network-based architectures to improve the performance of speech emotion recognition  An initial study utilized deep neural networks (DNNs) to extract high-level features from raw audio data and demonstrated its effectiveness in speech emotion recognition BIBREF14   With the advancement of deep learning methods, more complex neural-based architectures have been proposed  Convolutional neural network (CNN)-based models have been trained on information derived from raw audio signals using spectrograms or audio features such as Mel-frequency cepstral coefficients (MFCCs) and low-level descriptors (LLDs) BIBREF15 , BIBREF16 , BIBREF17   These neural network-based models are combined to produce higher-complexity models BIBREF18 , BIBREF19 , and these models achieved the best-recorded performance when applied to the IEMOCAP dataset 
Another line of research has focused on adopting variant machine learning techniques combined with neural network-based models  One researcher utilized the multiobject learning approach and used gender and naturalness as auxiliary tasks so that the neural network-based model learned more features from a given dataset BIBREF20   Another researcher investigated transfer learning methods, leveraging external data from related domains BIBREF21  
As emotional dialogue is composed of sound and spoken content, researchers have also investigated the combination of acoustic features and language information, built belief network-based methods of identifying emotional key phrases, and assessed the emotional salience of verbal cues from both phoneme sequences and words BIBREF22 , BIBREF23   However, none of these studies have utilized information from speech signals and text sequences simultaneously in an end-to-end learning neural network-based model to classify emotions 
Model
This section describes the methodologies that are applied to the speech emotion recognition task  We start by introducing the recurrent encoder model for the audio and text modalities individually  We then propose a multimodal approach that encodes both audio and textual information simultaneously via a dual recurrent encoder 
Audio Recurrent Encoder (ARE)
Motivated by the architecture used in BIBREF24 , BIBREF25 , we build an audio recurrent encoder (ARE) to predict the class of a given audio signal  Once MFCC features have been extracted from an audio signal, a subset of the sequential features is fed into the RNN (i e , gated recurrent units (GRUs)), which leads to the formation of the network's internal hidden state INLINEFORM0 to model the time series patterns  This internal hidden state is updated at each time step with the input data INLINEFORM1 and the hidden state of the previous time step INLINEFORM2 as follows: DISPLAYFORM0
where INLINEFORM0 is the RNN function with weight parameter INLINEFORM1 , INLINEFORM2 represents the hidden state at t- INLINEFORM3 time step, and INLINEFORM4 represents the t- INLINEFORM5 MFCC features in INLINEFORM6   After encoding the audio signal INLINEFORM7 with the RNN, the last hidden state of the RNN, INLINEFORM8 , is considered to be the representative vector that contains all of the sequential audio data  This vector is then concatenated with another prosodic feature vector, INLINEFORM9 , to generate a more informative vector representation of the signal, INLINEFORM10   The MFCC and the prosodic features are extracted from the audio signal using the openSMILE toolkit BIBREF26 , INLINEFORM11 , respectively  Finally, the emotion class is predicted by applying the softmax function to the vector INLINEFORM12   For a given audio sample INLINEFORM13 , we assume that INLINEFORM14 is the true label vector, which contains all zeros but contains a one at the correct class, and INLINEFORM15 is the predicted probability distribution from the softmax layer  The training objective then takes the following form: DISPLAYFORM0
where INLINEFORM0 is the calculated representative vector of the audio signal with dimensionality INLINEFORM1   The INLINEFORM2 and the bias INLINEFORM3 are learned model parameters  C is the total number of classes, and N is the total number of samples used in training  The upper part of Figure shows the architecture of the ARE model 
Text Recurrent Encoder (TRE)
We assume that speech transcripts can be extracted from audio signals with high accuracy, given the advancement of ASR technologies BIBREF7   We attempt to use the processed textual information as another modality in predicting the emotion class of a given signal  To use textual information, a speech transcript is tokenized and indexed into a sequence of tokens using the Natural Language Toolkit (NLTK) BIBREF27   Each token is then passed through a word-embedding layer that converts a word index to a corresponding 300-dimensional vector that contains additional contextual meaning between words  The sequence of embedded tokens is fed into a text recurrent encoder (TRE) in such a way that the audio MFCC features are encoded using the ARE represented by equation EQREF2   In this case, INLINEFORM0 is the t- INLINEFORM1 embedded token from the text input  Finally, the emotion class is predicted from the last hidden state of the text-RNN using the softmax function 
We use the same training objective as the ARE model, and the predicted probability distribution for the target class is as follows: DISPLAYFORM0
where INLINEFORM0 is last hidden state of the text-RNN, INLINEFORM1 , and the INLINEFORM2 and bias INLINEFORM3 are learned model parameters  The lower part of Figure indicates the architecture of the TRE model 
Multimodal Dual Recurrent Encoder (MDRE)
We present a novel architecture called the multimodal dual recurrent encoder (MDRE) to overcome the limitations of existing approaches  In this study, we consider multiple modalities, such as MFCC features, prosodic features and transcripts, which contain sequential audio information, statistical audio information and textual information, respectively  These types of data are the same as those used in the ARE and TRE cases  The MDRE model employs two RNNs to encode data from the audio signal and textual inputs independently  The audio-RNN encodes MFCC features from the audio signal using equation EQREF2   The last hidden state of the audio-RNN is concatenated with the prosodic features to form the final vector representation INLINEFORM0 , and this vector is then passed through a fully connected neural network layer to form the audio encoding vector A  On the other hand, the text-RNN encodes the word sequence of the transcript using equation EQREF2   The final hidden states of the text-RNN are also passed through another fully connected neural network layer to form a textual encoding vector T  Finally, the emotion class is predicted by applying the softmax function to the concatenation of the vectors A and T  We use the same training objective as the ARE model, and the predicted probability distribution for the target class is as follows: DISPLAYFORM0
where INLINEFORM0 is the feed-forward neural network with weight parameter INLINEFORM1 , and INLINEFORM2 , INLINEFORM3 are final encoding vectors from the audio-RNN and text-RNN, respectively  INLINEFORM4 and the bias INLINEFORM5 are learned model parameters 
Multimodal Dual Recurrent Encoder with Attention (MDREA)
Inspired by the concept of the attention mechanism used in neural machine translation BIBREF28 , we propose a novel multimodal attention method to focus on the specific parts of a transcript that contain strong emotional information, conditioning on the audio information  Figure shows the architecture of the MDREA model  First, the audio data and text data are encoded with the audio-RNN and text-RNN using equation EQREF2   We then consider the final audio encoding vector INLINEFORM0 as a context vector  As seen in equation EQREF9 , during each time step t, the dot product between the context vector e and the hidden state of the text-RNN at each t-th sequence INLINEFORM1 is evaluated to calculate a similarity score INLINEFORM2   Using this score INLINEFORM3 as a weight parameter, the weighted sum of the sequences of the hidden state of the text-RNN, INLINEFORM4 , is calculated to generate an attention-application vector Z  This attention-application vector is concatenated with the final encoding vector of the audio-RNN INLINEFORM5 (equation EQREF7 ), which will be passed through the softmax function to predict the emotion class  We use the same training objective as the ARE model, and the predicted probability distribution for the target class is as follows: DISPLAYFORM0
where INLINEFORM0 and the bias INLINEFORM1 are learned model parameters 
Dataset
We evaluate our model using the Interactive Emotional Dyadic Motion Capture (IEMOCAP) BIBREF18 dataset  This dataset was collected following theatrical theory in order to simulate natural dyadic interactions between actors  We use categorical evaluations with majority agreement  We use only four emotional categories happy, sad, angry, and neutral to compare the performance of our model with other research using the same categories  The IEMOCAP dataset includes five sessions, and each session contains utterances from two speakers (one male and one female)  This data collection process resulted in 10 unique speakers  For consistent comparison with previous work, we merge the excitement dataset with the happiness dataset  The final dataset contains a total of 5531 utterances (1636 happy, 1084 sad, 1103 angry, 1708 neutral) 
Feature extraction
To extract speech information from audio signals, we use MFCC values, which are widely used in analyzing audio signals  The MFCC feature set contains a total of 39 features, which include 12 MFCC parameters (1-12) from the 26 Mel-frequency bands and log-energy parameters, 13 delta and 13 acceleration coefficients The frame size is set to 25 ms at a rate of 10 ms with the Hamming function  According to the length of each wave file, the sequential step of the MFCC features is varied  To extract additional information from the data, we also use prosodic features, which show effectiveness in affective computing  The prosodic features are composed of 35 features, which include the F0 frequency, the voicing probability, and the loudness contours  All of these MFCC and prosodic features are extracted from the data using the OpenSMILE toolkit BIBREF26  
Implementation details
Among the variants of the RNN function, we use GRUs as they yield comparable performance to that of the LSTM and include a smaller number of weight parameters BIBREF29   We use a max encoder step of 750 for the audio input, based on the implementation choices presented in BIBREF30 and 128 for the text input because it covers the maximum length of the transcripts  The vocabulary size of the dataset is 3,747, including the “_UNK_" token, which represents unknown words, and the “_PAD_" token, which is used to indicate padding information added while preparing mini-batch data  The number of hidden units and the number of layers in the RNN for each model (ARE, TRE, MDRE and MDREA) are selected based on extensive hyperparameter search experiments  The weights of the hidden units are initialized using orthogonal weights BIBREF31 ], and the text embedding layer is initialized from pretrained word-embedding vectors BIBREF32  
In preparing the textual dataset, we first use the released transcripts of the IEMOCAP dataset for simplicity  To investigate the practical performance, we then process all of the IEMOCAP audio data using an ASR system (the Google Cloud Speech API) and retrieve the transcripts  The performance of the Google ASR system is reflected by its word error rate (WER) of 5 53% 
Performance evaluation
As the dataset is not explicitly split beforehand into training, development, and testing sets, we perform 5-fold cross validation to determine the overall performance of the model  The data in each fold are split into training, development, and testing datasets (8:0 5:1 5, respectively)  After training the model, we measure the weighted average precision (WAP) over the 5-fold dataset  We train and evaluate the model 10 times per fold, and the model performance is assessed in terms of the mean score and standard deviation 
We examine the WAP values, which are shown in Table 1  First, our ARE model shows the baseline performance because we use minimal audio features, such as the MFCC and prosodic features with simple architectures  On the other hand, the TRE model shows higher performance gain compared to the ARE  From this result, we note that textual data are informative in emotion prediction tasks, and the recurrent encoder model is effective in understanding these types of sequential data  Second, the newly proposed model, MDRE, shows a substantial performance gain  It thus achieves the state-of-the-art performance with a WAP value of 0 718  This result shows that multimodal information is a key factor in affective computing  Lastly, the attention model, MDREA, also outperforms the best existing research results (WAP 0 690 to 0 688) BIBREF19   However, the MDREA model does not match the performance of the MDRE model, even though it utilizes a more complex architecture  We believe that this result arises because insufficient data are available to properly determine the complex model parameters in the MDREA model  Moreover, we presume that this model will show better performance when the audio signals are aligned with the textual sequence while applying the attention mechanism  We leave the implementation of this point as a future research direction 
To investigate the practical performance of the proposed models, we conduct further experiments with the ASR-processed transcript data (see “-ASR” models in Table )  The label accuracy of the processed transcripts is 5 53% WER  The TRE-ASR, MDRE-ASR and MDREA-ASR models reflect degraded performance compared to that of the TRE, MDRE and MDREA models  However, the performance of these models is still competitive; in particular, the MDRE-ASR model outperforms the previous best-performing model, 3CNN-LSTM10H (WAP 0 691 to 0 688) 
Error analysis
We analyze the predictions of the ARE, TRE, and MDRE models  Figure shows the confusion matrix of each model  The ARE model (Fig  ) incorrectly classifies most instances of happy as neutral (43 51%); thus, it shows reduced accuracy (35 15%) in predicting the the happy class  Overall, most of the emotion classes are frequently confused with the neutral class  This observation is in line with the findings of BIBREF30 , who noted that the neutral class is located in the center of the activation-valence space, complicating its discrimination from the other classes 
Interestingly, the TRE model (Fig  ) shows greater prediction gains in predicting the happy class when compared to the ARE model (35 15% to 75 73%)  This result seems plausible because the model can benefit from the differences among the distributions of words in happy and neutral expressions, which gives more emotional information to the model than that of the audio signal data  On the other hand, it is striking that the TRE model incorrectly predicts instances of the sad class as the happy class 16 20% of the time, even though these emotional states are opposites of one another 
The MDRE model (Fig  ) compensates for the weaknesses of the previous two models (ARE and TRE) and benefits from their strengths to a surprising degree  The values arranged along the diagonal axis show that all of the accuracies of the correctly predicted class have increased  Furthermore, the occurrence of the incorrect “sad-to-happy" cases in the TRE model is reduced from 16 20% to 9 15% 
Conclusions
In this paper, we propose a novel multimodal dual recurrent encoder model that simultaneously utilizes text data, as well as audio signals, to permit the better understanding of speech data  Our model encodes the information from audio and text sequences using dual RNNs and then combines the information from these sources using a feed-forward neural model to predict the emotion class  Extensive experiments show that our proposed model outperforms other state-of-the-art methods in classifying the four emotion categories, and accuracies ranging from 68 8% to 71 8% are obtained when the model is applied to the IEMOCAP dataset  In particular, it resolves the issue in which predictions frequently incorrectly yield the neutral class, as occurs in previous models that focus on audio features 
In the future work, we aim to extend the modalities to audio, text and video inputs  Furthermore, we plan to investigate the application of the attention mechanism to data derived from multiple modalities  This approach seems likely to uncover enhanced learning schemes that will increase performance in both speech emotion recognition and other multimodal classification tasks 
Acknowledgments
K  Jung is with the Department of Electrical and Computer Engineering, ASRI, Seoul National University, Seoul, Korea  This work was supported by the Ministry of Trade, Industry & Energy (MOTIE, Korea) under Industrial Technology Innovation Program (No 10073144) 
Context 2: Introduction
We posses a wealth of prior knowledge about many natural language processing tasks  For example, in text categorization, we know that words such as NBA, player, and basketball are strong indicators of the sports category BIBREF0 , and words like terrible, boring, and messing indicate a negative polarity while words like perfect, exciting, and moving suggest a positive polarity in sentiment classification 
A key problem arisen here, is how to leverage such knowledge to guide the learning process, an interesting problem for both NLP and machine learning communities  Previous studies addressing the problem fall into several lines  First, to leverage prior knowledge to label data BIBREF1 , BIBREF2   Second, to encode prior knowledge with a prior on parameters, which can be commonly seen in many Bayesian approaches BIBREF3 , BIBREF4   Third, to formalise prior knowledge with additional variables and dependencies BIBREF5   Last, to use prior knowledge to control the distributions over latent output variables BIBREF6 , BIBREF7 , BIBREF8 , which makes the output variables easily interpretable 
However, a crucial problem, which has rarely been addressed, is the bias in the prior knowledge that we supply to the learning model  Would the model be robust or sensitive to the prior knowledge? Or, which kind of knowledge is appropriate for the task? Let's see an example: we may be a baseball fan but unfamiliar with hockey so that we can provide a few number of feature words of baseball, but much less of hockey for a baseball-hockey classification task  Such prior knowledge may mislead the model with heavy bias to baseball  If the model cannot handle this situation appropriately, the performance may be undesirable 
In this paper, we investigate into the problem in the framework of Generalized Expectation Criteria BIBREF7   The study aims to reveal the factors of reducing the sensibility of the prior knowledge and therefore to make the model more robust and practical  To this end, we introduce auxiliary regularization terms in which our prior knowledge is formalized as distribution over output variables  Recall the example just mentioned, though we do not have enough knowledge to provide features for class hockey, it is easy for us to provide some neutral words, namely words that are not strong indicators of any class, like player here  As one of the factors revealed in this paper, supplying neutral feature words can boost the performance remarkably, making the model more robust 
More attractively, we do not need manual annotation to label these neutral feature words in our proposed approach 
More specifically, we explore three regularization terms to address the problem: (1) a regularization term associated with neutral features; (2) the maximum entropy of class distribution regularization term; and (3) the KL divergence between reference and predicted class distribution  For the first manner, we simply use the most common features as neutral features and assume the neutral features are distributed uniformly over class labels  For the second and third one, we assume we have some knowledge about the class distribution which will be detailed soon later 
To summarize, the main contributions of this work are as follows:
The rest of the paper is structured as follows: In Section 2, we briefly describe the generalized expectation criteria and present the proposed regularization terms  In Section 3, we conduct extensive experiments to justify the proposed methods  We survey related work in Section 4, and summarize our work in Section 5 
Method
We address the robustness problem on top of GE-FL BIBREF0 , a GE method which leverages labeled features as prior knowledge  A labeled feature is a strong indicator of a specific class and is manually provided to the classifier  For example, words like amazing, exciting can be labeled features for class positive in sentiment classification 
Generalized Expectation Criteria
Generalized expectation (GE) criteria BIBREF7 provides us a natural way to directly constrain the model in the preferred direction  For example, when we know the proportion of each class of the dataset in a classification task, we can guide the model to predict out a pre-specified class distribution 
Formally, in a parameter estimation objective function, a GE term expresses preferences on the value of some constraint functions about the model's expectation  Given a constraint function $G({\rm x}, y)$ , a conditional model distribution $p_\theta (y|\rm x)$ , an empirical distribution $\tilde{p}({\rm x})$ over input samples and a score function $S$ , a GE term can be expressed as follows:
$$S(E_{\tilde{p}({\rm x})}[E_{p_\theta (y|{\rm x})}[G({\rm x}, y)]])$$   (Eq  4)
Learning from Labeled Features
Druck et al  ge-fl proposed GE-FL to learn from labeled features using generalized expectation criteria  When given a set of labeled features $K$ , the reference distribution over classes of these features is denoted by $\hat{p}(y| x_k), k \in K$   GE-FL introduces the divergence between this reference distribution and the model predicted distribution $p_\theta (y | x_k)$ , as a term of the objective function:
$$\mathcal {O} = \sum _{k \in K} KL(\hat{p}(y|x_k) || p_\theta (y | x_k)) + \sum _{y,i} \frac{\theta _{yi}^2}{2 \sigma ^2}$$   (Eq  6)
where $\theta _{yi}$ is the model parameter which indicates the importance of word $i$ to class $y$   The predicted distribution $p_\theta (y | x_k)$ can be expressed as follows: $ p_\theta (y | x_k) = \frac{1}{C_k} \sum _{\rm x} p_\theta (y|{\rm x})I(x_k) $
in which $I(x_k)$ is 1 if feature $k$ occurs in instance ${\rm x}$ and 0 otherwise, $C_k = \sum _{\rm x} I(x_k)$ is the number of instances with a non-zero value of feature $k$ , and $p_\theta (y|{\rm x})$ takes a softmax form as follows: $ p_\theta (y|{\rm x}) = \frac{1}{Z(\rm x)}\exp (\sum _i \theta _{yi}x_i)  $
To solve the optimization problem, L-BFGS can be used for parameter estimation 
In the framework of GE, this term can be obtained by setting the constraint function $G({\rm x}, y) = \frac{1}{C_k} \vec{I} (y)I(x_k)$ , where $\vec{I}(y)$ is an indicator vector with 1 at the index corresponding to label $y$ and 0 elsewhere 
Regularization Terms
GE-FL reduces the heavy load of instance annotation and performs well when we provide prior knowledge with no bias  In our experiments, we observe that comparable numbers of labeled features for each class have to be supplied  But as mentioned before, it is often the case that we are not able to provide enough knowledge for some of the classes  For the baseball-hockey classification task, as shown before, GE-FL will predict most of the instances as baseball  In this section, we will show three terms to make the model more robust 
Neutral features are features that are not informative indicator of any classes, for instance, word player to the baseball-hockey classification task  Such features are usually frequent words across all categories  When we set the preference distribution of the neutral features to be uniform distributed, these neutral features will prevent the model from biasing to the class that has a dominate number of labeled features 
Formally, given a set of neutral features $K^{^{\prime }}$ , the uniform distribution is $\hat{p}_u(y|x_k) = \frac{1}{|C|}, k \in K^{^{\prime }}$ , where $|C|$ is the number of classes  The objective function with the new term becomes
$$\mathcal {O}_{NE} = \mathcal {O} + \sum _{k \in K^{^{\prime }}} KL(\hat{p}_u(y|x_k) || p_\theta (y | x_k)) $$   (Eq  9)
Note that we do not need manual annotation to provide neutral features  One simple way is to take the most common features as neutral features  Experimental results show that this strategy works successfully 
Another way to prevent the model from drifting from the desired direction is to constrain the predicted class distribution on unlabeled data  When lacking knowledge about the class distribution of the data, one feasible way is to take maximum entropy principle, as below:
$$\mathcal {O}_{ME} = \mathcal {O} + \lambda \sum _{y} p(y) \log p(y)$$   (Eq  11)
where $p(y)$ is the predicted class distribution, given by $ p(y) = \frac{1}{|X|} \sum _{\rm x} p_\theta (y | \rm x)  $ To control the influence of this term on the overall objective function, we can tune $\lambda $ according to the difference in the number of labeled features of each class  In this paper, we simply set $\lambda $ to be proportional to the total number of labeled features, say $\lambda = \beta |K|$  
This maximum entropy term can be derived by setting the constraint function to $G({\rm x}, y) = \vec{I}(y)$   Therefore, $E_{p_\theta (y|{\rm x})}[G({\rm x}, y)]$ is just the model distribution $p_\theta (y|{\rm x})$ and its expectation with the empirical distribution $\tilde{p}(\rm x)$ is simply the average over input samples, namely $p(y)$   When $S$ takes the maximum entropy form, we can derive the objective function as above 
Sometimes, we have already had much knowledge about the corpus, and can estimate the class distribution roughly without labeling instances  Therefore, we introduce the KL divergence between the predicted and reference class distributions into the objective function 
Given the preference class distribution $\hat{p}(y)$ , we modify the objective function as follows:
$$\mathcal {O}_{KL} &= \mathcal {O} + \lambda KL(\hat{p}(y) || p(y))$$   (Eq  13)
Similarly, we set $\lambda = \beta |K|$  
This divergence term can be derived by setting the constraint function to $G({\rm x}, y) = \vec{I}(y)$ and setting the score function to $S(\hat{p}, p) = \sum _i \hat{p}_i \log \frac{\hat{p}_i}{p_i}$ , where $p$ and $\hat{p}$ are distributions  Note that this regularization term involves the reference class distribution which will be discussed later 
Experiments
In this section, we first justify the approach when there exists unbalance in the number of labeled features or in class distribution  Then, to test the influence of $\lambda $ , we conduct some experiments with the method which incorporates the KL divergence of class distribution  Last, we evaluate our approaches in 9 commonly used text classification datasets  We set $\lambda = 5|K|$ by default in all experiments unless there is explicit declaration  The baseline we choose here is GE-FL BIBREF0 , a method based on generalization expectation criteria 
Data Preparation
We evaluate our methods on several commonly used datasets whose themes range from sentiment, web-page, science to medical and healthcare  We use bag-of-words feature and remove stopwords in the preprocess stage  Though we have labels of all documents, we do not use them during the learning process, instead, we use the label of features 
The movie dataset, in which the task is to classify the movie reviews as positive or negtive, is used for testing the proposed approaches with unbalanced labeled features, unbalanced datasets or different $\lambda $ parameters  All unbalanced datasets are constructed based on the movie dataset by randomly removing documents of the positive class  For each experiment, we conduct 10-fold cross validation 
As described in BIBREF0 , there are two ways to obtain labeled features  The first way is to use information gain  We first calculate the mutual information of all features according to the labels of the documents and select the top 20 as labeled features for each class as a feature pool  Note that using information gain requires the document label, but this is only to simulate how we human provide prior knowledge to the model  The second way is to use LDA BIBREF9 to select features  We use the same selection process as BIBREF0 , where they first train a LDA on the dataset, and then select the most probable features of each topic (sorted by $P(w_i|t_j)$ , the probability of word $w_i$ given topic $t_j$ ) 
Similar to BIBREF10 , BIBREF0 , we estimate the reference distribution of the labeled features using a heuristic strategy  If there are $|C|$ classes in total, and $n$ classes are associated with a feature $k$ , the probability that feature $k$ is related with any one of the $n$ classes is $\frac{0 9}{n}$ and with any other class is $\frac{0 1}{|C| - n}$  
Neutral features are the most frequent words after removing stop words, and their reference distributions are uniformly distributed  We use the top 10 frequent words as neutral features in all experiments 
With Unbalanced Labeled Features
In this section, we evaluate our approach when there is unbalanced knowledge on the categories to be classified  The labeled features are obtained through information gain  Two settings are chosen:
(a) We randomly select $t \in [1, 20]$ features from the feature pool for one class, and only one feature for the other  The original balanced movie dataset is used (positive:negative=1:1) 
(b) Similar to (a), but the dataset is unbalanced, obtained by randomly removing 75% positive documents (positive:negative=1:4) 
As shown in Figure 1 , Maximum entropy principle shows improvement only on the balanced case

ground_truth: the attention model, MDREA, also outperforms the best existing research results (WAP 0.690 to 0.688)

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the three regularization terms?
Context 2:  An obvious reason is that maximum entropy only favors uniform distribution 
Incorporating Neutral features performs similarly to maximum entropy since we assume that neutral words are uniformly distributed  Its accuracy decreases slowly when the number of labeled features becomes larger ( $t>4$ ) (Figure 1 (a)), suggesting that the model gradually biases to the class with more labeled features, just like GE-FL 
Incorporating the KL divergence of class distribution performs much better than GE-FL on both balanced and unbalanced datasets  This shows that it is effective to control the unbalance in labeled features and in the dataset 
With Balanced Labeled Features
We also compare with the baseline when the labeled features are balanced  Similar to the experiment above, the labeled features are obtained by information gain  Two settings are experimented with:
(a) We randomly select $t \in [1, 20]$ features from the feature pool for each class, and conduct comparisons on the original balanced movie dataset (positive:negtive=1:1) 
(b) Similar to (a), but the class distribution is unbalanced, by randomly removing 75% positive documents (positive:negative=1:4) 
Results are shown in Figure 2   When the dataset is balanced (Figure 2 (a)), there is little difference between GE-FL and our methods  The reason is that the proposed regularization terms provide no additional knowledge to the model and there is no bias in the labeled features  On the unbalanced dataset (Figure 2 (b)), incorporating KL divergence is much better than GE-FL since we provide additional knowledge(the true class distribution), but maximum entropy and neutral features are much worse because forcing the model to approach the uniform distribution misleads it 
With Unbalanced Class Distributions
Our methods are also evaluated on datasets with different unbalanced class distributions  We manually construct several movie datasets with class distributions of 1:2, 1:3, 1:4 by randomly removing 50%, 67%, 75% positive documents  The original balanced movie dataset is used as a control group  We test with both balanced and unbalanced labeled features  For the balanced case, we randomly select 10 features from the feature pool for each class, and for the unbalanced case, we select 10 features for one class, and 1 feature for the other  Results are shown in Figure 3  
Figure 3 (a) shows that when the dataset and the labeled features are both balanced, there is little difference between our methods and GE-FL(also see Figure 2 (a))  But when the class distribution becomes more unbalanced, the difference becomes more remarkable  Performance of neutral features and maximum entropy decrease significantly but incorporating KL divergence increases remarkably  This suggests if we have more accurate knowledge about class distribution, KL divergence can guide the model to the right direction 
Figure 3 (b) shows that when the labeled features are unbalanced, our methods significantly outperforms GE-FL  Incorporating KL divergence is robust enough to control unbalance both in the dataset and in labeled features while the other three methods are not so competitive 
The Influence of λ\lambda 
We present the influence of $\lambda $ on the method that incorporates KL divergence in this section  Since we simply set $\lambda = \beta |K|$ , we just tune $\beta $ here  Note that when $\beta = 0$ , the newly introduced regularization term is disappeared, and thus the model is actually GE-FL  Again, we test the method with different $\lambda $ in two settings:
(a) We randomly select $t \in [1, 20]$ features from the feature pool for one class, and only one feature for the other class  The original balanced movie dataset is used (positive:negative=1:1) 
(b) Similar to (a), but the dataset is unbalanced, obtained by randomly removing 75% positive documents (positive:negative=1:4) 
Results are shown in Figure 4   As expected, $\lambda $ reflects how strong the regularization is  The model tends to be closer to our preferences with the increasing of $\lambda $ on both cases 
Using LDA Selected Features
We compare our methods with GE-FL on all the 9 datasets in this section  Instead of using features obtained by information gain, we use LDA to select labeled features  Unlike information gain, LDA does not employ any instance labels to find labeled features  In this setting, we can build classification models without any instance annotation, but just with labeled features 
Table 1 shows that our three methods significantly outperform GE-FL  Incorporating neutral features performs better than GE-FL on 7 of the 9 datasets, maximum entropy is better on 8 datasets, and KL divergence better on 7 datasets 
LDA selects out the most predictive features as labeled features without considering the balance among classes  GE-FL does not exert any control on such an issue, so the performance is severely suffered  Our methods introduce auxiliary regularization terms to control such a bias problem and thus promote the model significantly 
Related Work
There have been much work that incorporate prior knowledge into learning, and two related lines are surveyed here  One is to use prior knowledge to label unlabeled instances and then apply a standard learning algorithm  The other is to constrain the model directly with prior knowledge 
Liu et al text manually labeled features which are highly predictive to unsupervised clustering assignments and use them to label unlabeled data  Chang et al guiding proposed constraint driven learning  They first used constraints and the learned model to annotate unlabeled instances, and then updated the model with the newly labeled data  Daumé daume2008cross proposed a self training method in which several models are trained on the same dataset, and only unlabeled instances that satisfy the cross task knowledge constraints are used in the self training process 
MaCallum et al gec proposed generalized expectation(GE) criteria which formalised the knowledge as constraint terms about the expectation of the model into the objective function Graça et al pr proposed posterior regularization(PR) framework which projects the model's posterior onto a set of distributions that satisfy the auxiliary constraints  Druck et al ge-fl explored constraints of labeled features in the framework of GE by forcing the model's predicted feature distribution to approach the reference distribution  Andrzejewski et al andrzejewski2011framework proposed a framework in which general domain knowledge can be easily incorporated into LDA  Altendorf et al altendorf2012learning explored monotonicity constraints to improve the accuracy while learning from sparse data  Chen et al chen2013leveraging tried to learn comprehensible topic models by leveraging multi-domain knowledge 
Mann and McCallum simple,generalized incorporated not only labeled features but also other knowledge like class distribution into the objective function of GE-FL  But they discussed only from the semi-supervised perspective and did not investigate into the robustness problem, unlike what we addressed in this paper 
There are also some active learning methods trying to use prior knowledge  Raghavan et al feedback proposed to use feedback on instances and features interlacedly, and demonstrated that feedback on features boosts the model much  Druck et al active proposed an active learning method which solicits labels on features rather than on instances and then used GE-FL to train the model 
Conclusion and Discussions
This paper investigates into the problem of how to leverage prior knowledge robustly in learning models  We propose three regularization terms on top of generalized expectation criteria  As demonstrated by the experimental results, the performance can be considerably improved when taking into account these factors  Comparative results show that our proposed methods is more effective and works more robustly against baselines  To the best of our knowledge, this is the first work to address the robustness problem of leveraging knowledge, and may inspire other research 
We then present more detailed discussions about the three regularization methods  Incorporating neutral features is the simplest way of regularization, which doesn't require any modification of GE-FL but just finding out some common features  But as Figure 1 (a) shows, only using neutral features are not strong enough to handle extremely unbalanced labeled features 
The maximum entropy regularization term shows the strong ability of controlling unbalance 
This method doesn't need any extra knowledge, and is thus suitable when we know nothing about the corpus  But this method assumes that the categories are uniformly distributed, which may not be the case in practice, and it will have a degraded performance if the assumption is violated (see Figure 1 (b), Figure 2 (b), Figure 3 (a)) 
The KL divergence performs much better on unbalanced corpora than other methods  The reason is that KL divergence utilizes the reference class distribution and doesn't make any assumptions  The fact suggests that additional knowledge does benefit the model 
However, the KL divergence term requires providing the true class distribution  Sometimes, we may have the exact knowledge about the true distribution, but sometimes we may not  Fortunately, the model is insensitive to the true distribution and therefore a rough estimation of the true distribution is sufficient  In our experiments, when the true class distribution is 1:2, where the reference class distribution is set to 1:1 5/1:2/1:2 5, the accuracy is 0 755/0 756/0 760 respectively  This provides us the possibility to perform simple computing on the corpus to obtain the distribution in reality  Or, we can set the distribution roughly with domain expertise 
Context 3: Introduction
Pretrained Language Models (PTLMs) such as BERT BIBREF1 have spearheaded advances on many NLP tasks  Usually, PTLMs are pretrained on unlabeled general-domain and/or mixed-domain text, such as Wikipedia, digital books or the Common Crawl corpus 
When applying PTLMs to specific domains, it can be useful to domain-adapt them  Domain adaptation of PTLMs has typically been achieved by pretraining on target-domain text  One such model is BioBERT BIBREF2, which was initialized from general-domain BERT and then pretrained on biomedical scientific publications  The domain adaptation is shown to be helpful for target-domain tasks such as biomedical Named Entity Recognition (NER) or Question Answering (QA)  On the downside, the computational cost of pretraining can be considerable: BioBERTv1 0 was adapted for ten days on eight large GPUs (see Table TABREF1), which is expensive, environmentally unfriendly, prohibitive for small research labs and students, and may delay prototyping on emerging domains 
We therefore propose a fast, CPU-only domain-adaptation method for PTLMs: We train Word2Vec BIBREF3 on target-domain text and align the resulting word vectors with the wordpiece vectors of an existing general-domain PTLM  The PTLM thus gains domain-specific lexical knowledge in the form of additional word vectors, but its deeper layers remain unchanged  Since Word2Vec and the vector space alignment are efficient models, the process requires a fraction of the resources associated with pretraining the PTLM itself, and it can be done on CPU 
In Section SECREF4, we use the proposed method to domain-adapt BERT on PubMed+PMC (the data used for BioBERTv1 0) and/or CORD-19 (Covid-19 Open Research Dataset)  We improve over general-domain BERT on eight out of eight biomedical NER tasks, using a fraction of the compute cost associated with BioBERT  In Section SECREF5, we show how to quickly adapt an existing Question Answering model to text about the Covid-19 pandemic, without any target-domain Language Model pretraining or finetuning 
Related work ::: The BERT PTLM
For our purpose, a PTLM consists of three parts: A tokenizer $\mathcal {T}_\mathrm {LM} : \mathbb {L}^+ \rightarrow \mathbb {L}_\mathrm {LM}^+$, a wordpiece embedding function $\mathcal {E}_\mathrm {LM}: \mathbb {L}_\mathrm {LM} \rightarrow \mathbb {R}^{d_\mathrm {LM}}$ and an encoder function $\mathcal {F}_\mathrm {LM}$  $\mathbb {L}_\mathrm {LM}$ is a limited vocabulary of wordpieces  All words that are not in $\mathbb {L}_\mathrm {LM}$ are tokenized into sequences of shorter wordpieces, e g , tachycardia becomes ta ##chy ##card ##ia  Given a sentence $S = [w_1, \ldots , w_T]$, tokenized as $\mathcal {T}_\mathrm {LM}(S) = [\mathcal {T}_\mathrm {LM}(w_1); \ldots ; \mathcal {T}_\mathrm {LM}(w_T)]$, $\mathcal {E}_\mathrm {LM}$ embeds every wordpiece in $\mathcal {T}_\mathrm {LM}(S)$ into a real-valued, trainable wordpiece vector  The wordpiece vectors of the entire sequence are stacked and fed into $\mathcal {F}_\mathrm {LM}$  Note that we consider position and segment embeddings to be a part of $\mathcal {F}_\mathrm {LM}$ rather than $\mathcal {E}_\mathrm {LM}$ 
In the case of BERT, $\mathcal {F}_\mathrm {LM}$ is a Transformer BIBREF4, followed by a final Feed-Forward Net  During pretraining, the Feed-Forward Net predicts the identity of masked wordpieces  When finetuning on a supervised task, it is usually replaced with a randomly initialized task-specific layer 
Related work ::: Domain-adapted PTLMs
Domain adaptation of PTLMs is typically achieved by pretraining on unlabeled target-domain text  Some examples of such models are BioBERT BIBREF2, which was pretrained on the PubMed and/or PubMed Central (PMC) corpora, SciBERT BIBREF5, which was pretrained on papers from SemanticScholar, ClinicalBERT BIBREF6, BIBREF7 and ClinicalXLNet BIBREF8, which were pretrained on clinical patient notes, and AdaptaBERT BIBREF9, which was pretrained on Early Modern English text  In most cases, a domain-adapted PTLM is initialized from a general-domain PTLM (e g , standard BERT), though BIBREF5 report better results with a model that was pretrained from scratch with a custom wordpiece vocabulary  In this paper, we focus on BioBERT, as its domain adaptation corpora are publicly available 
Related work ::: Word vectors
Word vectors are distributed representations of words that are trained on unlabeled text  Contrary to PTLMs, word vectors are non-contextual, i e , a word type is always assigned the same vector, regardless of context  In this paper, we use Word2Vec BIBREF3 to train word vectors  We will denote the Word2Vec lookup function as $\mathcal {E}_\mathrm {W2V} : \mathbb {L}_\mathrm {W2V} \rightarrow \mathbb {R}^{d_\mathrm {W2V}}$ 
Related work ::: Word vector space alignment
Word vector space alignment has most frequently been explored in the context of cross-lingual word embeddings  For instance, BIBREF10 align English and Spanish Word2Vec spaces by a simple linear transformation  BIBREF11 use a related method to align cross-lingual word vectors and multilingual BERT wordpiece vectors 
Method
In the following, we assume access to a general-domain PTLM, as described in Section SECREF2, and a corpus of unlabeled target-domain text 
Method ::: Creating new input vectors
In a first step, we train Word2Vec on the target-domain corpus  In a second step, we take the intersection of $\mathbb {L}_\mathrm {LM}$ and $\mathbb {L}_\mathrm {W2V}$  In practice, the intersection mostly contains wordpieces from $\mathbb {L}_\mathrm {LM}$ that correspond to standalone words  It also contains single characters and other noise, however, we found that filtering them does not improve alignment quality  In a third step, we use the intersection to fit an unconstrained linear transformation $\mathbf {W} \in \mathbb {R}^{d_\mathrm {LM} \times d_\mathrm {W2V}}$ via least squares:
Intuitively, $\mathbf {W}$ makes Word2Vec vectors “look like” the PTLM's native wordpiece vectors, just like cross-lingual alignment makes word vectors from one language “look like” word vectors from another language  In Table TABREF7 (top), we show examples of within-space and cross-space nearest neighbors after alignment 
Method ::: Updating the wordpiece embedding layer
Next, we redefine the wordpiece embedding layer of the PTLM  The most radical strategy would be to replace the entire layer with the aligned Word2Vec vectors:
In initial experiments, this strategy led to a drop in performance, presumably because function words are not well represented by Word2Vec, and replacing them disrupts BERT's syntactic abilities  To prevent this problem, we leave existing wordpiece vectors intact and only add new ones:
Method ::: Updating the tokenizer
In a final step, we update the tokenizer to account for the added words  Let $\mathcal {T}_\mathrm {LM}$ be the standard BERT tokenizer, and let $\hat{\mathcal {T}}_\mathrm {LM}$ be the tokenizer that treats all words in $\mathbb {L}_\mathrm {LM} \cup \mathbb {L}_\mathrm {W2V}$ as one-wordpiece tokens, while tokenizing any other words as usual 
In practice, a given word may or may not benefit from being tokenized by $\hat{\mathcal {T}}_\mathrm {LM}$ instead of $\mathcal {T}_\mathrm {LM}$  To give a concrete example, 82% of the words in the BC5CDR NER dataset that end in the suffix -ia are inside a disease entity (e g , tachycardia)  $\mathcal {T}_\mathrm {LM}$ tokenizes this word as ta ##chy ##card ##ia, thereby exposing the orthographic cue to the model  As a result, $\mathcal {T}_\mathrm {LM}$ leads to higher recall on -ia diseases  But there are many cases where wordpiece tokenization is meaningless or misleading  For instance euthymia (not a disease) is tokenized by $\mathcal {T}_\mathrm {LM}$ as e ##uth ##ym ##ia, making it likely to be classified as a disease  By contrast, $\hat{\mathcal {T}}_\mathrm {LM}$ gives euthymia a one-wordpiece representation that depends only on distributional semantics  We find that using $\hat{\mathcal {T}}_\mathrm {LM}$ improves precision on -ia diseases 
To combine these complementary strengths, we use a 50/50 mixture of $\mathcal {T}_\mathrm {LM}$-tokenization and $\hat{\mathcal {T}}_\mathrm {LM}$-tokenization when finetuning the PTLM on a task  At test time, we use both tokenizers and mean-pool the outputs  Let $o(\mathcal {T}(S))$ be some output of interest (e g , a logit), given sentence $S$ tokenized by $\mathcal {T}$  We predict:
Experiment 1: Biomedical NER
In this section, we use the proposed method to create GreenBioBERT, an inexpensive and environmentally friendly alternative to BioBERT  Recall that BioBERTv1 0 (biobert_v1 0_pubmed_pmc) was initialized from general-domain BERT (bert-base-cased) and pretrained on PubMed+PMC 
Experiment 1: Biomedical NER ::: Domain adaptation
We train Word2Vec with vector size $d_\mathrm {W2V} = d_\mathrm {LM} = 768$ on PubMed+PMC (see Appendix for details)  Then, we follow the procedure described in Section SECREF3 to update the wordpiece embedding layer and tokenizer of general-domain BERT 
Experiment 1: Biomedical NER ::: Finetuning
We finetune GreenBioBERT on the eight publicly available NER tasks used in BIBREF2  We also do reproduction experiments with general-domain BERT and BioBERTv1 0, using the same setup as our model  We average results over eight random seeds  See Appendix for details on preprocessing, training and hyperparameters 
Experiment 1: Biomedical NER ::: Results and discussion
Table TABREF7 (bottom) shows entity-level precision, recall and F1  For ease of visualization, Figure FIGREF13 shows what portion of the BioBERT – BERT F1 delta is covered  We improve over general-domain BERT on all tasks with varying effect sizes

ground_truth: a regularization term associated with neutral features, the maximum entropy of class distribution regularization term, the KL divergence between reference and predicted class distribution

a regularization term associated with neutral features,  the maximum entropy of class distribution, KL divergence between reference and predicted class distribution

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which eight NER tasks did they evaluate on?
Context 3:  Depending on the points of reference, we cover an average 52% to 60% of the BioBERT – BERT F1 delta (54% for BioBERTv1 0, 60% for BioBERTv1 1 and 52% for our reproduction experiments)  Table TABREF17 (top) shows the importance of vector space alignment: If we replace the aligned Word2Vec vectors with their non-aligned counterparts (by setting $\mathbf {W} = \mathbf {1}$) or with randomly initialized vectors, F1 drops on all tasks 
Experiment 2: Covid-19 QA
In this section, we use the proposed method to quickly adapt an existing general-domain QA model to an emerging target domain: Covid-19  Our baseline model is SQuADBERT (bert-large-uncased-whole-word-masking-finetuned-squad), a version of BERT that was finetuned on general-domain SQuAD BIBREF19  We evaluate on Deepset-AI Covid-QA, a SQuAD-style dataset with 1380 questions (see Appendix for details on data and preprocessing)  We assume that there is no target-domain finetuning data, which is a realistic setup for a new domain 
Experiment 2: Covid-19 QA ::: Domain adaptation
We train Word2Vec with vector size $d_\mathrm {W2V} = d_\mathrm {LM} = 1024$ on CORD-19 (Covid-19 Open Research Dataset) and/or PubMed+PMC  The process takes less than an hour on CORD-19 and about one day on the combined corpus, again without the need for a GPU  Then, we update SQuADBERT's wordpiece embedding layer and tokenizer, as described in Section SECREF3  We refer to the resulting model as GreenCovidSQuADBERT 
Experiment 2: Covid-19 QA ::: Results and discussion
Table TABREF17 (bottom) shows that GreenCovidSQuADBERT outperforms general-domain SQuADBERT in all metrics  Most of the improvement can be achieved with just the small CORD-19 corpus, which is more specific to the target domain (compare “Cord-19 only” and “Cord-19+PubMed+PMC”) 
Conclusion
As a reaction to the trend towards high-resource models, we have proposed an inexpensive, CPU-only method for domain-adapting Pretrained Language Models: We train Word2Vec vectors on target-domain data and align them with the wordpiece vector space of a general-domain PTLM 
On eight biomedical NER tasks, we cover over 50% of the BioBERT – BERT F1 delta, at 5% of BioBERT's domain adaptation CO$_2$ footprint and 2% of its cloud compute cost  We have also shown how to rapidly adapt an existing BERT QA model to an emerging domain – the Covid-19 pandemic – without the need for target-domain Language Model pretraining or finetuning 
We hope that our approach will benefit practitioners with limited time or resources, and that it will encourage environmentally friendlier NLP 
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Word2Vec training
We downloaded the PubMed, PMC and CORD-19 corpora from:
https://ftp ncbi nlm nih gov/pub/pmc/oa_bulk/ [20 January 2020, 68GB raw text]
https://ftp ncbi nlm nih gov/pubmed/baseline/ [20 January 2020, 24GB raw text]
https://pages semanticscholar org/coronavirus-research [17 April 2020, 2GB raw text]
We extract all abstracts and text bodies and apply the BERT basic tokenizer (a word tokenizer that standard BERT uses before wordpiece tokenization)  Then, we train CBOW Word2Vec with negative sampling  We use default parameters except for the vector size (which we set to $d_\mathrm {W2V} = d_\mathrm {LM}$) 
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 1: Biomedical NER ::: Pretrained models
General-domain BERT and BioBERTv1 0 were downloaded from:
https://storage googleapis com/bert_models/2018_10_18/cased_L-12_H-768_A-12 zip
https://github com/naver/biobert-pretrained
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 1: Biomedical NER ::: Data
We downloaded the NER datasets by following instructions on https://github com/dmis-lab/biobert#Datasets  For detailed dataset statistics, see BIBREF2 
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 1: Biomedical NER ::: Preprocessing
We cut all sentences into chunks of 30 or fewer whitespace-tokenized words (without splitting inside labeled spans)  Then, we tokenize every chunk $S$ with $\mathcal {T} = \mathcal {T}_\mathrm {LM}$ or $\mathcal {T} = \hat{\mathcal {T}}_\mathrm {LM}$ and add special tokens:
Word-initial wordpieces in $\mathcal {T}(S)$ are labeled as B(egin), I(nside) or O(utside), while non-word-initial wordpieces are labeled as X(ignore) 
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 1: Biomedical NER ::: Modeling, training and inference
We follow BIBREF2's implementation (https://github com/dmis-lab/biobert): We add a randomly initialized softmax classifier on top of the last BERT layer to predict the labels  We finetune the entire model to minimize negative log likelihood, with the standard Adam optimizer BIBREF20 and a linear learning rate scheduler (10% warmup)  Like BIBREF2, we finetune on the concatenation of the training and development set  All finetuning runs were done on a GeForce Titan X GPU (12GB) 
Since we do not have the resources for an extensive hyperparameter search, we use defaults and recommendations from the BioBERT repository: Batch size of 32, peak learning rate of $1 \cdot 10^{-5}$, and 100 epochs 
At inference time, we gather the output logits of word-initial wordpieces only  Since the number of word-initial wordpieces is the same for $\mathcal {T}_\mathrm {LM}(S)$ and $\hat{\mathcal {T}}_\mathrm {LM}(S)$, this makes mean-pooling the logits straightforward 
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 1: Biomedical NER ::: Note on our reproduction experiments
We found it easier to reproduce or exceed BIBREF2's results for general-domain BERT, compared to their results for BioBERTv1 0 (see Figure FIGREF13, main paper)  While this may be due to hyperparameters, it suggests that BioBERTv1 0 was more strongly tuned than BERT in the original BioBERT paper  This observation does not affect our conclusions, as GreenBioBERT performs better than reproduced BERT as well 
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 2: Covid-19 QA ::: Pretrained model
We downloaded the SQuADBERT baseline from:
https://huggingface co/bert-large-uncased-whole-word-masking-finetuned-squad
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 2: Covid-19 QA ::: Data
We downloaded the Deepset-AI Covid-QA dataset from:
https://github com/deepset-ai/COVID-QA/blob/master/data/question-answering/200423_covidQA json [24 April 2020]
At the time of writing, the dataset contains 1380 questions and gold answer spans  Every question is associated with one of 98 research papers (contexts)  We treat the entire dataset as a test set 
Note that there are some important differences between the dataset and SQuAD, which make the task challenging:
The contexts are full documents rather than single paragraphs  Thus, the correct answer may appear several times, often with slightly different wordings  Only a single one of the occurrences is annotated as correct, e g :
What was the prevalence of Coronavirus OC43 in community samples in Ilorin, Nigeria?
13 3% (95% CI 6 9-23 6%) # from main text
(13 3%, 10/75)  # from abstract
SQuAD gold answers are defined as the “shortest span in the paragraph that answered the question” BIBREF19, but many Covid-QA gold answers are longer and contain non-essential context, e g :
When was the Middle East Respiratory Syndrome Coronavirus isolated first?
(MERS-CoV) was first isolated in 2012, in a 60-year-old man who died in Jeddah, KSA due to severe acute pneumonia and multiple organ failure
2012,
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 2: Covid-19 QA ::: Preprocessing
We tokenize every question-context pair $(Q, C)$ with $\mathcal {T} = \mathcal {T}_\mathrm {LM}$ or $\mathcal {T} = \hat{\mathcal {T}}_\mathrm {LM}$, which yields $(\mathcal {T}(Q), \mathcal {T}(C))$  Since $\mathcal {T}(C)$ is usually too long to be digested in a single forward pass, we define a sliding window with width and stride $N = \mathrm {floor}(\frac{509 - |\mathcal {T}(Q)|}{2})$  At step $n$, the “active” window is between $a^{(l)}_n = nN$ and $a^{(r)}_n = \mathrm {min}(|C|, nN+N)$  The input is defined as:
$p^{(l)}_n$ and $p^{(r)}_n$ are chosen such that $|X^{(n)}| = 512$, and such that the active window is in the center of the input (if possible) 
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 2: Covid-19 QA ::: Modeling and inference
Feeding $X^{(n)}$ into the pretrained QA model yields start logits $\mathbf {h^{\prime }}^{(\mathrm {start}, n)} \in \mathbb {R}^{|X^{(n)}|}$ and end logits $\mathbf {h^{\prime }}^{(\mathrm {end},n)} \in \mathbb {R}^{|X^{(n)}|}$  We extract and concatenate the slices that correspond to the active windows of all steps:
Next, we map the logits from the wordpiece level to the word level  This allows us to mean-pool the outputs of $\mathcal {T}_\mathrm {LM}$ and $\hat{\mathcal {T}}_\mathrm {LM}$ even when $|\mathcal {T}_\mathrm {LM}(C)| \ne |\hat{\mathcal {T}}_\mathrm {LM}(C)|$ 
Let $c_i$ be a whitespace-delimited word in $C$  Let $\mathcal {T}(C)_{j:j+|\mathcal {T}(c_i)|}$ be the corresponding wordpieces  The start and end logits of $c_i$ are derived as:
Finally, we return the answer span $C_{k:k^{\prime }}$ that maximizes $o^{(\mathrm {start})}_k + o^{(\mathrm {end})}_{k^{\prime }}$, subject to the constraints that $k^{\prime }$ does not precede $k$ and the answer span is not longer than 500 characters 

ground_truth: BC5CDR-disease, NCBI-disease, BC5CDR-chem, BC4CHEMD, BC2GM, JNLPBA, LINNAEUS, Species-800

BC5CDR-disease, NCBI-disease, BC5CDR-chem, BC4CHEMD, BC2GM, JNLPBA, LINNAEUS, Species-800