prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Were any of the pipeline components based on deep learning models?
Context 1: Introduction
The automatic identification, extraction and representation of the information conveyed in texts is a key task nowadays  In fact, this research topic is increasing its relevance with the exponential growth of social networks and the need to have tools that are able to automatically process them BIBREF0 
Some of the domains where it is more important to be able to perform this kind of action are the juridical and legal ones  Effectively, it is crucial to have the capability to analyse open access text sources, like social nets (Twitter and Facebook, for instance), blogs, online newspapers, and to be able to extract the relevant information and represent it in a knowledge base, allowing posterior inferences and reasoning 
In the context of this work, we will present results of the R&D project Agatha, where we developed a pipeline of processes that analyses texts (in Portuguese, Spanish, or English) and is able to populate a specialized ontology BIBREF1 (related to criminal law) for the representation of events, depicted in such texts  Events are represented by objects having associated actions, agents, elements, places and time  After having populated the event ontology, we have an automatic process linking the identified entities to external referents, creating, this way, a linked data knowledge base 
It is important to point out that, having the text information represented in an ontology allows us to perform complex queries and inferences, which can detect patterns of typical criminal actions 
Another axe of innovation in this research is the development, for the Portuguese language, of a pipeline of Natural Language Processing (NLP) processes, that allows us to fully process sentences and represent their content in an ontology  Although there are several tools for the processing of the Portuguese language, the combination of all these steps in a integrated tool is a new contribution 
Moreover, we have already explored other related research path, namely author profiling BIBREF2, aggression identification BIBREF3 and hate-speech detection BIBREF4 over social media, plus statute law retrieval and entailment for Japanese BIBREF5 
The remainder of this paper is organized as follows: Section SECREF2 describes our proposed architecture together with the Portuguese modules for its computational processing  Section SECREF3 discusses different design options and Section SECREF4 provides our conclusions together with some pointers for future work 
Framework for Processing Portuguese Text
The framework for processing Portuguese texts is depicted in Fig  FIGREF2, which illustrates how relevant pieces of information are extracted from the text  Namely, input files (Portuguese texts) go through a series of modules: part-of-speech tagging, named entity recognition, dependency parsing, semantic role labeling, subject-verb-object triple extraction, and lexicon matching 
The main goal of all the modules except lexicon matching is to identify events given in the text  These events are then used to populate an ontology 
The lexicon matching, on the other hand, was created to link words that are found in the text source with the data available not only on Eurovoc BIBREF6 thesaurus but also on the EU's terminology database IATE BIBREF7 (see Section SECREF12 for details) 
Most of these modules are deeply related and are detailed in the subsequent subsections 
Framework for Processing Portuguese Text ::: Part-Of-Speech Tagging
Part-of-speech tagging happens after language detection  It labels each word with a tag that indicates its syntactic role in the sentence  For instance, a word could be a noun, verb, adjective or adverb (or other syntactic tag)  We used Freeling BIBREF8 library to provide the tags  This library resorts to a Hidden Markov Model as described by Brants BIBREF9  The end result is a tag for each word as described by the EAGLES tagset  
Framework for Processing Portuguese Text ::: Named Entity Recognition
We use the named entity recognition module after part-of-speech tagging  This module labels each part of the sentence into different categories such as "PERSON", "LOCATION", or "ORGANIZATION"  We also used Freeling to label the named entities and the details of the algorithm are shown in the paper by Carreras et al BIBREF10  Aside from the three aforementioned categories, we also extracted "DATE/TIME" and "CURRENCY" values by looking at the part-of-speech tags: date/time words have a tag of "W", while currencies have "Zm" 
Framework for Processing Portuguese Text ::: Dependency Parsing
Dependency parsing involves tagging a word based on different features to indicate if it is dependent on another word  The Freeling library also has dependency parsing models for Portuguese  Since we wanted to build a SRL (Semantic Role Labeling) module on top of the dependency parser and the current released version of Freeling does not have an SRL module for Portuguese, we trained a different Portuguese dependency parsing model that was compatible (in terms of used tags) with the available annotated 
We used the dataset from System-T BIBREF11, which has SRL tags, as well as, the other preceding tags  It was necessary to do some pre-processing and tag mapping in order to make it viable to train a Portuguese model 
We made 589 tag conversions over 14 different categories  The breakdown of tag conversions per category is given by table TABREF7  These rules can be further seen in the corresponding Github repository BIBREF12  The modified training and development datasets are also available on another Github repositorypage BIBREF13 for further research and comparison purposes 
Framework for Processing Portuguese Text ::: Semantic Role Labeling
We execute the SRL (Semantic Role Labeling) module after obtaining the word dependencies  This module aims at giving a semantic role to a syntactic constituent of a sentence  The semantic role is always in relation to a verb and these roles could either be an actor, object, time, or location, which are then tagged as A0, A1, AM-TMP, AM-LOC, respectively  We trained a model for this module on top of the dependency parser described in the previous subsection using the modified dataset from System-T  The module also needs co-reference resolution to work and, to achieve this, we adapted the Spanish co-reference modules for Portuguese, changing the words that are equivalent (in total, we changed 253 words) 
Framework for Processing Portuguese Text ::: SVO Extraction
From the yield of the SRL (Semantic Role Labeling) module, our framework can distinguish actors, actions, places, time and objects from the sentences  Utilizing this extracted data, we can distinguish subject-verb-object (SVO) triples using the SVO extraction algorithm BIBREF14  The algorithm finds, for each sentence, the verb and the tuples related to that verb using Semantic Role Labeling (subsection SECREF8)  After the extraction of SVOs from texts, they are inserted into a specific event ontology (see section SECREF12 for the creation of a knowledge base) 
Framework for Processing Portuguese Text ::: Lexicon Matching
The sole purpose of this module is to find important terms and/or concepts from the extracted text  To do this, we use Euvovoc BIBREF6, a multilingual thesaurus that was developed for and by the European Union  The Euvovoc has 21 fields and each field is further divided into a variable number of micro-thesauri  Here, due to the application of this work in the Agatha project (mentioned in Section SECREF1), we use the terms of the criminal law BIBREF15 micro-thesaurus  Further, we classified each term of the criminal law micro-thesaurus into four categories namely, actor, event, place and object  The term classification can be seen in Table TABREF11 
After the classification of these terms, we implemented two different matching algorithms between the extracted words and the criminal law micro-thesaurus terms  The first is an exact string match wherein lowercase equivalents of the words of the input sentences are matched exactly with lower case equivalents of the predefined terms  The second matching algorithm uses Levenshtein distance BIBREF16, allowing some near-matches that are close enough to the target term 
Framework for Processing Portuguese Text ::: Linked Data: Ontology, Thesaurus and Terminology
In the computer science field, an ontology can be defined has:
a formal specification of a conceptualization;
shared vocabulary and taxonomy which models a domain with the definition of objects and/or concepts and their properties and relations;
the representation of entities, ideas, and events, along with their properties and relations, according to a system of categories 
A knowledge base is one kind of repository typically used to store answers to questions or solutions to problems enabling rapid search, retrieval, and reuse, either by an ontology or directly by those requesting support  For a more detailed description of ontologies and knowledge bases, see for instance BIBREF17 
For designing the ontology adequate for our goals, we referred to the Simple Event Model (SEM) BIBREF18 as a baseline model  A pictorial representation of this ontology is given in Figure FIGREF16
Considering the criminal law domain case study, we made a few changes to the original SEM ontology  The entities of the model are:
Actor: person involved with event
Place: location of the event
Time: time of the event
Object: that actor act upon
Organization: organization involved with event
Currency: money involved with event
The proposed ontology was designed in such a manner that it can incorporate information extracted from multiple documents  In this context, suppose that the source of documents is aare a legal police department, where each document isare under the hood of a particular case/crime; furthermoreFurther, a single case can have documents from multiple languages  Now, considering case 1 has 100 documents and case 2 has 100 documents then there is not only a connection among the documents of a single case but rather among all the cases with all the combined 200 documents  In this way, the proposed method is able to produce a detailed and well-connected knowledge base 
Figure FIGREF23 shows the proposed ontology, which, in our evaluation procedure, was populated with 3121 events entries from 51 documents 
Protege BIBREF19 tool was used for creating the ontology and GraphDB BIBREF20 for populating & querying the data  GraphDB is an enterprise-ready Semantic Graph Database, compliant with W3C Standards  Semantic Graph Databases (also called RDF triplestores) provide the core infrastructure for solutions where modeling agility, data integration, relationship exploration, and cross-enterprise data publishing and consumption are important  GraphDB has a SPARQL (SQL-like query language) interface for RDF graph databases with the following types:
SELECT: returns tabular results
CONSTRUCT: creates a new RDF graph based on query results
ASK: returns "YES", if the query has a solution, otherwise "NO"
DESCRIBE: returns RDF data about a resource  This is useful when the RDF data structure in the data source is not known
INSERT: inserts triples into a graph
DELETE: deletes triples from a graph
Furthermore, we have extended the ontology BIBREF21 to connect the extracted terms with Eurovoc criminal law (discussed in subsection SECREF10) and IATE BIBREF7 terms  IATE (Interactive Terminology for Europe) is the EU's general terminology database and its aim is to provide a web-based infrastructure for all EU terminology resources, enhancing the availability and standardization of the information  The extended ontology has a number of sub-classes for Actor, Event, Object and Place classes detailed in Table TABREF30 
Discussion
We have defined a major design principle for our architecture: it should be modular and not rely on human made rules allowing, as much as possible, its independence from a specific language  In this way, its potential application to another language would be easier, simply by changing the modules or the models of specific modules  In fact, we have explored the use of already existing modules and adopted and integrated several of these tools into our pipeline 
It is important to point out that, as far as we know, there is no integrated architecture supporting the full processing pipeline for the Portuguese language  We evaluated several systems like Rembrandt BIBREF22 or LinguaKit: the former only has the initial steps of our proposal (until NER) and the later performed worse than our system 
This framework, developed within the context of the Agatha project (described in Section SECREF1) has the full processing pipeline for Portuguese texts: it receives sentences as input and outputs ontological information: a) first performs all NLP typical tasks until semantic role labelling; b) then, it extracts subject-verb-object triples; c) and, then, it performs ontology matching procedures  As a final result, the obtained output is inserted into a specialized ontology 
We are aware that each of the architecture modules can, and should, be improved but our main goal was the creation of a full working text processing pipeline for the Portuguese language 
Conclusions and Future Work
Besides the end–to–end NLP pipeline for the Portuguese language, the other main contributions of this work can be summarize as follows:
Development of an ontology for the criminal law domain;
Alignment of the Eurovoc thesaurus and IATE terminology with the ontology created;
Representation of the extracted events from texts in the linked knowledge base defined 
The obtained results support our claim that the proposed system can be used as a base tool for information extraction for the Portuguese language  Being composed by several modules, each of them with a high level of complexity, it is certain that our approach can be improved and an overall better performance can be achieved 
As future work we intend, not only to continue improving the individual modules, but also plan to extend this work to the:
automatic creation of event timelines;
incorporation in the knowledge base of information obtained from videos or pictures describing scenes relevant to criminal investigations 
Acknowledgments
The authors would like to thank COMPETE 2020, PORTUGAL 2020 Program, the European Union, and ALENTEJO 2020 for supporting this research as part of Agatha Project SI & IDT number 18022 (Intelligent analysis system of open of sources information for surveillance/crime control)  The authors would also like to thank LISP - Laboratory of Informatics, Systems and Parallelism 
Context 2: Introduction
Distributed word representations, commonly referred to as word embeddings BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , serve as elementary building blocks in the course of algorithm design for an expanding range of applications in natural language processing (NLP), including named entity recognition BIBREF4 , BIBREF5 , parsing BIBREF6 , sentiment analysis BIBREF7 , BIBREF8 , and word-sense disambiguation BIBREF9   Although the empirical utility of word embeddings as an unsupervised method for capturing the semantic or syntactic features of a certain word as it is used in a given lexical resource is well-established BIBREF10 , BIBREF11 , BIBREF12 , an understanding of what these features mean remains an open problem BIBREF13 , BIBREF14 and as such word embeddings mostly remain a black box  It is desirable to be able to develop insight into this black box and be able to interpret what it means, while retaining the utility of word embeddings as semantically-rich intermediate representations  Other than the intrinsic value of this insight, this would not only allow us to explain and understand how algorithms work BIBREF15 , but also set a ground that would facilitate the design of new algorithms in a more deliberate way 
Recent approaches to generating word embeddings (e g  BIBREF0 , BIBREF2 ) are rooted linguistically in the field of distributed semantics BIBREF16 , where words are taken to assume meaning mainly by their degree of interaction (or lack thereof) with other words in the lexicon BIBREF17 , BIBREF18   Under this paradigm, dense, continuous vector representations are learned in an unsupervised manner from a large corpus, using the word cooccurrence statistics directly or indirectly, and such an approach is shown to result in vector representations that mathematically capture various semantic and syntactic relations between words BIBREF0 , BIBREF2 , BIBREF3   However, the dense nature of the learned embeddings obfuscate the distinct concepts encoded in the different dimensions, which renders the resulting vectors virtually uninterpretable  The learned embeddings make sense only in relation to each other and their specific dimensions do not carry explicit information that can be interpreted  However, being able to interpret a word embedding would illuminate the semantic concepts implicitly represented along the various dimensions of the embedding, and reveal its hidden semantic structures 
In the literature, researchers tackled interpretability problem of the word embeddings using different approaches  Several researchers BIBREF19 , BIBREF20 , BIBREF21 proposed algorithms based on non-negative matrix factorization (NMF) applied to cooccurrence variant matrices  Other researchers suggested to obtain interpretable word vectors from existing uninterpretable word vectors by applying sparse coding BIBREF22 , BIBREF23 , by training a sparse auto-encoder to transform the embedding space BIBREF24 , by rotating the original embeddings BIBREF25 , BIBREF26 or by applying transformations based on external semantic datasets BIBREF27  
Although the above-mentioned approaches provide better interpretability that is measured using a particular method such as word intrusion test, usually the improved interpretability comes with a cost of performance in the benchmark tests such as word similarity or word analogy  One possible explanation for this performance decrease is that the proposed transformations from the original embedding space distort the underlying semantic structure constructed by the original embedding algorithm  Therefore, it can be claimed that a method that learns dense and interpretable word embeddings without inflicting any damage to the underlying semantic learning mechanism is the key to achieve both high performing and interpretable word embeddings 
Especially after the introduction of the word2vec algorithm by Mikolov BIBREF0 , BIBREF1 , there has been a growing interest in algorithms that generate improved word representations under some performance metric  Significant effort is spent on appropriately modifying the objective functions of the algorithms in order to incorporate knowledge from external resources, with the purpose of increasing the performance of the resulting word representations BIBREF28 , BIBREF29 , BIBREF30 , BIBREF31 , BIBREF32 , BIBREF33 , BIBREF34 , BIBREF35 , BIBREF36 , BIBREF37   Inspired by the line of work reported in these studies, we propose to use modified objective functions for a different purpose: learning more interpretable dense word embeddings  By doing this, we aim to incorporate semantic information from an external lexical resource into the word embedding so that the embedding dimensions are aligned along predefined concepts  This alignment is achieved by introducing a modification to the embedding learning process  In our proposed method, which is built on top of the GloVe algorithm BIBREF2 , the cost function for any one of the words of concept word-groups is modified by the introduction of an additive term to the cost function  Each embedding vector dimension is first associated with a concept  For a word belonging to any one of the word-groups representing these concepts, the modified cost term favors an increase for the value of this word's embedding vector dimension corresponding to the concept that the particular word belongs to  For words that do not belong to any one of the word-groups, the cost term is left untouched  Specifically, Roget's Thesaurus BIBREF38 , BIBREF39 is used to derive the concepts and concept word-groups to be used as the external lexical resource for our proposed method  We quantitatively demonstrate the increase in interpretability by using the measure given in BIBREF27 , BIBREF40 as well as demonstrating qualitative results  We also show that the semantic structure of the original embedding has not been harmed in the process since there is no performance loss with standard word-similarity or word-analogy tests 
The paper is organized as follows  In Section SECREF2 , we discuss previous studies related to our work under two main categories: interpretability of word embeddings and joint-learning frameworks where the objective function is modified  In Section SECREF3 , we present the problem framework and provide the formulation within the GloVe BIBREF2 algorithm setting  In Section SECREF4 where our approach is proposed, we motivate and develop a modification to the original objective function with the aim of increasing representation interpretability  In Section SECREF5 , experimental results are provided and the proposed method is quantitatively and qualitatively evaluated  Additionally, in Section SECREF5 , results demonstrating the extent to which the original semantic structure of the embedding space is affected are presented by using word-analogy and word-similarity tests  We conclude the paper in Section SECREF6  
Related Work
Methodologically, our work is related to prior studies that aim to obtain “improved” word embeddings using external lexical resources, under some performance metric  Previous work in this area can be divided into two main categories: works that i) modify the word embedding learning algorithm to incorporate lexical information, ii) operate on pre-trained embeddings with a post-processing step 
Among works that follow the first approach, BIBREF28 extend the Skip-Gram model by incorporating the word similarity relations extracted from the Paraphrase Database (PPDB) and WordNet BIBREF29 , into the Skip-Gram predictive model as an additional cost term  In BIBREF30 , the authors extend the CBOW model by considering two types of semantic information, termed relational and categorical, to be incorporated into the embeddings during training  For the former type of semantic information, the authors propose the learning of explicit vectors for the different relations extracted from a semantic lexicon such that the word pairs that satisfy the same relation are distributed more homogeneously  For the latter, the authors modify the learning objective such that some weighted average distance is minimized for words under the same semantic category  In BIBREF31 , the authors represent the synonymy and hypernymy-hyponymy relations in terms of inequality constraints, where the pairwise similarity rankings over word triplets are forced to follow an order extracted from a lexical resource  Following their extraction from WordNet, the authors impose these constraints in the form of an additive cost term to the Skip-Gram formulation  Finally, BIBREF32 builds on top of the GloVe algorithm by introducing a regularization term to the objective function that encourages the vector representations of similar words as dictated by WordNet to be similar as well 
Turning our attention to the post-processing approach for enriching word embeddings with external lexical knowledge, BIBREF33 has introduced the retrofitting algorithm that acts on pre-trained embeddings such as Skip-Gram or GloVe  The authors propose an objective function that aims to balance out the semantic information captured in the pre-trained embeddings with the constraints derived from lexical resources such as WordNet, PPDB and FrameNet  One of the models proposed in BIBREF34 extends the retrofitting approach to incorporate the word sense information from WordNet  Similarly, BIBREF35 creates multi-sense embeddings by gathering the word sense information from a lexical resource and learning to decompose the pre-trained embeddings into a convex combination of sense embeddings  In BIBREF36 , the authors focus on improving word embeddings for capturing word similarity, as opposed to mere relatedness  To this end, they introduce the counter-fitting technique which acts on the input word vectors such that synonymous words are attracted to one another whereas antonymous words are repelled, where the synonymy-antonymy relations are extracted from a lexical resource  More recently, the ATTRACT-REPEL algorithm proposed by BIBREF37 improves on counter-fitting by a formulation which imparts the word vectors with external lexical information in mini-batches 
Most of the studies discussed above ( BIBREF30 , BIBREF31 , BIBREF32 , BIBREF33 , BIBREF34 , BIBREF36 , BIBREF37 ) report performance improvements in benchmark tests such as word similarity or word analogy, while BIBREF29 uses a different analysis method (mean reciprocal rank)  In sum, the literature is rich with studies aiming to obtain word embeddings that perform better under specific performance metrics  However, less attention has been directed to the issue of interpretability of the word embeddings  In the literature, the problem of interpretability has been tackled using different approaches  BIBREF19 proposed non-negative matrix factorization (NMF) for learning sparse, interpretable word vectors from co-occurrence variant matrices where the resulting vector space is called non-negative sparse embeddigns (NNSE)  However, since NMF methods require maintaining a global matrix for learning, they suffer from memory and scale issue  This problem has been addressed in BIBREF20 where an online method of learning interpretable word embeddings from corpora using a modified version of skip-gram model BIBREF0 is proposed  As a different approach, BIBREF21 combined text-based similarity information among words with brain activity based similarity information to improve interpretability using joint non-negative sparse embedding (JNNSE) 
A common alternative approach for learning interpretable embeddings is to learn transformations that map pre-trained state-of-the-art embeddings to new interpretable semantic spaces  To obtain sparse, higher dimensional and more interpretable vector spaces, BIBREF22 and BIBREF23 use sparse coding on conventional dense word embeddings  However, these methods learn the projection vectors that are used for the transformation from the word embeddings without supervision  For this reason, labels describing the corresponding semantic categories cannot be provided  An alternative approach was proposed in BIBREF25 , where orthogonal transformations were utilized to increase interpretability while preserving the performance of the underlying embedding  However, BIBREF25 has also shown that total interpretability of an embedding is kept constant under any orthogonal transformation and it can only be redistributed across the dimensions  Rotation algorithms based on exploratory factor analysis (EFA) to preserve the performance of the original word embeddings while improving their interpretability was proposed in BIBREF26   BIBREF24 proposed to deploy a sparse auto-encoder using pre-trained dense word embeddings to improve interpretability  More detailed investigation of semantic structure and interpretability of word embeddings can be found in BIBREF27 , where a metric was proposed to quantitatively measure the degree of interpretability already present in the embedding vector spaces 
Previous works on interpretability mentioned above, except BIBREF21 , BIBREF27 and our proposed method, do not need external resources, utilization of which has both advantages and disadvantages  Methods that do not use external resources require fewer resources but they also lack the aid of information extracted from these resources 
Problem Description
For the task of unsupervised word embedding extraction, we operate on a discrete collection of lexical units (words) INLINEFORM0 that is part of an input corpus INLINEFORM1 , with number of tokens INLINEFORM2 , sourced from a vocabulary INLINEFORM3 of size INLINEFORM4   In the setting of distributional semantics, the objective of a word embedding algorithm is to maximize some aggregate utility over the entire corpus so that some measure of “closeness” is maximized for pairs of vector representations INLINEFORM14 for words which, on the average, appear in proximity to one another  In the GloVe algorithm BIBREF2 , which we base our improvements upon, the following objective function is considered: DISPLAYFORM0
In ( EQREF6 ), INLINEFORM0 and INLINEFORM1 stand for word and context vector representations, respectively, for words INLINEFORM2 and INLINEFORM3 , while INLINEFORM4 represents the (possibly weighted) cooccurrence count for the word pair INLINEFORM5   Intuitively, ( EQREF6 ) represents the requirement that if some word INLINEFORM6 occurs often enough in the context (or vicinity) of another word INLINEFORM7 , then the corresponding word representations should have a large enough inner product in keeping with their large INLINEFORM8 value, up to some bias terms INLINEFORM9 ; and vice versa  INLINEFORM10 in ( EQREF6 ) is used as a discounting factor that prohibits rare cooccurrences from disproportionately influencing the resulting embeddings 
The objective ( EQREF6 ) is minimized using stochastic gradient descent by iterating over the matrix of cooccurrence records INLINEFORM0   In the GloVe algorithm, for a given word INLINEFORM1 , the final word representation is taken to be the average of the two intermediate vector representations obtained from ( EQREF6 ); i e, INLINEFORM2   In the next section, we detail the enhancements made to ( EQREF6 ) for the purposes of enhanced interpretability, using the aforementioned framework as our basis 
Imparting Interpretability
Our approach falls into a joint-learning framework where the distributional information extracted from the corpus is allowed to fuse with the external lexicon-based information  Word-groups extracted from Roget's Thesaurus are directly mapped to individual dimensions of word embeddings  Specifically, the vector representations of words that belong to a particular group are encouraged to have deliberately increased values in a particular dimension that corresponds to the word-group under consideration  This can be achieved by modifying the objective function of the embedding algorithm to partially influence vector representation distributions across their dimensions over an input vocabulary  To do this, we propose the following modification to the GloVe objective in ( EQREF6 ): rCl J = i,j=1V f(Xij)[ (wiTwj + bi + bj -Xij)2
+ k(l=1D INLINEFORM0 iFl g(wi,l) + l=1D INLINEFORM1 j Fl g(wj,l) ) ]  In ( SECREF4 ), INLINEFORM2 denotes the indices for the elements of the INLINEFORM3 th concept word-group which we wish to assign in the vector dimension INLINEFORM4   The objective ( SECREF4 ) is designed as a mixture of two individual cost terms: the original GloVe cost term along with a second term that encourages embedding vectors of a given concept word-group to achieve deliberately increased values along an associated dimension INLINEFORM5   The relative weight of the second term is controlled by the parameter INLINEFORM6   The simultaneous minimization of both objectives ensures that words that are similar to, but not included in, one of these concept word-groups are also “nudged” towards the associated dimension INLINEFORM7   The trained word vectors are thus encouraged to form a distribution where the individual vector dimensions align with certain semantic concepts represented by a collection of concept word-groups, one assigned to each vector dimension  To facilitate this behaviour, ( SECREF4 ) introduces a monotone decreasing function INLINEFORM8 defined as INLINEFORM9
which serves to increase the total cost incurred if the value of the INLINEFORM0 th dimension for the two vector representations INLINEFORM1 and INLINEFORM2 for a concept word INLINEFORM3 with INLINEFORM4 fails to be large enough  INLINEFORM5 is also shown in Fig  FIGREF7  
The objective ( SECREF4 ) is minimized using stochastic gradient descent over the cooccurrence records INLINEFORM0   Intuitively, the terms added to ( SECREF4 ) in comparison with ( EQREF6 ) introduce the effect of selectively applying a positive step-type input to the original descent updates of ( EQREF6 ) for concept words along their respective vector dimensions, which influences the dimension value in the positive direction  The parameter INLINEFORM1 in ( SECREF4 ) allows for the adjustment of the magnitude of this influence as needed 
In the next section, we demonstrate the feasibility of this approach by experiments with an example collection of concept word-groups extracted from Roget's Thesaurus 
Experiments and Results
We first identified 300 concepts, one for each dimension of the 300-dimensional vector representation, by employing Roget's Thesaurus  This thesaurus follows a tree structure which starts with a Root node that contains all the words and phrases in the thesaurus  The root node is successively split into Classes and Sections, which are then (optionally) split into Subsections of various depths, finally ending in Categories, which constitute the smallest unit of word/phrase collections in the structure  The actual words and phrases descend from these Categories, and make up the leaves of the tree structure  We note that a given word typically appears in multiple categories corresponding to the different senses of the word  We constructed concept word-groups from Roget's Thesaurus as follows: We first filtered out the multi-word phrases and the relatively obscure terms from the thesaurus  The obscure terms were identified by checking them against a vocabulary extracted from Wikipedia  We then obtained 300 word-groups as the result of a partitioning operation applied to the subtree that ends with categories as its leaves  The partition boundaries, hence the resulting word-groups, can be chosen in many different ways  In our proposed approach, we have chosen to determine this partitioning by traversing this tree structure from the root node in breadth-first order, and by employing a parameter INLINEFORM0 for the maximum size of a node  Here, the size of a node is defined as the number of unique words that ever-descend from that node  During the traversal, if the size of a given node is less than this threshold, we designate the words that ultimately descend from that node as a concept word-group  Otherwise, if the node has children, we discard the node, and queue up all its children for further consideration  If this node does not have any children, on the other hand, the node is truncated to INLINEFORM1 elements with the highest frequency-ranks, and the resulting words are designated as a concept word-group  We note that the choice of INLINEFORM2 greatly affects the resulting collection of word-groups: Excessively large values result in few word-groups that greatly overlap with one another, while overly small values result in numerous tiny word-groups that fail to adequately represent a concept  We experimentally determined that a INLINEFORM3 value of 452 results in the most healthy number of relatively large word-groups (113 groups with size INLINEFORM4 100), while yielding a preferably small overlap amongst the resulting word-groups (with average overlap size not exceeding 3 words)  A total of 566 word-groups were thus obtained  259 smallest word-groups (with size INLINEFORM5 38) were discarded to bring down the number of word-groups to 307  Out of these, 7 groups with the lowest median frequency-rank were further discarded, which yields the final 300 concept word-groups used in the experiments  We present some of the resulting word-groups in Table TABREF9  
By using the concept word-groups, we have trained the GloVe algorithm with the proposed modification given in Section SECREF4 on a snapshot of English Wikipedia measuring 8GB in size, with the stop-words filtered out  Using the parameters given in Table TABREF10 , this resulted in a vocabulary size of 287,847  For the weighting parameter in Eq  SECREF4 , we used a value of INLINEFORM0   The algorithm was trained over 20 iterations  The GloVe algorithm without any modifications was also trained as a baseline with the same parameters  In addition to the original GloVe algorithm, we compare our proposed method with previous studies that aim to obtain interpretable word vectors

ground_truth: No

No

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Do they report results only on English data?
Context 2:  We train the improved projected gradient model proposed in BIBREF20 to obtain word vectors (called OIWE-IPG) using the same corpus we use to train GloVe and our proposed method  Using the methods proposed in BIBREF23 , BIBREF26 , BIBREF24 on our baseline GloVe embeddings, we obtain SOV, SPINE and Parsimax (orthogonal) word representations, respectively  We train all the models with the proposed parameters  However, in BIBREF26 , the authors show results for a relatively small vocabulary of 15,000 words  When we trained their model on our baseline GloVe embeddings with a large vocabulary of size 287,847, the resulting vectors performed significantly poor on word similarity tasks compared to the results presented in their paper  In addition, Parsimax (orthogonal) word vectors obtained using method in BIBREF26 are nearly identical to the baseline vectors (i e  learned orthogonal transformation matrix is very close to identity)  Therefore, Parsimax (orthogonal) yields almost same results with baseline vectors in all evaluations  We evaluate the interpretability of the resulting embeddings qualitatively and quantitatively  We also test the performance of the embeddings on word similarity and word analogy tests 
In our experiments, vocabulary size is close to 300,000 while only 16,242 unique words of the vocabulary are present in the concept groups  Furthermore, only dimensions that correspond to the concept group of the word will be updated due to the additional cost term  Given that these concept words can belong to multiple concept groups (2 on average), only 33,319 parameters are updated  There are 90 million individual parameters present for the 300,000 word vectors of size 300  Of these parameters, only approximately 33,000 are updated by the additional cost term 
Qualitative Evaluation for Interpretability
In Fig  FIGREF13 , we demonstrate the particular way in which the proposed algorithm ( SECREF4 ) influences the vector representation distributions  Specifically, we consider, for illustration, the 32nd dimension values for the original GloVe algorithm and our modified version, restricting the plots to the top-1000 words with respect to their frequency ranks for clarity of presentation  In Fig  FIGREF13 , the words in the horizontal axis are sorted in descending order with respect to the values at the 32nd dimension of their word embedding vectors coming from the original GloVe algorithm  The dimension values are denoted with blue and red/green markers for the original and the proposed algorithms, respectively  Additionally, the top-50 words that achieve the greatest 32nd dimension values among the considered 1000 words are emphasized with enlarged markers, along with text annotations  In the presented simulation of the proposed algorithm, the 32nd dimension values are encoded with the concept JUDGMENT, which is reflected as an increase in the dimension values for words such as committee, academy, and article  We note that these words (red) are not part of the pre-determined word-group for the concept JUDGMENT, in contrast to words such as award, review and account (green) which are  This implies that the increase in the corresponding dimension values seen for these words is attributable to the joint effect of the first term in ( SECREF4 ) which is inherited from the original GloVe algorithm, in conjunction with the remaining terms in the proposed objective expression ( SECREF4 )  This experiment illustrates that the proposed algorithm is able to impart the concept of JUDGMENT on its designated vector dimension above and beyond the supplied list of words belonging to the concept word-group for that dimension  We also present the list of words with the greatest dimension value for the dimensions 11, 13, 16, 31, 36, 39, 41, 43 and 79 in Table TABREF11   These dimensions are aligned/imparted with the concepts that are given in the column headers  In Table TABREF11 , the words that are highlighted with green denote the words that exist in the corresponding word-group obtained from Roget's Thesaurus (and are thus explicitly forced to achieve increased dimension values), while the red words denote the words that achieve increased dimension values by virtue of their cooccurrence statistics with the thesaurus-based words (indirectly, without being explicitly forced)  This again illustrates that a semantic concept can indeed be coded to a vector dimension provided that a sensible lexical resource is used to guide semantically related words to the desired vector dimension via the proposed objective function in ( SECREF4 )  Even the words that do not appear in, but are semantically related to, the word-groups that we formed using Roget's Thesaurus, are indirectly affected by the proposed algorithm  They also reflect the associated concepts at their respective dimensions even though the objective functions for their particular vectors are not modified  This point cannot be overemphasized  Although the word-groups extracted from Roget's Thesaurus impose a degree of supervision to the process, the fact that the remaining words in the entire vocabulary are also indirectly affected makes the proposed method a semi-supervised approach that can handle words that are not in these chosen word-groups  A qualitative example of this result can be seen in the last column of Table TABREF11   It is interesting to note the appearance of words such as guerilla, insurgency, mujahideen, Wehrmacht and Luftwaffe in addition to the more obvious and straightforward army, soldiers and troops, all of which are not present in the associated word-group WARFARE 
Most of the dimensions we investigated exhibit similar behaviour to the ones presented in Table TABREF11   Thus generally speaking, we can say that the entries in Table TABREF11 are representative of the great majority  However, we have also specifically looked for dimensions that make less sense and determined a few such dimensions which are relatively less satisfactory  These less satisfactory examples are given in Table TABREF14   These examples are also interesting in that they shed insight into the limitations posed by polysemy and existence of very rare outlier words 
Quantitative Evaluation for Interpretability
One of the main goals of this study is to improve the interpretability of dense word embeddings by aligning the dimensions with predefined concepts from a suitable lexicon  A quantitative measure is required to reliably evaluate the achieved improvement  One of the methods proposed to measure the interpretability is the word intrusion test BIBREF41   But, this method is expensive to apply since it requires evaluations from multiple human evaluators for each embedding dimension  In this study, we use a semantic category-based approach based on the method and category dataset (SEMCAT) introduced in BIBREF27 to quantify interpretability  Specifically, we apply a modified version of the approach presented in BIBREF40 in order to consider possible sub-groupings within the categories in SEMCAT  Interpretability scores are calculated using Interpretability Score (IS) as given below:
DISPLAYFORM0
In ( EQREF17 ), INLINEFORM0 and INLINEFORM1 represents the interpretability scores in the positive and negative directions of the INLINEFORM2 dimension ( INLINEFORM3 , INLINEFORM4 number of dimensions in the embedding space) of word embedding space for the INLINEFORM5 category ( INLINEFORM6 , INLINEFORM7 is number of categories in SEMCAT, INLINEFORM8 ) in SEMCAT respectively  INLINEFORM9 is the set of words in the INLINEFORM10 category in SEMCAT and INLINEFORM11 is the number of words in INLINEFORM12   INLINEFORM13 corresponds to the minimum number of words required to construct a semantic category (i e  represent a concept)  INLINEFORM14 represents the set of INLINEFORM15 words that have the highest ( INLINEFORM16 ) and lowest ( INLINEFORM17 ) values in INLINEFORM18 dimension of the embedding space  INLINEFORM19 is the intersection operator and INLINEFORM20 is the cardinality operator (number of elements) for the intersecting set  In ( EQREF17 ), INLINEFORM21 gives the interpretability score for the INLINEFORM22 dimension and INLINEFORM23 gives the average interpretability score of the embedding space 
Fig  FIGREF18 presents the measured average interpretability scores across dimensions for original GloVe embeddings, for the proposed method and for the other four methods we compare, along with a randomly generated embedding  Results are calculated for the parameters INLINEFORM0 and INLINEFORM1   Our proposed method significantly improves the interpretability for all INLINEFORM2 compared to the original GloVe approach  Our proposed method is second to only SPINE in increasing interpretability  However, as we will experimentally demonstrate in the next subsection, in doing this, SPINE almost entirely destroys the underlying semantic structure of the word embeddings, which is the primary function of a word embedding 
The proposed method and interpretability measurements are both based on utilizing concepts represented by word-groups  Therefore it is expected that there will be higher interpretability scores for some of the dimensions for which the imparted concepts are also contained in SEMCAT  However, by design, word groups that they use are formed by using different sources and are independent  Interpretability measurements use SEMCAT while our proposed method utilizes Roget's Thesaurus 
Intrinsic Evaluation of the Embeddings
It is necessary to show that the semantic structure of the original embedding has not been damaged or distorted as a result of aligning the dimensions with given concepts, and that there is no substantial sacrifice involved from the performance that can be obtained with the original GloVe  To check this, we evaluate performances of the proposed embeddings on word similarity BIBREF42 and word analogy BIBREF0 tests  We compare the results with the original embeddings and the three alternatives excluding Parsimax BIBREF26 since orthogonal transformations will not affect the performance of the original embeddings on these tests 
Word similarity test measures the correlation between word similarity scores obtained from human evaluation (i e  true similarities) and from word embeddings (usually using cosine similarity)  In other words, this test quantifies how well the embedding space reflects human judgements in terms of similarities between different words  The correlation scores for 13 different similarity test sets are reported in Table TABREF20   We observe that, let alone a reduction in performance, the obtained scores indicate an almost uniform improvement in the correlation values for the proposed algorithm, outperforming all the alternatives in almost all test sets  Categories from Roget's thesaurus are groupings of words that are similar in some sense which the original embedding algorithm may fail to capture  These test results signify that the semantic information injected into the algorithm by the additional cost term is significant enough to result in a measurable improvement  It should also be noted that scores obtained by SPINE is unacceptably low on almost all tests indicating that it has achieved its interpretability performance at the cost of losing its semantic functions 
Word analogy test is introduced in BIBREF1 and looks for the answers of the questions that are in the form of "X is to Y, what Z is to ?" by applying simple arithmetic operations to vectors of words X, Y and Z  We present precision scores for the word analogy tests in Table TABREF21   It can be seen that the alternative approaches that aim to improve interpretability, have poor performance on the word analogy tests  However, our proposed method has comparable performance with the original GloVe embeddings  Our method outperforms GloVe in semantic analogy test set and in overall results, while GloVe performs slightly better in syntactic test set  This comparable performance is mainly due to the cost function of our proposed method that includes the original objective of the GloVe 
To investigate the effect of the additional cost term on the performance improvement in the semantic analogy test, we present Table TABREF22   In particular, we present results for the cases where i) all questions in the dataset are considered, ii) only the questions that contains at least one concept word are considered, iii) only the questions that consist entirely of concept words are considered  We note specifically that for the last case, only a subset of the questions under the semantic category family txt ended up being included  We observe that for all three scenarios, our proposed algorithm results in an improvement in the precision scores  However, the greatest performance increase is seen for the last scenario, which underscores the extent to which the semantic features captured by embeddings can be improved with a reasonable selection of the lexical resource from which the concept word-groups were derived 
Conclusion
We presented a novel approach to impart interpretability into word embeddings  We achieved this by encouraging different dimensions of the vector representation to align with predefined concepts, through the addition of an additional cost term in the optimization objective of the GloVe algorithm that favors a selective increase for a pre-specified input of concept words along each dimension 
We demonstrated the efficacy of this approach by applying qualitative and quantitative evaluations for interpretability  We also showed via standard word-analogy and word-similarity tests that the semantic coherence of the original vector space is preserved, even slightly improved  We have also performed and reported quantitative comparisons with several other methods for both interpretabilty increase and preservation of semantic coherence  Upon inspection of Fig  FIGREF18 and Tables TABREF20 , TABREF21 , and TABREF22 altogether, it should be noted that our proposed method achieves both of the objectives simultaneously, increased interpretability and preservation of the intrinsic semantic structure 
An important point was that, while it is expected for words that are already included in the concept word-groups to be aligned together since their dimensions are directly updated with the proposed cost term, it was also observed that words not in these groups also aligned in a meaningful manner without any direct modification to their cost function  This indicates that the cost term we added works productively with the original cost function of GloVe to handle words that are not included in the original concept word-groups, but are semantically related to those word-groups  The underlying mechanism can be explained as follows  While the outside lexical resource we introduce contains a relatively small number of words compared to the total number of words, these words and the categories they represent have been carefully chosen and in a sense, "densely span" all the words in the language  By saying "span", we mean they cover most of the concepts and ideas in the language without leaving too many uncovered areas  With "densely" we mean all areas are covered with sufficient strength  In other words, this subset of words is able to constitute a sufficiently strong skeleton, or scaffold  Now remember that GloVe works to align or bring closer related groups of words, which will include words from the lexical source  So the joint action of aligning the words with the predefined categories (introduced by us) and aligning related words (handled by GloVe) allows words not in the lexical groups to also be aligned meaningfully  We may say that the non-included words are "pulled along" with the included words by virtue of the "strings" or "glue" that is provided by GloVe  In numbers, the desired effect is achieved by manipulating less than only 0 05% of parameters of the entire word vectors  Thus, while there is a degree of supervision coming from the external lexical resource, the rest of the vocabulary is also aligned indirectly in an unsupervised way  This may be the reason why, unlike earlier proposed approaches, our method is able to achieve increasing interpretability without destroying underlying semantic structure, and consequently without sacrificing performance in benchmark tests 
Upon inspecting the 2nd column of Table TABREF14 , where qualitative results for concept TASTE are presented, another insight regarding the learning mechanism of our proposed approach can be made  Here it seems understandable that our proposed approach, along with GloVe, brought together the words taste and polish, and then the words Polish and, for instance, Warsaw are brought together by GloVe  These examples are interesting in that they shed insight into how GloVe works and the limitations posed by polysemy  It should be underlined that the present approach is not totally incapable of handling polysemy, but cannot do so perfectly  Since related words are being clustered, sufficiently well-connected words that do not meaningfully belong along with others will be appropriately "pulled away" from that group by several words, against the less effective, inappropriate pull of a particular word  Even though polish with lowercase "p" belongs where it is, it is attracting Warsaw to itself through polysemy and this is not meaningful  Perhaps because Warsaw is not a sufficiently well-connected word, it ends being dragged along, although words with greater connectedness to a concept group might have better resisted such inappropriate attractions 
In this study, we used the GloVe algorithm as the underlying dense word embedding scheme to demonstrate our approach  However, we stress that it is possible for our approach to be extended to other word embedding algorithms which have a learning routine consisting of iterations over cooccurrence records, by making suitable adjustments in the objective function  Since word2vec model is also based on the coocurrences of words in a sliding window through a large corpus, we expect that our approach can also be applied to word2vec after making suitable adjustments, which can be considered as an immediate future work for our approach  Although the semantic concepts are encoded in only one direction (positive) within the embedding dimensions, it might be beneficial to pursue future work that also encodes opposite concepts, such as good and bad, in two opposite directions of the same dimension 
The proposed methodology can also be helpful in computational cross-lingual studies, where the similarities are explored across the vector spaces of different languages BIBREF43 , BIBREF44  
Context 3: Introduction
Pretrained Language Models (PTLMs) such as BERT BIBREF1 have spearheaded advances on many NLP tasks  Usually, PTLMs are pretrained on unlabeled general-domain and/or mixed-domain text, such as Wikipedia, digital books or the Common Crawl corpus 
When applying PTLMs to specific domains, it can be useful to domain-adapt them  Domain adaptation of PTLMs has typically been achieved by pretraining on target-domain text  One such model is BioBERT BIBREF2, which was initialized from general-domain BERT and then pretrained on biomedical scientific publications  The domain adaptation is shown to be helpful for target-domain tasks such as biomedical Named Entity Recognition (NER) or Question Answering (QA)  On the downside, the computational cost of pretraining can be considerable: BioBERTv1 0 was adapted for ten days on eight large GPUs (see Table TABREF1), which is expensive, environmentally unfriendly, prohibitive for small research labs and students, and may delay prototyping on emerging domains 
We therefore propose a fast, CPU-only domain-adaptation method for PTLMs: We train Word2Vec BIBREF3 on target-domain text and align the resulting word vectors with the wordpiece vectors of an existing general-domain PTLM  The PTLM thus gains domain-specific lexical knowledge in the form of additional word vectors, but its deeper layers remain unchanged  Since Word2Vec and the vector space alignment are efficient models, the process requires a fraction of the resources associated with pretraining the PTLM itself, and it can be done on CPU 
In Section SECREF4, we use the proposed method to domain-adapt BERT on PubMed+PMC (the data used for BioBERTv1 0) and/or CORD-19 (Covid-19 Open Research Dataset)  We improve over general-domain BERT on eight out of eight biomedical NER tasks, using a fraction of the compute cost associated with BioBERT  In Section SECREF5, we show how to quickly adapt an existing Question Answering model to text about the Covid-19 pandemic, without any target-domain Language Model pretraining or finetuning 
Related work ::: The BERT PTLM
For our purpose, a PTLM consists of three parts: A tokenizer $\mathcal {T}_\mathrm {LM} : \mathbb {L}^+ \rightarrow \mathbb {L}_\mathrm {LM}^+$, a wordpiece embedding function $\mathcal {E}_\mathrm {LM}: \mathbb {L}_\mathrm {LM} \rightarrow \mathbb {R}^{d_\mathrm {LM}}$ and an encoder function $\mathcal {F}_\mathrm {LM}$  $\mathbb {L}_\mathrm {LM}$ is a limited vocabulary of wordpieces  All words that are not in $\mathbb {L}_\mathrm {LM}$ are tokenized into sequences of shorter wordpieces, e g , tachycardia becomes ta ##chy ##card ##ia  Given a sentence $S = [w_1, \ldots , w_T]$, tokenized as $\mathcal {T}_\mathrm {LM}(S) = [\mathcal {T}_\mathrm {LM}(w_1); \ldots ; \mathcal {T}_\mathrm {LM}(w_T)]$, $\mathcal {E}_\mathrm {LM}$ embeds every wordpiece in $\mathcal {T}_\mathrm {LM}(S)$ into a real-valued, trainable wordpiece vector  The wordpiece vectors of the entire sequence are stacked and fed into $\mathcal {F}_\mathrm {LM}$  Note that we consider position and segment embeddings to be a part of $\mathcal {F}_\mathrm {LM}$ rather than $\mathcal {E}_\mathrm {LM}$ 
In the case of BERT, $\mathcal {F}_\mathrm {LM}$ is a Transformer BIBREF4, followed by a final Feed-Forward Net  During pretraining, the Feed-Forward Net predicts the identity of masked wordpieces  When finetuning on a supervised task, it is usually replaced with a randomly initialized task-specific layer 
Related work ::: Domain-adapted PTLMs
Domain adaptation of PTLMs is typically achieved by pretraining on unlabeled target-domain text  Some examples of such models are BioBERT BIBREF2, which was pretrained on the PubMed and/or PubMed Central (PMC) corpora, SciBERT BIBREF5, which was pretrained on papers from SemanticScholar, ClinicalBERT BIBREF6, BIBREF7 and ClinicalXLNet BIBREF8, which were pretrained on clinical patient notes, and AdaptaBERT BIBREF9, which was pretrained on Early Modern English text  In most cases, a domain-adapted PTLM is initialized from a general-domain PTLM (e g , standard BERT), though BIBREF5 report better results with a model that was pretrained from scratch with a custom wordpiece vocabulary  In this paper, we focus on BioBERT, as its domain adaptation corpora are publicly available 
Related work ::: Word vectors
Word vectors are distributed representations of words that are trained on unlabeled text  Contrary to PTLMs, word vectors are non-contextual, i e , a word type is always assigned the same vector, regardless of context  In this paper, we use Word2Vec BIBREF3 to train word vectors  We will denote the Word2Vec lookup function as $\mathcal {E}_\mathrm {W2V} : \mathbb {L}_\mathrm {W2V} \rightarrow \mathbb {R}^{d_\mathrm {W2V}}$ 
Related work ::: Word vector space alignment
Word vector space alignment has most frequently been explored in the context of cross-lingual word embeddings  For instance, BIBREF10 align English and Spanish Word2Vec spaces by a simple linear transformation  BIBREF11 use a related method to align cross-lingual word vectors and multilingual BERT wordpiece vectors 
Method
In the following, we assume access to a general-domain PTLM, as described in Section SECREF2, and a corpus of unlabeled target-domain text 
Method ::: Creating new input vectors
In a first step, we train Word2Vec on the target-domain corpus  In a second step, we take the intersection of $\mathbb {L}_\mathrm {LM}$ and $\mathbb {L}_\mathrm {W2V}$  In practice, the intersection mostly contains wordpieces from $\mathbb {L}_\mathrm {LM}$ that correspond to standalone words  It also contains single characters and other noise, however, we found that filtering them does not improve alignment quality  In a third step, we use the intersection to fit an unconstrained linear transformation $\mathbf {W} \in \mathbb {R}^{d_\mathrm {LM} \times d_\mathrm {W2V}}$ via least squares:
Intuitively, $\mathbf {W}$ makes Word2Vec vectors “look like” the PTLM's native wordpiece vectors, just like cross-lingual alignment makes word vectors from one language “look like” word vectors from another language  In Table TABREF7 (top), we show examples of within-space and cross-space nearest neighbors after alignment 
Method ::: Updating the wordpiece embedding layer
Next, we redefine the wordpiece embedding layer of the PTLM  The most radical strategy would be to replace the entire layer with the aligned Word2Vec vectors:
In initial experiments, this strategy led to a drop in performance, presumably because function words are not well represented by Word2Vec, and replacing them disrupts BERT's syntactic abilities  To prevent this problem, we leave existing wordpiece vectors intact and only add new ones:
Method ::: Updating the tokenizer
In a final step, we update the tokenizer to account for the added words  Let $\mathcal {T}_\mathrm {LM}$ be the standard BERT tokenizer, and let $\hat{\mathcal {T}}_\mathrm {LM}$ be the tokenizer that treats all words in $\mathbb {L}_\mathrm {LM} \cup \mathbb {L}_\mathrm {W2V}$ as one-wordpiece tokens, while tokenizing any other words as usual 
In practice, a given word may or may not benefit from being tokenized by $\hat{\mathcal {T}}_\mathrm {LM}$ instead of $\mathcal {T}_\mathrm {LM}$  To give a concrete example, 82% of the words in the BC5CDR NER dataset that end in the suffix -ia are inside a disease entity (e g , tachycardia)  $\mathcal {T}_\mathrm {LM}$ tokenizes this word as ta ##chy ##card ##ia, thereby exposing the orthographic cue to the model  As a result, $\mathcal {T}_\mathrm {LM}$ leads to higher recall on -ia diseases  But there are many cases where wordpiece tokenization is meaningless or misleading  For instance euthymia (not a disease) is tokenized by $\mathcal {T}_\mathrm {LM}$ as e ##uth ##ym ##ia, making it likely to be classified as a disease  By contrast, $\hat{\mathcal {T}}_\mathrm {LM}$ gives euthymia a one-wordpiece representation that depends only on distributional semantics  We find that using $\hat{\mathcal {T}}_\mathrm {LM}$ improves precision on -ia diseases 
To combine these complementary strengths, we use a 50/50 mixture of $\mathcal {T}_\mathrm {LM}$-tokenization and $\hat{\mathcal {T}}_\mathrm {LM}$-tokenization when finetuning the PTLM on a task  At test time, we use both tokenizers and mean-pool the outputs  Let $o(\mathcal {T}(S))$ be some output of interest (e g , a logit), given sentence $S$ tokenized by $\mathcal {T}$  We predict:
Experiment 1: Biomedical NER
In this section, we use the proposed method to create GreenBioBERT, an inexpensive and environmentally friendly alternative to BioBERT  Recall that BioBERTv1 0 (biobert_v1 0_pubmed_pmc) was initialized from general-domain BERT (bert-base-cased) and pretrained on PubMed+PMC 
Experiment 1: Biomedical NER ::: Domain adaptation
We train Word2Vec with vector size $d_\mathrm {W2V} = d_\mathrm {LM} = 768$ on PubMed+PMC (see Appendix for details)  Then, we follow the procedure described in Section SECREF3 to update the wordpiece embedding layer and tokenizer of general-domain BERT 
Experiment 1: Biomedical NER ::: Finetuning
We finetune GreenBioBERT on the eight publicly available NER tasks used in BIBREF2  We also do reproduction experiments with general-domain BERT and BioBERTv1 0, using the same setup as our model  We average results over eight random seeds  See Appendix for details on preprocessing, training and hyperparameters 
Experiment 1: Biomedical NER ::: Results and discussion
Table TABREF7 (bottom) shows entity-level precision, recall and F1  For ease of visualization, Figure FIGREF13 shows what portion of the BioBERT – BERT F1 delta is covered  We improve over general-domain BERT on all tasks with varying effect sizes

ground_truth: Yes

Unanswerable

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which eight NER tasks did they evaluate on?
Context 3:  Depending on the points of reference, we cover an average 52% to 60% of the BioBERT – BERT F1 delta (54% for BioBERTv1 0, 60% for BioBERTv1 1 and 52% for our reproduction experiments)  Table TABREF17 (top) shows the importance of vector space alignment: If we replace the aligned Word2Vec vectors with their non-aligned counterparts (by setting $\mathbf {W} = \mathbf {1}$) or with randomly initialized vectors, F1 drops on all tasks 
Experiment 2: Covid-19 QA
In this section, we use the proposed method to quickly adapt an existing general-domain QA model to an emerging target domain: Covid-19  Our baseline model is SQuADBERT (bert-large-uncased-whole-word-masking-finetuned-squad), a version of BERT that was finetuned on general-domain SQuAD BIBREF19  We evaluate on Deepset-AI Covid-QA, a SQuAD-style dataset with 1380 questions (see Appendix for details on data and preprocessing)  We assume that there is no target-domain finetuning data, which is a realistic setup for a new domain 
Experiment 2: Covid-19 QA ::: Domain adaptation
We train Word2Vec with vector size $d_\mathrm {W2V} = d_\mathrm {LM} = 1024$ on CORD-19 (Covid-19 Open Research Dataset) and/or PubMed+PMC  The process takes less than an hour on CORD-19 and about one day on the combined corpus, again without the need for a GPU  Then, we update SQuADBERT's wordpiece embedding layer and tokenizer, as described in Section SECREF3  We refer to the resulting model as GreenCovidSQuADBERT 
Experiment 2: Covid-19 QA ::: Results and discussion
Table TABREF17 (bottom) shows that GreenCovidSQuADBERT outperforms general-domain SQuADBERT in all metrics  Most of the improvement can be achieved with just the small CORD-19 corpus, which is more specific to the target domain (compare “Cord-19 only” and “Cord-19+PubMed+PMC”) 
Conclusion
As a reaction to the trend towards high-resource models, we have proposed an inexpensive, CPU-only method for domain-adapting Pretrained Language Models: We train Word2Vec vectors on target-domain data and align them with the wordpiece vector space of a general-domain PTLM 
On eight biomedical NER tasks, we cover over 50% of the BioBERT – BERT F1 delta, at 5% of BioBERT's domain adaptation CO$_2$ footprint and 2% of its cloud compute cost  We have also shown how to rapidly adapt an existing BERT QA model to an emerging domain – the Covid-19 pandemic – without the need for target-domain Language Model pretraining or finetuning 
We hope that our approach will benefit practitioners with limited time or resources, and that it will encourage environmentally friendlier NLP 
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Word2Vec training
We downloaded the PubMed, PMC and CORD-19 corpora from:
https://ftp ncbi nlm nih gov/pub/pmc/oa_bulk/ [20 January 2020, 68GB raw text]
https://ftp ncbi nlm nih gov/pubmed/baseline/ [20 January 2020, 24GB raw text]
https://pages semanticscholar org/coronavirus-research [17 April 2020, 2GB raw text]
We extract all abstracts and text bodies and apply the BERT basic tokenizer (a word tokenizer that standard BERT uses before wordpiece tokenization)  Then, we train CBOW Word2Vec with negative sampling  We use default parameters except for the vector size (which we set to $d_\mathrm {W2V} = d_\mathrm {LM}$) 
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 1: Biomedical NER ::: Pretrained models
General-domain BERT and BioBERTv1 0 were downloaded from:
https://storage googleapis com/bert_models/2018_10_18/cased_L-12_H-768_A-12 zip
https://github com/naver/biobert-pretrained
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 1: Biomedical NER ::: Data
We downloaded the NER datasets by following instructions on https://github com/dmis-lab/biobert#Datasets  For detailed dataset statistics, see BIBREF2 
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 1: Biomedical NER ::: Preprocessing
We cut all sentences into chunks of 30 or fewer whitespace-tokenized words (without splitting inside labeled spans)  Then, we tokenize every chunk $S$ with $\mathcal {T} = \mathcal {T}_\mathrm {LM}$ or $\mathcal {T} = \hat{\mathcal {T}}_\mathrm {LM}$ and add special tokens:
Word-initial wordpieces in $\mathcal {T}(S)$ are labeled as B(egin), I(nside) or O(utside), while non-word-initial wordpieces are labeled as X(ignore) 
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 1: Biomedical NER ::: Modeling, training and inference
We follow BIBREF2's implementation (https://github com/dmis-lab/biobert): We add a randomly initialized softmax classifier on top of the last BERT layer to predict the labels  We finetune the entire model to minimize negative log likelihood, with the standard Adam optimizer BIBREF20 and a linear learning rate scheduler (10% warmup)  Like BIBREF2, we finetune on the concatenation of the training and development set  All finetuning runs were done on a GeForce Titan X GPU (12GB) 
Since we do not have the resources for an extensive hyperparameter search, we use defaults and recommendations from the BioBERT repository: Batch size of 32, peak learning rate of $1 \cdot 10^{-5}$, and 100 epochs 
At inference time, we gather the output logits of word-initial wordpieces only  Since the number of word-initial wordpieces is the same for $\mathcal {T}_\mathrm {LM}(S)$ and $\hat{\mathcal {T}}_\mathrm {LM}(S)$, this makes mean-pooling the logits straightforward 
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 1: Biomedical NER ::: Note on our reproduction experiments
We found it easier to reproduce or exceed BIBREF2's results for general-domain BERT, compared to their results for BioBERTv1 0 (see Figure FIGREF13, main paper)  While this may be due to hyperparameters, it suggests that BioBERTv1 0 was more strongly tuned than BERT in the original BioBERT paper  This observation does not affect our conclusions, as GreenBioBERT performs better than reproduced BERT as well 
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 2: Covid-19 QA ::: Pretrained model
We downloaded the SQuADBERT baseline from:
https://huggingface co/bert-large-uncased-whole-word-masking-finetuned-squad
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 2: Covid-19 QA ::: Data
We downloaded the Deepset-AI Covid-QA dataset from:
https://github com/deepset-ai/COVID-QA/blob/master/data/question-answering/200423_covidQA json [24 April 2020]
At the time of writing, the dataset contains 1380 questions and gold answer spans  Every question is associated with one of 98 research papers (contexts)  We treat the entire dataset as a test set 
Note that there are some important differences between the dataset and SQuAD, which make the task challenging:
The contexts are full documents rather than single paragraphs  Thus, the correct answer may appear several times, often with slightly different wordings  Only a single one of the occurrences is annotated as correct, e g :
What was the prevalence of Coronavirus OC43 in community samples in Ilorin, Nigeria?
13 3% (95% CI 6 9-23 6%) # from main text
(13 3%, 10/75)  # from abstract
SQuAD gold answers are defined as the “shortest span in the paragraph that answered the question” BIBREF19, but many Covid-QA gold answers are longer and contain non-essential context, e g :
When was the Middle East Respiratory Syndrome Coronavirus isolated first?
(MERS-CoV) was first isolated in 2012, in a 60-year-old man who died in Jeddah, KSA due to severe acute pneumonia and multiple organ failure
2012,
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 2: Covid-19 QA ::: Preprocessing
We tokenize every question-context pair $(Q, C)$ with $\mathcal {T} = \mathcal {T}_\mathrm {LM}$ or $\mathcal {T} = \hat{\mathcal {T}}_\mathrm {LM}$, which yields $(\mathcal {T}(Q), \mathcal {T}(C))$  Since $\mathcal {T}(C)$ is usually too long to be digested in a single forward pass, we define a sliding window with width and stride $N = \mathrm {floor}(\frac{509 - |\mathcal {T}(Q)|}{2})$  At step $n$, the “active” window is between $a^{(l)}_n = nN$ and $a^{(r)}_n = \mathrm {min}(|C|, nN+N)$  The input is defined as:
$p^{(l)}_n$ and $p^{(r)}_n$ are chosen such that $|X^{(n)}| = 512$, and such that the active window is in the center of the input (if possible) 
Inexpensive Domain Adaptation of Pretrained Language Models (Appendix) ::: Experiment 2: Covid-19 QA ::: Modeling and inference
Feeding $X^{(n)}$ into the pretrained QA model yields start logits $\mathbf {h^{\prime }}^{(\mathrm {start}, n)} \in \mathbb {R}^{|X^{(n)}|}$ and end logits $\mathbf {h^{\prime }}^{(\mathrm {end},n)} \in \mathbb {R}^{|X^{(n)}|}$  We extract and concatenate the slices that correspond to the active windows of all steps:
Next, we map the logits from the wordpiece level to the word level  This allows us to mean-pool the outputs of $\mathcal {T}_\mathrm {LM}$ and $\hat{\mathcal {T}}_\mathrm {LM}$ even when $|\mathcal {T}_\mathrm {LM}(C)| \ne |\hat{\mathcal {T}}_\mathrm {LM}(C)|$ 
Let $c_i$ be a whitespace-delimited word in $C$  Let $\mathcal {T}(C)_{j:j+|\mathcal {T}(c_i)|}$ be the corresponding wordpieces  The start and end logits of $c_i$ are derived as:
Finally, we return the answer span $C_{k:k^{\prime }}$ that maximizes $o^{(\mathrm {start})}_k + o^{(\mathrm {end})}_{k^{\prime }}$, subject to the constraints that $k^{\prime }$ does not precede $k$ and the answer span is not longer than 500 characters 

ground_truth: BC5CDR-disease, NCBI-disease, BC5CDR-chem, BC4CHEMD, BC2GM, JNLPBA, LINNAEUS, Species-800

BC5CDR-disease, NCBI-disease, BC5CDR-chem, BC4CHEMD, BC2GM, JNLPBA, LINNAEUS, Species-800