prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How do they obtain psychological dimensions of people?
Context 1: Introduction
Blogging gained momentum in 1999 and became especially popular after the launch of freely available, hosted platforms such as blogger com or livejournal com  Blogging has progressively been used by individuals to share news, ideas, and information, but it has also developed a mainstream role to the extent that it is being used by political consultants and news services as a tool for outreach and opinion forming as well as by businesses as a marketing tool to promote products and services BIBREF0  
For this paper, we compiled a very large geolocated collection of blogs, written by individuals located in the U S , with the purpose of creating insightful mappings of the blogging community  In particular, during May-July 2015, we gathered the profile information for all the users that have self-reported their location in the U S , along with a number of posts for all their associated blogs  We utilize this blog collection to generate maps of the U S  that reflect user demographics, language use, and distributions of psycholinguistic and semantic word classes  We believe that these maps can provide valuable insights and partial verification of previous claims in support of research in linguistic geography BIBREF1 , regional personality BIBREF2 , and language analysis BIBREF3 , BIBREF4 , as well as psychology and its relation to human geography BIBREF5  
Data Collection
Our premise is that we can generate informative maps using geolocated information available on social media; therefore, we guide the blog collection process with the constraint that we only accept blogs that have specific location information  Moreover, we aim to find blogs belonging to writers from all 50 U S  states, which will allow us to build U S  maps for various dimensions of interest 
We first started by collecting a set of profiles of bloggers that met our location specifications by searching individual states on the profile finder on http://www blogger com  Starting with this list, we can locate the profile page for a user, and subsequently extract additional information, which includes fields such as name, email, occupation, industry, and so forth  It is important to note that the profile finder only identifies users that have an exact match to the location specified in the query; we thus built and ran queries that used both state abbreviations (e g , TX, AL), as well as the states' full names (e g , Texas, Alabama) 
After completing all the processing steps, we identified 197,527 bloggers with state location information  For each of these bloggers, we found their blogs (note that a blogger can have multiple blogs), for a total of 335,698 blogs  For each of these blogs, we downloaded the 21 most recent blog postings, which were cleaned of HTML tags and tokenized, resulting in a collection of 4,600,465 blog posts 
Maps from Blogs
Our dataset provides mappings between location, profile information, and language use, which we can leverage to generate maps that reflect demographic, linguistic, and psycholinguistic properties of the population represented in the dataset 
People Maps
The first map we generate depicts the distribution of the bloggers in our dataset across the U S  Figure FIGREF1 shows the density of users in our dataset in each of the 50 states  For instance, the densest state was found to be California with 11,701 users  The second densest is Texas, with 9,252 users, followed by New York, with 9,136  The state with the fewest bloggers is Delaware with 1,217 users  Not surprisingly, this distribution correlates well with the population of these states, with a Spearman's rank correlation INLINEFORM0 of 0 91 and a p-value INLINEFORM1 0 0001, and is very similar to the one reported in Lin and Halavais Lin04 
Figure FIGREF1 also shows the cities mentioned most often in our dataset  In particular, it illustrates all 227 cities that have at least 100 bloggers  The bigger the dot on the map, the larger the number of users found in that city  The five top blogger-dense cities, in order, are: Chicago, New York, Portland, Seattle, and Atlanta 
We also generate two maps that delineate the gender distribution in the dataset  Overall, the blogging world seems to be dominated by females: out of 153,209 users who self-reported their gender, only 52,725 are men and 100,484 are women  Figures FIGREF1 and FIGREF1 show the percentage of male and female bloggers in each of the 50 states  As seen in this figure, there are more than the average number of male bloggers in states such as California and New York, whereas Utah and Idaho have a higher percentage of women bloggers 
Another profile element that can lead to interesting maps is the Industry field BIBREF6   Using this field, we created different maps that plot the geographical distribution of industries across the country  As an example, Figure FIGREF2 shows the percentage of the users in each state working in the automotive and tourism industries respectively 
Linguistic Maps
Another use of the information found in our dataset is to build linguistic maps, which reflect the geographic lexical variation across the 50 states BIBREF7   We generate maps that represent the relative frequency by which a word occurs in the different states  Figure FIGREF3 shows sample maps created for two different words  The figure shows the map generated for one location specific word, Maui, which unsurprisingly is found predominantly in Hawaii, and a map for a more common word, lake, which has a high occurrence rate in Minnesota (Land of 10,000 Lakes) and Utah (home of the Great Salt Lake)  Our demo described in Section SECREF4 , can also be used to generate maps for function words, which can be very telling regarding people's personality BIBREF8  
Psycholinguistic and Semantic Maps
LIWC  In addition to individual words, we can also create maps for word categories that reflect a certain psycholinguistic or semantic property  Several lexical resources, such as Roget or Linguistic Inquiry and Word Count BIBREF9 , group words into categories  Examples of such categories are Money, which includes words such as remuneration, dollar, and payment; or Positive feelings with words such as happy, cheerful, and celebration  Using the distribution of the individual words in a category, we can compile distributions for the entire category, and therefore generate maps for these word categories  For instance, figure FIGREF8 shows the maps created for two categories: Positive Feelings and Money  The maps are not surprising, and interestingly they also reflect an inverse correlation between Money and Positive Feelings  
Values  We also measure the usage of words related to people's core values as reported by Boyd et al  boyd2015  The sets of words, or themes, were excavated using the Meaning Extraction Method (MEM) BIBREF10   MEM is a topic modeling approach applied to a corpus of texts created by hundreds of survey respondents from the U S  who were asked to freely write about their personal values  To illustrate, Figure FIGREF9 shows the geographical distributions of two of these value themes: Religion and Hard Work  Southeastern states often considered as the nation's “Bible Belt” BIBREF11 were found to have generally higher usage of Religion words such as God, bible, and church  Another broad trend was that western-central states (e g , Wyoming, Nebraska, Iowa) commonly blogged about Hard Work, using words such as hard, work, and job more often than bloggers in other regions 
Web Demonstration
A prototype, interactive charting demo is available at http://lit eecs umich edu/~geoliwc/  In addition to drawing maps of the geographical distributions on the different LIWC categories, the tool can report the three most and least correlated LIWC categories in the U S  and compare the distributions of any two categories 
Conclusions
In this paper, we showed how we can effectively leverage a prodigious blog dataset  Not only does the dataset bring out the extensive linguistic content reflected in the blog posts, but also includes location information and rich metadata  These data allow for the generation of maps that reflect the demographics of the population, variations in language use, and differences in psycholinguistic and semantic categories  These mappings can be valuable to both psychologists and linguists, as well as lexicographers  A prototype demo has been made available together with the code used to collect our dataset 
Acknowledgments
This material is based in part upon work supported by the National Science Foundation (#1344257) and by the John Templeton Foundation (#48503)  Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation or the John Templeton Foundation  We would like to thank our colleagues Hengjing Wang, Jiatao Fan, Xinghai Zhang, and Po-Jung Huang who provided technical help with the implementation of the demo 
Context 2: Introduction
Decoding intended speech or motor activity from brain signals is one of the major research areas in Brain Computer Interface (BCI) systems BIBREF0 , BIBREF1   In particular, speech-related BCI technologies attempt to provide effective vocal communication strategies for controlling external devices through speech commands interpreted from brain signals BIBREF2   Not only do they provide neuro-prosthetic help for people with speaking disabilities and neuro-muscular disorders like locked-in-syndrome, nasopharyngeal cancer, and amytotropic lateral sclerosis (ALS), but also equip people with a better medium to communicate and express thoughts, thereby improving the quality of rehabilitation and clinical neurology BIBREF3 , BIBREF4   Such devices also have applications in entertainment, preventive treatments, personal communication, games, etc  Furthermore, BCI technologies can be utilized in silent communication, as in noisy environments, or situations where any sort of audio-visual communication is infeasible 
Among the various brain activity-monitoring modalities in BCI, electroencephalography (EEG) BIBREF5 , BIBREF6 has demonstrated promising potential to differentiate between various brain activities through measurement of related electric fields  EEG is non-invasive, portable, low cost, and provides satisfactory temporal resolution  This makes EEG suitable to realize BCI systems  EEG data, however, is challenging: these data are high dimensional, have poor SNR, and suffer from low spatial resolution and a multitude of artifacts  For these reasons, it is not particularly obvious how to decode the desired information from raw EEG signals  Although the area of BCI based speech intent recognition has received increasing attention among the research community in the past few years, most research has focused on classification of individual speech categories in terms of discrete vowels, phonemes and words BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 , BIBREF15   This includes categorization of imagined EEG signal into binary vowel categories like /a/, /u/ and rest BIBREF7 , BIBREF8 , BIBREF9 ; binary syllable classes like /ba/ and /ku/ BIBREF1 , BIBREF10 , BIBREF11 , BIBREF12 ; a handful of control words like 'up', 'down', 'left', 'right' and 'select' BIBREF15 or others like 'water', 'help', 'thanks', 'food', 'stop' BIBREF13 , Chinese characters BIBREF14 , etc  Such works mostly involve traditional signal processing or manual feature handcrafting along with linear classifiers (e g , SVMs)  In our recent work BIBREF16 , we introduced deep learning models for classification of vowels and words that achieved 23 45% improvement of accuracy over the baseline 
Production of articulatory speech is an extremely complicated process, thereby rendering understanding of the discriminative EEG manifold corresponding to imagined speech highly challenging  As a result, most of the existing approaches failed to achieve satisfactory accuracy on decoding speech tokens from the speech imagery EEG data  Perhaps, for these reasons, very little work has been devoted to relating the brain signals to the underlying articulation  The few exceptions include BIBREF17 , BIBREF18   In BIBREF17 , Zhao et al  used manually handcrafted features from EEG data, combined with speech audio and facial features to achieve classification of the phonological categories varying based on the articulatory steps  However, the imagined speech classification accuracy based on EEG data alone, as reported in BIBREF17 , BIBREF18 , are not satisfactory in terms of accuracy and reliability  We now turn to describing our proposed models 
Proposed Framework
Cognitive learning process underlying articulatory speech production involves incorporation of intermediate feedback loops and utilization of past information stored in the form of memory as well as hierarchical combination of several feature extractors  To this end, we develop our mixed neural network architecture composed of three supervised and a single unsupervised learning step, discussed in the next subsections and shown in Fig  FIGREF1   We formulate the problem of categorizing EEG data based on speech imagery as a non-linear mapping INLINEFORM0 of a multivariate time-series input sequence INLINEFORM1 to fixed output INLINEFORM2 , i e, mathematically INLINEFORM3 : INLINEFORM4 , where c and t denote the EEG channels and time instants respectively 
Preprocessing step
We follow similar pre-processing steps on raw EEG data as reported in BIBREF17 (ocular artifact removal using blind source separation, bandpass filtering and subtracting mean value from each channel) except that we do not perform Laplacian filtering step since such high-pass filtering may decrease information content from the signals in the selected bandwidth 
Joint variability of electrodes
Multichannel EEG data is high dimensional multivariate time series data whose dimensionality depends on the number of electrodes  It is a major hurdle to optimally encode information from these EEG data into lower dimensional space  In fact, our investigation based on a development set (as we explain later) showed that well-known deep neural networks (e g , fully connected networks such as convolutional neural networks, recurrent neural networks and autoencoders) fail to individually learn such complex feature representations from single-trial EEG data  Besides, we found that instead of using the raw multi-channel high-dimensional EEG requiring large training times and resource requirements, it is advantageous to first reduce its dimensionality by capturing the information transfer among the electrodes  Instead of the conventional approach of selecting a handful of channels as BIBREF17 , BIBREF18 , we address this by computing the channel cross-covariance, resulting in positive, semi-definite matrices encoding the connectivity of the electrodes  We define channel cross-covariance (CCV) between any two electrodes INLINEFORM0 and INLINEFORM1 as: INLINEFORM2   Next, we reject the channels which have significantly lower cross-covariance than auto-covariance values (where auto-covariance implies CCV on same electrode)  We found this measure to be essential as the higher cognitive processes underlying speech planning and synthesis involve frequent information exchange between different parts of the brain  Hence, such matrices often contain more discriminative features and hidden information than mere raw signals  This is essentially different than our previous work BIBREF16 where we extract per-channel 1-D covariance information and feed it to the networks  We present our sample 2-D EEG cross-covariance matrices (of two individuals) in Fig  FIGREF2  
CNN & LSTM
In order to decode spatial connections between the electrodes from the channel covariance matrix, we use a CNN BIBREF19 , in particular a four-layered 2D CNN stacking two convolutional and two fully connected hidden layers  The INLINEFORM0 feature map at a given CNN layer with input INLINEFORM1 , weight matrix INLINEFORM2 and bias INLINEFORM3 is obtained as: INLINEFORM4   At this first level of hierarchy, the network is trained with the corresponding labels as target outputs, optimizing a cross-entropy cost function  In parallel, we apply a four-layered recurrent neural network on the channel covariance matrices to explore the hidden temporal features of the electrodes  Namely, we exploit an LSTM BIBREF20 consisting of two fully connected hidden layers, stacked with two LSTM layers and trained in a similar manner as CNN 
Deep autoencoder for spatio-temporal information
As we found the individually-trained parallel networks (CNN and LSTM) to be useful (see Table TABREF12 ), we suspected the combination of these two networks could provide a more powerful discriminative spatial and temporal representation of the data than each independent network  As such, we concatenate the last fully-connected layer from the CNN with its counterpart in the LSTM to compose a single feature vector based on these two penultimate layers  Ultimately, this forms a joint spatio-temporal encoding of the cross-covariance matrix 
In order to further reduce the dimensionality of the spatio-temporal encodings and cancel background noise effects BIBREF21 , we train an unsupervised deep autoenoder (DAE) on the fused heterogeneous features produced by the combined CNN and LSTM information  The DAE forms our second level of hierarchy, with 3 encoding and 3 decoding layers, and mean squared error (MSE) as the cost function 
Classification with Extreme Gradient Boost
At the third level of hierarchy, the discrete latent vector representation of the deep autoencoder is fed into an Extreme Gradient Boost based classification layer BIBREF22 , BIBREF23 motivated by BIBREF21   It is a regularized gradient boosted decision tree that performs well on structured problems

ground_truth: using the Meaning Extraction Method

Unanswerable

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What data was presented to the subjects to elicit event-related responses?
Context 2:  Since our EEG-phonological pairwise classification has an internal structure involving individual phonemes and words, it seems to be a reasonable choice of classifier  The classifier receives its input from the latent vectors of the deep autoencoder and is trained in a supervised manner to output the final predicted classes corresponding to the speech imagery 
Dataset
We evaluate our model on a publicly available dataset, KARA ONE BIBREF17 , composed of multimodal data for stimulus-based, imagined and articulated speech state corresponding to 7 phonemic/syllabic ( /iy/, /piy/, /tiy/, /diy/, /uw/, /m/, /n/ ) as well as 4 words(pat, pot, knew and gnaw)  The dataset consists of 14 participants, with each prompt presented 11 times to each individual  Since our intention is to classify the phonological categories from human thoughts, we discard the facial and audio information and only consider the EEG data corresponding to imagined speech  It is noteworthy that given the mixed nature of EEG signals, it is reportedly challenging to attain a pairwise EEG-phoneme mapping BIBREF18   In order to explore the problem space, we thus specifically target five binary classification problems addressed in BIBREF17 , BIBREF18 , i e presence/absence of consonants, phonemic nasal, bilabial, high-front vowels and high-back vowels 
Training and hyperparameter selection
We performed two sets of experiments with the single-trial EEG data  In PHASE-ONE, our goals was to identify the best architectures and hyperparameters for our networks with a reasonable number of runs  For PHASE-ONE, we randomly shuffled and divided the data (1913 signals from 14 individuals) into train (80%), development (10%) and test sets (10%)  In PHASE-TWO, in order to perform a fair comparison with the previous methods reported on the same dataset, we perform a leave-one-subject out cross-validation experiment using the best settings we learn from PHASE-ONE 
The architectural parameters and hyperparameters listed in Table TABREF6 were selected through an exhaustive grid-search based on the validation set of PHASE-ONE  We conducted a series of empirical studies starting from single hidden-layered networks for each of the blocks and, based on the validation accuracy, we increased the depth of each given network and selected the optimal parametric set from all possible combinations of parameters  For the gradient boosting classification, we fixed the maximum depth at 10, number of estimators at 5000, learning rate at 0 1, regularization coefficient at 0 3, subsample ratio at 0 8, and column-sample/iteration at 0 4  We did not find any notable change of accuracy while varying other hyperparameters while training gradient boost classifier 
Performance analysis and discussion
To demonstrate the significance of the hierarchical CNN-LSTM-DAE method, we conducted separate experiments with the individual networks in PHASE-ONE of experiments and summarized the results in Table TABREF12 From the average accuracy scores, we observe that the mixed network performs much better than individual blocks which is in agreement with the findings in BIBREF21   A detailed analysis on repeated runs further shows that in most of the cases, LSTM alone does not perform better than chance  CNN, on the other hand, is heavily biased towards the class label which sees more training data corresponding to it  Though the situation improves with combined CNN-LSTM, our analysis clearly shows the necessity of a better encoding scheme to utilize the combined features rather than mere concatenation of the penultimate features of both networks 
The very fact that our combined network improves the classification accuracy by a mean margin of 14 45% than the CNN-LSTM network indeed reveals that the autoencoder contributes towards filtering out the unrelated and noisy features from the concatenated penultimate feature set  It also proves that the combined supervised and unsupervised neural networks, trained hierarchically, can learn the discriminative manifold better than the individual networks and it is crucial for improving the classification accuracy  In addition to accuracy, we also provide the kappa coefficients BIBREF24 of our method in Fig  FIGREF14   Here, a higher mean kappa value corresponding to a task implies that the network is able to find better discriminative information from the EEG data beyond random decisions  The maximum above-chance accuracy (75 92%) is recorded for presence/absence of the vowel task and the minimum (49 14%) is recorded for the INLINEFORM0  
To further investigate the feature representation achieved by our model, we plot T-distributed Stochastic Neighbor Embedding (tSNE) corresponding to INLINEFORM0 and V/C classification tasks in Fig  FIGREF8   We particularly select these two tasks as our model exhibits respectively minimum and maximum performance for these two  The tSNE visualization reveals that the second set of features are more easily separable than the first one, thereby giving a rationale for our performance 
Next, we provide performance comparison of the proposed approach with the baseline methods for PHASE-TWO of our study (cross-validation experiment) in Table TABREF15   Since the model encounters the unseen data of a new subject for testing, and given the high inter-subject variability of the EEG data, a reduction in the accuracy was expected  However, our network still managed to achieve an improvement of 18 91, 9 95, 67 15, 2 83 and 13 70 % over BIBREF17   Besides, our best model shows more reliability compared to previous works: The standard deviation of our model's classification accuracy across all the tasks is reduced from 22 59% BIBREF17 and 17 52% BIBREF18 to a mere 5 41% 
Conclusion and future direction
In an attempt to move a step towards understanding the speech information encoded in brain signals, we developed a novel mixed deep neural network scheme for a number of binary classification tasks from speech imagery EEG data  Unlike previous approaches which mostly deal with subject-dependent classification of EEG into discrete vowel or word labels, this work investigates a subject-invariant mapping of EEG data with different phonological categories, varying widely in terms of underlying articulator motions (eg: involvement or non-involvement of lips and velum, variation of tongue movements etc)  Our model takes an advantage of feature extraction capability of CNN, LSTM as well as the deep learning benefit of deep autoencoders  We took BIBREF17 , BIBREF18 as the baseline works investigating the same problem and compared our performance with theirs  Our proposed method highly outperforms the existing methods across all the five binary classification tasks by a large average margin of 22 51% 
Acknowledgments
This work was funded by the Natural Sciences and Engineering Research Council (NSERC) of Canada and Canadian Institutes for Health Research (CIHR) 
Context 3: Introduction
The automatic processing of medical texts and documents plays an increasingly important role in the recent development of the digital health area  To enable dedicated Natural Language Processing (NLP) that is highly accurate with respect to medically relevant categories, manually annotated data from this domain is needed  One category of high interest and relevance are medical entities  Only very few annotated corpora in the medical domain exist  Many of them focus on the relation between chemicals and diseases or proteins and diseases, such as the BC5CDR corpus BIBREF0, the Comparative Toxicogenomics Database BIBREF1, the FSU PRotein GEne corpus BIBREF2 or the ADE (adverse drug effect) corpus BIBREF3  The NCBI Disease Corpus BIBREF4 contains condition mention annotations along with annotations of symptoms  Several new corpora of annotated case reports were made available recently  grouin-etal-2019-clinical presented a corpus with medical entity annotations of clinical cases written in French, copdPhenotype presented a corpus focusing on phenotypic information for chronic obstructive pulmonary disease while 10 1093/database/bay143 presented a corpus focusing on identifying main finding sentences in case reports 
The corpus most comparable to ours is the French corpus of clinical case reports by grouin-etal-2019-clinical  Their annotations are based on UMLS semantic types  Even though there is an overlap in annotated entities, semantic classes are not the same  Lab results are subsumed under findings in our corpus and are not annotated as their own class  Factors extend beyond gender and age and describe any kind of risk factor that contributes to a higher probability of having a certain disease  Our corpus includes additional entity types  We annotate conditions, findings (including medical findings such as blood values), factors, and also modifiers which indicate the negation of other entities as well as case entities, i  e , entities specific to one case report  An overview is available in Table TABREF3 
A Corpus of Medical Case Reports with Medical Entity Annotation ::: Annotation tasks
Case reports are standardized in the CARE guidelines BIBREF5  They represent a detailed description of the symptoms, signs, diagnosis, treatment, and follow-up of an individual patient  We focus on documents freely available through PubMed Central (PMC)  The presentation of the patient's case can usually be found in a dedicated section or the abstract  We perform a manual annotation of all mentions of case entities, conditions, findings, factors and modifiers  The scope of our manual annotation is limited to the presentation of a patient's signs and symptoms  In addition, we annotate the title of the case report 
A Corpus of Medical Case Reports with Medical Entity Annotation ::: Annotation Guidelines
We annotate the following entities:
case entity marks the mention of a patient  A case report can contain more than one case description  Therefore, all the findings, factors and conditions related to one patient are linked to the respective case entity  Within the text, this entity is often represented by the first mention of the patient and overlaps with the factor annotations which can, e  g , mark sex and age (cf  Figure FIGREF12) 
condition marks a medical disease such as pneumothorax or dislocation of the shoulder 
factor marks a feature of a patient which might influence the probability for a specific diagnosis  It can be immutable (e  g , sex and age), describe a specific medical history (e  g , diabetes mellitus) or a behaviour (e  g , smoking) 
finding marks a sign or symptom a patient shows  This can be visible (e  g , rash), described by a patient (e  g , headache) or measurable (e  g , decreased blood glucose level) 
negation modifier explicitly negate the presence of a certain finding usually setting the case apart from common cases 
We also annotate relations between these entities, where applicable  Since we work on case descriptions, the anchor point of these relations is the case that is described  The following relations are annotated:
has relations exist between a case entity and factor, finding or condition entities 
modifies relations exist between negation modifiers and findings 
causes relations exist between conditions and findings 
Example annotations are shown in Figure FIGREF16 
A Corpus of Medical Case Reports with Medical Entity Annotation ::: Annotators
We asked medical doctors experienced in extracting knowledge related to medical entities from texts to annotate the entities described above  Initially, we asked four annotators to test our guidelines on two texts  Subsequently, identified issues were discussed and resolved  Following this pilot annotation phase, we asked two different annotators to annotate two case reports according to our guidelines  The same annotators annotated an overall collection of 53 case reports 
Inter-annotator agreement is calculated based on two case reports  We reach a Cohen's kappa BIBREF6 of 0 68  Disagreements mainly appear for findings that are rather unspecific such as She no longer eats out with friends which can be seen as a finding referring to “avoidance behaviour” 
A Corpus of Medical Case Reports with Medical Entity Annotation ::: Annotation Tools and Format
The annotation was performed using WebAnno BIBREF7, a web-based tool for linguistic annotation  The annotators could choose between a pre-annotated version or a blank version of each text  The pre-annotated versions contained suggested entity spans based on string matches from lists of conditions and findings synonym lists  Their quality varied widely throughout the corpus  The blank version was preferred by the annotators  We distribute the corpus in BioC JSON format  BioC was chosen as it allows us to capture the complexities of the annotations in the biomedical domain  It represented each documents properties ranging from full text, individual passages/sentences along with captured annotations and relationships in an organized manner  BioC is based on character offsets of annotations and allows the stacking of different layers 
A Corpus of Medical Case Reports with Medical Entity Annotation ::: Corpus Overview
The corpus consists of 53 documents, which contain an average number of 156 1 sentences per document, each with 19 55 tokens on average  The corpus comprises 8,275 sentences and 167,739 words in total  However, as mentioned above, only case presentation sections, headings and abstracts are annotated  The numbers of annotated entities are summarized in Table TABREF24 
Findings are the most frequently annotated type of entity  This makes sense given that findings paint a clinical picture of the patient's condition  The number of tokens per entity ranges from one token for all types to 5 tokens for cases (average length 3 1), nine tokens for conditions (average length 2 0), 16 tokens for factors (average length 2 5), 25 tokens for findings (average length 2 6) and 18 tokens for modifiers (average length 1 4) (cf  Table TABREF24)  Examples of rather long entities are given in Table TABREF25 
Entities can appear in a discontinuous way  We model this as a relation between two spans which we call “discontinuous” (cf  Figure FIGREF26)  Especially findings often appear as discontinuous entities, we found 543 discontinuous finding relations  The numbers for conditions and factors are lower with seven and two, respectively  Entities can also be nested within one another  This happens either when the span of one annotation is completely embedded in the span of another annotation (fully-nested; cf  Figure FIGREF12), or when there is a partial overlapping between the spans of two different entities (partially-nested; cf  Figure FIGREF12)  There is a high number of inter-sentential relations in the corpus (cf  Table TABREF27)  This can be explained by the fact that the case entity occurs early in each document; furthermore, it is related to finding and factor annotations that are distributed across different sentences 
The most frequently annotated relation in our corpus is the has-relation between a case entity and the findings related to that case  This correlates with the high number of finding entities  The relations contained in our corpus are summarized in Table TABREF27 
Baseline systems for Named Entity Recognition in medical case reports
We evaluate the corpus using Named Entity Recognition (NER), i  e , the task of finding mentions of concepts of interest in unstructured text  We focus on detecting cases, conditions, factors, findings and modifiers in case reports (cf  Section SECREF6)  We approach this as a sequence labeling problem  Four systems were developed to offer comparable robust baselines 
The original documents are pre-processed (sentence splitting and tokenization with ScispaCy)  We do not perform stop word removal or lower-casing of the tokens  The BIO labeling scheme is used to capture the order of tokens belonging to the same entity type and enable span-level detection of entities  Detection of nested and/or discontinuous entities is not supported  The annotated corpus is randomized and split in five folds using scikit-learn BIBREF9  Each fold has a train, test and dev split with the test split defined as  15% of the train split  This ensures comparability between the presented systems 
Baseline systems for Named Entity Recognition in medical case reports ::: Conditional Random Fields
Conditional Random Fields (CRF) BIBREF10 are a standard approach when dealing with sequential data in the context of sequence labeling  We use a combination of linguistic and semantic features, with a context window of size five, to describe each of the tokens and the dependencies between them  Hyper-parameter optimization is performed using randomized search and cross validation  Span-based F1 score is used as the optimization metric 
Baseline systems for Named Entity Recognition in medical case reports ::: BiLSTM-CRF
Prior to the emergence of deep neural language models, BiLSTM-CRF models BIBREF11 had achieved state-of-the-art results for the task of sequence labeling  We use a BiLSTM-CRF model with both word-level and character-level input  BioWordVec BIBREF12 pre-trained word embeddings are used in the embedding layer for the input representation  A bidirectional LSTM layer is applied to a multiplication of the two input representations  Finally, a CRF layer is applied to predict the sequence of labels  Dropout and L1/L2 regularization is used where applicable  He (uniform) initialization BIBREF13 is used to initialize the kernels of the individual layers  As the loss metric, CRF-based loss is used, while optimizing the model based on the CRF Viterbi accuracy  Additionally, span-based F1 score is used to serialize the best performing model  We train for a maximum of 100 epochs, or until an early stopping criterion is reached (no change in validation loss value grater than 0 01 for ten consecutive epochs)  Furthermore, Adam BIBREF14 is used as the optimizer  The learning rate is reduced by a factor of 0 3 in case no significant increase of the optimization metric is achieved in three consecutive epochs 
Baseline systems for Named Entity Recognition in medical case reports ::: Multi-Task Learning
Multi-Task Learning (MTL) BIBREF15 has become popular with the progress in deep learning  This model family is characterized by simultaneous optimization of multiple loss functions and transfer of knowledge achieved this way  The knowledge is transferred through the use of one or multiple shared layers  Through finding supporting patterns in related tasks, MTL provides better generalization on unseen cases and the main tasks we are trying to solve 
We rely on the model presented by bekoulis2018joint and reuse the implementation provided by the authors  The model jointly trains two objectives supported by the dataset: the main task of NER and a supporting task of Relation Extraction (RE)  Two separate models are developed for each of the tasks  The NER task is solved with the help of a BiLSTM-CRF model, similar to the one presented in Section SECREF32 The RE task is solved by using a multi-head selection approach, where each token can have none or more relationships to in-sentence tokens  Additionally, this model also leverages the output of the NER branch model (the CRF prediction) to learn label embeddings  Shared layers consist of a concatenation of word and character embeddings followed by two bidirectional LSTM layers  We keep most of the parameters suggested by the authors and change (1) the number of training epochs to 100 to allow the comparison to other deep learning approaches in this work, (2) use label embeddings of size 64, (3) allow gradient clipping and (4) use $d=0 8$ as the pre-trained word embedding dropout and $d=0 5$ for all other dropouts  $\eta =1^{-3}$ is used as the learning rate with the Adam optimizer and tanh activation functions across layers  Although it is possible to use adversarial training BIBREF16, we omit from using it  We also omit the publication of results for the task of RE as we consider it to be a supporting task and no other competing approaches have been developed 
Baseline systems for Named Entity Recognition in medical case reports ::: BioBERT
Deep neural language models have recently evolved to a successful method for representing text  In particular, Bidirectional Encoder Representations from Transformers (BERT) outperformed previous state-of-the-art methods by a large margin on various NLP tasks BIBREF17  For our experiments, we use BioBERT, an adaptation of BERT for the biomedical domain, pre-trained on PubMed abstracts and PMC full-text articles BIBREF18  The BERT architecture for deriving text representations uses 12 hidden layers, consisting of 768 units each  For NER, token level BIO-tag probabilities are computed with a single output layer based on the representations from the last layer of BERT  We fine-tune the model on the entity recognition task during four training epochs with batch size $b=32$, dropout probability $d=0 1$ and learning rate $\eta =2^{-5}$  These hyper-parameters are proposed by Devlin2018 for BERT fine-tuning 
Baseline systems for Named Entity Recognition in medical case reports ::: Evaluation
To evaluate the performance of the four systems, we calculate the span-level precision (P), recall (R) and F1 scores, along with corresponding micro and macro scores  The reported values are shown in Table TABREF29 and are averaged over five folds, utilising the seqeval framework 
With a macro avg  F1-score of 0 59, MTL achieves the best result with a significant margin compared to CRF, BiLSTM-CRF and BERT  This confirms the usefulness of jointly training multiple objectives (minimizing multiple loss functions), and enabling knowledge transfer, especially in a setting with limited data (which is usually the case in the biomedical NLP domain)  This result also suggest the usefulness of BioBERT for other biomedical datasets as reported by Lee2019  Despite being a rather standard approach, CRF outperforms the more elaborated BiLSTM-CRF, presumably due to data scarcity and class imbalance  We hypothesize that an increase in training data would yield better results for BiLSTM-CRF but not outperform transfer learning approach of MTL (or even BioBERT)  In contrast to other common NER corpora, like CoNLL 2003, even the best baseline system only achieves relatively low scores  This outcome is due to the inherent difficulty of the task (annotators are experienced medical doctors) and the small number of training samples 
Conclusion
We present a new corpus, developed to facilitate the processing of case reports  The corpus focuses on five distinct entity types: cases, conditions, factors, findings and modifiers  Where applicable, relationships between entities are also annotated  Additionally, we annotate discontinuous entities with a special relationship type (discontinuous)  The corpus presented in this paper is the very first of its kind and a valuable addition to the scarce number of corpora available in the field of biomedical NLP  Its complexity, given the discontinuous nature of entities and a high number of nested and multi-label entities, poses new challenges for NLP methods applied for NER and can, hence, be a valuable source for insights into what entities “look like in the wild”  Moreover, it can serve as a playground for new modelling techniques such as the resolution of discontinuous entities as well as multi-task learning given the combination of entities and their relations  We provide an evaluation of four distinct NER systems that will serve as robust baselines for future work but which are, as of yet, unable to solve all the complex challenges this dataset holds  A functional service based on the presented corpus is currently being integrated, as a NER service, in the QURATOR platform BIBREF20 
Acknowledgments
The research presented in this article is funded by the German Federal Ministry of Education and Research (BMBF) through the project QURATOR (Unternehmen Region, Wachstumskern, grant no  03WKDA1A), see http://qurator ai  We want to thank our medical experts for their help annotating the data set, especially Ashlee Finckh and Sophie Klopfenstein 

ground_truth: 7 phonemic/syllabic ( /iy/, /piy/, /tiy/, /diy/, /uw/, /m/, /n/ ) as well as 4 words(pat, pot, knew and gnaw)

KARA ONE BIBREF17 , composed of multimodal data for stimulus-based, imagined and articulated speech state corresponding to 7 phonemic/syllabic ( /iy/, /piy/, /tiy/, /diy/, /uw/, /m/, /n/ ) as well as 4 words(pat, pot, knew and gnaw)

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How large is the corpus?
Context 4:  Equation (DISPLAY_FORM13) can ensure the Gaussian weight equals 1 when $dis_{ij}$ is 0  The larger distance between charactersis, the smaller the weight is, which makes one character can affect its adjacent characters more compared with other characters 
To combine the Gaussian weight to the self-attention, we produce the Hadamard product of Gaussian weight matrix $G$ and the score matrix produced by $Q{K^{T}}$
where $AG$ is the Gaussian-masked attention  It ensures that the relationship between two characters with long distances is weaker than adjacent characters 
The scaled dot-product attention models the relationship between two characters without regard to their distances in one sequence  For CWS task, the weight between adjacent characters should be more important while it is hard for self-attention to achieve the effect explicitly because the self-attention cannot get the order of sentences directly  The Gaussian-masked attention adjusts the weight between characters and their adjacent character to a larger value which stands for the effect of adjacent characters 
For forward and backward encoder, the self-attention sublayer needs to use a triangular matrix mask to let the self-attention focus on different weights:
where $pos_i$ is the position of character $c_i$  The triangular matrix for forward and backward encode are:
$\left[ \begin{matrix} 1 & 0 & 0 & \cdots &0\\ 1 & 1 & 0 & \cdots &0\\ 1 & 1 & 1 & \cdots &0\\ \vdots &\vdots &\vdots &\ddots &\vdots \\ 1 & 1 & 1 & \cdots & 1\\ \end{matrix} \right]$ $\left[ \begin{matrix} 1 & 1 & 1 & \cdots &1 \\ 0 & 1 & 1 & \cdots &1 \\ 0 & 0& 1 & \cdots &1 \\ \vdots &\vdots &\vdots &\ddots &\vdots \\ 0 & 0 & 0 & \cdots & 1\\ \end{matrix}\right]$
Similar as BIBREF24, we use multi-head attention to capture information from different dimension positions as Figure FIGREF16 and get Gaussian-masked directional multi-head attention  With multi-head attention architecture, the representation of input can be captured by
where $MH$ is the Gaussian-masked multi-head attention, ${W_i^q, W_i^k,W_i^v} \in \mathbb {R}^{d_k \times d_h}$ is the parameter matrices to generate heads, $d_k$ is the dimension of model and $d_h$ is the dimension of one head 
Models ::: Bi-affinal Attention Scorer
Regarding word boundaries as gaps between any adjacent words converts the character labeling task to the gap labeling task  Different from character labeling task, gap labeling task requires information of two adjacent characters  The relationship between adjacent characters can be represented as the type of gap  The characteristic of word boundaries makes bi-affine attention an appropriate scorer for CWS task 
Bi-affinal attention scorer is the component that we use to label the gap  Bi-affinal attention is developed from bilinear attention which has been used in dependency parsing BIBREF26 and SRL BIBREF27  The distribution of labels in a labeling task is often uneven which makes the output layer often include a fixed bias term for the prior probability of different labels BIBREF27  Bi-affine attention uses bias terms to alleviate the burden of the fixed bias term and get the prior probability which makes it different from bilinear attention  The distribution of the gap is uneven that is similar as other labeling task which fits bi-affine 
Bi-affinal attention scorer labels the target depending on information of independent unit and the joint information of two units  In bi-affinal attention, the score $s_{ij}$ of characters $c_i$ and $c_j$ $(i < j)$ is calculated by:
where $v_i^f$ is the forward information of $c_i$ and $v_i^b$ is the backward information of $c_j$  In Equation (DISPLAY_FORM21), $W$, $U$ and $b$ are all parameters that can be updated in training  $W$ is a matrix with shape $(d_i \times N\times d_j)$ and $U$ is a $(N\times (d_i + d_j))$ matrix where $d_i$ is the dimension of vector $v_i^f$ and $N$ is the number of labels 
In our model, the biaffine scorer uses the forward information of character in front of the gap and the backward information of the character behind the gap to distinguish the position of characters  Figure FIGREF22 is an example of labeling gap  The method of using biaffine scorer ensures that the boundaries of words can be determined by adjacent characters with different directional information  The score vector of the gap is formed by the probability of being a boundary of word  Further, the model generates all boundaries using activation function in a greedy decoding way 
Experiments ::: Experimental Settings ::: Data
We train and evaluate our model on datasets from SIGHAN Bakeoff 2005 BIBREF21 which has four datasets, PKU, MSR, AS and CITYU  Table TABREF23 shows the statistics of train data  We use F-score to evaluate CWS models  To train model with pre-trained embeddings in AS and CITYU, we use OpenCC to transfer data from traditional Chinese to simplified Chinese 
Experiments ::: Experimental Settings ::: Pre-trained Embedding
We only use unigram feature so we only trained character embeddings  Our pre-trained embedding are pre-trained on Chinese Wikipedia corpus by word2vec BIBREF29 toolkit  The corpus used for pre-trained embedding is all transferred to simplified Chinese and not segmented  On closed test, we use embeddings initialized randomly 
Experiments ::: Experimental Settings ::: Hyperparameters
For different datasets, we use two kinds of hyperparameters which are presented in Table TABREF24  We use hyperparameters in Table TABREF24 for small corpora (PKU and CITYU) and normal corpora (MSR and AS)  We set the standard deviation of Gaussian function in Equation (DISPLAY_FORM13) to 2  Each training batch contains sentences with at most 4096 tokens 
Experiments ::: Experimental Settings ::: Optimizer
To train our model, we use the Adam BIBREF30 optimizer with $\beta _1=0 9$, $\beta _2=0 98$ and $\epsilon =10^{-9}$  The learning rate schedule is the same as BIBREF24:
where $d$ is the dimension of embeddings, $step$ is the step number of training and $warmup_step$ is the step number of warmup  When the number of steps is smaller than the step of warmup, the learning rate increases linearly and then decreases 
Experiments ::: Hardware and Implements
We trained our models on a single CPU (Intel i7-5960X) with an nVidia 1080 Ti GPU  We implement our model in Python with Pytorch 1 0 
Experiments ::: Results
Tables TABREF25 and TABREF26 reports the performance of recent models and ours in terms of closed test setting  Without the assistance of unsupervised segmentation features userd in BIBREF20, our model outperforms all the other models in MSR and AS except BIBREF18 and get comparable performance in PKU and CITYU  Note that all the other models for this comparison adopt various $n$-gram features while only our model takes unigram ones 
With unsupervised segmentation features introduced by BIBREF20, our model gets a higher result  Specially, the results in MSR and AS achieve new state-of-the-art and approaching previous state-of-the-art in CITYU and PKU  The unsupervised segmentation features are derived from the given training dataset, thus using them does not violate the rule of closed test of SIGHAN Bakeoff 
Table TABREF36 compares our model and recent neural models in terms of open test setting in which any external resources, especially pre-trained embeddings or language models can be used  In MSR and AS, our model gets a comparable result while our results in CITYU and PKU are not remarkable 
However, it is well known that it is always hard to compare models when using open test setting, especially with pre-trained embedding  Not all models may use the same method and data to pre-train  Though pre-trained embedding or language model can improve the performance, the performance improvement itself may be from multiple sources  It often that there is a success of pre-trained embedding to improve the performance, while it cannot prove that the model is better 
Compared with other LSTM models, our model performs better in AS and MSR than in CITYU and PKU  Considering the scale of different corpora, we believe that the size of corpus affects our model and the larger size is, the better model performs  For small corpus, the model tends to be overfitting 
Tables TABREF25 and TABREF26 also show the decoding time in different datasets  Our model finishes the segmentation with the least decoding time in all four datasets, thanks to the architecture of model which only takes attention mechanism as basic block 
Related Work ::: Chinese Word Segmentation
CWS is a task for Chinese natural language process to delimit word boundary  BIBREF0 for the first time formulize CWS as a sequence labeling task  BIBREF3 show that different character tag sets can make essential impact for CWS  BIBREF2 use CRFs as a model for CWS, achieving new state-of-the-art  Works of statistical CWS has built the basis for neural CWS 
Neural word segmentation has been widely used to minimize the efforts in feature engineering which was important in statistical CWS  BIBREF4 introduce the neural model with sliding-window based sequence labeling  BIBREF6 propose a gated recursive neural network (GRNN) for CWS to incorporate complicated combination of contextual character and n-gram features  BIBREF7 use LSTM to learn long distance information  BIBREF9 propose a neural framework that eliminates context windows and utilize complete segmentation history  BIBREF33 explore a joint model that performs segmentation, POS-Tagging and chunking simultaneously  BIBREF34 propose a feature-enriched neural model for joint CWS and part-of-speech tagging  BIBREF35 present a joint model to enhance the segmentation of Chinese microtext by performing CWS and informal word detection simultaneously  BIBREF17 propose a character-based convolutional neural model to capture $n$-gram features automatically and an effective approach to incorporate word embeddings  BIBREF11 improve the model in BIBREF9 and propose a greedy neural word segmenter with balanced word and character embedding inputs  BIBREF23 propose a novel neural network model to incorporate unlabeled and partially-labeled data  BIBREF36 propose two methods that extend the Bi-LSTM to perform incorporating dictionaries into neural networks for CWS  BIBREF37 propose Switch-LSTMs to segment words and provided a more flexible solution for multi-criteria CWS which is easy to transfer the learned knowledge to new criteria 
Related Work ::: Transformer
Transformer BIBREF24 is an attention-based neural machine translation model  The Transformer is one kind of self-attention networks (SANs) which is proposed in BIBREF38  Encoder of the Transformer consists of one self-attention layer and a position-wise feed-forward layer  Decoder of the Transformer contains one self-attention layer, one encoder-decoder attention layer and one position-wise feed-forward layer  The Transformer uses residual connections around the sublayers and then followed by a layer normalization layer 
Scaled dot-product attention is the key component in the Transformer  The input of attention contains queries, keys, and values of input sequences  The attention is generated using queries and keys like Equation (DISPLAY_FORM11)  Structure of scaled dot-product attention allows the self-attention layer generate the representation of sentences at once and contain the information of the sentence which is different from RNN that process characters of sentences one by one  Standard self-attention is similar as Gaussian-masked direction attention while it does not have directional mask and gaussian mask  BIBREF24 also propose multi-head attention which is better to generate representation of sentence by dividing queries, keys and values to different heads and get information from different subspaces 
Conclusion
In this paper, we propose an attention mechanism only based Chinese word segmentation model  Our model uses self-attention from the Transformer encoder to take sequence input and bi-affine attention scorer to predict the label of gaps  To improve the ability of capturing the localness and directional information of self-attention based encoder, we propose a variant of self-attention called Gaussian-masked directional multi-head attention to replace the standard self-attention  We also extend the Transformer encoder to capture directional features  Our model uses only unigram features instead of multiple $n$-gram features in previous work  Our model is evaluated on standard benchmark dataset, SIGHAN Bakeoff 2005, which shows not only our model performs segmentation faster than any previous models but also gives new higher or comparable segmentation performance against previous state-of-the-art models 

ground_truth: 8,275 sentences and 167,739 words in total

The corpus comprises 8,275 sentences and 167,739 words in total.