prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Is Arabic one of the 11 languages in CoVost?
Context 1: Introduction
End-to-end speech-to-text translation (ST) has attracted much attention recently BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6 given its simplicity against cascading automatic speech recognition (ASR) and machine translation (MT) systems  The lack of labeled data, however, has become a major blocker for bridging the performance gaps between end-to-end models and cascading systems  Several corpora have been developed in recent years  post2013improved introduced a 38-hour Spanish-English ST corpus by augmenting the transcripts of the Fisher and Callhome corpora with English translations  di-gangi-etal-2019-must created the largest ST corpus to date from TED talks but the language pairs involved are out of English only  beilharz2019librivoxdeen created a 110-hour German-English ST corpus from LibriVox audiobooks  godard-etal-2018-low created a Moboshi-French ST corpus as part of a rare language documentation effort  woldeyohannis provided an Amharic-English ST corpus in the tourism domain  boito2019mass created a multilingual ST corpus involving 8 languages from a multilingual speech corpus based on Bible readings BIBREF7  Previous work either involves language pairs out of English, very specific domains, very low resource languages or a limited set of language pairs  This limits the scope of study, including the latest explorations on end-to-end multilingual ST BIBREF8, BIBREF9  Our work is mostly similar and concurrent to iranzosnchez2019europarlst who created a multilingual ST corpus from the European Parliament proceedings  The corpus we introduce has larger speech durations and more translation tokens  It is diversified with multiple speakers per transcript/translation  Finally, we provide additional out-of-domain test sets 
In this paper, we introduce CoVoST, a multilingual ST corpus based on Common Voice BIBREF10 for 11 languages into English, diversified with over 11,000 speakers and over 60 accents  It includes a total 708 hours of French (Fr), German (De), Dutch (Nl), Russian (Ru), Spanish (Es), Italian (It), Turkish (Tr), Persian (Fa), Swedish (Sv), Mongolian (Mn) and Chinese (Zh) speeches, with French and German ones having the largest durations among existing public corpora  We also collect an additional evaluation corpus from Tatoeba for French, German, Dutch, Russian and Spanish, resulting in a total of 9 3 hours of speech  Both corpora are created at the sentence level and do not require additional alignments or segmentation  Using the official Common Voice train-development-test split, we also provide baseline models, including, to our knowledge, the first end-to-end many-to-one multilingual ST models  CoVoST is released under CC0 license and free to use  The Tatoeba evaluation samples are also available under friendly CC licenses  All the data can be acquired at https://github com/facebookresearch/covost 
Data Collection and Processing ::: Common Voice (CoVo)
Common Voice BIBREF10 is a crowdsourcing speech recognition corpus with an open CC0 license  Contributors record voice clips by reading from a bank of donated sentences  Each voice clip was validated by at least two other users  Most of the sentences are covered by multiple speakers, with potentially different genders, age groups or accents 
Raw CoVo data contains samples that passed validation as well as those that did not  To build CoVoST, we only use the former one and reuse the official train-development-test partition of the validated data  As of January 2020, the latest CoVo 2019-06-12 release includes 29 languages  CoVoST is currently built on that release and covers the following 11 languages: French, German, Dutch, Russian, Spanish, Italian, Turkish, Persian, Swedish, Mongolian and Chinese 
Validated transcripts were sent to professional translators  Note that the translators had access to the transcripts but not the corresponding voice clips since clips would not carry additional information  Since transcripts were duplicated due to multiple speakers, we deduplicated the transcripts before sending them to translators  As a result, different voice clips of the same content (transcript) will have identical translations in CoVoST for train, development and test splits 
In order to control the quality of the professional translations, we applied various sanity checks to the translations BIBREF11  1) For German-English, French-English and Russian-English translations, we computed sentence-level BLEU BIBREF12 with the NLTK BIBREF13 implementation between the human translations and the automatic translations produced by a state-of-the-art system BIBREF14 (the French-English system was a Transformer big BIBREF15 separately trained on WMT14)  We applied this method to these three language pairs only as we are confident about the quality of the corresponding systems  Translations with a score that was too low were manually inspected and sent back to the translators when needed  2) We manually inspected examples where the source transcript was identical to the translation  3) We measured the perplexity of the translations using a language model trained on a large amount of clean monolingual data BIBREF14  We manually inspected examples where the translation had a high perplexity and sent them back to translators accordingly  4) We computed the ratio of English characters in the translations  We manually inspected examples with a low ratio and sent them back to translators accordingly  5) Finally, we used VizSeq BIBREF16 to calculate similarity scores between transcripts and translations based on LASER cross-lingual sentence embeddings BIBREF17  Samples with low scores were manually inspected and sent back for translation when needed 
We also sanity check the overlaps of train, development and test sets in terms of transcripts and voice clips (via MD5 file hashing), and confirm they are totally disjoint 
Data Collection and Processing ::: Tatoeba (TT)
Tatoeba (TT) is a community built language learning corpus having sentences aligned across multiple languages with the corresponding speech partially available  Its sentences are on average shorter than those in CoVoST (see also Table TABREF2) given the original purpose of language learning  Sentences in TT are licensed under CC BY 2 0 FR and part of the speeches are available under various CC licenses 
We construct an evaluation set from TT (for French, German, Dutch, Russian and Spanish) as a complement to CoVoST development and test sets  We collect (speech, transcript, English translation) triplets for the 5 languages and do not include those whose speech has a broken URL or is not CC licensed  We further filter these samples by sentence lengths (minimum 4 words including punctuations) to reduce the portion of short sentences  This makes the resulting evaluation set closer to real-world scenarios and more challenging 
We run the same quality checks for TT as for CoVoST but we do not find poor quality translations according to our criteria  Finally, we report the overlap between CoVo transcripts and TT sentences in Table TABREF5  We found a minimal overlap, which makes the TT evaluation set a suitable additional test set when training on CoVoST 
Data Analysis ::: Basic Statistics
Basic statistics for CoVoST and TT are listed in Table TABREF2 including (unique) sentence counts, speech durations, speaker demographics (partially available) as well as vocabulary and token statistics (based on Moses-tokenized sentences by sacreMoses) on both transcripts and translations  We see that CoVoST has over 327 hours of German speeches and over 171 hours of French speeches, which, to our knowledge, corresponds to the largest corpus among existing public ST corpora (the second largest is 110 hours BIBREF18 for German and 38 hours BIBREF19 for French)  Moreover, CoVoST has a total of 18 hours of Dutch speeches, to our knowledge, contributing the first public Dutch ST resource  CoVoST also has around 27-hour Russian speeches, 37-hour Italian speeches and 67-hour Persian speeches, which is 1 8 times, 2 5 times and 13 3 times of the previous largest public one BIBREF7  Most of the sentences (transcripts) in CoVoST are covered by multiple speakers with potentially different accents, resulting in a rich diversity in the speeches  For example, there are over 1,000 speakers and over 10 accents in the French and German development / test sets  This enables good coverage of speech variations in both model training and evaluation 
Data Analysis ::: Speaker Diversity
As we can see from Table TABREF2, CoVoST is diversified with a rich set of speakers and accents  We further inspect the speaker demographics in terms of sample distributions with respect to speaker counts, accent counts and age groups, which is shown in Figure FIGREF6, FIGREF7 and FIGREF8  We observe that for 8 of the 11 languages, at least 60% of the sentences (transcripts) are covered by multiple speakers  Over 80% of the French sentences have at least 3 speakers  And for German sentences, even over 90% of them have at least 5 speakers  Similarly, we see that a large portion of sentences are spoken in multiple accents for French, German, Dutch and Spanish  Speakers of each language also spread widely across different age groups (below 20, 20s, 30s, 40s, 50s, 60s and 70s) 
Baseline Results
We provide baselines using the official train-development-test split on the following tasks: automatic speech recognition (ASR), machine translation (MT) and speech translation (ST) 
Baseline Results ::: Experimental Settings ::: Data Preprocessing
We convert raw MP3 audio files from CoVo and TT into mono-channel waveforms, and downsample them to 16,000 Hz  For transcripts and translations, we normalize the punctuation, we tokenize the text with sacreMoses and lowercase it  For transcripts, we further remove all punctuation markers except for apostrophes  We use character vocabularies on all the tasks, with 100% coverage of all the characters  Preliminary experimentation showed that character vocabularies provided more stable training than BPE  For MT, the vocabulary is created jointly on both transcripts and translations  We extract 80-channel log-mel filterbank features, computed with a 25ms window size and 10ms window shift using torchaudio  The features are normalized to 0 mean and 1 0 standard deviation  We remove samples having more than 3,000 frames or more than 256 characters for GPU memory efficiency (less than 25 samples are removed for all languages) 
Baseline Results ::: Experimental Settings ::: Model Training
Our ASR and ST models follow the architecture in berard2018end, but have 3 decoder layers like that in pino2019harnessing  For MT, we use a Transformer base architecture BIBREF15, but with 3 encoder layers, 3 decoder layers and 0 3 dropout  We use a batch size of 10,000 frames for ASR and ST, and a batch size of 4,000 tokens for MT  We train all models using Fairseq BIBREF20 for up to 200,000 updates  We use SpecAugment BIBREF21 for ASR and ST to alleviate overfitting 
Baseline Results ::: Experimental Settings ::: Inference and Evaluation
We use a beam size of 5 for all models  We use the best checkpoint by validation loss for MT, and average the last 5 checkpoints for ASR and ST  For MT and ST, we report case-insensitive tokenized BLEU BIBREF22 using sacreBLEU BIBREF23  For ASR, we report word error rate (WER) and character error rate (CER) using VizSeq 
Baseline Results ::: Automatic Speech Recognition (ASR)
For simplicity, we use the same model architecture for ASR and ST, although we do not leverage ASR models to pretrain ST model encoders later  Table TABREF18 shows the word error rate (WER) and character error rate (CER) for ASR models  We see that French and German perform the best given they are the two highest resource languages in CoVoST  The other languages are relatively low resource (especially Turkish and Swedish) and the ASR models are having difficulties to learn from this data 
Baseline Results ::: Machine Translation (MT)
MT models take transcripts (without punctuation) as inputs and outputs translations (with punctuation)  For simplicity, we do not change the text preprocessing methods for MT to correct this mismatch  Moreover, this mismatch also exists in cascading ST systems, where MT model inputs are the outputs of an ASR model  Table TABREF20 shows the BLEU scores of MT models  We notice that the results are consistent with what we see from ASR models  For example thanks to abundant training data, French has a decent BLEU score of 29 8/25 4  German doesn't perform well, because of less richness of content (transcripts)  The other languages are low resource in CoVoST and it is difficult to train decent models without additional data or pre-training techniques 
Baseline Results ::: Speech Translation (ST)
CoVoST is a many-to-one multilingual ST corpus  While end-to-end one-to-many and many-to-many multilingual ST models have been explored very recently BIBREF8, BIBREF9, many-to-one multilingual models, to our knowledge, have not  We hence use CoVoST to examine this setting  Table TABREF22 and TABREF23 show the BLEU scores for both bilingual and multilingual end-to-end ST models trained on CoVoST  We observe that combining speeches from multiple languages is consistently bringing gains to low-resource languages (all besides French and German)  This includes combinations of distant languages, such as Ru+Fr, Tr+Fr and Zh+Fr  Moreover, some combinations do bring gains to high-resource language (French) as well: Es+Fr, Tr+Fr and Mn+Fr  We simply provide the most basic many-to-one multilingual baselines here, and leave the full exploration of the best configurations to future work  Finally, we note that for some language pairs, absolute BLEU numbers are relatively low as we restrict model training to the supervised data  We encourage the community to improve upon those baselines, for example by leveraging semi-supervised training 
Baseline Results ::: Multi-Speaker Evaluation
In CoVoST, large portion of transcripts are covered by multiple speakers with different genders, accents and age groups  Besides the standard corpus-level BLEU scores, we also want to evaluate model output variance on the same content (transcript) but different speakers  We hence propose to group samples (and their sentence BLEU scores) by transcript, and then calculate average per-group mean and average coefficient of variation defined as follows:
and
where $G$ is the set of sentence BLEU scores grouped by transcript and $G^{\prime } = \lbrace g | g\in G, |g|>1, \textrm {Mean}(g) > 0 \rbrace $ 
$\textrm {BLEU}_{MS}$ provides a normalized quality score as oppose to corpus-level BLEU or unnormalized average of sentence BLEU  And $\textrm {CoefVar}_{MS}$ is a standardized measure of model stability against different speakers (the lower the better)  Table TABREF24 shows the $\textrm {BLEU}_{MS}$ and $\textrm {CoefVar}_{MS}$ of our ST models on CoVoST test set  We see that German and Persian have the worst $\textrm {CoefVar}_{MS}$ (least stable) given their rich speaker diversity in the test set and relatively small train set (see also Figure FIGREF6 and Table TABREF2)  Dutch also has poor $\textrm {CoefVar}_{MS}$ because of the lack of training data  Multilingual models are consistantly more stable on low-resource languages  Ru+Fr, Tr+Fr, Fa+Fr and Zh+Fr even have better $\textrm {CoefVar}_{MS}$ than all individual languages 
Conclusion
We introduce a multilingual speech-to-text translation corpus, CoVoST, for 11 languages into English, diversified with over 11,000 speakers and over 60 accents  We also provide baseline results, including, to our knowledge, the first end-to-end many-to-one multilingual model for spoken language translation  CoVoST is free to use with a CC0 license, and the additional Tatoeba evaluation samples are also CC-licensed 

ground_truth: No

No

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Are the automatically constructed datasets subject to quality control?
Context 2: Introduction
Automatically answering questions, especially in the open-domain setting (i e , where minimal or no contextual knowledge is explicitly provided), requires bringing to bear considerable amount of background knowledge and reasoning abilities  For example, knowing the answers to the two questions in Figure FIGREF1 requires identifying a specific ISA relation (i e , that cooking is a type of learned behavior) as well as recalling the definition of a concept (i e , that global warming is defined as a worldwide increase in temperature)  In the multiple-choice setting, which is the variety of question-answering (QA) that we focus on in this paper, there is also pragmatic reasoning involved in selecting optimal answer choices (e g , while greenhouse effect might in some other context be a reasonable answer to the second question in Figure FIGREF1, global warming is a preferable candidate) 
Recent successes in QA, driven largely by the creation of new resources BIBREF2, BIBREF3, BIBREF4, BIBREF5 and advances in model pre-training BIBREF6, BIBREF7, raise a natural question: do state-of-the-art multiple-choice QA (MCQA) models that excel at standard tasks really have basic knowledge and reasoning skills?
Most existing MCQA datasets are constructed through either expensive crowd-sourcing BIBREF8 or hand engineering effort, in the former case making it possible to collect large amounts of data at the cost of losing systematic control over the semantics of the target questions  Hence, doing a controlled experiment to answer such a question for QA is difficult given a lack of targeted challenge datasets 
Having definitive empirical evidence of model competence on any given phenomenon requires constructing a wide range of systematic tests  For example, in measuring competence of definitions, not only do we want to see that the model can handle individual questions such as Figure FIGREF1 1 inside of benchmark tasks, but that it can answer a wider range of questions that exhaustively cover a broad set of concepts and question perturbations (i e , systematic adjustments to how the questions are constructed)  The same applies to ISA reasoning; not only is it important to recognize in the question in Figure FIGREF1 1 that cooking is a learned behavior, but also that cooking is a general type of behavior or, through a few more inferential steps, a type of human activity 
In this paper, we look at systematically constructing such tests by exploiting the vast amounts of structured information contained in various types of expert knowledge such as knowledge graphs and lexical taxonomies  Our general methodology works as illustrated in Figure FIGREF1: given any MCQA model trained on a set of benchmark tasks, we systematically generate a set of synthetic dataset probes (i e , MCQA renderings of the target information) from information in expert knowledge sources  We then use these probes to ask two empirical questions: 1) how well do models trained on benchmark tasks perform on these probing tasks and; 2) can such models be re-trained to master new challenges with minimal performance loss on their original tasks?
While our methodology is amenable to any knowledge source and set of models/benchmark tasks, we focus on probing state-of-the-art transformer models BIBREF7, BIBREF9 in the domain of science MCQA  For sources of expert knowledge, we use WordNet, a comprehensive lexical ontology, and other publicly available dictionary resources  We devise probes that measure model competence in definition and taxonomic knowledge in different settings (including hypernymy, hyponymy, and synonymy detection, and word sense disambiguation)  This choice is motivated by fact that the science domain is considered particularly challenging for QA BIBREF10, BIBREF11, BIBREF12, and existing science benchmarks are known to involve widespread use of such knowledge (see BIBREF1, BIBREF13 for analysis), which is also arguably fundamental to more complex forms of reasoning 
We show that accurately probing QA models via synthetic datasets is not straightforward, as unexpected artifacts can easily arise in such data  This motivates our carefully constructed baselines and close data inspection to ensure probe quality 
Our results confirm that transformer-based QA models have a remarkable ability to recognize certain types of knowledge captured in our probes—even without additional fine-tuning  Such models can even outperform strong task-specific models trained directly on our probing tasks (e g , on definitions, our best model achieves 77% test accuracy without specialized training, as opposed to 51% for a task-specific LSTM-based model)  We also show that the same models can be effectively re-fine-tuned on small samples (even 100 examples) of probe data, and that high performance on the probes tends to correlate with a smaller drop in the model's performance on the original QA task 
Our comprehensive assessment reveals several interesting nuances to the overall positive trend  For example, the performance of even the best QA models degrades substantially on our hyponym probes (by 8-15%) when going from 1-hop links to 2-hops  Further, the accuracy of even our best models on the WordNetQA probe drops by 14-44% under our cluster-based analysis, which assesses whether a model knows several facts about each individual concept, rather than just being good at answering isolated questions  State-of-the-art QA models thus have much room to improve even in some fundamental building blocks, namely definitions and taxonomic hierarchies, of more complex forms of reasoning 
Related Work
We follow recent work on constructing challenge datasets for probing neural models, which has primarily focused on the task of natural language inference (NLI) BIBREF14, BIBREF15, BIBREF16, BIBREF17, BIBREF18  Most of this work looks at constructing data through adversarial generation methods, which have also been found useful for creating stronger models BIBREF19  There has also been work on using synthetic data of the type we consider in this paper BIBREF20, BIBREF21, BIBREF22  We closely follow the methodology of BIBREF22, who use hand-constructed linguistic fragments to probe NLI models and study model re-training using a variant of the inoculation by fine-tuning strategy of BIBREF23  In contrast, we focus on probing open-domain MCQA models (see BIBREF24 for a related study in the reading comprehension setting) as well as constructing data from much larger sources of structured knowledge 
Our main study focuses on probing the BERT model and fine-tuning approach of BIBREF7, and other variants thereof, which are all based on the transformer architecture of BIBREF25  Related to our efforts, there have been recent studies into the types of relational knowledge contained in large-scale knowledge models BIBREF26, BIBREF27, BIBREF28, which, similar to our work, probe models using structured knowledge sources  This prior work, however, primarily focuses on unearthing the knowledge contained in the underlying language models as is without further training, using simple (single token) cloze-style probing tasks and templates (similar to what we propose in Section SECREF3)  In contrast, we focus on understanding the knowledge contained in language models after they have been trained for a QA end-task using benchmark datasets in which such knowledge is expected to be widespread  Further, our evaluation is done before and after these models are fine-tuned on our probe QA tasks, using a more complex set of QA templates and target inferences 
The use of lexical resources and knowledge graphs such as WordNet to construct datasets has a long history, and has recently appeared in work on adversarial attacks BIBREF14, BIBREF29 and general task construction BIBREF30, BIBREF31  In the area of MCQA, there is related work on constructing questions from tuples BIBREF32, BIBREF3, both of which involve standard crowd annotation to elicit question-answer pairs (see also BIBREF33, BIBREF34)  In contrast to this work, we focus on generating data in an entirely automatic fashion, which obviates the need for expensive annotation and gives us the flexibility to construct much larger datasets that control a rich set of semantic aspects of the target questions 
Dataset Probes and Construction
Our probing methodology starts by constructing challenge datasets (Figure FIGREF1, yellow box) from a target set of knowledge resources  Each of our probing datasets consists of multiple-choice questions that include a question $\textbf {q}$ and a set of answer choices or candidates $\lbrace a_{1},   a_{N}\rbrace $  This section describes in detail the 5 different datasets we build, which are drawn from two sources of expert knowledge, namely WordNet BIBREF35 and the GNU Collaborative International Dictionary of English (GCIDE)  We describe each resource in turn, and explain how the resulting dataset probes, which we call WordNetQA and DictionaryQA, are constructed 
For convenience, we will describe each source of expert knowledge as a directed, edge-labeled graph $G$  The nodes of this graph are $\mathcal {V} = \mathcal {C} \cup \mathcal {W} \cup \mathcal {S} \cup \mathcal {D}$, where $\mathcal {C}$ is a set of atomic concepts, $\mathcal {W}$ a set of words, $\mathcal {S}$ a set of sentences, and $\mathcal {D}$ a set of definitions (see Table TABREF4 for details for WordNet and GCIDE)  Each edge of $G$ is directed from an atomic concept in $\mathcal {C}$ to another node in $V$, and is labeled with a relation, such as hypernym or isa$^\uparrow $, from a set of relations $\mathcal {R}$ (see Table TABREF4) 
When defining our probe question templates, it will be useful to view $G$ as a set of (relation, source, target) triples $\mathcal {T} \subseteq \mathcal {R} \times \mathcal {C} \times \mathcal {V}$  Due to their origin in an expert knowledge source, such triples preserve semantic consistency  For instance, when the relation in a triple is def, the corresponding edge maps a concept in $\mathcal {C}$ to a definition in $\mathcal {D}$ 
To construct probe datasets, we rely on two heuristic functions, defined below for each individual probe: $\textsc {gen}_{\mathcal {Q}}(\tau )$, which generates gold question-answer pairs $(\textbf {q},\textbf {a})$ from a set of triples $\tau \subseteq \mathcal {T}$ and question templates $\mathcal {Q}$, and $\textsc {distr}(\tau ^{\prime })$, which generates distractor answers choices $\lbrace a^{\prime }_{1},   a^{\prime }_{N-1} \rbrace $ based on another set of triples $\tau ^{\prime }$ (where usually $\tau \subset \tau ^{\prime }$)  For brevity, we will use $\textsc {gen}(\tau )$ to denote $\textsc {gen}_{\mathcal {Q}}(\tau )$, leaving question templates $\mathcal {Q}$ implicit 
Dataset Probes and Construction ::: WordNetQA
WordNet is an English lexical database consisting of around 117k concepts, which are organized into groups of synsets that each contain a gloss (i e , a definition of the target concept), a set of representative English words (called lemmas), and, in around 33k synsets, example sentences  In addition, many synsets have ISA links to other synsets that express complex taxonomic relations  Figure FIGREF6 shows an example and Table TABREF4 summarizes how we formulate WordNet as a set of triples $\mathcal {T}$ of various types  These triples together represent a directed, edge-labeled graph $G$  Our main motivation for using WordNet, as opposed to a resource such as ConceptNet BIBREF36, is the availability of glosses ($\mathcal {D}$) and example sentences ($\mathcal {S}$), which allows us to construct natural language questions that contextualize the types of concepts we want to probe 
Dataset Probes and Construction ::: WordNetQA ::: Example Generation @!START@$\textsc {gen}(\tau )$@!END@ 
We build 4 individual datasets based on semantic relations native to WordNet (see BIBREF37): hypernymy (i e , generalization or ISA reasoning up a taxonomy, ISA$^\uparrow $), hyponymy (ISA$^{\downarrow }$), synonymy, and definitions  To generate a set of questions in each case, we employ a number of rule templates $\mathcal {Q}$ that operate over tuples  A subset of such templates is shown in Table TABREF8  The templates were designed to mimic naturalistic questions we observed in our science benchmarks 
For example, suppose we wish to create a question $\textbf {q}$ about the definition of a target concept $c \in \mathcal {C}$  We first select a question template from $\mathcal {Q}$ that first introduces the concept $c$ and its lemma $l \in \mathcal {W}$ in context using the example sentence $s \in \mathcal {S}$, and then asks to identify the corresponding WordNet gloss $d \in \mathcal {D}$, which serves as the gold answer $\textbf {a}$  The same is done for ISA reasoning; each question about a hypernym/hyponym relation between two concepts $c \rightarrow ^{\uparrow /\downarrow } c^{\prime } \in \mathcal {T}_{i}$ (e g , $\texttt {dog} \rightarrow ^{\uparrow /\downarrow } \texttt {animal/terrier}$) first introduces a context for $c$ and then asks for an answer that identifies $c^{\prime }$ (which is also provided with a gloss so as to contain all available context) 
In the latter case, the rules $(\texttt {isa}^{r},c,c^{\prime }) \in \mathcal {T}_i$ in Table TABREF8 cover only direct ISA links from $c$ in direction $r \in \lbrace \uparrow ,\downarrow \rbrace $  In practice, for each $c$ and direction $r$, we construct tests that cover the set HOPS$(c,r)$ of all direct as well as derived ISA relations of $c$:
This allows us to evaluate the extent to which models are able to handle complex forms of reasoning that require several inferential steps or hops 
Dataset Probes and Construction ::: WordNetQA ::: Distractor Generation: @!START@$\textsc {distr}(\tau ^{\prime })$@!END@ 
An example of how distractors are generated is shown in Figure FIGREF6, which relies on similar principles as above  For each concept $c$, we choose 4 distractor answers that are close in the WordNet semantic space  For example, when constructing hypernymy tests for $c$ from the set hops$(c,\uparrow )$, we build distractors by drawing from $\textsc {hops}(c,\downarrow )$ (and vice versa), as well as from the $\ell $-deep sister family of $c$, defined as follows  The 1-deep sister family is simply $c$'s siblings or sisters, i e , the other children $\tilde{c} \ne c$ of the parent node $c^{\prime }$ of $c$  For $\ell > 1$, the $\ell $-deep sister family also includes all descendants of each $\tilde{c}$ up to $\ell -1$ levels deep, denoted $\textsc {hops}_{\ell -1}(\tilde{c},\downarrow )$  Formally:
For definitions and synonyms we build distractors from all of these sets (with a similar restriction on the depth of sister distractors as noted above)  In doing this, we can systematically investigate model performance on a wide range of distractor sets 
Dataset Probes and Construction ::: WordNetQA ::: Perturbations and Semantic Clusters
Based on how we generate data, for each concept $c$ (i e , atomic WordNet synset) and probe type (i e , definitions, hypernymy, etc ), we have a wide variety of questions related to $c$ that manipulate 1) the complexity of reasoning that is involved (e g , the number of inferential hops) and; 2) the types of distractors (or distractor perturbations) that are employed  We call such sets semantic clusters  As we describe in the next section, semantic clusters allow us to devise new types of evaluation that reveal whether models have comprehensive and consistent knowledge of target concepts (e g , evaluating whether a model can correctly answer several questions associated with a concept, as opposed to a few disjoint instances) 
Details of the individual datasets are shown in Table TABREF12  From these sets, we follow BIBREF22 in allocating a maximum of 3k examples for training and reserve the rest for development and testing  Since we are interested in probing, having large held-out sets allows us to do detailed analysis and cluster-based evaluation 
Dataset Probes and Construction ::: DictionaryQA
The DictionaryQA dataset is created from the GCIDE dictionary, which is a comprehensive open-source English dictionary built largely from the Webster's Revised Unabridged Dictionary BIBREF38  Each entry consists of a word, its part-of-speech, its definition, and an optional example sentence (see Table TABREF14)  Overall, 33k entries (out of a total of 155k) contain example sentences/usages  As with the WordNet probes, we focus on this subset so as to contextualize each word being probed  In contrast to WordNet, GCIDE does not have ISA relations or explicit synsets, so we take each unique entry to be a distinct sense  We then use the dictionary entries to create a probe that centers around word-sense disambiguation, as described below 
Dataset Probes and Construction ::: DictionaryQA ::: Example and Distractor Generation 
To generate gold questions and answers, we use the same generation templates for definitions exemplified in Figure TABREF8 for WordNetQA  To generate distractors, we simply take alternative definitions for the target words that represent a different word sense (e g , the alternative definitions of gift shown in Table TABREF14), as well as randomly chosen definitions if needed to create a 5-way multiple choice question  As above, we reserve a maximum of 3k examples for training  Since we have only 9k examples in total in this dataset (see WordSense in Table TABREF12), we also reserve 3k each for development and testing 
We note that initial attempts to build this dataset through standard random splitting gave rise to certain systematic biases that were exploited by the choice-only baseline models described in the next section, and hence inflated overall model scores  After several efforts at filtering we found that, among other factors, using definitions from entries without example sentences as distractors (e g , the first two entries in Table TABREF14) had a surprising correlation with such biases  This suggests that possible biases involving differences between dictionary entries with and without examples can taint the resulting automatically generated MCQA dataset (for more discussion on the pitfalls involved with automatic dataset construction, see Section SECREF5) 
Probing Methodology and Modeling
Given the probes above, we now can start to answer the empirical questions posed at the beginning  Our main focus is on looking at transformer-based MCQA models trained in the science domain (using the benchmarks shown in Table TABREF21)  In this section, we provide details of MCQA and the target models, as well as several baselines that we use to sanity check our new datasets  To evaluate model competence, we look at a combination of model performance after science pre-training and after additional model fine-tuning using the lossless inoculation strategy of BIBREF22 (Section SECREF22)  In Section SECREF24, we also discuss a cluster-level accuracy metric for measuring performance over semantic clusters 
Probing Methodology and Modeling ::: Task Definition and Modeling
Given a dataset $D =\lbrace (\textbf {q}^{(d)}, \lbrace a_{1}^{(d)},   , a_{N}^{(d)}\rbrace ) \rbrace _{d}^{\mid D \mid }$ consisting of pairs of questions stems $\textbf {q}$ and answer choices $a_{i}$, the goal is to find the correct answer $a_{i^{*}}$ that correctly answers each $\textbf {q}$  Throughout this paper, we look at 5-way multiple-choice problems (i e , where each $N=5$) 
Probing Methodology and Modeling ::: Task Definition and Modeling ::: Question+Answer Encoder 
To model this, our investigation centers around the use of the transformer-based BIBREF25 BERT encoder and fine-tuning approach of BIBREF7 (see also BIBREF6)  For each question and individual answer pair $q^{(j)}_{a_{i}}$, we assume the following rendering of this input:
which is run through the pre-trained BERT encoder to generate a representation for $ q^{(j)}_{a_{i}}$ using the hidden state representation for CLS (i e , the classifier token) $\textbf {c}_{i}$:
The probability of a given answer $p^{(j)}_{i}$ is then computed as $p^{(j)}_{i} \propto e^{\textbf {v}\cdot \textbf {c}^{(j)}_{i}}$, which uses an additional set of classification parameters $\textbf {v} \in \mathbb {R}^{H}$ that are optimized (along with the full transformer network) by taking the final loss of the probability of each correct answer $p_{i^{*}}$ over all answer choices:
We specifically use BERT-large uncased with whole-word masking, as well as the RoBERTa-large model from BIBREF9, which is a more robustly trained version of the original BERT model  Our system uses the implementations provided in AllenNLP BIBREF39 and Huggingface BIBREF40 
Probing Methodology and Modeling ::: Task Definition and Modeling ::: Baselines and Sanity Checks 
When creating synthetic datasets, it is important to ensure that systematic biases, or annotation artifacts BIBREF41, are not introduced into the resulting probes and that the target datasets are sufficiently challenging (or good, in the sense of BIBREF42)  To test for this, we use several of the MCQA baseline models first introduced in BIBREF0, which take inspiration from the LSTM-based models used in BIBREF43 for NLI and various partial-input baselines based on these models 
Following the notation from BIBREF0, for any given sequence $s$ of tokens in $\lbrace q^{(j)}, a_{1}^{(j)},   ,a_{N}^{(j)}\rbrace $ in $D$, an encoding of $s$ is given as $h_{s}^{(j)} = \textbf {BiLSTM}(\textsc {embed}(s)) \in \mathbb {R}^{|s| \times 2h}$ (where $h$ is the dimension of the hidden state in each directional network, and embed$(\cdot )$ is an embedding function that assigns token-level embeddings to each token in $s$)  A contextual representation for each $s$ is then built by applying an element-wise max operation over $h_{s}$ as follows:
With these contextual representations, different baseline models can be constructed  For example, a Choice-Only model, which is a variant of the well-known hypothesis-only baseline used in NLI BIBREF46, scores each choice $c_{i}$ in the following way:
for $\textbf {W}^{T} \in \mathbb {R}^{2h}$ independently of the question and assigns a probability to each answer $p_{i}^{(j)} \propto e^{\alpha _{i}^{(j)}}$ 
A slight variant of this model, the Choice-to-choice model, tries to single out a given answer choice relative to other choices by scoring all choice pairs $\alpha _{i,i^{\prime }}^{(j)} = \textsc {Att}(r^{(j)}_{c_{i}},r^{(j)}_{c_{i^{\prime }}}) \in \mathbb {R}$ using a learned attention mechanism Att and finding the choice with the minimal similarity to other options (for full details, see their original paper)  In using these partial-input baselines, which we train directly on each target probe, we can check whether systematic biases related to answer choices were introduced into the data creation process 
A Question-to-choice model, in contrast, uses the contextual representations for each question and individual choice and an attention model Att model to get a score $\alpha ^{(j)}_{q,i} = \textsc {Att}(r^{(j)}_{q},r^{(j)}_{c_{i}}) \in \mathbb {R}$ as above  Here we also experiment with using ESIM BIBREF47 to generate the contextual representations $r$, as well as a simpler VecSimilarity model that measures the average vector similarity between question and answer tokens: $\alpha ^{(j)}_{q,i} = \textsc {Sim}(\textsc {embed}(q^{(j)}),\textsc {embed}(c^{(j)}_{i}))$  In contrast to the models above, these sets of baselines are used to check for artifacts between questions and answers that are not captured in the partial-input baselines (see discussion in BIBREF49) and ensure that the overall MCQA tasks are sufficiently difficult for our transformer models 
Probing Methodology and Modeling ::: Inoculation and Pre-training
Using the various models introduced above, we train these models on benchmark tasks in the science domain and look at model performance on our probes with and without additional training on samples of probe data, building on the idea of inoculation from BIBREF23  Model inoculation is the idea of continuing to train models on new challenge tasks (in our cases, separately for each probe) using only a small amount of examples  Unlike in ordinary fine-tuning, the goal is not to learn an entirely re-purposed model, but to improve on (or vaccinate against) particular phenomena (e g , our synthetic probes) that potentially deviate from a model's original training distribution (but that nonetheless might involve knowledge already contained in the model) 
In the variant proposed in BIBREF22, for each pre-trained (science) model and architecture $M_{a}$ we continue training the model on $k$ new probe examples (with a maximum of $k=$ 3k) under a set of different hyper-parameter configurations $j \in \lbrace 1,    , J\rbrace $ and identify, for each $k$, the model $M_{*}^{a,k}$ with the best aggregate performance $S$ on the original (orig) and new task:
As in BIBREF22, we found all models to be especially sensitive to different learning rates, and performed comprehensive hyper-parameters searches that also manipulate the number of iterations and random seeds used 
Using this methodology, we can see how much exposure to new data it takes for a given model to master a new task, and whether there are phenomena that stress particular models (e g , lead to catastrophic forgetting of the original task)  Given the restrictions on the number of fine-tuning examples, our assumption is that when models are able to maintain good performance on their original task during inoculation, the quickness with which they are able to learn the inoculated task provides evidence of prior competence, which is precisely what we aim to probe  To measure past performance, we define a model's inoculation cost as the difference in the performance of this model on its original task before and after inoculation 
We pre-train on an aggregated training set of the benchmark science exams detailed in Table TABREF21, and created an aggregate development set of around 4k science questions for evaluating overall science performance and inoculation costs  To handle the mismatch between number of answer choices in these sets, we made all sets 5-way by adding empty answers as needed  We also experimented with a slight variant of inoculation, called add-some inoculation, which involves balancing the inoculation training sets with naturalistic science questions  We reserve the MCQL dataset in Table TABREF21 for this purpose, and experiment with balancing each probe example with a science example (x1 matching) and adding twice as many science questions (x2 matching, up to 3k) for each new example 
Probing Methodology and Modeling ::: Evaluating Model Competence
The standard way to evaluate our MCQA models is by looking at the overall accuracy of the correct answer prediction, or what we call instance-level accuracy (as in Table TABREF25)  Given the nature of our data and the existence of semantic clusters as detailed in Section SECREF11 (i e , sets of questions and answers under different distractor choices and inference complexity), we also measure a model's cluster-level (or strict cluster) accuracy, which requires correctly answering all questions in a cluster  Example semantic clusters are shown in Table TABREF30; in the first case, there are 6 ISA$^\uparrow $ questions (including perturbations) about the concept trouser n 01 (e g , involving knowing that trousers are a type of consumer good and garment/clothing), which a model must answer in order to receive full credit 
Our cluster-based analysis is motivated by the idea that if a model truly knows the meaning of a given concept, such as the concept of trousers, then it should be able to answer arbitrary questions about this concept without sensitivity to varied distractors  While our strict cluster metric is simplistic, it takes inspiration from work on visual QA BIBREF53, and allows us to evaluate how consistent and robust models are across our different probes, and to get insight into whether errors are concentrated on a small set of concepts or widespread across clusters 
Results and Findings
In this section, we provide the results of the empirical questions first introduced in Figure FIGREF1, starting with the results of our baseline models 
Results and Findings ::: Are our Probes Sufficiently Challenging?
As shown in Table TABREF25, most of our partial-input baselines (i e , Choice-Only and Choice-to-Choice models) failed to perform well on our dataset probes across a wide range of models, showing that such probes are generally immune from biases relating to how distractors were generated  As already discussed in Section SECREF13, however, initial versions of the DictionaryQA dataset had unforeseen biases partly related to whether distractors were sampled from entries without example sentences, which resulted in high Choice-Only-GloVe scores ranging around 56% accuracy before a filtering step was applied to remove these distractors 
We had similar issues with the hypernymy probe which, even after a filtering step that used our Choice-to-Choice-GloVe model, still leads to high results on the BERT and RoBERTa choice-only models  Given that several attempts were made to entirely de-duplicate the different splits (both in terms of gold answers and distractor types), the source of these biases is not at all obvious, which shows how easy it is for unintended biases in expert knowledge to appear in the resulting datasets and the importance of having rigorous baselines  We also note the large gap in some cases between the BERT and RoBERTa versus GloVe choice-only models, which highlights the need for having partial-input baselines that use the best available models 
Using a more conventional set of Task-Specific QA models (i e , the LSTM-based Question-to-Choice models trained directly on the probes), we can see that results are not particularly strong on any of the datasets, suggesting that our probes are indeed sufficiently challenging and largely immune from overt artifacts  The poor performance of the VecSimilarity (which uses pre-trained Word2Vec embeddings without additional training) provides additional evidence that elementary lexical matching strategies are insufficient for solving any of the probing tasks 
Results and Findings ::: How well do pre-trained MCQA models do?
Science models that use non-transformer based encoders, such as the ESIM model with GloVe and ELMO, perform poorly across all probes, in many cases scoring near random chance, showing limits to how well they generalize from science to other tasks even with pre-trained GloVe and ELMO embeddings  In sharp contrast, the transformer models have mixed results, the most striking result being the RoBERTa models on the definitions and synonymy probes (achieving a test accuracy of 77% and 61%, respectively), which outperform several of the task-specific LSTM models trained directly on the probes  At first glance, this suggests that RoBERTa, which generally far outpaces even BERT across most probes, has high competence of definitions and synonyms even without explicit training on our new tasks 
Given the controlled nature of our probes, we can get a more detailed view of how well the science models are performing across different reasoning and distractor types, as shown in the first column of Figure FIGREF28 for ESIM and RoBERTa  The ESIM science model without training has uniformly poor performance across all categories, whereas the performance of RoBERTa is more varied  Across all datasets and number of hops (i e , the rows in the heat maps), model performance for RoBERTa is consistently highest among examples with random distractors (i e , the first column), and lowest in cases involving distractors that are closest in WordNet space (e g , sister and ISA, or up/down, distractors of distance $k^{\prime }=1$)  This is not surprising, given that, in the first case, random distractors are likely to be the easiest category (and the opposite for distractors close in space), but suggests that RoBERTa might only be getting the easiest cases correct 
Model performance also clearly degrades for hypernymy and hyponymy across all models as the number of hops $k$ increases (see red dashed boxes)  For example, problems that involve hyponym reasoning with sister distractors of distance $k^{\prime }=1$ (i e , the second column) degrades from 47% to 15% when the number of hops $k$ increases from 1 to 4  This general tendency persists even after additional fine-tuning, as we discuss next, and gives evidence that models are limited in their capacity for certain types of multi-hop inferences 
As discussed by BIBREF26, the choice of generation templates can have a significant effect on model performance  The results so far should therefore be regarded as a lower bound on model competence  It is possible that model performance is high for definitions, for example, because the associated templates best align with the science training distribution (which we know little about)  For this reason, the subsequent inoculation step is important—it gives the model an opportunity to learn about our target templates and couple this learned knowledge with its general knowledge acquired during pre-training and science training (which is, again, what we aim to probe) 
Results and Findings ::: Can Models Be Effectively Inoculated?
Model performance after additional fine-tuning, or inoculation, is shown in the last 3 rows of Table TABREF25, along with learning curves shown in Figure FIGREF29 for a selection of probes and models  In the former case, the performance represents the model (and inoculation amount) with the highest aggregate performance over the old task and new probe  Here we again see the transformer-based models outperform non-transformer models, and that better models correlate with lower inoculation costs  For example, when inoculating on synonymy, the cost for ESIM is around 7% reduced accuracy on its original task, as opposed to $< 1$% and around 1% for BERT and RoBERTa, respectively  This shows the high capacity for transformer models to absorb new tasks with minimal costs, as also observed in BIBREF22 for NLI 
As shown in Figure FIGREF29, transformer models tend to learn most tasks fairly quickly while keeping constant scores on their original tasks (i e , the flat dashed lines observed in plots 1-4), which gives evidence of high competence  In both cases, add-some inoculation proves to be a cheap and easy way to 1) improve scores on the probing tasks (i e , the solid black and blue lines in plot 1) and; 2) minimize loss on science (e g , the blue and black dashed lines in plots 2-4)  The opposite is the case for ESIM (plots 5-6); models are generally unable to simultaneously learn individual probes without degrading on their original task, and adding more science data during inoculation confuses models on both tasks 
As shown in Figure FIGREF28, RoBERTa is able to significantly improve performance across most categories even after inoculation with a mere 100 examples (the middle plot), which again provides strong evidence of prior competence  As an example, RoBERTa improves on 2-hop hyponymy inference with random distractors by 18% (from 59% to 77%)  After 3k examples, the model has high performance on virtually all categories (the same score increases from 59% to 87%), however results still tends to degrade as a function of hop and distractor complexity, as discussed above 
Despite the high performance of our transformer models after inoculation, model performance on most probes (with the exception of Definitions) averages around 80% for our best models  This suggests that there is still considerable room for improvement, especially for synonymy and word sense, which is a topic that we discuss more in Section SECREF6 
Results and Findings ::: Are Models Consistent across Clusters?
Table TABREF32 shows cluster-level accuracies for the different WordNetQA probes  As with performance across the different inference/distractor categories, these results are mixed  For some probes, such as definitions, our best models appear to be rather robust; e g , our RoBERTa model has a cluster accuracy of $75\%$, meaning that it can answer all questions perfectly for 75% of the target concepts and that errors are concentrated on a small minority (25%) of concepts  On synonymy and hypernymy, both BERT and RoBERTa appear robust on the majority of concepts, showing that errors are similarly concentrated  In contrast, our best model on hyponymy has an accuracy of 36%, meaning that its errors are spread across many concepts, thus suggesting less robustness 
Table TABREF30 shows a selection of semantic clusters involving ISA reasoning, as well as the model performance over different answers (shown symbolically) and perturbations  For example, in the the second case, the cluster is based around the concept/synset oppose v 06 and involves 4 inferences and a total 24 questions (i e , inferences with perturbations)  Our weakest model, ESIM, answers only 5 out of 24 questions correctly, whereas RoBERTa gets 21/24  In the other cases, RoBERTa gets all clusters correct, whereas BERT and ESIM get none of them correct 
We emphasize that these results only provide a crude look into model consistency and robustness  Recalling again the details in Table TABREF12, probes differ in terms of average size of clusters  Hyponymy, in virtue of having many more questions per cluster, might simply be a much more difficult dataset  In addition, such a strict evaluation does not take into account potential errors inside of clusters, which is an important issue that we discuss in the next section  We leave addressing such issues and coming up with more insightful cluster-based metrics for future work 
Discussion and Conclusion
We presented several new challenge datasets and a novel methodology for automatically building such datasets from knowledge graphs and taxonomies  We used these to probe state-of-the-art open-domain QA models (centering around models based on variants of BERT)  While our general methodology is amendable to any target knowledge resource or QA model/domain, we focus on probing definitions and ISA knowledge using open-source dictionaries and MCQA models trained in the science domain 
We find, consistent with recent probing studies BIBREF26, that transformer-based models have a remarkable ability to answer questions that involve complex forms of relational knowledge, both with and without explicit exposure to our new target tasks  In the latter case, a newer RoBERTa model trained only on benchmark science tasks is able to outperform several task-specific LSTM-based models trained directly on our probing data  When re-trained on small samples (e g , 100 examples) of probing data using variations of the lossless inoculation strategy from BIBREF22, RoBERTa is able to master many aspects of our probes with virtually no performance loss on its original QA task 
These positive results suggest that transformer-based models, especially models additionally fine-tuned on small samples of synthetic data, can be used in place of task-specific models used for querying relational knowledge, as has already been done for targeted tasks such as word sense disambiguation BIBREF54  Since models seem to already contain considerable amounts of relational knowledge, our simple inoculation strategy, which tries to nudge models to bring out this knowledge explicitly, could serve as a cheaper alternative to recent attempts to build architectures that explicitly incorporate structured knowledge BIBREF55; we see many areas where our inoculation strategy could be improved for such purposes, including having more complex loss functions that manage old and new information, as well as using techniques that take into account network plasticity BIBREF56 
The main appeal of using automatically generate datasets is the ability to systematically manipulate and control the complexity of target questions, which allows for more controlled experimentation and new forms of evaluation  Despite the positive results described above, results that look directly at the effect of different types of distractors and the complexity of reasoning show that our best models, even after additional fine-tuning, struggle with certain categories of hard distractors and multi-hop inferences  For some probes, our cluster-based analysis also reveals that errors are widespread across concept clusters, suggesting that models are not always consistent and robust  These results, taken together with our findings about the vulnerability of synthetic datasets to systematic biases, suggest that there is much room for improvement and that the positive results should be taken with a grain of salt  Developing better ways to evaluate semantic clusters and model robustness would be a step in this direction 
We emphasize that using synthetic versus naturalistic QA data comes with important trade-offs  While we are able to generate large amounts of systematically controlled data at virtually no cost or need for manual annotation, it is much harder to validate the quality of such data at such a scale and such varying levels of complexity  Conversely, with benchmark QA datasets, it is much harder to perform the type of careful manipulations and cluster-based analyses we report here  While we assume that the expert knowledge we employ, in virtue of being hand-curated by human experts, is generally correct, we know that such resources are fallible and error-prone  Initial crowd-sourcing experiments that look at validating samples of our data show high agreement across probes and that human scores correlate with the model trends across the probe categories  More details of these studies are left for future work 
Context 3: Introduction
Performance appraisal (PA) is an important HR process, particularly for modern organizations that crucially depend on the skills and expertise of their workforce  The PA process enables an organization to periodically measure and evaluate every employee's performance  It also provides a mechanism to link the goals established by the organization to its each employee's day-to-day activities and performance  Design and analysis of PA processes is a lively area of research within the HR community BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3  
The PA process in any modern organization is nowadays implemented and tracked through an IT system (the PA system) that records the interactions that happen in various steps  Availability of this data in a computer-readable database opens up opportunities to analyze it using automated statistical, data-mining and text-mining techniques, to generate novel and actionable insights / patterns and to help in improving the quality and effectiveness of the PA process BIBREF4 , BIBREF5 , BIBREF6   Automated analysis of large-scale PA data is now facilitated by technological and algorithmic advances, and is becoming essential for large organizations containing thousands of geographically distributed employees handling a wide variety of roles and tasks 
A typical PA process involves purposeful multi-step multi-modal communication between employees, their supervisors and their peers  In most PA processes, the communication includes the following steps: (i) in self-appraisal, an employee records his/her achievements, activities, tasks handled etc ; (ii) in supervisor assessment, the supervisor provides the criticism, evaluation and suggestions for improvement of performance etc ; and (iii) in peer feedback (aka INLINEFORM0 view), the peers of the employee provide their feedback  There are several business questions that managers are interested in  Examples:
In this paper, we develop text mining techniques that can automatically produce answers to these questions  Since the intended users are HR executives, ideally, the techniques should work with minimum training data and experimentation with parameter setting  These techniques have been implemented and are being used in a PA system in a large multi-national IT company 
The rest of the paper is organized as follows  Section SECREF2 summarizes related work  Section SECREF3 summarizes the PA dataset used in this paper  Section SECREF4 applies sentence classification algorithms to automatically discover three important classes of sentences in the PA corpus viz , sentences that discuss strengths, weaknesses of employees and contain suggestions for improving her performance  Section SECREF5 considers the problem of mapping the actual targets mentioned in strengths, weaknesses and suggestions to a fixed set of attributes  In Section SECREF6 , we discuss how the feedback from peers for a particular employee can be summarized  In Section SECREF7 we draw conclusions and identify some further work 
Related Work
We first review some work related to sentence classification  Semantically classifying sentences (based on the sentence's purpose) is a much harder task, and is gaining increasing attention from linguists and NLP researchers  McKnight and Srinivasan BIBREF7 and Yamamoto and Takagi BIBREF8 used SVM to classify sentences in biomedical abstracts into classes such as INTRODUCTION, BACKGROUND, PURPOSE, METHOD, RESULT, CONCLUSION  Cohen et al  BIBREF9 applied SVM and other techniques to learn classifiers for sentences in emails into classes, which are speech acts defined by a verb-noun pair, with verbs such as request, propose, amend, commit, deliver and nouns such as meeting, document, committee; see also BIBREF10   Khoo et al  BIBREF11 uses various classifiers to classify sentences in emails into classes such as APOLOGY, INSTRUCTION, QUESTION, REQUEST, SALUTATION, STATEMENT, SUGGESTION, THANKING etc  Qadir and Riloff BIBREF12 proposes several filters and classifiers to classify sentences on message boards (community QA systems) into 4 speech acts: COMMISSIVE (speaker commits to a future action), DIRECTIVE (speaker expects listener to take some action), EXPRESSIVE (speaker expresses his or her psychological state to the listener), REPRESENTATIVE (represents the speaker's belief of something)  Hachey and Grover BIBREF13 used SVM and maximum entropy classifiers to classify sentences in legal documents into classes such as FACT, PROCEEDINGS, BACKGROUND, FRAMING, DISPOSAL; see also BIBREF14   Deshpande et al  BIBREF15 proposes unsupervised linguistic patterns to classify sentences into classes SUGGESTION, COMPLAINT 
There is much work on a closely related problem viz , classifying sentences in dialogues through dialogue-specific categories called dialogue acts BIBREF16 , which we will not review here  Just as one example, Cotterill BIBREF17 classifies questions in emails into the dialogue acts of YES_NO_QUESTION, WH_QUESTION, ACTION_REQUEST, RHETORICAL, MULTIPLE_CHOICE etc 
We could not find much work related to mining of performance appraisals data  Pawar et al  BIBREF18 uses kernel-based classification to classify sentences in both performance appraisal text and product reviews into classes SUGGESTION, APPRECIATION, COMPLAINT  Apte et al  BIBREF6 provides two algorithms for matching the descriptions of goals or tasks assigned to employees to a standard template of model goals  One algorithm is based on the co-training framework and uses goal descriptions and self-appraisal comments as two separate perspectives  The second approach uses semantic similarity under a weak supervision framework  Ramrakhiyani et al  BIBREF5 proposes label propagation algorithms to discover aspects in supervisor assessments in performance appraisals, where an aspect is modelled as a verb-noun pair (e g  conduct training, improve coding) 
Dataset
In this paper, we used the supervisor assessment and peer feedback text produced during the performance appraisal of 4528 employees in a large multi-national IT company  The corpus of supervisor assessment has 26972 sentences  The summary statistics about the number of words in a sentence is: min:4 max:217 average:15 5 STDEV:9 2 Q1:9 Q2:14 Q3:19 
Sentence Classification
The PA corpus contains several classes of sentences that are of interest  In this paper, we focus on three important classes of sentences viz , sentences that discuss strengths (class STRENGTH), weaknesses of employees (class WEAKNESS) and suggestions for improving her performance (class SUGGESTION)  The strengths or weaknesses are mostly about the performance in work carried out, but sometimes they can be about the working style or other personal qualities  The classes WEAKNESS and SUGGESTION are somewhat overlapping; e g , a suggestion may address a perceived weakness  Following are two example sentences in each class 
STRENGTH:
WEAKNESS:
SUGGESTION:
Several linguistic aspects of these classes of sentences are apparent  The subject is implicit in many sentences  The strengths are often mentioned as either noun phrases (NP) with positive adjectives (Excellent technology leadership) or positive nouns (engineering strength) or through verbs with positive polarity (dedicated) or as verb phrases containing positive adjectives (delivers innovative solutions)  Similarly for weaknesses, where negation is more frequently used (presentations are not his forte), or alternatively, the polarities of verbs (avoid) or adjectives (poor) tend to be negative  However, sometimes the form of both the strengths and weaknesses is the same, typically a stand-alone sentiment-neutral NP, making it difficult to distinguish between them; e g , adherence to timing or timely closure  Suggestions often have an imperative mood and contain secondary verbs such as need to, should, has to  Suggestions are sometimes expressed using comparatives (better process compliance)  We built a simple set of patterns for each of the 3 classes on the POS-tagged form of the sentences  We use each set of these patterns as an unsupervised sentence classifier for that class  If a particular sentence matched with patterns for multiple classes, then we have simple tie-breaking rules for picking the final class  The pattern for the STRENGTH class looks for the presence of positive words / phrases like takes ownership, excellent, hard working, commitment, etc  Similarly, the pattern for the WEAKNESS class looks for the presence of negative words / phrases like lacking, diffident, slow learner, less focused, etc  The SUGGESTION pattern not only looks for keywords like should, needs to but also for POS based pattern like “a verb in the base form (VB) in the beginning of a sentence” 
We randomly selected 2000 sentences from the supervisor assessment corpus and manually tagged them (dataset D1)  This labelled dataset contained 705, 103, 822 and 370 sentences having the class labels STRENGTH, WEAKNESS, SUGGESTION or OTHER respectively  We trained several multi-class classifiers on this dataset  Table TABREF10 shows the results of 5-fold cross-validation experiments on dataset D1  For the first 5 classifiers, we used their implementation from the SciKit Learn library in Python (scikit-learn org)  The features used for these classifiers were simply the sentence words along with their frequencies  For the last 2 classifiers (in Table TABREF10 ), we used our own implementation  The overall accuracy for a classifier is defined as INLINEFORM0 , where the denominator is 2000 for dataset D1  Note that the pattern-based approach is unsupervised i e , it did not use any training data  Hence, the results shown for it are for the entire dataset and not based on cross-validation 
Comparison with Sentiment Analyzer
We also explored whether a sentiment analyzer can be used as a baseline for identifying the class labels STRENGTH and WEAKNESS  We used an implementation of sentiment analyzer from TextBlob to get a polarity score for each sentence  Table TABREF13 shows the distribution of positive, negative and neutral sentiments across the 3 class labels STRENGTH, WEAKNESS and SUGGESTION  It can be observed that distribution of positive and negative sentiments is almost similar in STRENGTH as well as SUGGESTION sentences, hence we can conclude that the information about sentiments is not much useful for our classification problem 
Discovering Clusters within Sentence Classes
After identifying sentences in each class, we can now answer question (1) in Section SECREF1   From 12742 sentences predicted to have label STRENGTH, we extract nouns that indicate the actual strength, and cluster them using a simple clustering algorithm which uses the cosine similarity between word embeddings of these nouns  We repeat this for the 9160 sentences with predicted label WEAKNESS or SUGGESTION as a single class  Tables TABREF15 and TABREF16 show a few representative clusters in strengths and in weaknesses, respectively  We also explored clustering 12742 STRENGTH sentences directly using CLUTO BIBREF19 and Carrot2 Lingo BIBREF20 clustering algorithms  Carrot2 Lingo discovered 167 clusters and also assigned labels to these clusters  We then generated 167 clusters using CLUTO as well  CLUTO does not generate cluster labels automatically, hence we used 5 most frequent words within the cluster as its labels  Table TABREF19 shows the largest 5 clusters by both the algorithms  It was observed that the clusters created by CLUTO were more meaningful and informative as compared to those by Carrot2 Lingo  Also, it was observed that there is some correspondence between noun clusters and sentence clusters  E g  the nouns cluster motivation expertise knowledge talent skill (Table TABREF15 ) corresponds to the CLUTO sentence cluster skill customer management knowledge team (Table TABREF19 )  But overall, users found the nouns clusters to be more meaningful than the sentence clusters 
PA along Attributes
In many organizations, PA is done from a predefined set of perspectives, which we call attributes  Each attribute covers one specific aspect of the work done by the employees  This has the advantage that we can easily compare the performance of any two employees (or groups of employees) along any given attribute  We can correlate various performance attributes and find dependencies among them  We can also cluster employees in the workforce using their supervisor ratings for each attribute to discover interesting insights into the workforce  The HR managers in the organization considered in this paper have defined 15 attributes (Table TABREF20 )  Each attribute is essentially a work item or work category described at an abstract level  For example, FUNCTIONAL_EXCELLENCE covers any tasks, goals or activities related to the software engineering life-cycle (e g , requirements analysis, design, coding, testing etc ) as well as technologies such as databases, web services and GUI 
In the example in Section SECREF4 , the first sentence (which has class STRENGTH) can be mapped to two attributes: FUNCTIONAL_EXCELLENCE and BUILDING_EFFECTIVE_TEAMS  Similarly, the third sentence (which has class WEAKNESS) can be mapped to the attribute INTERPERSONAL_EFFECTIVENESS and so forth  Thus, in order to answer the second question in Section SECREF1 , we need to map each sentence in each of the 3 classes to zero, one, two or more attributes, which is a multi-class multi-label classification problem 
We manually tagged the same 2000 sentences in Dataset D1 with attributes, where each sentence may get 0, 1, 2, etc  up to 15 class labels (this is dataset D2)  This labelled dataset contained 749, 206, 289, 207, 91, 223, 191, 144, 103, 80, 82, 42, 29, 15, 24 sentences having the class labels listed in Table TABREF20 in the same order  The number of sentences having 0, 1, 2, or more than 2 attributes are: 321, 1070, 470 and 139 respectively  We trained several multi-class multi-label classifiers on this dataset  Table TABREF21 shows the results of 5-fold cross-validation experiments on dataset D2 
Precision, Recall and F-measure for this multi-label classification are computed using a strategy similar to the one described in BIBREF21   Let INLINEFORM0 be the set of predicted labels and INLINEFORM1 be the set of actual labels for the INLINEFORM2 instance  Precision and recall for this instance are computed as follows: INLINEFORM3
It can be observed that INLINEFORM0 would be undefined if INLINEFORM1 is empty and similarly INLINEFORM2 would be undefined when INLINEFORM3 is empty  Hence, overall precision and recall are computed by averaging over all the instances except where they are undefined  Instance-level F-measure can not be computed for instances where either precision or recall are undefined  Therefore, overall F-measure is computed using the overall precision and recall 
Summarization of Peer Feedback using ILP
The PA system includes a set of peer feedback comments for each employee  To answer the third question in Section SECREF1 , we need to create a summary of all the peer feedback comments about a given employee  As an example, following are the feedback comments from 5 peers of an employee 
The individual sentences in the comments written by each peer are first identified and then POS tags are assigned to each sentence  We hypothesize that a good summary of these multiple comments can be constructed by identifying a set of important text fragments or phrases  Initially, a set of candidate phrases is extracted from these comments and a subset of these candidate phrases is chosen as the final summary, using Integer Linear Programming (ILP)  The details of the ILP formulation are shown in Table TABREF36   As an example, following is the summary generated for the above 5 peer comments 
humble nature, effective communication, technical expertise, always supportive, vast knowledge
Following rules are used to identify candidate phrases:
Various parameters are used to evaluate a candidate phrase for its importance  A candidate phrase is more important:
A complete list of parameters is described in detail in Table TABREF36  
There is a trivial constraint INLINEFORM0 which makes sure that only INLINEFORM1 out of INLINEFORM2 candidate phrases are chosen  A suitable value of INLINEFORM3 is used for each employee depending on number of candidate phrases identified across all peers (see Algorithm SECREF6 )  Another set of constraints ( INLINEFORM4 to INLINEFORM5 ) make sure that at least one phrase is selected for each of the leadership attributes  The constraint INLINEFORM6 makes sure that multiple phrases sharing the same headword are not chosen at a time  Also, single word candidate phrases are chosen only if they are adjectives or nouns with lexical category noun attribute  This is imposed by the constraint INLINEFORM7   It is important to note that all the constraints except INLINEFORM8 are soft constraints, i e  there may be feasible solutions which do not satisfy some of these constraints  But each constraint which is not satisfied, results in a penalty through the use of slack variables  These constraints are described in detail in Table TABREF36  
The objective function maximizes the total importance score of the selected candidate phrases  At the same time, it also minimizes the sum of all slack variables so that the minimum number of constraints are broken 
INLINEFORM0 : No  of candidate phrases INLINEFORM1 : No  of phrases to select as part of summary
INLINEFORM0 INLINEFORM1 INLINEFORM2 INLINEFORM3 INLINEFORM4 INLINEFORM5 INLINEFORM6 INLINEFORM7 INLINEFORM8
INLINEFORM0 and INLINEFORM1 INLINEFORM2 INLINEFORM3 INLINEFORM4 INLINEFORM5 INLINEFORM6
INLINEFORM0 (For determining number of phrases to select to include in summary)
Evaluation of auto-generated summaries
We considered a dataset of 100 employees, where for each employee multiple peer comments were recorded  Also, for each employee, a manual summary was generated by an HR personnel  The summaries generated by our ILP-based approach were compared with the corresponding manual summaries using the ROUGE BIBREF22 unigram score  For comparing performance of our ILP-based summarization algorithm, we explored a few summarization algorithms provided by the Sumy package  A common parameter which is required by all these algorithms is number of sentences keep in the final summary  ILP-based summarization requires a similar parameter K, which is automatically decided based on number of total candidate phrases  Assuming a sentence is equivalent to roughly 3 phrases, for Sumy algorithms, we set number of sentences parameter to the ceiling of K/3  Table TABREF51 shows average and standard deviation of ROUGE unigram f1 scores for each algorithm, over the 100 summaries  The performance of ILP-based summarization is comparable with the other algorithms, as the two sample t-test does not show statistically significant difference  Also, human evaluators preferred phrase-based summary generated by our approach to the other sentence-based summaries 
Conclusions and Further Work
In this paper, we presented an analysis of the text generated in Performance Appraisal (PA) process in a large multi-national IT company  We performed sentence classification to identify strengths, weaknesses and suggestions for improvements found in the supervisor assessments and then used clustering to discover broad categories among them  As this is non-topical classification, we found that SVM with ADWS kernel BIBREF18 produced the best results  We also used multi-class multi-label classification techniques to match supervisor assessments to predefined broad perspectives on performance  Logistic Regression classifier was observed to produce the best results for this topical classification  Finally, we proposed an ILP-based summarization technique to produce a summary of peer feedback comments for a given employee and compared it with manual summaries 
The PA process also generates much structured data, such as supervisor ratings  It is an interesting problem to compare and combine the insights from discovered from structured data and unstructured text  Also, we are planning to automatically discover any additional performance attributes to the list of 15 attributes currently used by HR 

ground_truth: No

No

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What evaluation metrics are looked at for classification tasks?
Context 4:  Therefore, for every word in our training set, we define our newly introduced ranking method, the Aggressive Feature Ranking (AFR), as: INLINEFORM0
In Figure FIGREF3 we illustrate the performance of all four methods in our industry prediction task on the development data  Note that for each method, we provide both the accuracy ( INLINEFORM0 ) and the average per-class accuracy ( INLINEFORM1 )  The Majority and All Words methods apply to all the features; therefore, they are represented as a straight line in the figure  The IGR and AFR methods are applied to varying subsets of the features using a 5% step 
Our experiments demonstrate that the word choice that the users make in their posts correlates with their industry  The first observation in Figure FIGREF3 is that the INLINEFORM0 is proportional to INLINEFORM1 ; as INLINEFORM2 increases, so does INLINEFORM3   Secondly, the best result on the development set is achieved by using the top 90% of the features using the AFR method  Lastly, the improvements of the IGR and AFR feature selections are not substantially better in comparison to All Words (at most 5% improvement between All Words and AFR), which suggest that only a few noisy features exist and most of the words play some role in shaping the “language" of an industry 
As a final evaluation, we apply on the test data the classifier found to work best on the development data (AFR feature selection, top 90% features), for an INLINEFORM0 of 0 534 and INLINEFORM1 of 0 477 
Leveraging User Metadata
Together with the industry information and the most recent postings of each blogger, we also download a number of accompanying profile elements  Using these additional elements, we explore the potential of incorporating users' metadata in our classifiers 
Table TABREF7 shows the different user metadata we consider together with their coverage percentage (not all users provide a value for all of the profile elements)  With the exception of the gender field, the remaining metadata elements shown in Table TABREF7 are completed by the users as a freely editable text field  This introduces a considerable amount of noise in the set of possible metadata values  Examples of noise in the occupation field include values such as “Retired”, “I work ”, or “momma” which are not necessarily informative for our industry prediction task 
To examine whether the metadata fields can help in the prediction of a user's industry, we build classifiers using the different metadata elements  For each metadata element that has a textual value, we use all the words in the training set for that field as features  The only two exceptions are the state field, which is encoded as one feature that can take one out of 50 different values representing the 50 U S  states; and the gender field, which is encoded as a feature with a distinct value for each user gender option: undefined, male, or female 
As shown in Table TABREF9 , we build four different classifiers using the multinomial NB algorithm: Occu (which uses the words found in the occupation profile element), Intro (introduction), Inter (interests), and Gloc (combined gender, city, state) 
In general, all the metadata classifiers perform better than our majority baseline ( INLINEFORM0 of 18 88%)  For the Gloc classifier, this result is in alignment with previous studies BIBREF24   However, the only metadata classifier that outperforms the content classifier is the Occu classifier, which despite missing and noisy occupation values exceeds the content classifier's performance by an absolute 3 2% 
To investigate the promise of combining the five different classifiers we have built so far, we calculate their inter-prediction agreement using Fleiss's Kappa BIBREF25 , as well as the lower prediction bounds using the double fault measure BIBREF26   The Kappa values, presented in the lower left side of Table TABREF10 , express the classification agreement for categorical items, in this case the users' industry  Lower values, especially values below 30%, mean smaller agreement  Since all five classifiers have better-than-baseline accuracy, this low agreement suggests that their predictions could potentially be combined to achieve a better accumulated result 
Moreover, the double fault measure values, which are presented in the top-right hand side of Table TABREF10 , express the proportion of test cases for which both of the two respective classifiers make false predictions, essentially providing the lowest error bound for the pairwise ensemble classifier performance  The lower those numbers are, the greater the accuracy potential of any meta-classification scheme that combines those classifiers  Once again, the low double fault measure values suggest potential gain from a combination of the base classifiers into an ensemble of models 
After establishing the promise of creating an ensemble of classifiers, we implement two meta-classification approaches  First, we combine our classifiers using features concatenation (or early fusion)  Starting with our content-based classifier (Text), we successively add the features derived from each metadata element  The results, both micro- and macro-accuracy, are presented in Table TABREF12   Even though all these four feature concatenation ensembles outperform the content-based classifier in the development set, they fail to outperform the Occu classifier 
Second, we explore the potential of using stacked generalization (or late fusion) BIBREF27   The base classifiers, referred to as L0 classifiers, are trained on different folds of the training set and used to predict the class of the remaining instances  Those predictions are then used together with the true label of the training instances to train a second classifier, referred to as the L1 classifier: this L1 is used to produce the final prediction on both the development data and the test data  Traditionally, stacking uses different machine learning algorithms on the same training data  However in our case, we use the same algorithm (multinomial NB) on heterogeneous data (i e , different types of data such as content, occupation, introduction, interests, gender, city and state) in order to exploit all available sources of information 
The ensemble learning results on the development set are shown in Table TABREF12   We notice a constant improvement for both metrics when adding more classifiers to our ensemble except for the Gloc classifier, which slightly reduces the performance  The best result is achieved using an ensemble of the Text, Occu, Intro, and Inter L0 classifiers; the respective performance on the test set is an INLINEFORM0 of 0 643 and an INLINEFORM1 of 0 564  Finally, we present in Figure FIGREF11 the prediction accuracy for the final classifier for each of the different industries in our test dataset  Evidently, some industries are easier to predict than others  For example, while the Real Estate and Religion industries achieve accuracy figures above 80%, other industries, such as the Banking industry, are predicted correctly in less than 17% of the time  Anecdotal evidence drawn from the examination of the confusion matrix does not encourage any strong association of the Banking class with any other  The misclassifications are roughly uniform across all other classes, suggesting that the users in the Banking industry use language in a non-distinguishing way 
Qualitative Analysis
In this section, we provide a qualitative analysis of the language of the different industries 
Top-Ranked Words
To conduct a qualitative exploration of which words indicate the industry of a user, Table TABREF14 shows the three top-ranking content words for the different industries using the AFR method 
Not surprisingly, the top ranked words align well with what we would intuitively expect for each industry  Even though most of these words are potentially used by many users regardless of their industry in our dataset, they are still distinguished by the AFR method because of the different frequencies of these words in the text of each industry 
Industry-specific Word Similarities
Next, we examine how the meaning of a word is shaped by the context in which it is uttered  In particular, we qualitatively investigate how the speakers' industry affects meaning by learning vector-space representations of words that take into account such contextual information  To achieve this, we apply the contextualized word embeddings proposed by Bamman et al  Bamman14, which are based on an extension of the “skip-gram" language model BIBREF28  
In addition to learning a global representation for each word, these contextualized embeddings compute one deviation from the common word embedding representation for each contextual variable, in this case, an industry option  These deviations capture the terms' meaning variations (shifts in the INLINEFORM0 -dimensional space of the representations, where INLINEFORM1 in our experiments) in the text of the different industries, however all the embeddings are in the same vector space to allow for comparisons to one another 
Using the word representations learned for each industry, we present in Table TABREF16 the terms in the Technology and the Tourism industries that have the highest cosine similarity with a job-related word, customers  Similarly, Table TABREF17 shows the words in the Environment and the Tourism industries that are closest in meaning to a general interest word, food  More examples are given in the Appendix SECREF8  
The terms that rank highest in each industry are noticeably different  For example, as seen in Table TABREF17 , while food in the Environment industry is similar to nutritionally and locally, in the Tourism industry the same word relates more to terms such as delicious and pastries  These results not only emphasize the existing differences in how people in different industries perceive certain terms, but they also demonstrate that those differences can effectively be captured in the resulting word embeddings 
Emotional Orientation per Industry and Gender
As a final analysis, we explore how words that are emotionally charged relate to different industries  To quantify the emotional orientation of a text, we use the Positive Emotion and Negative Emotion categories in the Linguistic Inquiry and Word Count (LIWC) dictionary BIBREF29   The LIWC dictionary contains lists of words that have been shown to correlate with the psychological states of people that use them; for example, the Positive Emotion category contains words such as “happy,” “pretty,” and “good ”
For the text of all the users in each industry we measure the frequencies of Positive Emotion and Negative Emotion words normalized by the text's length  Table TABREF20 presents the industries' ranking for both categories of words based on their relative frequencies in the text of each industry 
We further perform a breakdown per-gender, where we once again calculate the proportion of emotionally charged words in each industry, but separately for each gender  We find that the industry rankings of the relative frequencies INLINEFORM0 of emotionally charged words for the two genders are statistically significantly correlated, which suggests that regardless of their gender, users use positive (or negative) words with a relative frequency that correlates with their industry  (In other words, even if e g , Fashion has a larger number of women users, both men and women working in Fashion will tend to use more positive words than the corresponding gender in another industry with a larger number of men users such as Automotive )
Finally, motivated by previous findings of correlations between job satisfaction and gender dominance in the workplace BIBREF30 , we explore the relationship between the usage of Positive Emotion and Negative Emotion words and the gender dominance in an industry  Although we find that there are substantial gender imbalances in each industry (Appendix SECREF9 ), we did not find any statistically significant correlation between the gender dominance ratio in the different industries and the usage of positive (or negative) emotional words in either gender in our dataset 
Conclusion
In this paper, we examined the task of predicting a social media user's industry  We introduced an annotated dataset of over 20,000 blog users and applied a content-based classifier in conjunction with two feature selection methods for an overall accuracy of up to 0 534, which represents a large improvement over the majority class baseline of 0 188 
We also demonstrated how the user metadata can be incorporated in our classifiers  Although concatenation of features drawn both from blog content and profile elements did not yield any clear improvements over the best individual classifiers, we found that stacking improves the prediction accuracy to an overall accuracy of 0 643, as measured on our test dataset  A more in-depth analysis showed that not all industries are equally easy to predict: while industries such as Real Estate and Religion are clearly distinguishable with accuracy figures over 0 80, others such as Banking are much harder to predict 
Finally, we presented a qualitative analysis to provide some insights into the language of different industries, which highlighted differences in the top-ranked words in each industry, word semantic similarities, and the relative frequency of emotionally charged words 
Acknowledgments
This material is based in part upon work supported by the National Science Foundation (#1344257) and by the John Templeton Foundation (#48503)  Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation or the John Templeton Foundation 
Additional Examples of Word Similarities

ground_truth: Precision, Recall, F-measure, accuracy

Precision, Recall and F-measure