prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which hyperparameters were varied in the experiments on the four tasks?
Context 1: Introduction
Many research attempts have proposed novel features that improve the performance of learning algorithms in particular tasks  Such features are often motivated by domain knowledge or manual labor  Although useful and often state-of-the-art, adapting such solutions on NLP systems across tasks can be tricky and time-consuming BIBREF0   Therefore, simple yet general and powerful methods that perform well across several datasets are valuable BIBREF1  
An approach that has become extremely popular lately in NLP tasks, is to train word embeddings in an unsupervised way  These embeddings are dense vectors that project words or short text spans like phrases in a vector space where dimensions are supposed to capture text properties  Such embeddings can then be used either as features with off-the-shelf algorithms like Support Vector Machines, or to initialize deep learning systems BIBREF2   However, as shown in BIBREF3 linear architectures perform better in high-dimensional discrete spaces compared to continuous ones  The latter is probably the main reason of the high performance of the vector space model BIBREF4 in tasks like text classification with linear models like SVMs  Using linear algorithms, while taking advantage of the expressiveness of text embeddings is the focus of this work 
In this paper, we explore a hybrid approach, that uses text embeddings as a proxy to create features  Motivated by the argument that text embeddings manage to encode the semantics of text, we explore how clustering text embeddings can impact the performance of different NLP tasks  Although such an approach has been used in different studies during feature engineering, the selection of word vectors and the number of clusters remain a trial-end-error procedure  In this work we present an empirical evaluation across diverse tasks to verify whether and when such features are useful 
Word clusters have been used as features in various tasks like Part-of-Speech tagging and NER  Owoputi et al  Owoputi13 use Brown clusters BIBREF5 in a POS tagger showing that this type of features carry rich lexical knowledge as they can substitute lexical resources like gazetteers  Kiritchenko et al  KiritchenkoZM14 discusses their use on sentiment classification while Hee et al  HeeLH16 incorporate them in the task of irony detection in Twitter  Ritter et al  Ritter2011 inject also word clusters in a NER tagger  While these works show that word clusters are beneficial no clear guidelines can be concluded of how and when to use them 
In this work, we empirically demonstrate that using different types of embeddings on three NLP tasks with twitter data we manage to achieve better or near to the state-of-the art performance on three NLP tasks: (i) Named Entity Recognition (NER) segmentation, (ii) NER classification, (iii) fine-grained sentiment analysis and (iv) fine-grained sentiment quantification  For each of the three tasks, we achieve higher performance than without using features which indicates the effectiveness of the cluster membership features  Importantly, our evaluation compared to previous work BIBREF6 who focus on old and well studied datasets uses recent and challenging datasets composed by tweets  The obtained results across all the tasks permits us to reveal important aspects of the use of word clusters and therefore provide guidelines  Although our obtained scores are state-of-the-art, our analysis reveals that the performance in such tasks is far from perfect and, hence, identifies that there is still much space for improvement and future work 
Word Clusters
Word embeddings associate words with dense, low-dimensional vectors  Recently, several models have been proposed in order to obtain these embeddings  Among others, the skipgram (skipgram) model with negative sampling BIBREF7 , the continuous bag-of-words (cbow) model BIBREF7 and Glove (glove) BIBREF8 have been shown to be effective  Training those models requires no annotated data and can be done using big amounts of text  Such a model can be seen as a function INLINEFORM0 that projects a word INLINEFORM1 in a INLINEFORM2 -dimensional space: INLINEFORM3 , where INLINEFORM4 is predefined  Here, we focus on applications using data from Twitter, which pose several difficulties due to being particularly short, using creative vocabulary, abbreviations and slang 
For all the tasks in our experimental study, we use 36 millions English tweets collected between August and September 2017  A pre-processing step has been applied to replace URLs with a placeholder and to pad punctuation  The final vocabulary size was around 1 6 millions words  Additionally to the in-domain corpus we collected, we use GloVe vectors trained on Wikipedia articles in order to investigate the impact of out-of-domain word-vectors 
We cluster the embeddings with INLINEFORM0 -Means  The k-means clusters are initialized using “k-means++” as proposed in BIBREF9 , while the algorithm is run for 300 iterations  We try different values for INLINEFORM1   For each INLINEFORM2 , we repeat the clustering experiment with different seed initialization for 10 times and we select the clustering result that minimizes the cluster inertia 
Experimental Evaluation
We evaluate the proposed approach for augmenting the feature space in four tasks: (i) NER segmentation, (ii) NER classification, (iii) fine-grained sentiment classification and (iv) fine-grained sentiment quantification  The next sections present the evaluation settings we used  For each of the tasks, we use the designated training sets to train the learning algorithms, and we report the scores of the evaluation measures used in the respective test parts 
Named-Entity Recognition in Twitter
NER concerns the classification of textual segments in a predefined set of categories, like persons, organization and locations  We use the data of the last competition in NER for Twitter which released as a part of the 2nd Workshop on Noisy User-generated Text BIBREF10   More specifically, the organizers provided annotated tweets with 10 named-entity types (person, movie, sportsteam, product etc ) and the task comprised two sub-tasks: 1) the detection of entity bounds and 2) the classification of an entity into one of the 10 types  The evaluation measure for both sub-tasks is the F INLINEFORM0 measure 
The following is an example of a tweet which contains two named entities  Note that named entities may span several words in the text:
INLINEFORM0 tonite     90 's music    oldskool night wiith INLINEFORM1
Our model for solving the task is a learning to search approach  More specifically we follow BIBREF11 which has been ranked 2nd among 10 participants in the aforementioned competition BIBREF10   The model uses handcrafted features like n-grams, part-of-speech tags, capitalization and membership in gazetteers  The algorithm used belongs to the family of learning to search for structured prediction tasks BIBREF12   These methods decompose the problem in a search space with states, actions and policies and then learn a hypothesis controlling a policy over the state-action space  The BIO encoding is used for attributing the corresponding labels to the tokens where B-type is used for the first token of the entity, I-type for inside tokens in case of multi-term entities and O for non entity tokens 
Tables TABREF6 and TABREF7 present the results for the different number of clusters across the three vector models used to induce the clusters  For all the experiments we keep the same parametrization for the learning algorithm and we present the performance of each run on the official test set 
Regarding the segmentation task we notice that adding word clusters as features improve the performance of the best model up to 1 1 F-score points while it boosts performance in the majority of cases  In only one case, for glove INLINEFORM0 vectors, there is a drop across all number of clusters used 
As for the number of clusters, the best results are generally obtained between 250 and 1000 classes for all word vector models  These dimensions seem to be sufficient for the three-class sub-task that we deal with  The different models of word vectors perform similarly and thus one cannot privilege a certain type of word vectors  Interestingly, the clusters learned on the Wikipedia GloVe vectors offer competitive performance with respect to the in-domain word vectors used for the other cases showing that one can rely to out-of-domain data for constructing such representations 
Concerning the classification task (Table TABREF7 ) we generally observe a drop in the performance of the tagger as we deal with 10 classes  This essentially corresponds to a multi-class problem with 21 classes: one for the non-entity type and two classes for each entity type  In this setting we notice that the best results are obtained in most cases for higher number of classes (1000 or 2000) possibly due to a better discriminatory power in higher dimensions  Note also, that in some cases the addition of word cluster features does not necessarily improve the performance  Contrary, it may degrade it as it is evident in the case of glove INLINEFORM0 word clusters  Like in the case of segmentation we do not observe a word vector model that clearly outperforms the rest  Finally, we note the same competitive performance of the Wikipedia word clusters and notably for the glove INLINEFORM1 clusters which obtain the best F1-score 
Fine-grained Sentiment Analysis
The task of fine grained sentiment classification consists in predicting the sentiment of an input text according to a five point scale (sentiment INLINEFORM0 {VeryNegative, Negative, Neutral, Positive, VeryPositive})  We use the setting of task 4 of SemEval2016 “Sentiment Analysis in Twitter” and the dataset released by the organizers for subtask 4 BIBREF13  
In total, the training (resp  test) data consist of 9,070 (resp  20,632) tweets 
The evaluation measure selected in BIBREF13 for the task in the macro-averaged Mean Absolute Error (MAE INLINEFORM0 )  It is a measure of error, hence lower values are better  The measure's goal is to take into account the order of the classes when penalizing the decision of a classifier  For instance, misclassifying a very negative example as very positive is a bigger mistake than classifying it as negative or neutral  Penalizing a classifier according to how far the predictions are from the true class is captured by MAE INLINEFORM1 BIBREF14   Also, the advantage of using the macro- version instead of the standard version of the measure is the robustness against the class imbalance in the data 
Learning algorithm To demonstrate the efficiency of cluster membership features we rely on the system of BIBREF15 which was ranked 1st among 11 participants and uses a Logistic Regression as a learning algorithm  We follow the same feature extraction steps which consist of extracting n-gram and character n-gram features, part-of-speech counts as well as sentiment scores using standard sentiment lexicons such as the Bing Liu's BIBREF16 and the MPQA lexicons BIBREF17   For the full description, we refer the interested reader to BIBREF15  
To evaluate the performance of the proposed feature augmentation technique, we present in Table TABREF10 the macro-averaged Mean Absolute Error scores for different settings on the official test set of BIBREF13   First, notice that the best score in the test data is achieved using cluster membership features, where the word embeddings are trained using the skipgram model  The achieved score improves the state-of-the art on the dataset, which to the best of our knowledge was by BIBREF15   Also, note that the score on the test data improves for each type of embeddings used, which means that augmenting the feature space using cluster membership features helps the sentiment classification task 
Note, also, that using the clusters produced by the out-of-domain embeddings trained on wikipedia that were released as part of BIBREF8 performs surprisingly well  One might have expected their addition to hurt the performance  However, their value probably stems from the sheer amount of data used for their training as well as the relatively simple type of words (like awesome, terrible) which are discriminative for this task  Lastly, note that in each of the settings, the best results are achieved when the number of clusters is within INLINEFORM0 as in the NER tasks  Comparing the performance across the different embeddings, one cannot claim that a particular embedding performs better  It is evident though that augmenting the feature space with feature derived using the proposed method, preferably with in-domain data, helps the classification performance and reduces MAE INLINEFORM1  
From the results of Table TABREF10 it is clear that the addition of the cluster membership features improves the sentiment classification performance  To better understand though why these clusters help, we manually examined a sample of the words associated with the clusters  To improve the eligibility of those results we first removed the hashtags and we filter the results using an English vocabulary  In Table TABREF11 we present sample words from two of the most characteristic clusters with respect to the task of sentiment classification  Notice how words with positive and negative meanings are put in the respective clusters 
Fine-Grained Sentiment Quantification
Quantification is the problem of estimating the prevalence of a class in a dataset  While classification concerns assigning a category to a single instance, like labeling a tweet with the sentiment it conveys, the goal of quantification is, given a set of instances, to estimate the relative frequency of single class  Therefore, sentiment quantification tries to answer questions like “Given a set of tweets about the new iPhone, what is the fraction of VeryPositive ones?”  In the rest, we show the effect of the features derived from the word embeddings clusters in the fine-grained classification problem, which was also part of the SemEval-2016 “Sentiment Analysis in Twitter” task BIBREF13  
Learning Algorithm To perform the quantification task, we rely on a classify and count approach, which was shown effective in a related binary quantification problem BIBREF15   The idea is that given a set of instances on a particular subject, one first classifies the instances and then aggregates the counts  To this end, we use the same feature representation steps and data with the ones used for fine grained classification (Section 3 2)  Note that the data of the task are associated with subjects (described in full detail at BIBREF13 ), and, hence, quantification is performed for the tweets of a subject  For each of the five categories, the output of the approach is a 5-dimensional vector with the estimated prevalence of the categories 
The evaluation measure for the problem is the Earth Movers Distance (EMD) BIBREF18   EMD is a measure of error, hence lower values are better  It assumes ordered categories, which in our problem is naturally defined  Further assuming that the distance of consecutive categories (e g , Positive and VeryPositive) is 1, the measure is calculated by: INLINEFORM0
where INLINEFORM0 is number of categories (five in our case) and INLINEFORM1 and INLINEFORM2 are the true and predicted prevalence respectively BIBREF19  
Results Table TABREF13 presents the results of augmenting the feature set with the proposed features  We use Logistic Regression as a base classifier for the classify and count approach  Notice the positive impact of the features in the performance in the task  Adding the features derived from clustering the embeddings consistently improves the performance  Interestingly, the best performance ( INLINEFORM0 ) is achieved using the out-of-domain vectors, as in the NER classification task  Also, notice how the approach improves over the state-of-the-art performance in the challenge ( INLINEFORM1 ) BIBREF13 , held by the method of BIBREF20   The improvement over the method of BIBREF20 however, does not necessarily mean that classify and count performs better in the task  It implies that the feature set we used is richer, that in turn highlights the value of robust feature extraction mechanisms which is the subject of this paper 
Conclusion
We have shown empirically the effectiveness of incorporating cluster membership features in the feature extraction pipeline of Named-Entity recognition, sentiment classification and quantification tasks  Our results strongly suggest that incorporating cluster membership features benefit the performance in the tasks  The fact that the performance improvements are consistent in the four tasks we investigated, further highlights their usefulness, both for practitioners and researchers 
Although our study does not identify a clear winner with respect to the type of word vectors (skipgram, cbow, or GloVe), our findings suggest that one should first try skip-gram embeddings of low dimensionality ( INLINEFORM0 ) and high number of clusters (e g , INLINEFORM1 ) as the results obtained using these settings are consistently competitive  Our results also suggest that using out-of-domain data, like Wikipedia articles in this case, to construct the word embeddings is a good practice, as the results we obtained with these vectors are also competitive  The positive of out-of-domain embeddings and their combination with in-domain ones remains to be further studied 
Context 3: Introduction
Suppose a user wants to write a sentence “I will be 10 minutes late ” Ideally, she would type just a few keywords such as “10 minutes late” and an autocomplete system would be able to infer the intended sentence (Figure FIGREF1)  Existing left-to-right autocomplete systems BIBREF0, BIBREF1 can often be inefficient, as the prefix of a sentence (e g  “I will be”) fails to capture the core meaning of the sentence  Besides the practical goal of building a better autocomplete system, we are interested in exploring the tradeoffs inherent to such communication schemes between the efficiency of typing keywords, accuracy of reconstruction, and interpretability of keywords 
One approach to learn such schemes is to collect a supervised dataset of keywords-sentence pairs as a training set, but (i) it would be expensive to collect such data from users, and (ii) a static dataset would not capture a real user's natural predilection to adapt to the system BIBREF2  Another approach is to avoid supervision and jointly learn a user-system communication scheme to directly optimize the combination of efficiency and accuracy  However, learning in this way can lead to communication schemes that are uninterpretable to humans BIBREF3, BIBREF4 (see Appendix for additional related work) 
In this work, we propose a simple, unsupervised approach to an autocomplete system that is efficient, accurate, and interpretable  For interpretability, we restrict keywords to be subsequences of their source sentences based on the intuition that humans can infer most of the original meaning from a few keywords  We then apply multi-objective optimization approaches to directly control and achieve desirable tradeoffs between efficiency and accuracy 
We observe that naively optimizing a linear combination of efficiency and accuracy terms is unstable and leads to suboptimal schemes  Thus, we propose a new objective which optimizes for communication efficiency under an accuracy constraint  We show this new objective is more stable and efficient than the linear objective at all accuracy levels 
As a proof-of-concept, we build an autocomplete system within this framework which allows a user to write sentences by specifying keywords  We empirically show that our framework produces communication schemes that are 52 16% more accurate than rule-based baselines when specifying 77 37% of sentences, and 11 73% more accurate than a naive, weighted optimization approach when specifying 53 38% of sentences  Finally, we demonstrate that humans can easily adapt to the keyword-based autocomplete system and save nearly 50% of time compared to typing a full sentence in our user study 
Approach
Consider a communication game in which the goal is for a user to communicate a target sequence $x= (x_1,    , x_m)$ to a system by passing a sequence of keywords $z= (z_1,    , z_n)$  The user generates keywords $z$ using an encoding strategy $q_{\alpha }(z\mid x)$, and the system attempts to guess the target sequence $x$ via a decoding strategy $p_{\beta }(x\mid z)$ 
A good communication scheme $(q_{\alpha }, p_{\beta })$ should be both efficient and accurate  Specifically, we prefer schemes that use fewer keywords (cost), and the target sentence $x$ to be reconstructed with high probability (loss) where
Based on our assumption that humans have an intuitive sense of retaining important keywords, we restrict the set of schemes to be a (potentially noncontiguous) subsequence of the target sentence  Our hypothesis is that such subsequence schemes naturally ensure interpretability, as efficient human and machine communication schemes are both likely to involve keeping important content words 
Approach ::: Modeling with autoencoders 
To learn communication schemes without supervision, we model the cooperative communication between a user and system through an encoder-decoder framework  Concretely, we model the user's encoding strategy $q_{\alpha }(z\mid x)$ with an encoder which encodes the target sentence $x$ into the keywords $z$ by keeping a subset of the tokens  This stochastic encoder $q_{\alpha }(z\mid x)$ is defined by a model which returns the probability of each token retained in the final subsequence $z$  Then, we sample from Bernoulli distributions according to these probabilities to either keep or drop the tokens independently (see Appendix for an example) 
We model the autocomplete system's decoding strategy $p_{\beta }(x\mid z)$ as a probabilistic model which conditions on the keywords $z$ and returns a distribution over predictions $x$  We use a standard sequence-to-sequence model with attention and copying for the decoder, but any model architecture can be used (see Appendix for details) 
Approach ::: Multi-objective optimization 
Our goal now is to learn encoder-decoder pairs which optimally balance the communication cost and reconstruction loss  The simplest approach to balancing efficiency and accuracy is to weight $\mathrm {cost}(x, \alpha )$ and $\mathrm {loss}(x, \alpha , \beta )$ linearly using a weight $\lambda $ as follows,
where the expectation is taken over the population distribution of source sentences $x$, which is omitted to simplify notation  However, we observe that naively weighting and searching over $\lambda $ is suboptimal and highly unstable—even slight changes to the weighting results in degenerate schemes which keep all or none of its tokens  This instability motivates us to develop a new stable objective 
Our main technical contribution is to draw inspiration from the multi-objective optimization literature and view the tradeoff as a sequence of constrained optimization problems, where we minimize the expected cost subject to varying expected reconstruction error constraints $\epsilon $,
This greatly improves the stability of the training procedure  We empirically observe that the model initially keeps most of the tokens to meet the constraints, and slowly learns to drop uninformative words from the keywords to minimize the cost  Furthermore, $\epsilon $ in Eq (DISPLAY_FORM6) allows us to directly control the maximum reconstruction error of resulting schemes, whereas $\lambda $ in Eq (DISPLAY_FORM5) is not directly related to any of our desiderata 
To optimize the constrained objective, we consider the Lagrangian of Eq (DISPLAY_FORM6),
Much like the objective in Eq (DISPLAY_FORM5) we can compute unbiased gradients by replacing the expectations with their averages over random minibatches  Although gradient descent guarantees convergence on Eq (DISPLAY_FORM7) only when the objective is convex, we find that not only is the optimization stable, the resulting solution achieves better performance than the weighting approach in Eq (DISPLAY_FORM5) 
Approach ::: Optimization 
Optimization with respect to $q_{\alpha }(z\mid x)$ is challenging as $z$ is discrete, and thus, we cannot differentiate $\alpha $ through $z$ via the chain rule  Because of this, we use the stochastic REINFORCE estimate BIBREF5 as follows:
We perform joint updates on $(\alpha , \beta , \lambda )$, where $\beta $ and $\lambda $ are updated via standard gradient computations, while $\alpha $ uses an unbiased, stochastic gradient estimate where we approximate the expectation in Eq (DISPLAY_FORM9)  We use a single sample from $q_{\alpha }(z\mid x)$ and moving-average of rewards as a baseline to reduce variance 
Experiments
We evaluate our approach by training an autocomplete system on 500K randomly sampled sentences from Yelp reviews BIBREF6 (see Appendix for details)  We quantify the efficiency of a communication scheme $(q_{\alpha },p_{\beta })$ by the retention rate of tokens, which is measured as the fraction of tokens that are kept in the keywords  The accuracy of a scheme is measured as the fraction of sentences generated by greedily decoding the model that exactly matches the target sentence 
Experiments ::: Effectiveness of constrained objective 
We first show that the linear objective in Eq (DISPLAY_FORM5) is suboptimal compared to the constrained objective in Eq (DISPLAY_FORM6)  Figure FIGREF10 compares the achievable accuracy and efficiency tradeoffs for the two objectives, which shows that the constrained objective results in more efficient schemes than the linear objective at every accuracy level (e g  11 73% more accurate at a 53 38% retention rate) 
We also observe that the linear objective is highly unstable as a function of the tradeoff parameter $\lambda $ and requires careful tuning  Even slight changes to $\lambda $ results in degenerate schemes that keep all or none of the tokens (e g  $\lambda \le 4 2$ and $\lambda \ge 4 4$)  On the other hand, the constrained objective is substantially more stable as a function of $\epsilon $ (e g  points for $\epsilon $ are more evenly spaced than $\lambda $) 
Experiments ::: Efficiency-accuracy tradeoff 
We quantify the efficiency-accuracy tradeoff compared to two rule-based baselines: Unif and Stopword  The Unif encoder randomly keeps tokens to generate keywords with the probability $\delta $  The Stopword encoder keeps all tokens but drops stop words (e g  `the', `a', `or') all the time ($\delta =0$) or half of the time ($\delta =0 5$)  The corresponding decoders for these encoders are optimized using gradient descent to minimize the reconstruction error (i e  $\mathrm {loss}(x, \alpha , \beta )$) 
Figure FIGREF10 shows that two baselines achieve similar tradeoff curves, while the constrained model achieves a substantial 52 16% improvement in accuracy at a 77 37% retention rate compared to Unif, thereby showing the benefits of jointly training the encoder and decoder 
Experiments ::: Robustness and analysis 
We provide additional experimental results on the robustness of learned communication schemes as well as in-depth analysis on the correlation between the retention rates of tokens and their properties, which we defer to Appendix and for space 
Experiments ::: User study 
We recruited 100 crowdworkers on Amazon Mechanical Turk (AMT) and measured completion times and accuracies for typing randomly sampled sentences from the Yelp corpus  Each user was shown alternating autocomplete and writing tasks across 50 sentences (see Appendix for user interface)  For the autocomplete task, we gave users a target sentence and asked them to type a set of keywords into the system  The users were shown the top three suggestions from the autocomplete system, and were asked to mark whether each of these three suggestions was semantically equivalent to the target sentence  For the writing task, we gave users a target sentence and asked them to either type the sentence verbatim or a sentence that preserves the meaning of the target sentence 
Table TABREF13 shows two examples of the autocomplete task and actual user-provided keywords  Each column contains a set of keywords and its corresponding top three suggestions generated by the autocomplete system with beam search  We observe that the system is likely to propose generic sentences for under-specified keywords (left column) and almost the same sentences for over-specified keywords (right column)  For properly specified keywords (middle column), the system completes sentences accordingly by adding a verb, adverb, adjective, preposition, capitalization, and punctuation 
Overall, the autocomplete system achieved high accuracy in reconstructing the keywords  Users marked the top suggestion from the autocomplete system to be semantically equivalent to the target $80 6$% of the time, and one of the top 3 was semantically equivalent $90 11$% of the time  The model also achieved a high exact match accuracy of 18 39%  Furthermore, the system was efficient, as users spent $3 86$ seconds typing keywords compared to $5 76$ seconds for full sentences on average  The variance of the typing time was $0 08$ second for keywords and $0 12$ second for full sentences, indicating that choosing and typing keywords for the system did not incur much overhead 
Experiments ::: Acknowledgments
We thank the reviewers and Yunseok Jang for their insightful comments  This work was supported by NSF CAREER Award IIS-1552635 and an Intuit Research Award 
Experiments ::: Reproducibility
All code, data and experiments are available on CodaLab at https://bit ly/353fbyn 

ground_truth: number of clusters, seed value in clustering, selection of word vectors, window size and dimension of embedding

different number of clusters, different embeddings

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What features are used?
Context 2: Introduction
Cancer is one of the leading causes of death in the world, with over 80,000 deaths registered in Canada in 2017 (Canadian Cancer Statistics 2017)  A computer-aided system for cancer diagnosis usually involves a pathologist rendering a descriptive report after examining the tissue glass slides obtained from the biopsy of a patient  A pathology report contains specific analysis of cells and tissues, and other histopathological indicators that are crucial for diagnosing malignancies  An average sized laboratory may produces a large quantity of pathology reports annually (e g , in excess of 50,000), but these reports are written in mostly unstructured text and with no direct link to the tissue sample  Furthermore, the report for each patient is a personalized document and offers very high variability in terminology due to lack of standards and may even include misspellings and missing punctuation, clinical diagnoses interspersed with complex explanations, different terminology to label the same malignancy, and information about multiple carcinoma appearances included in a single report BIBREF0  
In Canada, each Provincial and Territorial Cancer Registry (PTCR) is responsible for collecting the data about cancer diseases and reporting them to Statistics Canada (StatCan)  Every year, Canadian Cancer Registry (CCR) uses the information sources of StatCan to compile an annual report on cancer and tumor diseases  Many countries have their own cancer registry programs  These programs rely on the acquisition of diagnostic, treatment, and outcome information through manual processing and interpretation from various unstructured sources (e g , pathology reports, autopsy/laboratory reports, medical billing summaries)  The manual classification of cancer pathology reports is a challenging, time-consuming task and requires extensive training BIBREF0  
With the continued growth in the number of cancer patients, and the increase in treatment complexity, cancer registries face a significant challenge in manually reviewing the large quantity of reports BIBREF1 , BIBREF0   In this situation, Natural Language Processing (NLP) systems can offer a unique opportunity to automatically encode the unstructured reports into structured data  Since, the registries already have access to the large quantity of historically labeled and encoded reports, a supervised machine learning approach of feature extraction and classification is a compelling direction for making their workflow more effective and streamlined  If successful, such a solution would enable processing reports in much lesser time allowing trained personnel to focus on their research and analysis  However, developing an automated solution with high accuracy and consistency across wide variety of reports is a challenging problem 
For cancer registries, an important piece of information in a pathology report is the associated ICD-O code which describes the patient's histological diagnosis, as described by the World Health Organization's (WHO) International Classification of Diseases for Oncology BIBREF2   Prediction of the primary diagnosis from a pathology report provides a valuable starting point for exploration of machine learning techniques for automated cancer surveillance  A major application for this purpose would be “auto-reporting” based on analysis of whole slide images, the digitization of the biopsy glass slides  Structured, summarized and categorized reports can be associated with the image content when searching in large archives  Such as system would be able to drastically increase the efficiency of diagnostic processes for the majority of cases where in spite of obvious primary diagnosis, still time and effort is required from the pathologists to write a descriptive report 
The primary objective of our study is to analyze the efficacy of existing machine learning approaches for the automated classification of pathology reports into different diagnosis categories  We demonstrate that TF-IDF feature vectors combined with linear SVM or XGBoost classifier can be an effective method for classification of the reports, achieving up to 83% accuracy  We also show that TF-IDF features are capable of identifying important keywords within a pathology report  Furthermore, we have created a new dataset consisting of 1,949 pathology reports across 37 primary diagnoses  Taken together, our exploratory experiments with a newly introduced dataset on pathology reports opens many new opportunities for researchers to develop a scalable and automatic information extraction from unstructured pathology reports 
Background
NLP approaches for information extraction within the biomedical research areas range from rule-based systems BIBREF3 , to domain-specific systems using feature-based classification BIBREF1 , to the recent deep networks for end-to-end feature extraction and classification BIBREF0   NLP has had varied degree of success with free-text pathology reports BIBREF4   Various studies have acknowledge the success of NLP in interpreting pathology reports, especially for classification tasks or extracting a single attribute from a report BIBREF4 , BIBREF5  
The Cancer Text Information Extraction System (caTIES) BIBREF6 is a framework developed in a caBIG project focuses on information extraction from pathology reports  Specifically, caTIES extracts information from surgical pathology reports (SPR) with good precision as well as recall 
Another system known as Open Registry BIBREF7 is capable of filtering the reports with disease codes containing cancer  In BIBREF8 , an approach called Automated Retrieval Console (ARC) is proposed which uses machine learning models to predict the degree of association of a given pathology or radiology with the cancer  The performance ranges from an F-measure of 0 75 for lung cancer to 0 94 for colorectal cancer  However, ARC uses domain-specific rules which hiders with the generalization of the approach to variety of pathology reports 
This research work is inspired by themes emerging in many of the above studies  Specifically, we are evaluating the task of predicting the primary diagnosis from the pathology report  Unlike previous approaches, the system does not rely on custom rule-based knowledge, domain specific features, balanced dataset with fewer number of classes 
Materials and Methods
We assembled a dataset of 1,949 cleaned pathology reports  Each report is associated with one of the 37 different primary diagnoses based on IDC-O codes  The reports are collected from four different body parts or primary sites from multiple patients  The distribution of reports across different primary diagnoses and primary sites is reported in tab:report-distribution  The dataset was developed in three steps as follows 
Collecting pathology reports: The total of 11,112 pathology reports were downloaded from NCI's Genomic Data Commons (GDC) dataset in PDF format BIBREF9   Out of all PDF files, 1,949 reports were selected across multiple patients from four specific primary sites—thymus, testis, lung, and kidney  The selection was primarily made based on the quality of PDF files 
Cleaning reports: The next step was to extract the text content from these reports  Due to the significant time expense of manually re-typing all the pathology reports, we developed a new strategy to prepare our dataset  We applied an Optical Character Recognition (OCR) software to convert the PDF reports to text files  Then, we manually inspected all generated text files to fix any grammar/spelling issues and irrelevant characters as an artefact produced by the OCR system 
Splitting into training-testing data: We split the cleaned reports into 70% and 30% for training and testing, respectively  This split resulted in 1,364 training, and 585 testing reports 
Pre-Processing of Reports
We pre-processed the reports by setting their text content to lowercase and filtering out any non-alphanumeric characters  We used NLTK library to remove stopping words, e g , `the', `an', `was', `if' and so on BIBREF10   We then analyzed the reports to find common bigrams, such as “lung parenchyma”, “microscopic examination”, “lymph node” etc  We joined the biagrams with a hyphen, converting them into a single word  We further removed the words that occur less than 2% in each of the diagnostic category  As well, we removed the words that occur more than 90% across all the categories  We stored each pre-processed report in a separate text file 
TF-IDF features
TF-IDF stands for Term Frequency-Inverse Document Frequency, and it is a useful weighting scheme in information retrieval and text mining  TF-IDF signifies the importance of a term in a document within a corpus  It is important to note that a document here refers to a pathology report, a corpus refers to the collection of reports, and a term refers to a single word in a report  The TF-IDF weight for a term INLINEFORM0 in a document INLINEFORM1 is given by DISPLAYFORM0
We performed the following steps to transform a pathology report into a feature vector:
Create a set of vocabulary containing all unique words from all the pre-processed training reports 
Create a zero vector INLINEFORM0 of the same length as the vocabulary 
For each word INLINEFORM0 in a report INLINEFORM1 , set the corresponding index in INLINEFORM2 to INLINEFORM3  
The resultant INLINEFORM0 is a feature vector for the report INLINEFORM1 and it is a highly sparse vector 
Keyword extraction and topic modelling
The keyword extraction involves identifying important words within reports that summarizes its content  In contrast, the topic modelling allows grouping these keywords using an intelligent scheme, enabling users to further focus on certain aspects of a document  All the words in a pathology report are sorted according to their TF-IDF weights  The top INLINEFORM0 sorted words constitute the top INLINEFORM1 keywords for the report  The INLINEFORM2 is empirically set to 50 within this research  The extracted keywords are further grouped into different topics by using latent Dirichlet allocation (LDA) BIBREF11   The keywords in a report are highlighted using the color theme based on their topics 
Evaluation metrics
Each model is evaluated using two standard NLP metrics—micro and macro averaged F-scores, the harmonic mean of related metrics precision and recall  For each diagnostic category INLINEFORM0 from a set of 37 different classes INLINEFORM1 , the number of true positives INLINEFORM2 , false positives INLINEFORM3 , and false negatives INLINEFORM4 , the micro F-score is defined as DISPLAYFORM0
whereas macro F-score is given by DISPLAYFORM0
In summary, micro-averaged metrics have class representation roughly proportional to their test set representation (same as accuracy for classification problem with a single label per data point), whereas macro-averaged metrics are averaged by class without weighting by class prevalence BIBREF12  
Experimental setting
In this study, we performed two different series of experiments: i) evaluating the performance of TF-IDF features and various machine learning classifiers on the task of predicting primary diagnosis from the text content of a given report, and ii) using TF-IDF and LDA techniques to highlight the important keywords within a report  For the first experiment series, training reports are pre-processed, then their TF-IDF features are extracted  The TF-IDF features and the training labels are used to train different classification models  These different classification models and their hyper-parameters are reported in tab:classifier  The performance of classifiers is measured quantitatively on the test dataset using the evaluation metrics discussed in the previous section  For the second experiment series, a random report is selected and its top 50 keywords are extracted using TF-IDF weights  These 50 keywords are highlighted using different colors based on their associated topic, which are extracted through LDA  A non-expert based qualitative inspection is performed on the extracted keywords and their corresponding topics 
Experiment Series 1
A classification model is trained to predict the primary diagnosis given the content of the cancer pathology report  The performance results on this task are reported in tab:results  We can observe that the XGBoost classifier outperformed all other models for both the micro F-score metric, with a score of 0 92, and the macro F-score metric, with a score of 0 31  This was an improvement of 7% for the micro F-score over the next best model, SVM-L, and a marginal improvement of 5% for macro F-score  It is interesting to note that SVM with linear kernels performs much better than SVM with RBF kernel, scoring 9% on the macro F-score and 12% more on the micro F-score  It is suspected that since words used in primary diagnosis itself occur in some reports, thus enabling the linear models to outperform complex models 
Experiment Series 2
fig:keywords shows the top 50 keywords highlighted using TF-IDF and LDA  The proposed approach has performed well in highlighting the important regions, for example the topic highlighted with a red color containing “presence range tumor necrosis” provides useful biomarker information to readers 
Conclusions
We proposed a simple yet efficient TF-IDF method to extract and corroborate useful keywords from pathology cancer reports  Encoding a pathology report for cancer and tumor surveillance is a laborious task, and sometimes it is subjected to human errors and variability in the interpretation  One of the most important aspects of encoding a pathology report involves extracting the primary diagnosis  This may be very useful for content-based image retrieval to combine with visual information  We used existing classification model and TF-IDF features to predict the primary diagnosis  We achieved up to 92% accuracy using XGBoost classifier  The prediction accuracy empowers the adoption of machine learning methods for automated information extraction from pathology reports 

ground_truth: Unanswerable

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How are models evaluated in this human-machine communication game?
Context 4:  Incorporating this extra information improves the translation task  The large number of shared word types between the source and target sentences indicates that sharing the representation between them is beneficial 
Methods ::: Shakespearizing Poetic Captions ::: Seq2Seq with Attention
We use a sequence-to-sequence model which consists of a single layer unidrectional LSTM encoder and a single layer LSTM decoder and pre-trained retrofitted word embeddings shared between source and target sentences  We experimented with two different types of attention: global attention BIBREF9, in which the model makes use of the output from the encoder and decoder for the current time step only, and Bahdanau attention BIBREF10, where computing attention requires the output of the decoder from the prior time step  We found that global attention performs better in practice for our task of text style transfer 
Methods ::: Shakespearizing Poetic Captions ::: Seq2Seq with a Pointer Network
Since a pair of corresponding Shakespeare and modern English sentences have significant vocabulary overlap we extend the sequence-to-sequence model mentioned above using pointer networks BIBREF11 that provide location based attention and have been used to enable copying of tokens directly from the input  Moreover, there are lot of proper nouns and rare words which might not be predicted by a vanilla sequence to sequence model 
Methods ::: Shakespearizing Poetic Captions ::: Prediction
For both seq2seq models, we use the attention matrices returned at each decoder time step during inference, to compute the next word in the translated sequence if the decoder output at the current time step is the UNK token  We replace the UNKs in the target output with the highest aligned, maximum attention, source word  The seq2seq model with global attention gives the best results with an average target BLEU score of 29 65 on the style transfer dataset, compared with an average target BLEU score of 26 97 using the seq2seq model with pointer networks 
Results
We perform a qualitative analysis of the Shakespearean prose generated for the input paintings  We conducted a survey, in which we presented famous paintings including those shown in Figures FIGREF1 and FIGREF10 and the corresponding Shakespearean prose generated by the model, and asked 32 students to rate them on the basis of content, creativity and similarity to Shakespearean style on a Likert scale of 1-5  Figure FIGREF12 shows the result of our human evaluation 
The average content score across the paintings is 3 7 which demonstrates that the prose generated is relevant to the painting  The average creativity score is 3 9 which demonstrates that the model captures more than basic objects in the painting successfully using poetic clues in the scene  The average style score is 3 9 which demonstrates that the prose generated is perceived to be in the style of Shakespeare 
We also perform a quantitative analysis of style transfer by generating BLEU scores for the model output using the style transfer dataset  The variation of the BLEU scores with the source sentence lengths is shown in Figure FIGREF11  As expected, the BLEU scores decrease with increase in source sentence lengths 
Results ::: Implementation
All models were trained on Google Colab with a single GPU using Python 3 6 and Tensorflow 2 0  The number of hidden units for the encoder and decoder is 1,576 and 256 for seq2seq with global attention and seq2seq with pointer networks respectively  Adam optimizer was used with the default learning rate of 0 001  The model was trained for 25 epochs  We use pre-trained retrofitted word embeddings of dimension 192 
Results ::: Limitations
Since we do not have an end-to-end dataset, the generated English poem may not work well with Shakespeare style transfer as shown in Figure FIGREF12 for "Starry Night" with a low average content score  This happens when the style transfer dataset does not have similar words in the training set of sentences  A solution would be to expand the style transfer dataset, for a better representation of the poem data 
Results ::: Conclusions and Future Work
In conclusion, combining two pipelines with an intermediate representation works well in practice  We observe that a CNN-RNN based image-to-poem net combined with a seq2seq model with parallel text corpus for text style transfer synthesizes Shakespeare-style prose for a given painting  For the seq2seq model used, we observe that it performs better in practice using global attention as compared with local attention  We make our models and code publicly available BIBREF12  In future work we would like to experiment with GANs in the absence of non-parallel datasets, so that we can use varied styles for text style transfer  We would also like to experiment with cross aligned auto-encoders, which form a latent content representation, to efficiently separate style and content 

ground_truth: by training an autocomplete system on 500K randomly sampled sentences from Yelp reviews

efficiency of a communication scheme $(q_{\alpha },p_{\beta })$ by the retention rate of tokens, which is measured as the fraction of tokens that are kept in the keywords, accuracy of a scheme is measured as the fraction of sentences generated by greedily decoding the model that exactly matches the target sentence