prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What were their results on the new dataset?
Context 1: Introduction
In the kitchen, we increasingly rely on instructions from cooking websites: recipes  A cook with a predilection for Asian cuisine may wish to prepare chicken curry, but may not know all necessary ingredients apart from a few basics  These users with limited knowledge cannot rely on existing recipe generation approaches that focus on creating coherent recipes given all ingredients and a recipe name BIBREF0  Such models do not address issues of personal preference (e g  culinary tastes, garnish choices) and incomplete recipe details  We propose to approach both problems via personalized generation of plausible, user-specific recipes using user preferences extracted from previously consumed recipes 
Our work combines two important tasks from natural language processing and recommender systems: data-to-text generation BIBREF1 and personalized recommendation BIBREF2  Our model takes as user input the name of a specific dish, a few key ingredients, and a calorie level  We pass these loose input specifications to an encoder-decoder framework and attend on user profiles—learned latent representations of recipes previously consumed by a user—to generate a recipe personalized to the user's tastes  We fuse these `user-aware' representations with decoder output in an attention fusion layer to jointly determine text generation  Quantitative (perplexity, user-ranking) and qualitative analysis on user-aware model outputs confirm that personalization indeed assists in generating plausible recipes from incomplete ingredients 
While personalized text generation has seen success in conveying user writing styles in the product review BIBREF3, BIBREF4 and dialogue BIBREF5 spaces, we are the first to consider it for the problem of recipe generation, where output quality is heavily dependent on the content of the instructions—such as ingredients and cooking techniques 
To summarize, our main contributions are as follows:
We explore a new task of generating plausible and personalized recipes from incomplete input specifications by leveraging historical user preferences;
We release a new dataset of 180K+ recipes and 700K+ user reviews for this task;
We introduce new evaluation strategies for generation quality in instructional texts, centering on quantitative measures of coherence  We also show qualitatively and quantitatively that personalized models generate high-quality and specific recipes that align with historical user preferences 
Related Work
Large-scale transformer-based language models have shown surprising expressivity and fluency in creative and conditional long-text generation BIBREF6, BIBREF7  Recent works have proposed hierarchical methods that condition on narrative frameworks to generate internally consistent long texts BIBREF8, BIBREF9, BIBREF10  Here, we generate procedurally structured recipes instead of free-form narratives 
Recipe generation belongs to the field of data-to-text natural language generation BIBREF1, which sees other applications in automated journalism BIBREF11, question-answering BIBREF12, and abstractive summarization BIBREF13, among others  BIBREF14, BIBREF15 model recipes as a structured collection of ingredient entities acted upon by cooking actions  BIBREF0 imposes a `checklist' attention constraint emphasizing hitherto unused ingredients during generation  BIBREF16 attend over explicit ingredient references in the prior recipe step  Similar hierarchical approaches that infer a full ingredient list to constrain generation will not help personalize recipes, and would be infeasible in our setting due to the potentially unconstrained number of ingredients (from a space of 10K+) in a recipe  We instead learn historical preferences to guide full recipe generation 
A recent line of work has explored user- and item-dependent aspect-aware review generation BIBREF3, BIBREF4  This work is related to ours in that it combines contextual language generation with personalization  Here, we attend over historical user preferences from previously consumed recipes to generate recipe content, rather than writing styles 
Approach
Our model's input specification consists of: the recipe name as a sequence of tokens, a partial list of ingredients, and a caloric level (high, medium, low)  It outputs the recipe instructions as a token sequence: $\mathcal {W}_r=\lbrace w_{r,0}, \dots , w_{r,T}\rbrace $ for a recipe $r$ of length $T$  To personalize output, we use historical recipe interactions of a user $u \in \mathcal {U}$ 
Encoder: Our encoder has three embedding layers: vocabulary embedding $\mathcal {V}$, ingredient embedding $\mathcal {I}$, and caloric-level embedding $\mathcal {C}$  Each token in the (length $L_n$) recipe name is embedded via $\mathcal {V}$; the embedded token sequence is passed to a two-layered bidirectional GRU (BiGRU) BIBREF17, which outputs hidden states for names $\lbrace \mathbf {n}_{\text{enc},j} \in \mathbb {R}^{2d_h}\rbrace $, with hidden size $d_h$  Similarly each of the $L_i$ input ingredients is embedded via $\mathcal {I}$, and the embedded ingredient sequence is passed to another two-layered BiGRU to output ingredient hidden states as $\lbrace \mathbf {i}_{\text{enc},j} \in \mathbb {R}^{2d_h}\rbrace $  The caloric level is embedded via $\mathcal {C}$ and passed through a projection layer with weights $W_c$ to generate calorie hidden representation $\mathbf {c}_{\text{enc}} \in \mathbb {R}^{2d_h}$ 
Ingredient Attention: We apply attention BIBREF18 over the encoded ingredients to use encoder outputs at each decoding time step  We define an attention-score function $\alpha $ with key $K$ and query $Q$:
with trainable weights $W_{\alpha }$, bias $\mathbf {b}_{\alpha }$, and normalization term $Z$  At decoding time $t$, we calculate the ingredient context $\mathbf {a}_{t}^{i} \in \mathbb {R}^{d_h}$ as:
Decoder: The decoder is a two-layer GRU with hidden state $h_t$ conditioned on previous hidden state $h_{t-1}$ and input token $w_{r, t}$ from the original recipe text  We project the concatenated encoder outputs as the initial decoder hidden state:
To bias generation toward user preferences, we attend over a user's previously reviewed recipes to jointly determine the final output token distribution  We consider two different schemes to model preferences from user histories: (1) recipe interactions, and (2) techniques seen therein (defined in data)  BIBREF19, BIBREF20, BIBREF21 explore similar schemes for personalized recommendation 
Prior Recipe Attention: We obtain the set of prior recipes for a user $u$: $R^+_u$, where each recipe can be represented by an embedding from a recipe embedding layer $\mathcal {R}$ or an average of the name tokens embedded by $\mathcal {V}$  We attend over the $k$-most recent prior recipes, $R^{k+}_u$, to account for temporal drift of user preferences BIBREF22  These embeddings are used in the `Prior Recipe' and `Prior Name' models, respectively 
Given a recipe representation $\mathbf {r} \in \mathbb {R}^{d_r}$ (where $d_r$ is recipe- or vocabulary-embedding size depending on the recipe representation) the prior recipe attention context $\mathbf {a}_{t}^{r_u}$ is calculated as
Prior Technique Attention: We calculate prior technique preference (used in the `Prior Tech` model) by normalizing co-occurrence between users and techniques seen in $R^+_u$, to obtain a preference vector $\rho _{u}$  Each technique $x$ is embedded via a technique embedding layer $\mathcal {X}$ to $\mathbf {x}\in \mathbb {R}^{d_x}$  Prior technique attention is calculated as
where, inspired by copy mechanisms BIBREF23, BIBREF24, we add $\rho _{u,x}$ for technique $x$ to emphasize the attention by the user's prior technique preference 
Attention Fusion Layer: We fuse all contexts calculated at time $t$, concatenating them with decoder GRU output and previous token embedding:
We then calculate the token probability:
and maximize the log-likelihood of the generated sequence conditioned on input specifications and user preferences  fig:ex shows a case where the Prior Name model attends strongly on previously consumed savory recipes to suggest the usage of an additional ingredient (`cilantro') 
Recipe Dataset: Food com
We collect a novel dataset of 230K+ recipe texts and 1M+ user interactions (reviews) over 18 years (2000-2018) from Food com  Here, we restrict to recipes with at least 3 steps, and at least 4 and no more than 20 ingredients  We discard users with fewer than 4 reviews, giving 180K+ recipes and 700K+ reviews, with splits as in tab:recipeixnstats 
Our model must learn to generate from a diverse recipe space: in our training data, the average recipe length is 117 tokens with a maximum of 256  There are 13K unique ingredients across all recipes  Rare words dominate the vocabulary: 95% of words appear $<$100 times, accounting for only 1 65% of all word usage  As such, we perform Byte-Pair Encoding (BPE) tokenization BIBREF25, BIBREF26, giving a training vocabulary of 15K tokens across 19M total mentions  User profiles are similarly diverse: 50% of users have consumed $\le $6 recipes, while 10% of users have consumed $>$45 recipes 
We order reviews by timestamp, keeping the most recent review for each user as the test set, the second most recent for validation, and the remainder for training (sequential leave-one-out evaluation BIBREF27)  We evaluate only on recipes not in the training set 
We manually construct a list of 58 cooking techniques from 384 cooking actions collected by BIBREF15; the most common techniques (bake, combine, pour, boil) account for 36 5% of technique mentions  We approximate technique adherence via string match between the recipe text and technique list 
Experiments and Results
For training and evaluation, we provide our model with the first 3-5 ingredients listed in each recipe  We decode recipe text via top-$k$ sampling BIBREF7, finding $k=3$ to produce satisfactory results  We use a hidden size $d_h=256$ for both the encoder and decoder  Embedding dimensions for vocabulary, ingredient, recipe, techniques, and caloric level are 300, 10, 50, 50, and 5 (respectively)  For prior recipe attention, we set $k=20$, the 80th %-ile for the number of user interactions  We use the Adam optimizer BIBREF28 with a learning rate of $10^{-3}$, annealed with a decay rate of 0 9 BIBREF29  We also use teacher-forcing BIBREF30 in all training epochs 
In this work, we investigate how leveraging historical user preferences can improve generation quality over strong baselines in our setting  We compare our personalized models against two baselines  The first is a name-based Nearest-Neighbor model (NN)  We initially adapted the Neural Checklist Model of BIBREF0 as a baseline; however, we ultimately use a simple Encoder-Decoder baseline with ingredient attention (Enc-Dec), which provides comparable performance and lower complexity  All personalized models outperform baseline in BPE perplexity (tab:metricsontest) with Prior Name performing the best  While our models exhibit comparable performance to baseline in BLEU-1/4 and ROUGE-L, we generate more diverse (Distinct-1/2: percentage of distinct unigrams and bigrams) and acceptable recipes  BLEU and ROUGE are not the most appropriate metrics for generation quality  A `correct' recipe can be written in many ways with the same main entities (ingredients)  As BLEU-1/4 capture structural information via n-gram matching, they are not correlated with subjective recipe quality  This mirrors observations from BIBREF31, BIBREF8 
We observe that personalized models make more diverse recipes than baseline  They thus perform better in BLEU-1 with more key entities (ingredient mentions) present, but worse in BLEU-4, as these recipes are written in a personalized way and deviate from gold on the phrasal level  Similarly, the `Prior Name' model generates more unigram-diverse recipes than other personalized models and obtains a correspondingly lower BLEU-1 score 
Qualitative Analysis: We present sample outputs for a cocktail recipe in tab:samplerecipes, and additional recipes in the appendix  Generation quality progressively improves from generic baseline output to a blended cocktail produced by our best performing model  Models attending over prior recipes explicitly reference ingredients  The Prior Name model further suggests the addition of lemon and mint, which are reasonably associated with previously consumed recipes like coconut mousse and pork skewers 
Personalization: To measure personalization, we evaluate how closely the generated text corresponds to a particular user profile  We compute the likelihood of generated recipes using identical input specifications but conditioned on ten different user profiles—one `gold' user who consumed the original recipe, and nine randomly generated user profiles  Following BIBREF8, we expect the highest likelihood for the recipe conditioned on the gold user  We measure user matching accuracy (UMA)—the proportion where the gold user is ranked highest—and Mean Reciprocal Rank (MRR) BIBREF32 of the gold user  All personalized models beat baselines in both metrics, showing our models personalize generated recipes to the given user profiles  The Prior Name model achieves the best UMA and MRR by a large margin, revealing that prior recipe names are strong signals for personalization  Moreover, the addition of attention mechanisms to capture these signals improves language modeling performance over a strong non-personalized baseline 
Recipe Level Coherence: A plausible recipe should possess a coherent step order, and we evaluate this via a metric for recipe-level coherence  We use the neural scoring model from BIBREF33 to measure recipe-level coherence for each generated recipe  Each recipe step is encoded by BERT BIBREF34  Our scoring model is a GRU network that learns the overall recipe step ordering structure by minimizing the cosine similarity of recipe step hidden representations presented in the correct and reverse orders  Once pretrained, our scorer calculates the similarity of a generated recipe to the forward and backwards ordering of its corresponding gold label, giving a score equal to the difference between the former and latter  A higher score indicates better step ordering (with a maximum score of 2)  tab:coherencemetrics shows that our personalized models achieve average recipe-level coherence scores of 1 78-1 82, surpassing the baseline at 1 77 
Recipe Step Entailment: Local coherence is also crucial to a user following a recipe: it is crucial that subsequent steps are logically consistent with prior ones  We model local coherence as an entailment task: predicting the likelihood that a recipe step follows the preceding  We sample several consecutive (positive) and non-consecutive (negative) pairs of steps from each recipe  We train a BERT BIBREF34 model to predict the entailment score of a pair of steps separated by a [SEP] token, using the final representation of the [CLS] token  The step entailment score is computed as the average of scores for each set of consecutive steps in each recipe, averaged over every generated recipe for a model, as shown in tab:coherencemetrics 
Human Evaluation: We presented 310 pairs of recipes for pairwise comparison BIBREF8 (details in appendix) between baseline and each personalized model, with results shown in tab:metricsontest  On average, human evaluators preferred personalized model outputs to baseline 63% of the time, confirming that personalized attention improves the semantic plausibility of generated recipes  We also performed a small-scale human coherence survey over 90 recipes, in which 60% of users found recipes generated by personalized models to be more coherent and preferable to those generated by baseline models 
Conclusion
In this paper, we propose a novel task: to generate personalized recipes from incomplete input specifications and user histories  On a large novel dataset of 180K recipes and 700K reviews, we show that our personalized generative models can generate plausible, personalized, and coherent recipes preferred by human evaluators for consumption  We also introduce a set of automatic coherence measures for instructional texts as well as personalization metrics to support our claims  Our future work includes generating structured representations of recipes to handle ingredient properties, as well as accounting for references to collections of ingredients (e g  “dry mix") 
Acknowledgements  This work is partly supported by NSF #1750063  We thank all reviewers for their constructive suggestions, as well as Rei M , Sujoy P , Alicia L , Eric H , Tim S , Kathy C , Allen C , and Micah I  for their feedback 
Appendix ::: Food com: Dataset Details
Our raw data consists of 270K recipes and 1 4M user-recipe interactions (reviews) scraped from Food com, covering a period of 18 years (January 2000 to December 2018)  See tab:int-stats for dataset summary statistics, and tab:samplegk for sample information about one user-recipe interaction and the recipe involved 
Appendix ::: Generated Examples
See tab:samplechx for a sample recipe for chicken chili and tab:samplewaffle for a sample recipe for sweet waffles 
Human Evaluation
We prepared a set of 15 pairwise comparisons per evaluation session, and collected 930 pairwise evaluations (310 per personalized model) over 62 sessions  For each pair, users were given a partial recipe specification (name and 3-5 key ingredients), as well as two generated recipes labeled `A' and `B'  One recipe is generated from our baseline encoder-decoder model and one recipe is generated by one of our three personalized models (Prior Tech, Prior Name, Prior Recipe)  The order of recipe presentation (A/B) is randomly selected for each question  A screenshot of the user evaluation interface is given in fig:exeval  We ask the user to indicate which recipe they find more coherent, and which recipe best accomplishes the goal indicated by the recipe name  A screenshot of this survey interface is given in fig:exeval2 
Context 2: Introduction
Sarcasm is an intensive, indirect and complex construct that is often intended to express contempt or ridicule   Sarcasm, in speech, is multi-modal, involving tone, body-language and gestures along with linguistic artifacts used in speech  Sarcasm in text, on the other hand, is more restrictive when it comes to such non-linguistic modalities  This makes recognizing textual sarcasm more challenging for both humans and machines 
Sarcasm detection plays an indispensable role in applications like online review summarizers, dialog systems, recommendation systems and sentiment analyzers  This makes automatic detection of sarcasm an important problem  However, it has been quite difficult to solve such a problem with traditional NLP tools and techniques  This is apparent from the results reported by the survey from DBLP:journals/corr/JoshiBC16  The following discussion brings more insights into this 
Consider a scenario where an online reviewer gives a negative opinion about a movie through sarcasm: “This is the kind of movie you see because the theater has air conditioning”  It is difficult for an automatic sentiment analyzer to assign a rating to the movie and, in the absence of any other information, such a system may not be able to comprehend that prioritizing the air-conditioning facilities of the theater over the movie experience indicates a negative sentiment towards the movie  This gives an intuition to why, for sarcasm detection, it is necessary to go beyond textual analysis 
We aim to address this problem by exploiting the psycholinguistic side of sarcasm detection, using cognitive features extracted with the help of eye-tracking  A motivation to consider cognitive features comes from analyzing human eye-movement trajectories that supports the conjecture: Reading sarcastic texts induces distinctive eye movement patterns, compared to literal texts  The cognitive features, derived from human eye movement patterns observed during reading, include two primary feature types:
The cognitive features, along with textual features used in best available sarcasm detectors, are used to train binary classifiers against given sarcasm labels  Our experiments show significant improvement in classification accuracy over the state of the art, by performing such augmentation 
Related Work
Sarcasm, in general, has been the focus of research for quite some time  In one of the pioneering works jorgensen1984test explained how sarcasm arises when a figurative meaning is used opposite to the literal meaning of the utterance  In the word of clark1984pretense, sarcasm processing involves canceling the indirectly negated message and replacing it with the implicated one  giora1995irony, on the other hand, define sarcasm as a mode of indirect negation that requires processing of both negated and implicated messages  ivanko2003context define sarcasm as a six tuple entity consisting of a speaker, a listener, Context, Utterance, Literal Proposition and Intended Proposition and study the cognitive aspects of sarcasm processing 
Computational linguists have previously addressed this problem using rule based and statistical techniques, that make use of : (a) Unigrams and Pragmatic features BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 (b) Stylistic patterns BIBREF4 and patterns related to situational disparity BIBREF5 and (c) Hastag interpretations BIBREF6 , BIBREF7  
Most of the previously done work on sarcasm detection uses distant supervision based techniques (ex: leveraging hashtags) and stylistic/pragmatic features (emoticons, laughter expressions such as “lol” etc)  But, detecting sarcasm in linguistically well-formed structures, in absence of explicit cues or information (like emoticons), proves to be hard using such linguistic/stylistic features alone 
With the advent of sophisticated eye-trackers and electro/magneto-encephalographic (EEG/MEG) devices, it has been possible to delve deep into the cognitive underpinnings of sarcasm understanding  Filik2014, using a series of eye-tracking and EEG experiments try to show that for unfamiliar ironies, the literal interpretation would be computed first  They also show that a mismatch with context would lead to a re-interpretation of the statement, as being ironic  Camblin2007103 show that in multi-sentence passages, discourse congruence has robust effects on eye movements  This also implies that disrupted processing occurs for discourse incongruent words, even though they are perfectly congruous at the sentence level  In our previous work BIBREF8 , we augment cognitive features, derived from eye-movement patterns of readers, with textual features to detect whether a human reader has realized the presence of sarcasm in text or not 
The recent advancements in the literature discussed above, motivate us to explore gaze-based cognition for sarcasm detection  As far as we know, our work is the first of its kind 
Eye-tracking Database for Sarcasm Analysis
Sarcasm often emanates from incongruity BIBREF9 , which enforces the brain to reanalyze it BIBREF10   This, in turn, affects the way eyes move through the text  Hence, distinctive eye-movement patterns may be observed in the case of successful processing of sarcasm in text in contrast to literal texts  This hypothesis forms the crux of our method for sarcasm detection and we validate this using our previously released freely available sarcasm dataset BIBREF8 enriched with gaze information 
Document Description
The database consists of 1,000 short texts, each having 10-40 words  Out of these, 350 are sarcastic and are collected as follows: (a) 103 sentences are from two popular sarcastic quote websites, (b) 76 sarcastic short movie reviews are manually extracted from the Amazon Movie Corpus BIBREF11 by two linguists  (c) 171 tweets are downloaded using the hashtag #sarcasm from Twitter  The 650 non-sarcastic texts are either downloaded from Twitter or extracted from the Amazon Movie Review corpus  The sentences do not contain words/phrases that are highly topic or culture specific  The tweets were normalized to make them linguistically well formed to avoid difficulty in interpreting social media lingo  Every sentence in our dataset carries positive or negative opinion about specific “aspects”  For example, the sentence “The movie is extremely well cast” has positive sentiment about the aspect “cast” 
The annotators were seven graduate students with science and engineering background, and possess good English proficiency  They were given a set of instructions beforehand and are advised to seek clarifications before they proceed  The instructions mention the nature of the task, annotation input method, and necessity of head movement minimization during the experiment 
Task Description
The task assigned to annotators was to read sentences one at a time and label them with with binary labels indicating the polarity (i e , positive/negative)  Note that, the participants were not instructed to annotate whether a sentence is sarcastic or not , to rule out the Priming Effect (i e , if sarcasm is expected beforehand, processing incongruity becomes relatively easier BIBREF12 )  The setup ensures its “ecological validity” in two ways: (1) Readers are not given any clue that they have to treat sarcasm with special attention  This is done by setting the task to polarity annotation (instead of sarcasm detection)  (2) Sarcastic sentences are mixed with non sarcastic text, which does not give prior knowledge about whether the forthcoming text will be sarcastic or not 
The eye-tracking experiment is conducted by following the standard norms in eye-movement research BIBREF13   At a time, one sentence is displayed to the reader along with the “aspect” with respect to which the annotation has to be provided  While reading, an SR-Research Eyelink-1000 eye-tracker (monocular remote mode, sampling rate 500Hz) records several eye-movement parameters like fixations (a long stay of gaze) and saccade (quick jumping of gaze between two positions of rest) and pupil size 
The accuracy of polarity annotation varies between 72%-91% for sarcastic texts and 75%-91% for non-sarcastic text, showing the inherent difficulty of sentiment annotation, when sarcasm is present in the text under consideration  Annotation errors may be attributed to: (a) lack of patience/attention while reading, (b) issues related to text comprehension, and (c) confusion/indecisiveness caused due to lack of context 
For our analysis, we do not discard the incorrect annotations present in the database  Since our system eventually aims to involve online readers for sarcasm detection, it will be hard to segregate readers who misinterpret the text  We make a rational assumption that, for a particular text, most of the readers, from a fairly large population, will be able to identify sarcasm  Under this assumption, the eye-movement parameters, averaged across all readers in our setting, may not be significantly distorted by a few readers who would have failed to identify sarcasm  This assumption is applicable for both regular and multi-instance based classifiers explained in section SECREF6  
Analysis of Eye-movement Data
We observe distinct behavior during sarcasm reading, by analyzing the “fixation duration on the text” (also referred to as “dwell time” in the literature) and “scanpaths” of the readers 
Variation in the Average Fixation Duration per Word
Since sarcasm in text can be expected to induce cognitive load, it is reasonable to believe that it would require more processing time BIBREF14   Hence, fixation duration normalized over total word count should usually be higher for a sarcastic text than for a non-sarcastic one  We observe this for all participants in our dataset, with the average fixation duration per word for sarcastic texts being at least 1 5 times more than that of non-sarcastic texts  To test the statistical significance, we conduct a two-tailed t-test (assuming unequal variance) to compare the average fixation duration per word for sarcastic and non-sarcastic texts  The hypothesized mean difference is set to 0 and the error tolerance limit ( INLINEFORM0 ) is set to 0 05  The t-test analysis, presented in Table TABREF11 , shows that for all participants, a statistically significant difference exists between the average fixation duration per word for sarcasm (higher average fixation duration) and non-sarcasm (lower average fixation duration)  This affirms that the presence of sarcasm affects the duration of fixation on words 
It is important to note that longer fixations may also be caused by other linguistic subtleties (such as difficult words, ambiguity and syntactically complex structures) causing delay in comprehension, or occulomotor control problems forcing readers to spend time adjusting eye-muscles  So, an elevated average fixation duration per word may not sufficiently indicate the presence of sarcasm  But we would also like to share that, for our dataset, when we considered readability (Flesch readability ease-score BIBREF15 ), number of words in a sentence and average character per word along with the sarcasm label as the predictors of average fixation duration following a linear mixed effect model BIBREF16 , sarcasm label turned out to be the most significant predictor with a maximum slope  This indicates that average fixation duration per word has a strong connection with the text being sarcastic, at least in our dataset 
We now analyze scanpaths to gain more insights into the sarcasm comprehension process 
Analysis of Scanpaths
Scanpaths are line-graphs that contain fixations as nodes and saccades as edges; the radii of the nodes represent the fixation duration  A scanpath corresponds to a participant's eye-movement pattern while reading a particular sentence  Figure FIGREF14 presents scanpaths of three participants for the sarcastic sentence S1 and the non-sarcastic sentence S2  The x-axis of the graph represents the sequence of words a reader reads, and the y-axis represents a temporal sequence in milliseconds 
Consider a sarcastic text containing incongruous phrases A and B  Our qualitative scanpath-analysis reveals that scanpaths with respect to sarcasm processing have two typical characteristics  Often, a long regression - a saccade that goes to a previously visited segment - is observed when a reader starts reading B after skimming through A  In a few cases, the fixation duration on A and B are significantly higher than the average fixation duration per word  In sentence S1, we see long and multiple regressions from the two incongruous phrases “misconception” and “cherish”, and a few instances where phrases “always cherish” and “original misconception” are fixated longer than usual  Such eye-movement behaviors are not seen for S2 
Though sarcasm induces distinctive scanpaths like the ones depicted in Figure FIGREF14 in the observed examples, presence of such patterns is not sufficient to guarantee sarcasm; such patterns may also possibly arise from literal texts  We believe that a combination of linguistic features, readability of text and features derived from scanpaths would help discriminative machine learning models learn sarcasm better 
Features for Sarcasm Detection
We describe the features used for sarcasm detection in Table   The features enlisted under lexical,implicit incongruity and explicit incongruity are borrowed from various literature (predominantly from joshi2015harnessing)  These features are essential to separate sarcasm from other forms semantic incongruity in text (for example ambiguity arising from semantic ambiguity or from metaphors)  Two additional textual features viz  readability and word count of the text are also taken under consideration  These features are used to reduce the effect of text hardness and text length on the eye-movement patterns 
Simple Gaze Based Features
Readers' eye-movement behavior, characterized by fixations, forward saccades, skips and regressions, can be directly quantified by simple statistical aggregation (i e , either computing features for individual participants and then averaging or performing a multi-instance based learning as explained in section SECREF6 )  Since these eye-movement attributes relate to the cognitive process in reading BIBREF17 , we consider these as features in our model  Some of these features have been reported by sarcasmunderstandability for modeling sarcasm understandability of readers  However, as far as we know, these features are being introduced in NLP tasks like textual sarcasm detection for the first time  The values of these features are believed to increase with the increase in the degree of surprisal caused by incongruity in text (except skip count, which will decrease) 
Complex Gaze Based Features
For these features, we rely on a graph structure, namely “saliency graphs", derived from eye-gaze information and word sequences in the text 
For each reader and each sentence, we construct a “saliency graph”, representing the reader's attention characteristics  A saliency graph for a sentence INLINEFORM0 for a reader INLINEFORM1 , represented as INLINEFORM2 , is a graph with vertices ( INLINEFORM3 ) and edges ( INLINEFORM4 ) where each vertex INLINEFORM5 corresponds to a word in INLINEFORM6 (may not be unique) and there exists an edge INLINEFORM7 between vertices INLINEFORM8 and INLINEFORM9 if R performs at least one saccade between the words corresponding to INLINEFORM10 and INLINEFORM11  
Figure FIGREF15 shows an example of a saliency graph A saliency graph may be weighted, but not necessarily connected, for a given text (as there may be words in the given text with no fixation on them)  The “complex” gaze features derived from saliency graphs are also motivated by the theory of incongruity  For instance, Edge Density of a saliency graph increases with the number of distinct saccades, which could arise from the complexity caused by presence of sarcasm  Similarly, the highest weighted degree of a graph is expected to be higher, if the node corresponds to a phrase, incongruous to some other phrase in the text 
The Sarcasm Classifier
We interpret sarcasm detection as a binary classification problem  The training data constitutes 994 examples created using our eye-movement database for sarcasm detection  To check the effectiveness of our feature set, we observe the performance of multiple classification techniques on our dataset through a stratified 10-fold cross validation  We also compare the classification accuracy of our system and the best available systems proposed by riloff2013sarcasm and joshi2015harnessing on our dataset  Using Weka BIBREF18 and LibSVM BIBREF19 APIs, we implement the following classifiers:
Results
Table TABREF17 shows the classification results considering various feature combinations for different classifiers and other systems  These are:
Unigram (with principal components of unigram feature vectors),
Sarcasm (the feature-set reported by joshi2015harnessing subsuming unigram features and features from other reported systems)
Gaze (the simple and complex cognitive features we introduce, along with readability and word count features), and
Gaze+Sarcasm (the complete set of features) 
For all regular classifiers, the gaze features are averaged across participants and augmented with linguistic and sarcasm related features  For the MILR classifier, the gaze features derived from each participant are augmented with linguistic features and thus, a multi instance “bag” of features is formed for each sentence in the training data  This multi-instance dataset is given to an MILR classifier, which follows the standard multi instance assumption to derive class-labels for each bag 
For all the classifiers, our feature combination outperforms the baselines (considering only unigram features) as well as BIBREF3 , with the MILR classifier getting an F-score improvement of 3 7% and Kappa difference of 0 08  We also achieve an improvement of 2% over the baseline, using SVM classifier, when we employ our feature set  We also observe that the gaze features alone, also capture the differences between sarcasm and non-sarcasm classes with a high-precision but a low recall 
To see if the improvement obtained is statistically significant over the state-of-the art system with textual sarcasm features alone, we perform McNemar test  The output of the SVM classifier using only linguistic features used for sarcasm detection by joshi2015harnessing and the output of the MILR classifier with the complete set of features are compared, setting threshold INLINEFORM0   There was a significant difference in the classifier's accuracy with p(two-tailed) = 0 02 with an odds-ratio of 1 43, showing that the classification accuracy improvement is unlikely to be observed by chance in 95% confidence interval 
Considering Reading Time as a Cognitive Feature along with Sarcasm Features
One may argue that, considering simple measures of reading effort like “reading time” as cognitive feature instead of the expensive eye-tracking features for sarcasm detection may be a cost-effective solution  To examine this, we repeated our experiments with “reading time” considered as the only cognitive feature, augmented with the textual features  The F-scores of all the classifiers turn out to be close to that of the classifiers considering sarcasm feature alone and the difference in the improvement is not statistically significant ( INLINEFORM0 )  One the other hand, F-scores with gaze features are superior to the F-scores when reading time is considered as a cognitive feature 
How Effective are the Cognitive Features
We examine the effectiveness of cognitive features on the classification accuracy by varying the input training data size  To examine this, we create a stratified (keeping the class ratio constant) random train-test split of 80%:20%  We train our classifier with 100%, 90%, 80% and 70% of the training data with our whole feature set, and the feature combination from joshi2015harnessing  The goodness of our system is demonstrated by improvements in F-score and Kappa statistics, shown in Figure FIGREF22  
We further analyze the importance of features by ranking the features based on (a) Chi squared test, and (b) Information Gain test, using Weka's attribute selection module  Figure FIGREF23 shows the top 20 ranked features produced by both the tests  For both the cases, we observe 16 out of top 20 features to be gaze features  Further, in each of the cases, Average Fixation Duration per Word and Largest Regression Position are seen to be the two most significant features 
Example Cases
Table TABREF21 shows a few example cases from the experiment with stratified 80%-20% train-test split 
Example sentence 1 is sarcastic, and requires extra-linguistic knowledge (about poor living conditions at Manchester)  Hence, the sarcasm detector relying only on textual features is unable to detect the underlying incongruity  However, our system predicts the label successfully, possibly helped by the gaze features 
Similarly, for sentence 2, the false sense of presence of incongruity (due to phrases like “Helped me” and “Can't stop”) affects the system with only linguistic features  Our system, though, performs well in this case also 
Sentence 3 presents a false-negative case where it was hard for even humans to get the sarcasm  This is why our gaze features (and subsequently the complete set of features) account for erroneous prediction 
In sentence 4, gaze features alone false-indicate presence of incongruity, whereas the system predicts correctly when gaze and linguistic features are taken together 
From these examples, it can be inferred that, only gaze features would not have sufficed to rule out the possibility of detecting other forms of incongruity that do not result in sarcasm 
Error Analysis
Errors committed by our system arise from multiple factors, starting from limitations of the eye-tracker hardware to errors committed by linguistic tools and resources  Also, aggregating various eye-tracking parameters to extract the cognitive features may have caused information loss in the regular classification setting 
Conclusion
In the current work, we created a novel framework to detect sarcasm, that derives insights from human cognition, that manifests over eye movement patterns  We hypothesized that distinctive eye-movement patterns, associated with reading sarcastic text, enables improved detection of sarcasm  We augmented traditional linguistic features with cognitive features obtained from readers' eye-movement data in the form of simple gaze-based features and complex features derived from a graph structure  This extended feature-set improved the success rate of the sarcasm detector by 3 7%, over the best available system  Using cognitive features in an NLP Processing system like ours is the first proposal of its kind 
Our general approach may be useful in other NLP sub-areas like sentiment and emotion analysis, text summarization and question answering, where considering textual clues alone does not prove to be sufficient  We propose to augment this work in future by exploring deeper graph and gaze features  We also propose to develop models for the purpose of learning complex gaze feature representation, that accounts for the power of individual eye movement patterns along with the aggregated patterns of eye movements 
Acknowledgments
We thank the members of CFILT Lab, especially Jaya Jha and Meghna Singh, and the students of IIT Bombay for their help and support 
Context 3: Introduction
Performance appraisal (PA) is an important HR process, particularly for modern organizations that crucially depend on the skills and expertise of their workforce  The PA process enables an organization to periodically measure and evaluate every employee's performance  It also provides a mechanism to link the goals established by the organization to its each employee's day-to-day activities and performance  Design and analysis of PA processes is a lively area of research within the HR community BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3  
The PA process in any modern organization is nowadays implemented and tracked through an IT system (the PA system) that records the interactions that happen in various steps  Availability of this data in a computer-readable database opens up opportunities to analyze it using automated statistical, data-mining and text-mining techniques, to generate novel and actionable insights / patterns and to help in improving the quality and effectiveness of the PA process BIBREF4 , BIBREF5 , BIBREF6   Automated analysis of large-scale PA data is now facilitated by technological and algorithmic advances, and is becoming essential for large organizations containing thousands of geographically distributed employees handling a wide variety of roles and tasks 
A typical PA process involves purposeful multi-step multi-modal communication between employees, their supervisors and their peers  In most PA processes, the communication includes the following steps: (i) in self-appraisal, an employee records his/her achievements, activities, tasks handled etc ; (ii) in supervisor assessment, the supervisor provides the criticism, evaluation and suggestions for improvement of performance etc ; and (iii) in peer feedback (aka INLINEFORM0 view), the peers of the employee provide their feedback  There are several business questions that managers are interested in  Examples:
In this paper, we develop text mining techniques that can automatically produce answers to these questions  Since the intended users are HR executives, ideally, the techniques should work with minimum training data and experimentation with parameter setting  These techniques have been implemented and are being used in a PA system in a large multi-national IT company 
The rest of the paper is organized as follows  Section SECREF2 summarizes related work  Section SECREF3 summarizes the PA dataset used in this paper  Section SECREF4 applies sentence classification algorithms to automatically discover three important classes of sentences in the PA corpus viz , sentences that discuss strengths, weaknesses of employees and contain suggestions for improving her performance  Section SECREF5 considers the problem of mapping the actual targets mentioned in strengths, weaknesses and suggestions to a fixed set of attributes  In Section SECREF6 , we discuss how the feedback from peers for a particular employee can be summarized  In Section SECREF7 we draw conclusions and identify some further work 
Related Work
We first review some work related to sentence classification  Semantically classifying sentences (based on the sentence's purpose) is a much harder task, and is gaining increasing attention from linguists and NLP researchers  McKnight and Srinivasan BIBREF7 and Yamamoto and Takagi BIBREF8 used SVM to classify sentences in biomedical abstracts into classes such as INTRODUCTION, BACKGROUND, PURPOSE, METHOD, RESULT, CONCLUSION  Cohen et al  BIBREF9 applied SVM and other techniques to learn classifiers for sentences in emails into classes, which are speech acts defined by a verb-noun pair, with verbs such as request, propose, amend, commit, deliver and nouns such as meeting, document, committee; see also BIBREF10   Khoo et al  BIBREF11 uses various classifiers to classify sentences in emails into classes such as APOLOGY, INSTRUCTION, QUESTION, REQUEST, SALUTATION, STATEMENT, SUGGESTION, THANKING etc  Qadir and Riloff BIBREF12 proposes several filters and classifiers to classify sentences on message boards (community QA systems) into 4 speech acts: COMMISSIVE (speaker commits to a future action), DIRECTIVE (speaker expects listener to take some action), EXPRESSIVE (speaker expresses his or her psychological state to the listener), REPRESENTATIVE (represents the speaker's belief of something)  Hachey and Grover BIBREF13 used SVM and maximum entropy classifiers to classify sentences in legal documents into classes such as FACT, PROCEEDINGS, BACKGROUND, FRAMING, DISPOSAL; see also BIBREF14   Deshpande et al  BIBREF15 proposes unsupervised linguistic patterns to classify sentences into classes SUGGESTION, COMPLAINT 
There is much work on a closely related problem viz , classifying sentences in dialogues through dialogue-specific categories called dialogue acts BIBREF16 , which we will not review here  Just as one example, Cotterill BIBREF17 classifies questions in emails into the dialogue acts of YES_NO_QUESTION, WH_QUESTION, ACTION_REQUEST, RHETORICAL, MULTIPLE_CHOICE etc 
We could not find much work related to mining of performance appraisals data  Pawar et al  BIBREF18 uses kernel-based classification to classify sentences in both performance appraisal text and product reviews into classes SUGGESTION, APPRECIATION, COMPLAINT  Apte et al  BIBREF6 provides two algorithms for matching the descriptions of goals or tasks assigned to employees to a standard template of model goals  One algorithm is based on the co-training framework and uses goal descriptions and self-appraisal comments as two separate perspectives  The second approach uses semantic similarity under a weak supervision framework  Ramrakhiyani et al  BIBREF5 proposes label propagation algorithms to discover aspects in supervisor assessments in performance appraisals, where an aspect is modelled as a verb-noun pair (e g  conduct training, improve coding) 
Dataset
In this paper, we used the supervisor assessment and peer feedback text produced during the performance appraisal of 4528 employees in a large multi-national IT company  The corpus of supervisor assessment has 26972 sentences  The summary statistics about the number of words in a sentence is: min:4 max:217 average:15 5 STDEV:9 2 Q1:9 Q2:14 Q3:19 
Sentence Classification
The PA corpus contains several classes of sentences that are of interest  In this paper, we focus on three important classes of sentences viz , sentences that discuss strengths (class STRENGTH), weaknesses of employees (class WEAKNESS) and suggestions for improving her performance (class SUGGESTION)  The strengths or weaknesses are mostly about the performance in work carried out, but sometimes they can be about the working style or other personal qualities  The classes WEAKNESS and SUGGESTION are somewhat overlapping; e g , a suggestion may address a perceived weakness

ground_truth: average recipe-level coherence scores of 1.78-1.82, human evaluators preferred personalized model outputs to baseline 63% of the time

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What kind of stylistic features are obtained?
Context 3:  Following are two example sentences in each class 
STRENGTH:
WEAKNESS:
SUGGESTION:
Several linguistic aspects of these classes of sentences are apparent  The subject is implicit in many sentences  The strengths are often mentioned as either noun phrases (NP) with positive adjectives (Excellent technology leadership) or positive nouns (engineering strength) or through verbs with positive polarity (dedicated) or as verb phrases containing positive adjectives (delivers innovative solutions)  Similarly for weaknesses, where negation is more frequently used (presentations are not his forte), or alternatively, the polarities of verbs (avoid) or adjectives (poor) tend to be negative  However, sometimes the form of both the strengths and weaknesses is the same, typically a stand-alone sentiment-neutral NP, making it difficult to distinguish between them; e g , adherence to timing or timely closure  Suggestions often have an imperative mood and contain secondary verbs such as need to, should, has to  Suggestions are sometimes expressed using comparatives (better process compliance)  We built a simple set of patterns for each of the 3 classes on the POS-tagged form of the sentences  We use each set of these patterns as an unsupervised sentence classifier for that class  If a particular sentence matched with patterns for multiple classes, then we have simple tie-breaking rules for picking the final class  The pattern for the STRENGTH class looks for the presence of positive words / phrases like takes ownership, excellent, hard working, commitment, etc  Similarly, the pattern for the WEAKNESS class looks for the presence of negative words / phrases like lacking, diffident, slow learner, less focused, etc  The SUGGESTION pattern not only looks for keywords like should, needs to but also for POS based pattern like “a verb in the base form (VB) in the beginning of a sentence” 
We randomly selected 2000 sentences from the supervisor assessment corpus and manually tagged them (dataset D1)  This labelled dataset contained 705, 103, 822 and 370 sentences having the class labels STRENGTH, WEAKNESS, SUGGESTION or OTHER respectively  We trained several multi-class classifiers on this dataset  Table TABREF10 shows the results of 5-fold cross-validation experiments on dataset D1  For the first 5 classifiers, we used their implementation from the SciKit Learn library in Python (scikit-learn org)  The features used for these classifiers were simply the sentence words along with their frequencies  For the last 2 classifiers (in Table TABREF10 ), we used our own implementation  The overall accuracy for a classifier is defined as INLINEFORM0 , where the denominator is 2000 for dataset D1  Note that the pattern-based approach is unsupervised i e , it did not use any training data  Hence, the results shown for it are for the entire dataset and not based on cross-validation 
Comparison with Sentiment Analyzer
We also explored whether a sentiment analyzer can be used as a baseline for identifying the class labels STRENGTH and WEAKNESS  We used an implementation of sentiment analyzer from TextBlob to get a polarity score for each sentence  Table TABREF13 shows the distribution of positive, negative and neutral sentiments across the 3 class labels STRENGTH, WEAKNESS and SUGGESTION  It can be observed that distribution of positive and negative sentiments is almost similar in STRENGTH as well as SUGGESTION sentences, hence we can conclude that the information about sentiments is not much useful for our classification problem 
Discovering Clusters within Sentence Classes
After identifying sentences in each class, we can now answer question (1) in Section SECREF1   From 12742 sentences predicted to have label STRENGTH, we extract nouns that indicate the actual strength, and cluster them using a simple clustering algorithm which uses the cosine similarity between word embeddings of these nouns  We repeat this for the 9160 sentences with predicted label WEAKNESS or SUGGESTION as a single class  Tables TABREF15 and TABREF16 show a few representative clusters in strengths and in weaknesses, respectively  We also explored clustering 12742 STRENGTH sentences directly using CLUTO BIBREF19 and Carrot2 Lingo BIBREF20 clustering algorithms  Carrot2 Lingo discovered 167 clusters and also assigned labels to these clusters  We then generated 167 clusters using CLUTO as well  CLUTO does not generate cluster labels automatically, hence we used 5 most frequent words within the cluster as its labels  Table TABREF19 shows the largest 5 clusters by both the algorithms  It was observed that the clusters created by CLUTO were more meaningful and informative as compared to those by Carrot2 Lingo  Also, it was observed that there is some correspondence between noun clusters and sentence clusters  E g  the nouns cluster motivation expertise knowledge talent skill (Table TABREF15 ) corresponds to the CLUTO sentence cluster skill customer management knowledge team (Table TABREF19 )  But overall, users found the nouns clusters to be more meaningful than the sentence clusters 
PA along Attributes
In many organizations, PA is done from a predefined set of perspectives, which we call attributes  Each attribute covers one specific aspect of the work done by the employees  This has the advantage that we can easily compare the performance of any two employees (or groups of employees) along any given attribute  We can correlate various performance attributes and find dependencies among them  We can also cluster employees in the workforce using their supervisor ratings for each attribute to discover interesting insights into the workforce  The HR managers in the organization considered in this paper have defined 15 attributes (Table TABREF20 )  Each attribute is essentially a work item or work category described at an abstract level  For example, FUNCTIONAL_EXCELLENCE covers any tasks, goals or activities related to the software engineering life-cycle (e g , requirements analysis, design, coding, testing etc ) as well as technologies such as databases, web services and GUI 
In the example in Section SECREF4 , the first sentence (which has class STRENGTH) can be mapped to two attributes: FUNCTIONAL_EXCELLENCE and BUILDING_EFFECTIVE_TEAMS

ground_truth: Unanswerable

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What summarization algorithms did the authors experiment with?
Context 3:  Similarly, the third sentence (which has class WEAKNESS) can be mapped to the attribute INTERPERSONAL_EFFECTIVENESS and so forth  Thus, in order to answer the second question in Section SECREF1 , we need to map each sentence in each of the 3 classes to zero, one, two or more attributes, which is a multi-class multi-label classification problem 
We manually tagged the same 2000 sentences in Dataset D1 with attributes, where each sentence may get 0, 1, 2, etc  up to 15 class labels (this is dataset D2)  This labelled dataset contained 749, 206, 289, 207, 91, 223, 191, 144, 103, 80, 82, 42, 29, 15, 24 sentences having the class labels listed in Table TABREF20 in the same order  The number of sentences having 0, 1, 2, or more than 2 attributes are: 321, 1070, 470 and 139 respectively  We trained several multi-class multi-label classifiers on this dataset  Table TABREF21 shows the results of 5-fold cross-validation experiments on dataset D2 
Precision, Recall and F-measure for this multi-label classification are computed using a strategy similar to the one described in BIBREF21   Let INLINEFORM0 be the set of predicted labels and INLINEFORM1 be the set of actual labels for the INLINEFORM2 instance  Precision and recall for this instance are computed as follows: INLINEFORM3
It can be observed that INLINEFORM0 would be undefined if INLINEFORM1 is empty and similarly INLINEFORM2 would be undefined when INLINEFORM3 is empty  Hence, overall precision and recall are computed by averaging over all the instances except where they are undefined  Instance-level F-measure can not be computed for instances where either precision or recall are undefined  Therefore, overall F-measure is computed using the overall precision and recall 
Summarization of Peer Feedback using ILP
The PA system includes a set of peer feedback comments for each employee  To answer the third question in Section SECREF1 , we need to create a summary of all the peer feedback comments about a given employee  As an example, following are the feedback comments from 5 peers of an employee 
The individual sentences in the comments written by each peer are first identified and then POS tags are assigned to each sentence  We hypothesize that a good summary of these multiple comments can be constructed by identifying a set of important text fragments or phrases  Initially, a set of candidate phrases is extracted from these comments and a subset of these candidate phrases is chosen as the final summary, using Integer Linear Programming (ILP)  The details of the ILP formulation are shown in Table TABREF36   As an example, following is the summary generated for the above 5 peer comments 
humble nature, effective communication, technical expertise, always supportive, vast knowledge
Following rules are used to identify candidate phrases:
Various parameters are used to evaluate a candidate phrase for its importance  A candidate phrase is more important:
A complete list of parameters is described in detail in Table TABREF36  
There is a trivial constraint INLINEFORM0 which makes sure that only INLINEFORM1 out of INLINEFORM2 candidate phrases are chosen  A suitable value of INLINEFORM3 is used for each employee depending on number of candidate phrases identified across all peers (see Algorithm SECREF6 )  Another set of constraints ( INLINEFORM4 to INLINEFORM5 ) make sure that at least one phrase is selected for each of the leadership attributes  The constraint INLINEFORM6 makes sure that multiple phrases sharing the same headword are not chosen at a time  Also, single word candidate phrases are chosen only if they are adjectives or nouns with lexical category noun attribute  This is imposed by the constraint INLINEFORM7   It is important to note that all the constraints except INLINEFORM8 are soft constraints, i e  there may be feasible solutions which do not satisfy some of these constraints  But each constraint which is not satisfied, results in a penalty through the use of slack variables  These constraints are described in detail in Table TABREF36  
The objective function maximizes the total importance score of the selected candidate phrases  At the same time, it also minimizes the sum of all slack variables so that the minimum number of constraints are broken 
INLINEFORM0 : No  of candidate phrases INLINEFORM1 : No  of phrases to select as part of summary
INLINEFORM0 INLINEFORM1 INLINEFORM2 INLINEFORM3 INLINEFORM4 INLINEFORM5 INLINEFORM6 INLINEFORM7 INLINEFORM8
INLINEFORM0 and INLINEFORM1 INLINEFORM2 INLINEFORM3 INLINEFORM4 INLINEFORM5 INLINEFORM6
INLINEFORM0 (For determining number of phrases to select to include in summary)
Evaluation of auto-generated summaries
We considered a dataset of 100 employees, where for each employee multiple peer comments were recorded  Also, for each employee, a manual summary was generated by an HR personnel  The summaries generated by our ILP-based approach were compared with the corresponding manual summaries using the ROUGE BIBREF22 unigram score  For comparing performance of our ILP-based summarization algorithm, we explored a few summarization algorithms provided by the Sumy package  A common parameter which is required by all these algorithms is number of sentences keep in the final summary  ILP-based summarization requires a similar parameter K, which is automatically decided based on number of total candidate phrases  Assuming a sentence is equivalent to roughly 3 phrases, for Sumy algorithms, we set number of sentences parameter to the ceiling of K/3  Table TABREF51 shows average and standard deviation of ROUGE unigram f1 scores for each algorithm, over the 100 summaries  The performance of ILP-based summarization is comparable with the other algorithms, as the two sample t-test does not show statistically significant difference  Also, human evaluators preferred phrase-based summary generated by our approach to the other sentence-based summaries 
Conclusions and Further Work
In this paper, we presented an analysis of the text generated in Performance Appraisal (PA) process in a large multi-national IT company  We performed sentence classification to identify strengths, weaknesses and suggestions for improvements found in the supervisor assessments and then used clustering to discover broad categories among them  As this is non-topical classification, we found that SVM with ADWS kernel BIBREF18 produced the best results  We also used multi-class multi-label classification techniques to match supervisor assessments to predefined broad perspectives on performance  Logistic Regression classifier was observed to produce the best results for this topical classification  Finally, we proposed an ILP-based summarization technique to produce a summary of peer feedback comments for a given employee and compared it with manual summaries 
The PA process also generates much structured data, such as supervisor ratings  It is an interesting problem to compare and combine the insights from discovered from structured data and unstructured text  Also, we are planning to automatically discover any additional performance attributes to the list of 15 attributes currently used by HR 

ground_truth: LSA, TextRank, LexRank and ILP-based summary.

LSA, TextRank, LexRank