prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Was PolyReponse evaluated against some baseline?
Context 1: Introduction and Background
Task-oriented dialogue systems are primarily designed to search and interact with large databases which contain information pertaining to a certain dialogue domain: the main purpose of such systems is to assist the users in accomplishing a well-defined task such as flight booking BIBREF0, tourist information BIBREF1, restaurant search BIBREF2, or booking a taxi BIBREF3  These systems are typically constructed around rigid task-specific ontologies BIBREF1, BIBREF4 which enumerate the constraints the users can express using a collection of slots (e g , price range for restaurant search) and their slot values (e g , cheap, expensive for the aforementioned slots)  Conversations are then modelled as a sequence of actions that constrain slots to particular values  This explicit semantic space is manually engineered by the system designer  It serves as the output of the natural language understanding component as well as the input to the language generation component both in traditional modular systems BIBREF5, BIBREF6 and in more recent end-to-end task-oriented dialogue systems BIBREF7, BIBREF8, BIBREF9, BIBREF3 
Working with such explicit semantics for task-oriented dialogue systems poses several critical challenges on top of the manual time-consuming domain ontology design  First, it is difficult to collect domain-specific data labelled with explicit semantic representations  As a consequence, despite recent data collection efforts to enable training of task-oriented systems across multiple domains BIBREF0, BIBREF3, annotated datasets are still few and far between, as well as limited in size and the number of domains covered  Second, the current approach constrains the types of dialogue the system can support, resulting in artificial conversations, and breakdowns when the user does not understand what the system can and cannot support  In other words, training a task-based dialogue system for voice-controlled search in a new domain always implies the complex, expensive, and time-consuming process of collecting and annotating sufficient amounts of in-domain dialogue data 
In this paper, we present a demo system based on an alternative approach to task-oriented dialogue  Relying on non-generative response retrieval we describe the PolyResponse conversational search engine and its application in the task of restaurant search and booking  The engine is trained on hundreds of millions of real conversations from a general domain (i e , Reddit), using an implicit representation of semantics that directly optimizes the task at hand  It learns what responses are appropriate in different conversational contexts, and consequently ranks a large pool of responses according to their relevance to the current user utterance and previous dialogue history (i e , dialogue context) 
The technical aspects of the underlying conversational search engine are explained in detail in our recent work BIBREF11, while the details concerning the Reddit training data are also available in another recent publication BIBREF12  In this demo, we put focus on the actual practical usefulness of the search engine by demonstrating its potential in the task of restaurant search, and extending it to deal with multi-modal data  We describe a PolyReponse system that assists the users in finding a relevant restaurant according to their preference, and then additionally helps them to make a booking in the selected restaurant  Due to its retrieval-based design, with the PolyResponse engine there is no need to engineer a structured ontology, or to solve the difficult task of general language generation  This design also bypasses the construction of dedicated decision-making policy modules  The large ranking model already encapsulates a lot of knowledge about natural language and conversational flow 
Since retrieved system responses are presented visually to the user, the PolyResponse restaurant search engine is able to combine text responses with relevant visual information (e g , photos from social media associated with the current restaurant and related to the user utterance), effectively yielding a multi-modal response  This setup of using voice as input, and responding visually is becoming more and more prevalent with the rise of smart screens like Echo Show and even mixed reality  Finally, the PolyResponse restaurant search engine is multilingual: it is currently deployed in 8 languages enabling search over restaurants in 8 cities around the world  System snapshots in four different languages are presented in Figure FIGREF16, while screencast videos that illustrate the dialogue flow with the PolyResponse engine are available at: https://tinyurl com/y3evkcfz 
PolyResponse: Conversational Search
The PolyResponse system is powered by a single large conversational search engine, trained on a large amount of conversational and image data, as shown in Figure FIGREF2  In simple words, it is a ranking model that learns to score conversational replies and images in a given conversational context  The highest-scoring responses are then retrieved as system outputs  The system computes two sets of similarity scores: 1) $S(r,c)$ is the score of a candidate reply $r$ given a conversational context $c$, and 2) $S(p,c)$ is the score of a candidate photo $p$ given a conversational context $c$  These scores are computed as a scaled cosine similarity of a vector that represents the context ($h_c$), and a vector that represents the candidate response: a text reply ($h_r$) or a photo ($h_p$)  For instance, $S(r,c)$ is computed as $S(r,c)=C cos(h_r,h_c)$, where $C$ is a learned constant  The part of the model dealing with text input (i e , obtaining the encodings $h_c$ and $h_r$) follows the architecture introduced recently by Henderson:2019acl  We provide only a brief recap here; see the original paper for further details 
PolyResponse: Conversational Search ::: Text Representation 
The model, implemented as a deep neural network, learns to respond by training on hundreds of millions context-reply $(c,r)$ pairs  First, similar to Henderson:2017arxiv, raw text from both $c$ and $r$ is converted to unigrams and bigrams  All input text is first lower-cased and tokenised, numbers with 5 or more digits get their digits replaced by a wildcard symbol #, while words longer than 16 characters are replaced by a wildcard token LONGWORD  Sentence boundary tokens are added to each sentence  The vocabulary consists of the unigrams that occur at least 10 times in a random 10M subset of the Reddit training set (see Figure FIGREF2) plus the 200K most frequent bigrams in the same random subset 
During training, we obtain $d$-dimensional feature representations ($d=320$) shared between contexts and replies for each unigram and bigram jointly with other neural net parameters  A state-of-the-art architecture based on transformers BIBREF13 is then applied to unigram and bigram vectors separately, which are then averaged to form the final 320-dimensional encoding  That encoding is then passed through three fully-connected non-linear hidden layers of dimensionality $1,024$  The final layer is linear and maps the text into the final $l$-dimensional ($l=512$) representation: $h_c$ and $h_r$  Other standard and more sophisticated encoder models can also be used to provide final encodings $h_c$ and $h_r$, but the current architecture shows a good trade-off between speed and efficacy with strong and robust performance in our empirical evaluations on the response retrieval task using Reddit BIBREF14, OpenSubtitles BIBREF15, and AmazonQA BIBREF16 conversational test data, see BIBREF12 for further details 
In training the constant $C$ is constrained to lie between 0 and $\sqrt{l}$  Following Henderson:2017arxiv, the scoring function in the training objective aims to maximise the similarity score of context-reply pairs that go together, while minimising the score of random pairings: negative examples  Training proceeds via SGD with batches comprising 500 pairs (1 positive and 499 negatives) 
PolyResponse: Conversational Search ::: Photo Representation 
Photos are represented using convolutional neural net (CNN) models pretrained on ImageNet BIBREF17  We use a MobileNet model with a depth multiplier of 1 4, and an input dimension of $224 \times 224$ pixels as in BIBREF18  This provides a $1,280 \times 1 4 = 1,792$-dimensional representation of a photo, which is then passed through a single hidden layer of dimensionality $1,024$ with ReLU activation, before being passed to a hidden layer of dimensionality 512 with no activation to provide the final representation $h_p$ 
PolyResponse: Conversational Search ::: Data Source 1: Reddit 
For training text representations we use a Reddit dataset similar to AlRfou:2016arxiv  Our dataset is large and provides natural conversational structure: all Reddit data from January 2015 to December 2018, available as a public BigQuery dataset, span almost 3 7B comments BIBREF12  We preprocess the dataset to remove uninformative and long comments by retaining only sentences containing more than 8 and less than 128 word tokens  After pairing all comments/contexts $c$ with their replies $r$, we obtain more than 727M context-reply $(c,r)$ pairs for training, see Figure FIGREF2 
PolyResponse: Conversational Search ::: Data Source 2: Yelp 
Once the text encoding sub-networks are trained, a photo encoder is learned on top of a pretrained MobileNet CNN, using data taken from the Yelp Open dataset: it contains around 200K photos and their captions  Training of the multi-modal sub-network then maximises the similarity of captions encoded with the response encoder $h_r$ to the photo representation $h_p$  As a result, we can compute the score of a photo given a context using the cosine similarity of the respective vectors  A photo will be scored highly if it looks like its caption would be a good response to the current context 
PolyResponse: Conversational Search ::: Index of Responses 
The Yelp dataset is used at inference time to provide text and photo candidates to display to the user at each step in the conversation  Our restaurant search is currently deployed separately for each city, and we limit the responses to a given city  For instance, for our English system for Edinburgh we work with 396 restaurants, 4,225 photos (these include additional photos obtained using the Google Places API without captions), 6,725 responses created from the structured information about restaurants that Yelp provides, converted using simple templates to sentences of the form such as “Restaurant X accepts credit cards ”, 125,830 sentences extracted from online reviews 
PolyResponse: Conversational Search ::: PolyResponse in a Nutshell 
The system jointly trains two encoding functions (with shared word embeddings) $f(context)$ and $g(reply)$ which produce encodings $h_c$ and $h_r$, so that the similarity $S(c,r)$ is high for all $(c,r)$ pairs from the Reddit training data and low for random pairs  The encoding function $g()$ is then frozen, and an encoding function $t(photo)$ is learnt which makes the similarity between a photo and its associated caption high for all (photo, caption) pairs from the Yelp dataset, and low for random pairs  $t$ is a CNN pretrained on ImageNet, with a shallow one-layer DNN on top  Given a new context/query, we then provide its encoding $h_c$ by applying $f()$, and find plausible text replies and photo responses according to functions $g()$ and $t()$, respectively  These should be responses that look like answers to the query, and photos that look like they would have captions that would be answers to the provided query 
At inference, finding relevant candidates given a context reduces to computing $h_c$ for the context $c$ , and finding nearby $h_r$ and $h_p$ vectors  The response vectors can all be pre-computed, and the nearest neighbour search can be further optimised using standard libraries such as Faiss BIBREF19 or approximate nearest neighbour retrieval BIBREF20, giving an efficient search that scales to billions of candidate responses 
The system offers both voice and text input and output  Speech-to-text and text-to-speech conversion in the PolyResponse system is currently supported by the off-the-shelf Google Cloud tools 
Dialogue Flow
The ranking model lends itself to the one-shot task of finding the most relevant responses in a given context  However, a restaurant-browsing system needs to support a dialogue flow where the user finds a restaurant, and then asks questions about it  The dialogue state for each search scenario is represented as the set of restaurants that are considered relevant  This starts off as all the restaurants in the given city, and is assumed to monotonically decrease in size as the conversation progresses until the user converges to a single restaurant  A restaurant is only considered valid in the context of a new user input if it has relevant responses corresponding to it  This flow is summarised here:
S1  Initialise $R$ as the set of all restaurants in the city  Given the user's input, rank all the responses in the response pool pertaining to restaurants in $R$ 
S2  Retrieve the top $N$ responses $r_1, r_2, \ldots , r_N$ with corresponding (sorted) cosine similarity scores: $s_1 \ge s_2 \ge \ldots \ge s_N$ 
S3  Compute probability scores $p_i \propto \exp (a \cdot s_i)$ with $\sum _{i=1}^N p_i$, where $a>0$ is a tunable constant 
S4  Compute a score $q_e$ for each restaurant/entity $e \in R$, $q_e = \sum _{i: r_i \in e} p_i$ 
S5  Update $R$ to the smallest set of restaurants with highest $q$ whose $q$-values sum up to more than a predefined threshold $t$ 
S6  Display the most relevant responses associated with the updated $R$, and return to S2 
If there are multiple relevant restaurants, one response is shown from each  When only one restaurant is relevant, the top $N$ responses are all shown, and relevant photos are also displayed  The system does not require dedicated understanding, decision-making, and generation modules, and this dialogue flow does not rely on explicit task-tailored semantics  The set of relevant restaurants is kept internally while the system narrows it down across multiple dialogue turns  A simple set of predefined rules is used to provide a templatic spoken system response: e g , an example rule is “One review of $e$ said $r$”, where $e$ refers to the restaurant, and $r$ to a relevant response associated with $e$  Note that while the demo is currently focused on the restaurant search task, the described “narrowing down” dialogue flow is generic and applicable to a variety of applications dealing with similar entity search 
The system can use a set of intent classifiers to allow resetting the dialogue state, or to activate the separate restaurant booking dialogue flow  These classifiers are briefly discussed in §SECREF4 
Other Functionality ::: Multilinguality 
The PolyResponse restaurant search is currently available in 8 languages and for 8 cities around the world: English (Edinburgh), German (Berlin), Spanish (Madrid), Mandarin (Taipei), Polish (Warsaw), Russian (Moscow), Korean (Seoul), and Serbian (Belgrade)  Selected snapshots are shown in Figure FIGREF16, while we also provide videos demonstrating the use and behaviour of the systems at: https://tinyurl com/y3evkcfz  A simple MT-based translate-to-source approach at inference time is currently used to enable the deployment of the system in other languages: 1) the pool of responses in each language is translated to English by Google Translate beforehand, and pre-computed encodings of their English translations are used as representations of each foreign language response; 2) a provided user utterance (i e , context) is translated to English on-the-fly and its encoding $h_c$ is then learned  We plan to experiment with more sophisticated multilingual models in future work 
Other Functionality ::: Voice-Controlled Menu Search 
An additional functionality enables the user to get parts of the restaurant menu relevant to the current user utterance as responses  This is achieved by performing an additional ranking step of available menu items and retrieving the ones that are semantically relevant to the user utterance using exactly the same methodology as with ranking other responses  An example of this functionality is shown in Figure FIGREF21 
Other Functionality ::: Resetting and Switching to Booking 
The restaurant search system needs to support the discrete actions of restarting the conversation (i e , resetting the set $R$), and should enable transferring to the slot-based table booking flow  This is achieved using two binary intent classifiers, that are run at each step in the dialogue  These classifiers make use of the already-computed $h_c$ vector that represents the user's latest text  A single-layer neural net is learned on top of the 512-dimensional encoding, with a ReLU activation and 100 hidden nodes  To train the classifiers, sets of 20 relevant paraphrases (e g , “Start again”) are provided as positive examples  Finally, when the system successfully switches to the booking scenario, it proceeds to the slot filling task: it aims to extract all the relevant booking information from the user (e g , date, time, number of people to dine)  The entire flow of the system illustrating both the search phase and the booking phase is provided as the supplemental video material 
Conclusion and Future Work
This paper has presented a general approach to search-based dialogue that does not rely on explicit semantic representations such as dialogue acts or slot-value ontologies, and allows for multi-modal responses  In future work, we will extend the current demo system to more tasks and languages, and work with more sophisticated encoders and ranking functions  Besides the initial dialogue flow from this work (§SECREF3), we will also work with more complex flows dealing, e g , with user intent shifts 
Context 2: Introduction
Privacy policies are the documents which disclose the ways in which a company gathers, uses, shares and manages a user's data  As legal documents, they function using the principle of notice and choice BIBREF0, where companies post their policies, and theoretically, users read the policies and decide to use a company's products or services only if they find the conditions outlined in its privacy policy acceptable  Many legal jurisdictions around the world accept this framework, including the United States and the European Union BIBREF1, BIBREF2  However, the legitimacy of this framework depends upon users actually reading and understanding privacy policies to determine whether company practices are acceptable to them BIBREF3  In practice this is seldom the case BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10  This is further complicated by the highly individual and nuanced compromises that users are willing to make with their data BIBREF11, discouraging a `one-size-fits-all' approach to notice of data practices in privacy documents 
With devices constantly monitoring our environment, including our personal space and our bodies, lack of awareness of how our data is being used easily leads to problematic situations where users are outraged by information misuse, but companies insist that users have consented  The discovery of increasingly egregious uses of data by companies, such as the scandals involving Facebook and Cambridge Analytica BIBREF12, have further brought public attention to the privacy concerns of the internet and ubiquitous computing  This makes privacy a well-motivated application domain for NLP researchers, where advances in enabling users to quickly identify the privacy issues most salient to them can potentially have large real-world impact 
[1]https://play google com/store/apps/details?id=com gotokeep keep intl [2]https://play google com/store/apps/details?id=com viber voip [3]A question might not have any supporting evidence for an answer within the privacy policy 
Motivated by this need, we contribute PrivacyQA, a corpus consisting of 1750 questions about the contents of privacy policies, paired with over 3500 expert annotations  The goal of this effort is to kickstart the development of question-answering methods for this domain, to address the (unrealistic) expectation that a large population should be reading many policies per day  In doing so, we identify several understudied challenges to our ability to answer these questions, with broad implications for systems seeking to serve users' information-seeking intent  By releasing this resource, we hope to provide an impetus to develop systems capable of language understanding in this increasingly important domain 
Related Work
Prior work has aimed to make privacy policies easier to understand  Prescriptive approaches towards communicating privacy information BIBREF21, BIBREF22, BIBREF23 have not been widely adopted by industry  Recently, there have been significant research effort devoted to understanding privacy policies by leveraging NLP techniques BIBREF24, BIBREF25, BIBREF26, BIBREF27, BIBREF28, especially by identifying specific data practices within a privacy policy  We adopt a personalized approach to understanding privacy policies, that allows users to query a document and selectively explore content salient to them  Most similar is the PolisisQA corpus BIBREF29, which examines questions users ask corporations on Twitter  Our approach differs in several ways: 1) The PrivacyQA dataset is larger, containing 10x as many questions and answers  2) Answers are formulated by domain experts with legal training  3) PrivacyQA includes diverse question types, including unanswerable and subjective questions 
Our work is also related to reading comprehension in the open domain, which is frequently based upon Wikipedia passages BIBREF16, BIBREF17, BIBREF15, BIBREF30 and news articles BIBREF20, BIBREF31, BIBREF32  Table TABREF4 presents the desirable attributes our dataset shares with past approaches  This work is also tied into research in applying NLP approaches to legal documents BIBREF33, BIBREF34, BIBREF35, BIBREF36, BIBREF37, BIBREF38, BIBREF39  While privacy policies have legal implications, their intended audience consists of the general public rather than individuals with legal expertise  This arrangement is problematic because the entities that write privacy policies often have different goals than the audience  feng2015applying, tan-EtAl:2016:P16-1 examine question answering in the insurance domain, another specialized domain similar to privacy, where the intended audience is the general public 
Data Collection
We describe the data collection methodology used to construct PrivacyQA  With the goal of achieving broad coverage across application types, we collect privacy policies from 35 mobile applications representing a number of different categories in the Google Play Store  One of our goals is to include both policies from well-known applications, which are likely to have carefully-constructed privacy policies, and lesser-known applications with smaller install bases, whose policies might be considerably less sophisticated  Thus, setting 5 million installs as a threshold, we ensure each category includes applications with installs on both sides of this threshold  All policies included in the corpus are in English, and were collected before April 1, 2018, predating many companies' GDPR-focused BIBREF41 updates  We leave it to future studies BIBREF42 to look at the impact of the GDPR (e g , to what extent GDPR requirements contribute to making it possible to provide users with more informative answers, and to what extent their disclosures continue to omit issues that matter to users) 
Data Collection ::: Crowdsourced Question Elicitation
The intended audience for privacy policies consists of the general public  This informs the decision to elicit questions from crowdworkers on the contents of privacy policies  We choose not to show the contents of privacy policies to crowdworkers, a procedure motivated by a desire to avoid inadvertent biases BIBREF43, BIBREF44, BIBREF45, BIBREF46, BIBREF47, and encourage crowdworkers to ask a variety of questions beyond only asking questions based on practices described in the document 
Instead, crowdworkers are presented with public information about a mobile application available on the Google Play Store including its name, description and navigable screenshots  Figure FIGREF9 shows an example of our user interface  Crowdworkers are asked to imagine they have access to a trusted third-party privacy assistant, to whom they can ask any privacy question about a given mobile application  We use the Amazon Mechanical Turk platform and recruit crowdworkers who have been conferred “master” status and are located within the United States of America  Turkers are asked to provide five questions per mobile application, and are paid $2 per assignment, taking ~eight minutes to complete the task 
Data Collection ::: Answer Selection
To identify legally sound answers, we recruit seven experts with legal training to construct answers to Turker questions  Experts identify relevant evidence within the privacy policy, as well as provide meta-annotation on the question's relevance, subjectivity, OPP-115 category BIBREF49, and how likely any privacy policy is to contain the answer to the question asked 
Data Collection ::: Analysis
Table TABREF17 presents aggregate statistics of the PrivacyQA dataset  1750 questions are posed to our imaginary privacy assistant over 35 mobile applications and their associated privacy documents  As an initial step, we formulate the problem of answering user questions as an extractive sentence selection task, ignoring for now background knowledge, statistical data and legal expertise that could otherwise be brought to bear  The dataset is partitioned into a training set featuring 27 mobile applications and 1350 questions, and a test set consisting of 400 questions over 8 policy documents  This ensures that documents in training and test splits are mutually exclusive  Every question is answered by at least one expert  In addition, in order to estimate annotation reliability and provide for better evaluation, every question in the test set is answered by at least two additional experts 
Table TABREF14 describes the distribution over first words of questions posed by crowdworkers  We also observe low redundancy in the questions posed by crowdworkers over each policy, with each policy receiving ~49 94 unique questions despite crowdworkers independently posing questions  Questions are on average 8 4 words long  As declining to answer a question can be a legally sound response but is seldom practically useful, answers to questions where a minority of experts abstain to answer are filtered from the dataset  Privacy policies are ~3000 words long on average  The answers to the question asked by the users typically have ~100 words of evidence in the privacy policy document 
Data Collection ::: Analysis ::: Categories of Questions
Questions are organized under nine categories from the OPP-115 Corpus annotation scheme BIBREF49:
First Party Collection/Use: What, why and how information is collected by the service provider
Third Party Sharing/Collection: What, why and how information shared with or collected by third parties
Data Security: Protection measures for user information
Data Retention: How long user information will be stored
User Choice/Control: Control options available to users
User Access, Edit and Deletion: If/how users can access, edit or delete information
Policy Change: Informing users if policy information has been changed
International and Specific Audiences: Practices pertaining to a specific group of users
Other: General text, contact information or practices not covered by other categories 
For each question, domain experts indicate one or more relevant OPP-115 categories  We mark a category as relevant to a question if it is identified as such by at least two annotators  If no such category exists, the category is marked as `Other' if atleast one annotator has identified the `Other' category to be relevant  If neither of these conditions is satisfied, we label the question as having no agreement  The distribution of questions in the corpus across OPP-115 categories is as shown in Table TABREF16  First party and third party related questions are the largest categories, forming nearly 66 4% of all questions asked to the privacy assistant 
Data Collection ::: Analysis ::: Answer Validation
When do experts disagree? We would like to analyze the reasons for potential disagreement on the annotation task, to ensure disagreements arise due to valid differences in opinion rather than lack of adequate specification in annotation guidelines  It is important to note that the annotators are experts rather than crowdworkers  Accordingly, their judgements can be considered valid, legally-informed opinions even when their perspectives differ  For the sake of this question we randomly sample 100 instances in the test data and analyze them for likely reasons for disagreements  We consider a disagreement to have occurred when more than one expert does not agree with the majority consensus  By disagreement we mean there is no overlap between the text identified as relevant by one expert and another 
We find that the annotators agree on the answer for 74% of the questions, even if the supporting evidence they identify is not identical i e full overlap  They disagree on the remaining 26%  Sources of apparent disagreement correspond to situations when different experts: have differing interpretations of question intent (11%) (for example, when a user asks 'who can contact me through the app', the questions admits multiple interpretations, including seeking information about the features of the app, asking about first party collection/use of data or asking about third party collection/use of data), identify different sources of evidence for questions that ask if a practice is performed or not (4%), have differing interpretations of policy content (3%), identify a partial answer to a question in the privacy policy (2%) (for example, when the user asks `who is allowed to use the app' a majority of our annotators decline to answer, but the remaining annotators highlight partial evidence in the privacy policy which states that children under the age of 13 are not allowed to use the app), and other legitimate sources of disagreement (6%) which include personal subjective views of the annotators (for example, when the user asks `is my DNA information used in any way other than what is specified', some experts consider the boilerplate text of the privacy policy which states that it abides to practices described in the policy document as sufficient evidence to answer this question, whereas others do not) 
Experimental Setup
We evaluate the ability of machine learning methods to identify relevant evidence for questions in the privacy domain  We establish baselines for the subtask of deciding on the answerability (§SECREF33) of a question, as well as the overall task of identifying evidence for questions from policies (§SECREF37)  We describe aspects of the question that can render it unanswerable within the privacy domain (§SECREF41) 
Experimental Setup ::: Answerability Identification Baselines
We define answerability identification as a binary classification task, evaluating model ability to predict if a question can be answered, given a question in isolation  This can serve as a prior for downstream question-answering  We describe three baselines on the answerability task, and find they considerably improve performance over a majority-class baseline 
SVM: We define 3 sets of features to characterize each question  The first is a simple bag-of-words set of features over the question (SVM-BOW), the second is bag-of-words features of the question as well as length of the question in words (SVM-BOW + LEN), and lastly we extract bag-of-words features, length of the question in words as well as part-of-speech tags for the question (SVM-BOW + LEN + POS)  This results in vectors of 200, 201 and 228 dimensions respectively, which are provided to an SVM with a linear kernel 
CNN: We utilize a CNN neural encoder for answerability prediction  We use GloVe word embeddings BIBREF50, and a filter size of 5 with 64 filters to encode questions 
BERT: BERT BIBREF51 is a bidirectional transformer-based language-model BIBREF52  We fine-tune BERT-base on our binary answerability identification task with a learning rate of 2e-5 for 3 epochs, with a maximum sequence length of 128
Context 3: Introduction
The irony is a kind of figurative language, which is widely used on social media BIBREF0   The irony is defined as a clash between the intended meaning of a sentence and its literal meaning BIBREF1   As an important aspect of language, irony plays an essential role in sentiment analysis BIBREF2 , BIBREF0 and opinion mining BIBREF3 , BIBREF4  
Although some previous studies focus on irony detection, little attention is paid to irony generation  As ironies can strengthen sentiments and express stronger emotions, we mainly focus on generating ironic sentences  Given a non-ironic sentence, we implement a neural network to transfer it to an ironic sentence and constrain the sentiment polarity of the two sentences to be the same  For example, the input is “I hate it when my plans get ruined" which is negative in sentiment polarity and the output should be ironic and negative in sentiment as well, such as “I like it when my plans get ruined"  The speaker uses “like" to be ironic and express his or her negative sentiment  At the same time, our model can preserve contents which are irrelevant to sentiment polarity and irony  According to the categories mentioned in BIBREF5 , irony can be classified into 3 classes: verbal irony by means of a polarity contrast, the sentences containing expression whose polarity is inverted between the intended and the literal evaluation; other types of verbal irony, the sentences that show no polarity contrast between the literal and intended meaning but are still ironic; and situational irony, the sentences that describe situations that fail to meet some expectations  As ironies in the latter two categories are obscure and hard to understand, we decide to only focus on ironies in the first category in this work  For example, our work can be specifically described as: given a sentence “I hate to be ignored", we train our model to generate an ironic sentence such as “I love to be ignored"  Although there is “love" in the generated sentence, the speaker still expresses his or her negative sentiment by irony  We also make some explorations in the transformation from ironic sentences to non-ironic sentences at the end of our work  Because of the lack of previous work and baselines on irony generation, we implement our model based on style transfer  Our work will not only provide the first large-scale irony dataset but also make our model as a benchmark for the irony generation 
Recently, unsupervised style transfer becomes a very popular topic  Many state-of-the-art studies try to solve the task with sequence-to-sequence (seq2seq) framework  There are three main ways to build up models  The first is to learn a latent style-independent content representation and generate sentences with the content representation and another style BIBREF6 , BIBREF7   The second is to directly transfer sentences from one style to another under the control of classifiers and reinforcement learning BIBREF8   The third is to remove style attribute words from the input sentence and combine the remaining content with new style attribute words BIBREF9 , BIBREF10   The first method usually obtains better performances via adversarial training with discriminators  The style-independent content representation, nevertheless, is not easily obtained BIBREF11 , which results in poor performances  The second method is suitable for complex styles which are difficult to model and describe  The model can learn the deep semantic features by itself but sometimes the model is sensitive to parameters and hard to train  The third method succeeds to preserve content but cannot work for some complex styles such as democratic and republican  Sentences with those styles usually do not have specific style attribute words  Unfortunately, due to the lack of large irony dataset and difficulties of modeling ironies, there has been little work trying to generate ironies based on seq2seq framework as far as we know  Inspired by methods for style transfer, we decide to implement a specifically designed model based on unsupervised style transfer to explore irony generation 
In this paper, in order to address the lack of irony data, we first crawl over 2M tweets from twitter to build a dataset with 262,755 ironic and 112,330 non-ironic tweets  Then, due to the lack of parallel data, we propose a novel model to transfer non-ironic sentences to ironic sentences in an unsupervised way  As ironic style is hard to model and describe, we implement our model with the control of classifiers and reinforcement learning  Different from other studies in style transfer, the transformation from non-ironic to ironic sentences has to preserve sentiment polarity as mentioned above  Therefore, we not only design an irony reward to control the irony accuracy and implement denoising auto-encoder and back-translation to control content preservation but also design a sentiment reward to control sentiment preservation 
Experimental results demonstrate that our model achieves a high irony accuracy with well-preserved sentiment and content  The contributions of our work are as follows:
Related Work
Style Transfer: As irony is a complicated style and hard to model with some specific style attribute words, we mainly focus on studies without editing style attribute words 
Some studies are trying to disentangle style representation from content representation  In BIBREF12 , authors leverage adversarial networks to learn separate content representations and style representations  In BIBREF13 and BIBREF6 , researchers combine variational auto-encoders (VAEs) with style discriminators 
However, some recent studies BIBREF11 reveal that the disentanglement of content and style representations may not be achieved in practice  Therefore, some other research studies BIBREF9 , BIBREF10 strive to separate content and style by removing stylistic words  Nonetheless, many non-ironic sentences do not have specific stylistic words and as a result, we find it difficult to transfer non-ironic sentences to ironic sentences through this way in practice 
Besides, some other research studies do not disentangle style from content but directly learn representations of sentences  In BIBREF8 , authors propose a dual reinforcement learning framework without separating content and style representations  In BIBREF7 , researchers utilize a machine translation model to learn a sentence representation preserving the meaning of the sentence but reducing stylistic properties  In this method, the quality of generated sentences relies on the performance of classifiers to a large extent  Meanwhile, such models are usually sensitive to parameters and difficult to train  In contrast, we combine a pre-training process with reinforcement learning to build up a stable language model and design special rewards for our task 
Irony Detection: With the development of social media, irony detection becomes a more important task  Methods for irony detection can be mainly divided into two categories: methods based on feature engineering and methods based on neural networks 
As for methods based on feature engineering, In BIBREF1 , authors investigate pragmatic phenomena and various irony markers  In BIBREF14 , researchers leverage a combination of sentiment, distributional semantic and text surface features  Those models rely on hand-crafted features and are hard to implement 
When it comes to methods based on neural networks, long short-term memory (LSTM) BIBREF15 network is widely used and is very efficient for irony detection  In BIBREF16 , a tweet is divided into two segments and a subtract layer is implemented to calculate the difference between two segments in order to determine whether the tweet is ironic  In BIBREF17 , authors utilize a recurrent neural network with Bi-LSTM and self-attention without hand-crafted features  In BIBREF18 , researchers propose a system based on a densely connected LSTM network 
Our Dataset
In this section, we describe how we build our dataset with tweets  First, we crawl over 2M tweets from twitter using GetOldTweets-python  We crawl English tweets from 04/09/2012 to /12/18/2018  We first remove all re-tweets and use langdetect to remove all non-English sentences  Then, we remove hashtags attached at the end of the tweets because they are usually not parts of sentences and will confuse our language model  After that, we utilize Ekphrasis to process tweets  We remove URLs and restore remaining hashtags, elongated words, repeated words, and all-capitalized words  To simplify our dataset, We replace all “ INLINEFORM0 money INLINEFORM1 " and “ INLINEFORM2 time INLINEFORM3 " tokens with “ INLINEFORM4 number INLINEFORM5 " token when using Ekphrasis  And we delete sentences whose lengths are less than 10 or greater than 40  In order to restore abbreviations, we download an abbreviation dictionary from webopedia and restore abbreviations to normal words or phrases according to the dictionary  Finally, we remove sentences which have more than two rare words (appearing less than three times) in order to constrain the size of vocabulary  Finally, we get 662,530 sentences after pre-processing 
As neural networks are proved effective in irony detection, we decide to implement a neural classifier in order to classify the sentences into ironic and non-ironic sentences  However, the only high-quality irony dataset we can obtain is the dataset of Semeval-2018 Task 3 and the dataset is pretty small, which will cause overfitting to complex models  Therefore, we just implement a simple one-layer RNN with LSTM cell to classify pre-processed sentences into ironic sentences and non-ironic sentences because LSTM networks are widely used in irony detection  We train the model with the dataset of Semeval-2018 Task 3  After classification, we get 262,755 ironic sentences and 399,775 non-ironic sentences  According to our observation, not all non-ironic sentences are suitable to be transferred into ironic sentences  For example, “just hanging out   watching   is it monday yet" is hard to transfer because it does not have an explicit sentiment polarity  So we remove all interrogative sentences from the non-ironic sentences and only obtain the sentences which have words expressing strong sentiments  We evaluate the sentiment polarity of each word with TextBlob and we view those words with sentiment scores greater than 0 5 or less than -0 5 as words expressing strong sentiments  Finally, we build our irony dataset with 262,755 ironic sentences and 102,330 non-ironic sentences 
[t] Irony Generation Algorithm
INLINEFORM0 pre-train with auto-encoder Pre-train INLINEFORM1 , INLINEFORM2 with INLINEFORM3 using MLE based on Eq  EQREF16 Pre-train INLINEFORM4 , INLINEFORM5 with INLINEFORM6 using MLE based on Eq  EQREF17 INLINEFORM7 pre-train with back-translation Pre-train INLINEFORM8 , INLINEFORM9 , INLINEFORM10 , INLINEFORM11 with INLINEFORM12 using MLE based on Eq  EQREF19 Pre-train INLINEFORM13 , INLINEFORM14 , INLINEFORM15 , INLINEFORM16 with INLINEFORM17 using MLE based on Eq  EQREF20
INLINEFORM0 train with RL each epoch e = 1, 2,    , INLINEFORM1 INLINEFORM2 train non-irony2irony with RL INLINEFORM3 in N INLINEFORM4 update INLINEFORM5 , INLINEFORM6 , using INLINEFORM7 based on Eq  EQREF29 INLINEFORM8 back-translation INLINEFORM9 INLINEFORM10 INLINEFORM11 update INLINEFORM12 , INLINEFORM13 , INLINEFORM14 , INLINEFORM15 using MLE based on Eq  EQREF19 INLINEFORM16 train irony2non-irony with RL INLINEFORM17 in I INLINEFORM18 update INLINEFORM19 , INLINEFORM20 , using INLINEFORM21 similar to Eq  EQREF29 INLINEFORM22 back-translation INLINEFORM23 INLINEFORM24 INLINEFORM25 update INLINEFORM26 , INLINEFORM27 , INLINEFORM28 , INLINEFORM29 using MLE based on Eq  EQREF20
Our Method
Given two non-parallel corpora: non-ironic corpus N={ INLINEFORM0 , INLINEFORM1 ,    , INLINEFORM2 } and ironic corpus I={ INLINEFORM3 , INLINEFORM4 ,    , INLINEFORM5 }, the goal of our irony generation model is to generate an ironic sentence from a non-ironic sentence while preserving the content and sentiment polarity of the source input sentence  We implement an encoder-decoder framework where two encoders are utilized to encode ironic sentences and non-ironic sentences respectively and two decoders are utilized to decode ironic sentences and non-ironic sentences from latent representations respectively  In order to enforce a shared latent space, we share two layers on both the encoder side and the decoder side  Our model architecture is illustrated in Figure FIGREF13   We denote irony encoder as INLINEFORM6 , irony decoder as INLINEFORM7 and non-irony encoder as INLINEFORM8 , non-irony decoder as INLINEFORM9   Their parameters are INLINEFORM10 , INLINEFORM11 , INLINEFORM12 and INLINEFORM13  
Our irony generation algorithm is shown in Algorithm SECREF3   We first pre-train our model using denoising auto-encoder and back-translation to build up language models for both styles (section SECREF14 )  Then we implement reinforcement learning to train the model to transfer sentences from one style to another (section SECREF21 )  Meanwhile, to achieve content preservation, we utilize back-translation for one time in every INLINEFORM0 time steps 
Pretraining
In order to build up our language model and preserve the content, we apply the auto-encoder model  To prevent the model from simply copying the input sentence, we randomly add some noises in the input sentence  Specifically, for every word in the input sentence, there is 10% chance that we delete it, 10 % chance that we duplicate it, 10% chance that we swap it with the next word, or it remains unchanged  We first encode the input sentence INLINEFORM0 or INLINEFORM1 with respective encoder INLINEFORM2 or INLINEFORM3 to obtain its latent representation INLINEFORM4 or INLINEFORM5 and reconstruct the input sentence with the latent representation and respective decoder  So we can get the reconstruction loss for auto-encoder INLINEFORM6 : DISPLAYFORM0 DISPLAYFORM1
In addition to denoising auto-encoder, we implement back-translation BIBREF19 to generate a pseudo-parallel corpus  Suppose our model takes non-ironic sentence INLINEFORM0 as input  We first encode INLINEFORM1 with INLINEFORM2 to obtain its latent representation INLINEFORM3 and decode the latent representation with INLINEFORM4 to get a transferred sentence INLINEFORM5   Then we encode INLINEFORM6 with INLINEFORM7 and decode its latent representation with INLINEFORM8 to reconstruct the original input sentence INLINEFORM9   Therefore, our reconstruction loss for back-translation INLINEFORM10 : DISPLAYFORM0
And if our model takes ironic sentence INLINEFORM0 as input, we can get the reconstruction loss for back-translation as: DISPLAYFORM0
Reinforcement Learning
Since the gold transferred result of input is unavailable, we cannot evaluate the quality of the generated sentence directly  Therefore, we implement reinforcement learning and elaborately design two rewards to describe the irony accuracy and sentiment preservation, respectively 
A pre-trained binary irony classifier based on CNN BIBREF20 is used to evaluate how ironic a sentence is  We denote the parameter of the classifier as INLINEFORM0 and it is fixed during the training process 
In order to facilitate the transformation, we design the irony reward as the difference between the irony score of the input sentence and that of the output sentence  Formally, when we input a non-ironic sentence INLINEFORM0 and transfer it to an ironic sentence INLINEFORM1 , our irony reward is defined as: DISPLAYFORM0
where INLINEFORM0 denotes ironic style and INLINEFORM1 is the probability of that a sentence INLINEFORM2 is ironic 
To preserve the sentiment polarity of the input sentence, we also need to use classifiers to evaluate the sentiment polarity of the sentences  However, the sentiment analysis of ironic sentences and non-ironic sentences are different  In the case of figurative languages such as irony, sarcasm or metaphor, the sentiment polarity of the literal meaning may differ significantly from that of the intended figurative meaning BIBREF0   As we aim to train our model to transfer sentences from non-ironic to ironic, using only one classifier is not enough  As a result, we implement two pre-trained sentiment classifiers for non-ironic sentences and ironic sentences respectively  We denote the parameter of the sentiment classifier for ironic sentences as INLINEFORM0 and that of the sentiment classifier for non-ironic sentences as INLINEFORM1  
A challenge, when we implement two classifiers to evaluate the sentiment polarity, is that the two classifiers trained with different datasets may have different distributions of scores  That means we cannot directly calculate the sentiment reward with scores applied by two classifiers  To alleviate this problem and standardize the prediction results of two classifiers, we set a threshold for each classifier and subtract the respective threshold from scores applied by the classifier to obtain the comparative sentiment polarity score  We get the optimal threshold by maximizing the ability of the classifier according to the distribution of our training data 
We denote the threshold of ironic sentiment classifier as INLINEFORM0 and the threshold of non-ironic sentiment classifier as INLINEFORM1   The standardized sentiment score is defined as INLINEFORM2 and INLINEFORM3 where INLINEFORM4 denotes the positive sentiment polarity and INLINEFORM5 is the probability of that a sentence is positive in sentiment polarity 
As mentioned above, the input sentence and the generated sentence should express the same sentiment  For example, if we input a non-ironic sentence “I hate to be ignored" which is negative in sentiment polarity, the generated ironic sentence should be also negative, such as “I love to be ignored"  To achieve sentiment preservation, we design the sentiment reward as that one minus the absolute value of the difference between the standardized sentiment score of the input sentence and that of the generated sentence  Formally, when we input a non-ironic sentence INLINEFORM0 and transfer it to an ironic sentence INLINEFORM1 , our sentiment reward is defined as: DISPLAYFORM0
To encourage our model to focus on both the irony accuracy and the sentiment preservation, we apply the harmonic mean of irony reward and sentiment reward: DISPLAYFORM0
Policy Gradient
The policy gradient algorithm BIBREF21 is a simple but widely-used algorithm in reinforcement learning  It is used to maximize the expected reward INLINEFORM0   The objective function to minimize is defined as: DISPLAYFORM0
where INLINEFORM0 , INLINEFORM1 is the reward of INLINEFORM2 and INLINEFORM3 is the input size 
Training Details
INLINEFORM0 , INLINEFORM1 , INLINEFORM2 and INLINEFORM3 in our model are Transformers BIBREF22 with 4 layers and 2 shared layers  The word embeddings of 128 dimensions are learned during the training process  Our maximum sentence length is set as 40  The optimizer is Adam BIBREF23 and the learning rate is INLINEFORM4   The batch size is 32 and harmonic weight INLINEFORM5 in Eq 9 is 0 5  We set the interval INLINEFORM6 as 200  The model is pre-trained for 6 epochs and trained for 15 epochs for reinforcement learning 
Irony Classifier: We implement a CNN classifier trained with our irony dataset  All the CNN classifiers we utilize in this paper use the same parameters as BIBREF20  
Sentiment Classifier for Irony: We first implement a one-layer LSTM network to classify ironic sentences in our dataset into positive and negative ironies  The LSTM network is trained with the dataset of Semeval 2015 Task 11 BIBREF0 which is used for the sentiment analysis of figurative language in twitter  Then, we use the positive ironies and negative ironies to train the CNN sentiment classifier for irony 
Sentiment Classifier for Non-irony: Similar to the training process of the sentiment classifier for irony, we first implement a one-layer LSTM network trained with the dataset for the sentiment analysis of common twitters to classify the non-ironies into positive and negative non-ironies  Then we use the positive and negative non-ironies to train the sentiment classifier for non-irony 
Baselines
We compare our model with the following state-of-art generative models:
BackTrans BIBREF7 : In BIBREF7 , authors propose a model using machine translation in order to preserve the meaning of the sentence while reducing stylistic properties 
Unpaired BIBREF10 : In BIBREF10 , researchers implement a method to remove emotional words and add desired sentiment controlled by reinforcement learning 
CrossAlign BIBREF6 : In BIBREF6 , authors leverage refined alignment of latent representations to perform style transfer and a cross-aligned auto-encoder is implemented 
CPTG BIBREF24 : An interpolated reconstruction loss is introduced in BIBREF24 and a discriminator is implemented to control attributes in this work 
DualRL BIBREF8 : In BIBREF8 , researchers use two reinforcement rewards simultaneously to control style accuracy and content preservation 
Evaluation Metrics
In order to evaluate sentiment preservation, we use the absolute value of the difference between the standardized sentiment score of the input sentence and that of the generated sentence  We call the value as sentiment delta (senti delta)  Besides, we report the sentiment accuracy (Senti ACC) which measures whether the output sentence has the same sentiment polarity as the input sentence based on our standardized sentiment classifiers  The BLEU score BIBREF25 between the input sentences and the output sentences is calculated to evaluate the content preservation performance  In order to evaluate the overall performance of different models, we also report the geometric mean (G2) and harmonic mean (H2) of the sentiment accuracy and the BLEU score  As for the irony accuracy, we only report it in human evaluation results because it is more accurate for the human to evaluate the quality of irony as it is very complicated 
We first sample 50 non-ironic input sentences and their corresponding output sentences of different models  Then, we ask four annotators who are proficient in English to evaluate the qualities of the generated sentences of different models  They are required to rank the output sentences of our model and baselines from the best to the worst in terms of irony accuracy (Irony), Sentiment preservation (Senti) and content preservation (Content)  The best output is ranked with 1 and the worst output is ranked with 6  That means that the smaller our human evaluation value is, the better the corresponding model is 
Results and Discussions
Table TABREF35 shows the automatic evaluation results of the models in the transformation from non-ironic sentences to ironic sentences  From the results, our model obtains the best result in sentiment delta  The DualRL model achieves the highest result in other metrics, but most of its outputs are the almost same as the input sentences  So it is reasonable that DualRL system outperforms ours in these metrics but it actually does not transfer the non-ironic sentences to ironic sentences at all  From this perspective, we cannot view DualRL as an effective model for irony generation  In contrast, our model gets results close to those of DualRL and obtains a balance between irony accuracy, sentiment preservation, and content preservation if we also consider the irony accuracy discussed below 
And from human evaluation results shown in Table TABREF36 , our model gets the best average rank in irony accuracy  And as mentioned above, the DualRL model usually does not change the input sentence and outputs the same sentence  Therefore, it is reasonable that it obtains the best rank in sentiment and content preservation and ours is the second  However, it still demonstrates that our model, instead of changing nothing, transfers the style of the input sentence with content and sentiment preservation at the same time 
Case Study
In the section, we present some example outputs of different models  Table TABREF37 shows the results of the transformation from non-ironic sentences to ironic sentences  We can observe that: (1) The BackTrans system, the Unpaired system, the CrossAlign system and the CPTG system tends to generate sentences which are towards irony but do not preserve content  (2) The DualRL system preserves content and sentiment very well but even does not change the input sentence  (3) Our model considers both aspects and achieves a better balance among irony accuracy, sentiment and content preservation 
Error Analysis
Although our model outperforms other style transfer baselines according to automatic and human evaluation results, there are still some failure cases because irony generation is still a very challenging task  We would like to share the issues we meet during our experiments and our solutions to some of them in this section 
No Change: As mentioned above, many style transfer models, such as DualRL, tend to make few changes to the input sentence and output the same sentence  Actually, this is a common issue for unsupervised style transfer systems and we also meet it during our experiments  The main reason for the issue is that rewards for content preservation are too prominent and rewards for style accuracy cannot work well  In contrast, in order to guarantee the readability and fluency of the output sentence, we also cannot emphasize too much on rewards for style accuracy because it may cause some other issues such as word repetition mentioned below  A method to solve the problem is tuning hyperparameters and this is also the method we implement in this work  As for content preservation, maybe MLE methods such as back-translation are not enough because they tend to force models to generate specific words  In the future, we should further design some more suitable methods to control content preservation for models without disentangling style and content representations, such as DualRL and ours 
Word Repetition: During our experiments, we observe that some of the outputs prefer to repeat the same word as shown in Table TABREF38   This is because reinforcement learning rewards encourage the model to generate words which can get high scores from classifiers and even back-translation cannot stop it  Our solution is that we can lower the probability of decoding a word in decoders if the word has been generated in the previous time steps during testing  We also try to implement this method during training time but obtain worse performances because it may limit the effects of training  Some previous studies utilize language models to control the fluency of the output sentence and we also try this method  Nonetheless, pre-training a language model with tweets and using it to generate rewards is difficult because tweets are more casual and have more noise  Rewards from that kind of language model are usually not accurate and may confuse the model  In the future, we should come up with better methods to model language fluency with the consideration of irony accuracy, sentiment and content preservation, especially for tweets 
Improper Words: As ironic style is hard for our model to learn, it may generate some improper words which make the sentence strange  As the example shown in the Table TABREF38 , the sentiment word in the input sentence is “wonderful" and the model should change it into a negative word such as “sad" to make the output sentence ironic  However, the model changes “friday" and “fifa" which are not related to ironic styles  We have not found a very effective method to address this issue and maybe we should further explore stronger models to learn ironic styles better 
Additional Experiments
In this section, we describe some additional experiments on the transformation from ironic sentences to non-ironic sentences  Sometimes ironies are hard to understand and may cause misunderstanding, for which our task also explores the transformation from ironic sentences to non-ironic sentences 
As shown in Table TABREF46 , we also conduct automatic evaluations and the conclusions are similar to those of the transformation from non-ironic sentences to ironic sentences  As for human evaluation results in Table TABREF47 , our model still can achieve the second-best results in sentiment and content preservation  Nevertheless, DualRL system and ours get poor performances in irony accuracy  The reason may be that the other four baselines tend to generate common and even not fluent sentences which are irrelevant to the input sentences and are hard to be identified as ironies  So annotators usually mark these output sentences as non-ironic sentences, which causes these models to obtain better performances than DualRL and ours but much poorer results in sentiment and content preservation  Some examples are shown in Table TABREF52  
Conclusion and Future Work
In this paper, we first systematically define irony generation based on style transfer  Because of the lack of irony data, we make use of twitter and build a large-scale dataset  In order to control irony accuracy, sentiment preservation and content preservation at the same time, we also design a combination of rewards for reinforcement learning and incorporate reinforcement learning with a pre-training process  Experimental results demonstrate that our model outperforms other generative models and our rewards are effective  Although our model design is effective, there are still many errors and we systematically analyze them  In the future, we are interested in exploring these directions and our work may extend to other kinds of ironies which are more difficult to model 

ground_truth: No

No

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Were other baselines tested to compare with the neural baseline?
Context 2: 
Experimental Setup ::: Privacy Question Answering
Our goal is to identify evidence within a privacy policy for questions asked by a user  This is framed as an answer sentence selection task, where models identify a set of evidence sentences from all candidate sentences in each policy 
Experimental Setup ::: Privacy Question Answering ::: Evaluation Metric
Our evaluation metric for answer-sentence selection is sentence-level F1, implemented similar to BIBREF30, BIBREF16  Precision and recall are implemented by measuring the overlap between predicted sentences and sets of gold-reference sentences  We report the average of the maximum F1 from each n$-$1 subset, in relation to the heldout reference 
Experimental Setup ::: Privacy Question Answering ::: Baselines
We describe baselines on this task, including a human performance baseline 
No-Answer Baseline (NA) : Most of the questions we receive are difficult to answer in a legally-sound way on the basis of information present in the privacy policy  We establish a simple baseline to quantify the effect of identifying every question as unanswerable 
Word Count Baseline : To quantify the effect of using simple lexical matching to answer the questions, we retrieve the top candidate policy sentences for each question using a word count baseline BIBREF53, which counts the number of question words that also appear in a sentence  We include the top 2, 3 and 5 candidates as baselines 
BERT: We implement two BERT-based baselines BIBREF51 for evidence identification  First, we train BERT on each query-policy sentence pair as a binary classification task to identify if the sentence is evidence for the question or not (Bert)  We also experiment with a two-stage classifier, where we separately train the model on questions only to predict answerability  At inference time, if the answerable classifier predicts the question is answerable, the evidence identification classifier produces a set of candidate sentences (Bert + Unanswerable) 
Human Performance: We pick each reference answer provided by an annotator, and compute the F1 with respect to the remaining references, as described in section 4 2 1  Each reference answer is treated as the prediction, and the remaining n-1 answers are treated as the gold reference  The average of the maximum F1 across all reference answers is computed as the human baseline 
Results and Discussion
The results of the answerability baselines are presented in Table TABREF31, and on answer sentence selection in Table TABREF32  We observe that bert exhibits the best performance on a binary answerability identification task  However, most baselines considerably exceed the performance of a majority-class baseline  This suggests considerable information in the question, indicating it's possible answerability within this domain 
Table TABREF32 describes the performance of our baselines on the answer sentence selection task  The No-answer (NA) baseline performs at 28 F1, providing a lower bound on performance at this task  We observe that our best-performing baseline, Bert + Unanswerable achieves an F1 of 39 8  This suggest that bert is capable of making some progress towards answering questions in this difficult domain, while still leaving considerable headroom for improvement to reach human performance  Bert + Unanswerable performance suggests that incorporating information about answerability can help in this difficult domain  We examine this challenging phenomena of unanswerability further in Section  
Results and Discussion ::: Error Analysis
Disagreements are analyzed based on the OPP-115 categories of each question (Table TABREF34)  We compare our best performing BERT variant against the NA model and human performance  We observe significant room for improvement across all categories of questions but especially for first party, third party and data retention categories 
We analyze the performance of our strongest BERT variant, to identify classes of errors and directions for future improvement (Table 8)  We observe that a majority of answerability mistakes made by the BERT model are questions which are in fact answerable, but are identified as unanswerable by BERT  We observe that BERT makes 124 such mistakes on the test set  We collect expert judgments on relevance, subjectivity , silence and information about how likely the question is to be answered from the privacy policy from our experts  We find that most of these mistakes are relevant questions  However many of them were identified as subjective by the annotators, and at least one annotator marked 19 of these questions as having no answer within the privacy policy  However, only 6 of these questions were unexpected or do not usually have an answer in privacy policies  These findings suggest that a more nuanced understanding of answerability might help improve model performance in his challenging domain 
Results and Discussion ::: What makes Questions Unanswerable?
We further ask legal experts to identify potential causes of unanswerability of questions  This analysis has considerable implications  While past work BIBREF17 has treated unanswerable questions as homogeneous, a question answering system might wish to have different treatments for different categories of `unanswerable' questions  The following factors were identified to play a role in unanswerability:
Incomprehensibility: If a question is incomprehensible to the extent that its meaning is not intelligible 
Relevance: Is this question in the scope of what could be answered by reading the privacy policy 
Ill-formedness: Is this question ambiguous or vague  An ambiguous statement will typically contain expressions that can refer to multiple potential explanations, whereas a vague statement carries a concept with an unclear or soft definition 
Silence: Other policies answer this type of question but this one does not 
Atypicality: The question is of a nature such that it is unlikely for any policy policy to have an answer to the question 
Our experts attempt to identify the different `unanswerable' factors for all 573 such questions in the corpus  4 18% of the questions were identified as being incomprehensible (for example, `any difficulties to occupy the privacy assistant')  Amongst the comprehendable questions, 50% were identified as likely to have an answer within the privacy policy, 33 1% were identified as being privacy-related questions but not within the scope of a privacy policy (e g , 'has Viber had any privacy breaches in the past?') and 16 9% of questions were identified as completely out-of-scope (e g , `'will the app consume much space?')  In the questions identified as relevant, 32% were ill-formed questions that were phrased by the user in a manner considered vague or ambiguous  Of the questions that were both relevant as well as `well-formed', 95 7% of the questions were not answered by the policy in question but it was reasonable to expect that a privacy policy would contain an answer  The remaining 4 3% were described as reasonable questions, but of a nature generally not discussed in privacy policies  This suggests that the answerability of questions over privacy policies is a complex issue, and future systems should consider each of these factors when serving user's information seeking intent 
We examine a large-scale dataset of “natural” unanswerable questions BIBREF54 based on real user search engine queries to identify if similar unanswerability factors exist  It is important to note that these questions have previously been filtered, according to a criteria for bad questions defined as “(questions that are) ambiguous, incomprehensible, dependent on clear false presuppositions, opinion-seeking, or not clearly a request for factual information ” Annotators made the decision based on the content of the question without viewing the equivalent Wikipedia page  We randomly sample 100 questions from the development set which were identified as unanswerable, and find that 20% of the questions are not questions (e g , “all I want for christmas is you mariah carey tour”)  12% of questions are unlikely to ever contain an answer on Wikipedia, corresponding closely to our atypicality category  3% of questions are unlikely to have an answer anywhere (e g , `what guides Santa home after he has delivered presents?')  7% of questions are incomplete or open-ended (e g , `the south west wind blows across nigeria between')  3% of questions have an unresolvable coreference (e g , `how do i get to Warsaw Missouri from here')  4% of questions are vague, and a further 7% have unknown sources of error  2% still contain false presuppositions (e g , `what is the only fruit that does not have seeds?') and the remaining 42% do not have an answer within the document  This reinforces our belief that though they have been understudied in past work, any question answering system interacting with real users should expect to receive such unanticipated and unanswerable questions 
Conclusion
We present PrivacyQA, the first significant corpus of privacy policy questions and more than 3500 expert annotations of relevant answers  The goal of this work is to promote question-answering research in the specialized privacy domain, where it can have large real-world impact  Strong neural baselines on PrivacyQA achieve a performance of only 39 8 F1 on this corpus, indicating considerable room for future research  Further, we shed light on several important considerations that affect the answerability of questions  We hope this contribution leads to multidisciplinary efforts to precisely understand user intent and reconcile it with information in policy documents, from both the privacy and NLP communities 
Acknowledgements
This research was supported in part by grants from the National Science Foundation Secure and Trustworthy Computing program (CNS-1330596, CNS-1330214, CNS-15-13957, CNS-1801316, CNS-1914486, CNS-1914444) and a DARPA Brandeis grant on Personalized Privacy Assistants (FA8750-15-2-0277)  The US Government is authorized to reproduce and distribute reprints for Governmental purposes not withstanding any copyright notation  The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the NSF, DARPA, or the US Government  The authors would like to extend their gratitude to Elias Wright, Gian Mascioli, Kiara Pillay, Harrison Kay, Eliel Talo, Alexander Fagella and N  Cameron Russell for providing their valuable expertise and insight to this effort  The authors are also grateful to Eduard Hovy, Lorrie Cranor, Florian Schaub, Joel Reidenberg, Aditya Potukuchi and Igor Shalyminov for helpful discussions related to this work, and to the three anonymous reviewers of this draft for their constructive feedback  Finally, the authors would like to thank all crowdworkers who consented to participate in this study 

ground_truth: SVM, No-Answer Baseline (NA) , Word Count Baseline, Human Performance

No-Answer Baseline (NA), Word Count Baseline, Human Performance

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What experiments are conducted?
Context 4: 
Better IE with Difficulty Prediction
We next present experiments in which we attempt to use the predicted difficulty during training to improve models for information extraction of descriptions of Population, Interventions and Outcomes from medical article abstracts  We investigate two uses: (1) simply removing the most difficult sentences from the training set, and, (2) re-weighting the most difficult sentences 
We again use LSTM-CRF-Pattern as the base model and experimenting on the EBM-NLP corpus BIBREF5   This is trained on either (1) the training set with difficult sentences removed, or (2) the full training set but with instances re-weighted in proportion to their predicted difficulty score  Following BIBREF5 , we use the Adam optimizer with learning rate of 0 001, decay 0 9, batch size 20 and dropout 0 5  We use pretrained 200d GloVe vectors BIBREF21 to initialize word embeddings, and use 100d hidden char representations  Each word is thus represented with 300 dimensions in total  The hidden size is 100 for the LSTM in the character representation component, and 200 for the LSTM in the information extraction component  We train for 15 epochs, saving parameters that achieve the best F1 score on a nested development set 
Removing Difficult Examples
We first evaluate changes in performance induced by training the sequence labeling model using less data by removing difficult sentences prior to training  The hypothesis here is that these difficult instances are likely to introduce more noise than signal  We used a cross-fold approach to predict sentence difficulties, training on 9/10ths of the data and scoring the remaining 1/10th at a time  We then sorted sentences by predicted difficulty scores, and experimented with removing increasing numbers of these (in order of difficulty) prior to training the LSTM-CRF-Pattern model 
Figure 3 shows the results achieved by the LSTM-CRF-Pattern model after discarding increasing amounts of the training data: the $x$ and $y$ axes correspond to the the percentage of data removed and F1 scores, respectively  We contrast removing sentences predicted to be difficult with removing them (a) randomly (i i d ), and, (b) in inverse order of predicted inter-annotator agreement  The agreement prediction model is trained exactly the same like difficult prediction model, with simply changing the difficult score to annotation agreement  F1 scores actually improve (marginally) when we remove the most difficult sentences, up until we drop 4% of the data for Population and Interventions, and 6% for Outcomes  Removing training points at i i d  random degrades performance, as expected  Removing sentences in order of disagreement seems to have similar effect as removing them by difficulty score when removing small amount of the data, but the F1 scores drop much faster when removing more data  These findings indicate that sentences predicted to be difficult are indeed noisy, to the extent that they do not seem to provide the model useful signal 
Re-weighting by Difficulty
We showed above that removing a small number of the most difficult sentences does not harm, and in fact modestly improves, medical IE model performance  However, using the available data we are unable to test if this will be useful in practice, as we would need additional data to determine how many difficult sentences should be dropped 
We instead explore an alternative, practical means of exploiting difficulty predictions: we re-weight sentences during training inversely to their predicted difficulty  Formally, we weight sentence $i$ with difficulty scores above $\tau $ according to: $1-a\cdot (d_i-\tau )/(1-\tau )$ , where $d_i$ is the difficulty score for sentence $i$ , and $a$ is a parameter codifying the minimum weight value  We set $\tau $ to 0 8 so as to only re-weight sentences with difficulty in the top 20th percentile, and we set $a$ to 0 5  The re-weighting is equivalent to down-sampling the difficult sentences  LSTM-CRF-Pattern is our base model 
Table 4 reports the precision, recall and F1 achieved both with and without sentence re-weighting  Re-weighting improves all metrics modestly but consistently  All F1 differences are statistically significant under a sign test ( $p<0 01$ )  The model with best precision is different for Patient, Intervention and Outcome labels  However re-weighting by difficulty does consistently yield the best recall for all three extraction types, with the most notable improvement for i and o, where recall improved by 10 percentage points  This performance increase translated to improvements in F1 across all types, as compared to the base model and to re-weighting by agreement 
Involving Expert Annotators
The preceding experiments demonstrate that re-weighting difficult sentences annotated by the crowd generally improves the extraction models  Presumably the performance is influenced by the annotation quality 
We now examine the possibility that the higher quality and more consistent annotations of domain experts on the difficult instances will benefit the extraction model  This simulates an annotation strategy in which we route difficult instances to domain experts and easier ones to crowd annotators  We also contrast the value of difficult data to that of an i i d  random sample of the same size, both annotated by experts 
Expert annotations of Random and Difficult Instances
We re-annotate by experts a subset of most difficult instances and the same number of random instances  As collecting annotations from experts is slow and expensive, we only re-annotate the difficult instances for the interventions extraction task  We re-annotate the abstracts which cover the sentences with predicted difficulty scores in the top 5 percentile  We rank the abstracts from the training set by the count of difficult sentences, and re-annotate the abstracts that contain the most difficult sentences  Constrained by time and budget, we select only 2000 abstracts for re-annotation; 1000 of these are top-ranked, and 1000 are randomly sampled  This re-annotation cost $3,000  We have released the new annotation data at: https://github com/bepnye/EBM-NLP 
Following BIBREF5 , we recruited five medical experts via Up-work with advanced medical training and strong technical reading/writing skills  The expert annotator were asked to read the entire abstract and highlight, using the BRAT toolkit BIBREF26 , all spans describing medical Interventions  Each abstract is only annotated by one expert  We examined 30 re-annotated abstracts to ensure the annotation quality before hiring the annotator 
Table 5 presents the results of LSTM-CRF-Pattern model trained on the reannotated difficult subset and the random subset  The first two rows show the results for models trained with expert annotations  The model trained on random data has a slightly better F1 than that trained on the same amount of difficult data  The model trained on random data has higher precision but lower recall 
Rows 3 and 4 list the results for models trained on the same data but with crowd annotation  Models trained with expert-annotated data are clearly superior to those trained with crowd labels with respect to F1, indicating that the experts produced higher quality annotations  For crowdsourced annotations, training the model with data sampled at i i d  random achieves 2% higher F1 than when difficult instances are used  When expert annotations are used, this difference is less than 1%  This trend in performance may be explained by differences in annotation quality: the randomly sampled set was more consistently annotated by both experts and crowd because the difficult set is harder  However, in both cases expert annotations are better, with a bigger difference between the expert and crowd models on the difficult set 
The last row is the model trained on all 5k abstracts with crowd annotations  Its F1 score is lower than either expert model trained on only 20% of data, suggesting that expert annotations should be collected whenever possible  Again the crowd model on complete data has higher precision than expert models but its recall is much lower 
Routing To Experts or Crowd
So far a system was trained on one type of data, either labeled by crowd or experts  We now examine the performance of a system trained on data that was routed to either experts or crowd annotators depending on their predicted difficult  Given the results presented so far mixing annotators may be beneficial given their respective trade-offs of precision and recall  We use the annotations from experts for an abstract if it exists otherwise use crowd annotations  The results are presented in Table 6  
Rows 1 and 2 repeat the performance of the models trained on difficult subset and random subset with expert annotations only respectively  The third row is the model trained by combining difficult and random subsets with expert annotations  There are around 250 abstracts in the overlap of these two sets, so there are total 1 75k abstracts used for training the D+R model  Rows 4 to 6 are the models trained on all 5k abstracts with mixed annotations, where Other means the rest of the abstracts with crowd annotation only 
The results show adding more training data with crowd annotation still improves at least 1 point F1 score in all three extraction tasks  The improvement when the difficult subset with expert annotations is mixed with the remaining crowd annotation is 3 5 F1 score, much larger than when a random set of expert annotations are added  The model trained with re-annotating the difficult subset (D+Other) also outperforms the model with re-annotating the random subset (R+Other) by 2 points in F1  The model trained with re-annotating both of difficult and random subsets (D+R+Other), however, achieves only marginally higher F1 than the model trained with the re-annotated difficult subset (D+Other)  In sum, the results clearly indicate that mixing expert and crowd annotations leads to better models than using solely crowd data, and better than using expert data alone  More importantly, there is greater gain in performance when instances are routed according to difficulty, as compared to randomly selecting the data for expert annotators  These findings align with our motivating hypothesis that annotation quality for difficult instances is important for final model performance  They also indicate that mixing annotations from expert and crowd could be an effective way to achieve acceptable model performance given a limited budget 
How Many Expert Annotations?
We established that crowd annotation are still useful in supplementing expert annotations for medical IE  Obtaining expert annotations for the one thousand most difficult instances greatly improved the model performance  However the choice of how many difficult instances to annotate was an uninformed choice  Here we check if less expert data would have yielded similar gains  Future work will need to address how best to choose this parameter for a routing system 
We simulate a routing scenario in which we send consecutive batches of the most difficult examples to the experts for annotation  We track changes in performance as we increase the number of most-difficult-articles sent to domain experts  As shown in Figure 4 , adding expert annotations for difficult articles consistently increases F1 scores  The performance gain is mostly from increased recall; the precision changes only a bit with higher quality annotation  This observation implies that crowd workers often fail to mark target tokens, but do not tend to produce large numbers of false positives  We suspect such failures to identify relevant spans/tokens are due to insufficient domain knowledge possessed by crowd workers 
The F1 score achieved after re-annotating the 600 most-difficult articles reaches 68 1%, which is close to the performance when re-annotating 1000 random articles  This demonstrates the effectiveness of recognizing difficult instances  The trend when we use up all expert data is still upward, so adding even more expert data is likely to further improve performance  Unfortunately we exhausted our budget and were not able to obtain additional expert annotations  It is likely that as the size of the expert annotations increases, the value of crowd annotations will diminish  This investigation is left for future work 
Conclusions
We have introduced the task of predicting annotation difficulty for biomedical information extraction (IE)  We trained neural models using different learned representations to score texts in terms of their difficulty  Results from all models were strong with Pearson’s correlation coefficients higher than 0 45 in almost all evaluations, indicating the feasibility of this task  An ensemble model combining universal and task specific feature sentence vectors yielded the best results 
Experiments on biomedical IE tasks show that removing up to $\sim $ 10% of the sentences predicted to be most difficult did not decrease model performance, and that re-weighting sentences inversely to their difficulty score during training improves predictive performance  Simulations in which difficult examples are routed to experts and other instances to crowd annotators yields the best results, outperforming the strategy of randomly selecting data for expert annotation, and substantially improving upon the approach of relying exclusively on crowd annotations  In future work, routing strategies based on instance difficulty could be further investigated for budget-quality trade-off 
Acknowledgements
This work has been partially supported by NSF1748771 grant  Wallace was support in part by NIH/NLM R01LM012086 

ground_truth: Irony Classifier, Sentiment Classifier for Irony, Sentiment Classifier for Non-irony, transformation from ironic sentences to non-ironic sentences