prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Was PolyReponse evaluated against some baseline?
Context 1: Introduction and Background
Task-oriented dialogue systems are primarily designed to search and interact with large databases which contain information pertaining to a certain dialogue domain: the main purpose of such systems is to assist the users in accomplishing a well-defined task such as flight booking BIBREF0, tourist information BIBREF1, restaurant search BIBREF2, or booking a taxi BIBREF3  These systems are typically constructed around rigid task-specific ontologies BIBREF1, BIBREF4 which enumerate the constraints the users can express using a collection of slots (e g , price range for restaurant search) and their slot values (e g , cheap, expensive for the aforementioned slots)  Conversations are then modelled as a sequence of actions that constrain slots to particular values  This explicit semantic space is manually engineered by the system designer  It serves as the output of the natural language understanding component as well as the input to the language generation component both in traditional modular systems BIBREF5, BIBREF6 and in more recent end-to-end task-oriented dialogue systems BIBREF7, BIBREF8, BIBREF9, BIBREF3 
Working with such explicit semantics for task-oriented dialogue systems poses several critical challenges on top of the manual time-consuming domain ontology design  First, it is difficult to collect domain-specific data labelled with explicit semantic representations  As a consequence, despite recent data collection efforts to enable training of task-oriented systems across multiple domains BIBREF0, BIBREF3, annotated datasets are still few and far between, as well as limited in size and the number of domains covered  Second, the current approach constrains the types of dialogue the system can support, resulting in artificial conversations, and breakdowns when the user does not understand what the system can and cannot support  In other words, training a task-based dialogue system for voice-controlled search in a new domain always implies the complex, expensive, and time-consuming process of collecting and annotating sufficient amounts of in-domain dialogue data 
In this paper, we present a demo system based on an alternative approach to task-oriented dialogue  Relying on non-generative response retrieval we describe the PolyResponse conversational search engine and its application in the task of restaurant search and booking  The engine is trained on hundreds of millions of real conversations from a general domain (i e , Reddit), using an implicit representation of semantics that directly optimizes the task at hand  It learns what responses are appropriate in different conversational contexts, and consequently ranks a large pool of responses according to their relevance to the current user utterance and previous dialogue history (i e , dialogue context) 
The technical aspects of the underlying conversational search engine are explained in detail in our recent work BIBREF11, while the details concerning the Reddit training data are also available in another recent publication BIBREF12  In this demo, we put focus on the actual practical usefulness of the search engine by demonstrating its potential in the task of restaurant search, and extending it to deal with multi-modal data  We describe a PolyReponse system that assists the users in finding a relevant restaurant according to their preference, and then additionally helps them to make a booking in the selected restaurant  Due to its retrieval-based design, with the PolyResponse engine there is no need to engineer a structured ontology, or to solve the difficult task of general language generation  This design also bypasses the construction of dedicated decision-making policy modules  The large ranking model already encapsulates a lot of knowledge about natural language and conversational flow 
Since retrieved system responses are presented visually to the user, the PolyResponse restaurant search engine is able to combine text responses with relevant visual information (e g , photos from social media associated with the current restaurant and related to the user utterance), effectively yielding a multi-modal response  This setup of using voice as input, and responding visually is becoming more and more prevalent with the rise of smart screens like Echo Show and even mixed reality  Finally, the PolyResponse restaurant search engine is multilingual: it is currently deployed in 8 languages enabling search over restaurants in 8 cities around the world  System snapshots in four different languages are presented in Figure FIGREF16, while screencast videos that illustrate the dialogue flow with the PolyResponse engine are available at: https://tinyurl com/y3evkcfz 
PolyResponse: Conversational Search
The PolyResponse system is powered by a single large conversational search engine, trained on a large amount of conversational and image data, as shown in Figure FIGREF2  In simple words, it is a ranking model that learns to score conversational replies and images in a given conversational context  The highest-scoring responses are then retrieved as system outputs  The system computes two sets of similarity scores: 1) $S(r,c)$ is the score of a candidate reply $r$ given a conversational context $c$, and 2) $S(p,c)$ is the score of a candidate photo $p$ given a conversational context $c$  These scores are computed as a scaled cosine similarity of a vector that represents the context ($h_c$), and a vector that represents the candidate response: a text reply ($h_r$) or a photo ($h_p$)  For instance, $S(r,c)$ is computed as $S(r,c)=C cos(h_r,h_c)$, where $C$ is a learned constant  The part of the model dealing with text input (i e , obtaining the encodings $h_c$ and $h_r$) follows the architecture introduced recently by Henderson:2019acl  We provide only a brief recap here; see the original paper for further details 
PolyResponse: Conversational Search ::: Text Representation 
The model, implemented as a deep neural network, learns to respond by training on hundreds of millions context-reply $(c,r)$ pairs  First, similar to Henderson:2017arxiv, raw text from both $c$ and $r$ is converted to unigrams and bigrams  All input text is first lower-cased and tokenised, numbers with 5 or more digits get their digits replaced by a wildcard symbol #, while words longer than 16 characters are replaced by a wildcard token LONGWORD  Sentence boundary tokens are added to each sentence  The vocabulary consists of the unigrams that occur at least 10 times in a random 10M subset of the Reddit training set (see Figure FIGREF2) plus the 200K most frequent bigrams in the same random subset 
During training, we obtain $d$-dimensional feature representations ($d=320$) shared between contexts and replies for each unigram and bigram jointly with other neural net parameters  A state-of-the-art architecture based on transformers BIBREF13 is then applied to unigram and bigram vectors separately, which are then averaged to form the final 320-dimensional encoding  That encoding is then passed through three fully-connected non-linear hidden layers of dimensionality $1,024$  The final layer is linear and maps the text into the final $l$-dimensional ($l=512$) representation: $h_c$ and $h_r$  Other standard and more sophisticated encoder models can also be used to provide final encodings $h_c$ and $h_r$, but the current architecture shows a good trade-off between speed and efficacy with strong and robust performance in our empirical evaluations on the response retrieval task using Reddit BIBREF14, OpenSubtitles BIBREF15, and AmazonQA BIBREF16 conversational test data, see BIBREF12 for further details 
In training the constant $C$ is constrained to lie between 0 and $\sqrt{l}$  Following Henderson:2017arxiv, the scoring function in the training objective aims to maximise the similarity score of context-reply pairs that go together, while minimising the score of random pairings: negative examples  Training proceeds via SGD with batches comprising 500 pairs (1 positive and 499 negatives) 
PolyResponse: Conversational Search ::: Photo Representation 
Photos are represented using convolutional neural net (CNN) models pretrained on ImageNet BIBREF17  We use a MobileNet model with a depth multiplier of 1 4, and an input dimension of $224 \times 224$ pixels as in BIBREF18  This provides a $1,280 \times 1 4 = 1,792$-dimensional representation of a photo, which is then passed through a single hidden layer of dimensionality $1,024$ with ReLU activation, before being passed to a hidden layer of dimensionality 512 with no activation to provide the final representation $h_p$ 
PolyResponse: Conversational Search ::: Data Source 1: Reddit 
For training text representations we use a Reddit dataset similar to AlRfou:2016arxiv  Our dataset is large and provides natural conversational structure: all Reddit data from January 2015 to December 2018, available as a public BigQuery dataset, span almost 3 7B comments BIBREF12  We preprocess the dataset to remove uninformative and long comments by retaining only sentences containing more than 8 and less than 128 word tokens  After pairing all comments/contexts $c$ with their replies $r$, we obtain more than 727M context-reply $(c,r)$ pairs for training, see Figure FIGREF2 
PolyResponse: Conversational Search ::: Data Source 2: Yelp 
Once the text encoding sub-networks are trained, a photo encoder is learned on top of a pretrained MobileNet CNN, using data taken from the Yelp Open dataset: it contains around 200K photos and their captions  Training of the multi-modal sub-network then maximises the similarity of captions encoded with the response encoder $h_r$ to the photo representation $h_p$  As a result, we can compute the score of a photo given a context using the cosine similarity of the respective vectors  A photo will be scored highly if it looks like its caption would be a good response to the current context 
PolyResponse: Conversational Search ::: Index of Responses 
The Yelp dataset is used at inference time to provide text and photo candidates to display to the user at each step in the conversation  Our restaurant search is currently deployed separately for each city, and we limit the responses to a given city  For instance, for our English system for Edinburgh we work with 396 restaurants, 4,225 photos (these include additional photos obtained using the Google Places API without captions), 6,725 responses created from the structured information about restaurants that Yelp provides, converted using simple templates to sentences of the form such as “Restaurant X accepts credit cards ”, 125,830 sentences extracted from online reviews 
PolyResponse: Conversational Search ::: PolyResponse in a Nutshell 
The system jointly trains two encoding functions (with shared word embeddings) $f(context)$ and $g(reply)$ which produce encodings $h_c$ and $h_r$, so that the similarity $S(c,r)$ is high for all $(c,r)$ pairs from the Reddit training data and low for random pairs  The encoding function $g()$ is then frozen, and an encoding function $t(photo)$ is learnt which makes the similarity between a photo and its associated caption high for all (photo, caption) pairs from the Yelp dataset, and low for random pairs  $t$ is a CNN pretrained on ImageNet, with a shallow one-layer DNN on top  Given a new context/query, we then provide its encoding $h_c$ by applying $f()$, and find plausible text replies and photo responses according to functions $g()$ and $t()$, respectively  These should be responses that look like answers to the query, and photos that look like they would have captions that would be answers to the provided query 
At inference, finding relevant candidates given a context reduces to computing $h_c$ for the context $c$ , and finding nearby $h_r$ and $h_p$ vectors  The response vectors can all be pre-computed, and the nearest neighbour search can be further optimised using standard libraries such as Faiss BIBREF19 or approximate nearest neighbour retrieval BIBREF20, giving an efficient search that scales to billions of candidate responses 
The system offers both voice and text input and output  Speech-to-text and text-to-speech conversion in the PolyResponse system is currently supported by the off-the-shelf Google Cloud tools 
Dialogue Flow
The ranking model lends itself to the one-shot task of finding the most relevant responses in a given context  However, a restaurant-browsing system needs to support a dialogue flow where the user finds a restaurant, and then asks questions about it  The dialogue state for each search scenario is represented as the set of restaurants that are considered relevant  This starts off as all the restaurants in the given city, and is assumed to monotonically decrease in size as the conversation progresses until the user converges to a single restaurant  A restaurant is only considered valid in the context of a new user input if it has relevant responses corresponding to it  This flow is summarised here:
S1  Initialise $R$ as the set of all restaurants in the city  Given the user's input, rank all the responses in the response pool pertaining to restaurants in $R$ 
S2  Retrieve the top $N$ responses $r_1, r_2, \ldots , r_N$ with corresponding (sorted) cosine similarity scores: $s_1 \ge s_2 \ge \ldots \ge s_N$ 
S3  Compute probability scores $p_i \propto \exp (a \cdot s_i)$ with $\sum _{i=1}^N p_i$, where $a>0$ is a tunable constant 
S4  Compute a score $q_e$ for each restaurant/entity $e \in R$, $q_e = \sum _{i: r_i \in e} p_i$ 
S5  Update $R$ to the smallest set of restaurants with highest $q$ whose $q$-values sum up to more than a predefined threshold $t$ 
S6  Display the most relevant responses associated with the updated $R$, and return to S2 
If there are multiple relevant restaurants, one response is shown from each  When only one restaurant is relevant, the top $N$ responses are all shown, and relevant photos are also displayed  The system does not require dedicated understanding, decision-making, and generation modules, and this dialogue flow does not rely on explicit task-tailored semantics  The set of relevant restaurants is kept internally while the system narrows it down across multiple dialogue turns  A simple set of predefined rules is used to provide a templatic spoken system response: e g , an example rule is “One review of $e$ said $r$”, where $e$ refers to the restaurant, and $r$ to a relevant response associated with $e$  Note that while the demo is currently focused on the restaurant search task, the described “narrowing down” dialogue flow is generic and applicable to a variety of applications dealing with similar entity search 
The system can use a set of intent classifiers to allow resetting the dialogue state, or to activate the separate restaurant booking dialogue flow  These classifiers are briefly discussed in §SECREF4 
Other Functionality ::: Multilinguality 
The PolyResponse restaurant search is currently available in 8 languages and for 8 cities around the world: English (Edinburgh), German (Berlin), Spanish (Madrid), Mandarin (Taipei), Polish (Warsaw), Russian (Moscow), Korean (Seoul), and Serbian (Belgrade)  Selected snapshots are shown in Figure FIGREF16, while we also provide videos demonstrating the use and behaviour of the systems at: https://tinyurl com/y3evkcfz  A simple MT-based translate-to-source approach at inference time is currently used to enable the deployment of the system in other languages: 1) the pool of responses in each language is translated to English by Google Translate beforehand, and pre-computed encodings of their English translations are used as representations of each foreign language response; 2) a provided user utterance (i e , context) is translated to English on-the-fly and its encoding $h_c$ is then learned  We plan to experiment with more sophisticated multilingual models in future work 
Other Functionality ::: Voice-Controlled Menu Search 
An additional functionality enables the user to get parts of the restaurant menu relevant to the current user utterance as responses  This is achieved by performing an additional ranking step of available menu items and retrieving the ones that are semantically relevant to the user utterance using exactly the same methodology as with ranking other responses  An example of this functionality is shown in Figure FIGREF21 
Other Functionality ::: Resetting and Switching to Booking 
The restaurant search system needs to support the discrete actions of restarting the conversation (i e , resetting the set $R$), and should enable transferring to the slot-based table booking flow  This is achieved using two binary intent classifiers, that are run at each step in the dialogue  These classifiers make use of the already-computed $h_c$ vector that represents the user's latest text  A single-layer neural net is learned on top of the 512-dimensional encoding, with a ReLU activation and 100 hidden nodes  To train the classifiers, sets of 20 relevant paraphrases (e g , “Start again”) are provided as positive examples  Finally, when the system successfully switches to the booking scenario, it proceeds to the slot filling task: it aims to extract all the relevant booking information from the user (e g , date, time, number of people to dine)  The entire flow of the system illustrating both the search phase and the booking phase is provided as the supplemental video material 
Conclusion and Future Work
This paper has presented a general approach to search-based dialogue that does not rely on explicit semantic representations such as dialogue acts or slot-value ontologies, and allows for multi-modal responses  In future work, we will extend the current demo system to more tasks and languages, and work with more sophisticated encoders and ranking functions  Besides the initial dialogue flow from this work (§SECREF3), we will also work with more complex flows dealing, e g , with user intent shifts 
Context 2: Introduction
This work is licenced under a Creative Commons Attribution 4 0 International License  License details: http://creativecommons org/licenses/by/4 0/ Deep neural networks have been widely used in text classification and have achieved promising results BIBREF0 , BIBREF1 , BIBREF2   Most focus on content information and use models such as convolutional neural networks (CNN) BIBREF3 or recursive neural networks BIBREF4   However, for user-generated posts on social media like Facebook or Twitter, there is more information that should not be ignored  On social media platforms, a user can act either as the author of a post or as a reader who expresses his or her comments about the post 
In this paper, we classify posts taking into account post authorship, likes, topics, and comments  In particular, users and their “likes” hold strong potential for text mining  For example, given a set of posts that are related to a specific topic, a user's likes and dislikes provide clues for stance labeling  From a user point of view, users with positive attitudes toward the issue leave positive comments on the posts with praise or even just the post's content; from a post point of view, positive posts attract users who hold positive stances  We also investigate the influence of topics: different topics are associated with different stance labeling tendencies and word usage  For example we discuss women's rights and unwanted babies on the topic of abortion, but we criticize medicine usage or crime when on the topic of marijuana BIBREF5   Even for posts on a specific topic like nuclear power, a variety of arguments are raised: green energy, radiation, air pollution, and so on  As for comments, we treat them as additional text information  The arguments in the comments and the commenters (the users who leave the comments) provide hints on the post's content and further facilitate stance classification 
In this paper, we propose the user-topic-comment neural network (UTCNN), a deep learning model that utilizes user, topic, and comment information  We attempt to learn user and topic representations which encode user interactions and topic influences to further enhance text classification, and we also incorporate comment information  We evaluate this model on a post stance classification task on forum-style social media platforms  The contributions of this paper are as follows: 1  We propose UTCNN, a neural network for text in modern social media channels as well as legacy social media, forums, and message boards — anywhere that reveals users, their tastes, as well as their replies to posts  2  When classifying social media post stances, we leverage users, including authors and likers  User embeddings can be generated even for users who have never posted anything  3  We incorporate a topic model to automatically assign topics to each post in a single topic dataset  4  We show that overall, the proposed method achieves the highest performance in all instances, and that all of the information extracted, whether users, topics, or comments, still has its contributions 
Extra-Linguistic Features for Stance Classification
In this paper we aim to use text as well as other features to see how they complement each other in a deep learning model  In the stance classification domain, previous work has showed that text features are limited, suggesting that adding extra-linguistic constraints could improve performance BIBREF6 , BIBREF7 , BIBREF8   For example, Hasan and Ng as well as Thomas et al  require that posts written by the same author have the same stance BIBREF9 , BIBREF10   The addition of this constraint yields accuracy improvements of 1–7% for some models and datasets  Hasan and Ng later added user-interaction constraints and ideology constraints BIBREF7 : the former models the relationship among posts in a sequence of replies and the latter models inter-topic relationships, e g , users who oppose abortion could be conservative and thus are likely to oppose gay rights 
For work focusing on online forum text, since posts are linked through user replies, sequential labeling methods have been used to model relationships between posts  For example, Hasan and Ng use hidden Markov models (HMMs) to model dependent relationships to the preceding post BIBREF9 ; Burfoot et al  use iterative classification to repeatedly generate new estimates based on the current state of knowledge BIBREF11 ; Sridhar et al  use probabilistic soft logic (PSL) to model reply links via collaborative filtering BIBREF12   In the Facebook dataset we study, we use comments instead of reply links  However, as the ultimate goal in this paper is predicting not comment stance but post stance, we treat comments as extra information for use in predicting post stance 
Deep Learning on Extra-Linguistic Features
In recent years neural network models have been applied to document sentiment classification BIBREF13 , BIBREF4 , BIBREF14 , BIBREF15 , BIBREF2   Text features can be used in deep networks to capture text semantics or sentiment  For example, Dong et al  use an adaptive layer in a recursive neural network for target-dependent Twitter sentiment analysis, where targets are topics such as windows 7 or taylor swift BIBREF16 , BIBREF17 ; recursive neural tensor networks (RNTNs) utilize sentence parse trees to capture sentence-level sentiment for movie reviews BIBREF4 ; Le and Mikolov predict sentiment by using paragraph vectors to model each paragraph as a continuous representation BIBREF18   They show that performance can thus be improved by more delicate text models 
Others have suggested using extra-linguistic features to improve the deep learning model  The user-word composition vector model (UWCVM) BIBREF19 is inspired by the possibility that the strength of sentiment words is user-specific; to capture this they add user embeddings in their model  In UPNN, a later extension, they further add a product-word composition as product embeddings, arguing that products can also show different tendencies of being rated or reviewed BIBREF20   Their addition of user information yielded 2–10% improvements in accuracy as compared to the above-mentioned RNTN and paragraph vector methods  We also seek to inject user information into the neural network model  In comparison to the research of Tang et al  on sentiment classification for product reviews, the difference is two-fold  First, we take into account multiple users (one author and potentially many likers) for one post, whereas only one user (the reviewer) is involved in a review  Second, we add comment information to provide more features for post stance classification  None of these two factors have been considered previously in a deep learning model for text stance classification  Therefore, we propose UTCNN, which generates and utilizes user embeddings for all users — even for those who have not authored any posts — and incorporates comments to further improve performance 
Method
In this section, we first describe CNN-based document composition, which captures user- and topic-dependent document-level semantic representation from word representations  Then we show how to add comment information to construct the user-topic-comment neural network (UTCNN) 
User- and Topic-dependent Document Composition
As shown in Figure FIGREF4 , we use a general CNN BIBREF3 and two semantic transformations for document composition   We are given a document with an engaged user INLINEFORM0 , a topic INLINEFORM1 , and its composite INLINEFORM2 words, each word INLINEFORM3 of which is associated with a word embedding INLINEFORM4 where INLINEFORM5 is the vector dimension  For each word embedding INLINEFORM6 , we apply two dot operations as shown in Equation EQREF6 : DISPLAYFORM0
where INLINEFORM0 models the user reading preference for certain semantics, and INLINEFORM1 models the topic semantics; INLINEFORM2 and INLINEFORM3 are the dimensions of transformed user and topic embeddings respectively  We use INLINEFORM4 to model semantically what each user prefers to read and/or write, and use INLINEFORM5 to model the semantics of each topic  The dot operation of INLINEFORM6 and INLINEFORM7 transforms the global representation INLINEFORM8 to a user-dependent representation  Likewise, the dot operation of INLINEFORM9 and INLINEFORM10 transforms INLINEFORM11 to a topic-dependent representation 
After the two dot operations on INLINEFORM0 , we have user-dependent and topic-dependent word vectors INLINEFORM1 and INLINEFORM2 , which are concatenated to form a user- and topic-dependent word vector INLINEFORM3   Then the transformed word embeddings INLINEFORM4 are used as the CNN input  Here we apply three convolutional layers on the concatenated transformed word embeddings INLINEFORM5 : DISPLAYFORM0
where INLINEFORM0 is the index of words; INLINEFORM1 is a non-linear activation function (we use INLINEFORM2 ); INLINEFORM5 is the convolutional filter with input length INLINEFORM6 and output length INLINEFORM7 , where INLINEFORM8 is the window size of the convolutional operation; and INLINEFORM9 and INLINEFORM10 are the output and bias of the convolution layer INLINEFORM11 , respectively  In our experiments, the three window sizes INLINEFORM12 in the three convolution layers are one, two, and three, encoding unigram, bigram, and trigram semantics accordingly 
After the convolutional layer, we add a maximum pooling layer among convolutional outputs to obtain the unigram, bigram, and trigram n-gram representations  This is succeeded by an average pooling layer for an element-wise average of the three maximized convolution outputs 
UTCNN Model Description
Figure FIGREF10 illustrates the UTCNN model  As more than one user may interact with a given post, we first add a maximum pooling layer after the user matrix embedding layer and user vector embedding layer to form a moderator matrix embedding INLINEFORM0 and a moderator vector embedding INLINEFORM1 for moderator INLINEFORM2 respectively, where INLINEFORM3 is used for the semantic transformation in the document composition process, as mentioned in the previous section  The term moderator here is to denote the pseudo user who provides the overall semantic/sentiment of all the engaged users for one document  The embedding INLINEFORM4 models the moderator stance preference, that is, the pattern of the revealed user stance: whether a user is willing to show his preference, whether a user likes to show impartiality with neutral statements and reasonable arguments, or just wants to show strong support for one stance  Ideally, the latent user stance is modeled by INLINEFORM5 for each user  Likewise, for topic information, a maximum pooling layer is added after the topic matrix embedding layer and topic vector embedding layer to form a joint topic matrix embedding INLINEFORM6 and a joint topic vector embedding INLINEFORM7 for topic INLINEFORM8 respectively, where INLINEFORM9 models the semantic transformation of topic INLINEFORM10 as in users and INLINEFORM11 models the topic stance tendency  The latent topic stance is also modeled by INLINEFORM12 for each topic 
As for comments, we view them as short documents with authors only but without likers nor their own comments  Therefore we apply document composition on comments although here users are commenters (users who comment)  It is noticed that the word embeddings INLINEFORM0 for the same word in the posts and comments are the same, but after being transformed to INLINEFORM1 in the document composition process shown in Figure FIGREF4 , they might become different because of their different engaged users  The output comment representation together with the commenter vector embedding INLINEFORM2 and topic vector embedding INLINEFORM3 are concatenated and a maximum pooling layer is added to select the most important feature for comments  Instead of requiring that the comment stance agree with the post, UTCNN simply extracts the most important features of the comment contents; they could be helpful, whether they show obvious agreement or disagreement  Therefore when combining comment information here, the maximum pooling layer is more appropriate than other pooling or merging layers  Indeed, we believe this is one reason for UTCNN's performance gains 
Finally, the pooled comment representation, together with user vector embedding INLINEFORM0 , topic vector embedding INLINEFORM1 , and document representation are fed to a fully connected network, and softmax is applied to yield the final stance label prediction for the post 
Experiment
We start with the experimental dataset and then describe the training process as well as the implementation of the baselines  We also implement several variations to reveal the effects of features: authors, likers, comment, and commenters  In the results section we compare our model with related work 
Dataset
We tested the proposed UTCNN on two different datasets: FBFans and CreateDebate  FBFans is a privately-owned, single-topic, Chinese, unbalanced, social media dataset, and CreateDebate is a public, multiple-topic, English, balanced, forum dataset  Results using these two datasets show the applicability and superiority for different topics, languages, data distributions, and platforms 
The FBFans dataset contains data from anti-nuclear-power Chinese Facebook fan groups from September 2013 to August 2014, including posts and their author and liker IDs  There are a total of 2,496 authors, 505,137 likers, 33,686 commenters, and 505,412 unique users  Two annotators were asked to take into account only the post content to label the stance of the posts in the whole dataset as supportive, neutral, or unsupportive (hereafter denoted as Sup, Neu, and Uns)  Sup/Uns posts were those in support of or against anti-reconstruction; Neu posts were those evincing a neutral standpoint on the topic, or were irrelevant  Raw agreement between annotators is 0 91, indicating high agreement  Specifically, Cohen’s Kappa for Neu and not Neu labeling is 0 58 (moderate), and for Sup or Uns labeling is 0 84 (almost perfect)  Posts with inconsistent labels were filtered out, and the development and testing sets were randomly selected from what was left  Posts in the development and testing sets involved at least one user who appeared in the training set  The number of posts for each stance is shown on the left-hand side of Table TABREF12   About twenty percent of the posts were labeled with a stance, and the number of supportive (Sup) posts was much larger than that of the unsupportive (Uns) ones: this is thus highly skewed data, which complicates stance classification  On average, 161
Context 3: Introduction
Cyberbullying has been defined by the National Crime Prevention Council as the use of the Internet, cell phones or other devices to send or post text or images intended to hurt or embarrass another person  Various studies have estimated that between to 10% to 40% of internet users are victims of cyberbullying BIBREF0   Effects of cyberbullying can range from temporary anxiety to suicide BIBREF1   Many high profile incidents have emphasized the prevalence of cyberbullying on social media  Most recently in October 2017, a Swedish model Arvida Byström was cyberbullied to the extent of receiving rape threats after she appeared in an advertisement with hairy legs 
Detection of cyberbullying in social media is a challenging task  Definition of what constitutes cyberbullying is quite subjective  For example, frequent use of swear words might be considered as bullying by the general population  However, for teen oriented social media platforms such as Formspring, this does not necessarily mean bullying (Table TABREF9 )  Across multiple SMPs, cyberbullies attack victims on different topics such as race, religion, and gender  Depending on the topic of cyberbullying, vocabulary and perceived meaning of words vary significantly across SMPs  For example, in our experiments we found that for word `fat', the most similar words as per Twitter dataset are `female' and `woman' (Table TABREF23 )  However, other two datasets do not show such particular bias against women  This platform specific semantic similarity between words is a key aspect of cyberbullying detection across SMPs  Style of communication varies significantly across SMPs  For example, Twitter posts are short and lack anonymity  Whereas posts on Q&A oriented SMPs are long and have option of anonymity (Table TABREF7 )  Fast evolving words and hashtags in social media make it difficult to detect cyberbullying using swear word list based simple filtering approaches  The option of anonymity in certain social networks also makes it harder to identify cyberbullying as profile and history of the bully might not be available 
Past works on cyberbullying detection have at least one of the following three bottlenecks  First (Bottleneck B1), they target only one particular social media platform  How these methods perform across other SMPs is unknown  Second (Bottleneck B2), they address only one topic of cyberbullying such as racism, and sexism  Depending on the topic, vocabulary and nature of cyberbullying changes  These models are not flexible in accommodating changes in the definition of cyberbullying  Third (Bottleneck B3), they rely on carefully handcrafted features such as swear word list and POS tagging  However, these handcrafted features are not robust against variations in writing style  In contrast to existing bottlenecks, this work targets three different types of social networks (Formspring: a Q&A forum, Twitter: microblogging, and Wikipedia: collaborative knowledge repository) for three topics of cyberbullying (personal attack, racism, and sexism) without doing any explicit feature engineering by developing deep learning based models along with transfer learning 
We experimented with diverse traditional machine learning models (logistic regression, support vector machine, random forest, naive Bayes) and deep neural network models (CNN, LSTM, BLSTM, BLSTM with Attention) using variety of representation methods for words (bag of character n-gram, bag of word unigram, GloVe embeddings, SSWE embeddings)  Summary of our findings and research contributions is as follows 
Datasets
Please refer to Table TABREF7 for summary of datasets used  We performed experiments using large, diverse, manually annotated, and publicly available datasets for cyberbullying detection in social media  We cover three different types of social networks: teen oriented Q&A forum (Formspring), large microblogging platform (Twitter), and collaborative knowledge repository (Wikipedia talk pages)  Each dataset addresses a different topic of cyberbullying  Twitter dataset contains examples of racism and sexism  Wikipedia dataset contains examples of personal attack  However, Formspring dataset is not specifically about any single topic  All three datasets have the problem of class imbalance where posts labeled as cyberbullying are in the minority as compared to neutral posts  Variation in the number of posts across datasets also affects vocabulary size that represents the number of distinct words encountered in the dataset  We measure the size of a post in terms of the number of words in the post  For each dataset, there are only a few posts with large size  We truncate such large posts to the size of post ranked at 95 percentile in that dataset  For example, in Wikipedia dataset, the largest post has 2846 words  However, size of post ranked at 95 percentile in that dataset is only 231  Any post larger than size 231 in Wikipedia dataset will be truncated by considering only first 231 words  This truncation affects only a small minority of posts in each dataset  However, it is required for efficiently training various models in our experiments  Details of each dataset are as follows 
Formspring BIBREF2 : It was a question and answer based website where users could openly invite others to ask and answer questions  The dataset includes 12K annotated question and answer pairs  Each post is manually labeled by three workers  Among these pairs, 825 were labeled as containing cyberbullying content by at least two Amazon Mechanical turk workers 
Twitter BIBREF3 : This dataset includes 16K annotated tweets  The authors bootstrapped the corpus collection, by performing an initial manual search of common slurs and terms used pertaining to religious, sexual, gender, and ethnic minorities  Of the 16K tweets, 3117 are labeled as sexist, 1937 as racist, and the remaining are marked as neither sexist nor racist 
Wikipedia BIBREF4 : For each page in Wikipedia, a corresponding talk page maintains the history of discussion among users who participated in its editing  This data set includes over 100k labeled discussion comments from English Wikipedia's talk pages  Each comment was labeled by 10 annotators via Crowdflower on whether it contains a personal attack  There are total 13590 comments labeled as personal attack 
Use of Swear Words and Anonymity
Please refer to Table TABREF9   We use the following short forms in this section: B=Bullying, S=Swearing, A=Anonymous  Some of the values for Twitter dataset are undefined as Twitter does not allow anonymous postings  Use of swear words has been repeatedly linked to cyberbullying  However, preliminary analysis of datasets reveals that depending on swear word usage can neither lead to high precision nor high recall for cyberbullying detection  Swear word list based methods will have low precision as P(B INLINEFORM0 S) is not close to 1  In fact, for teen oriented social network Formspring, 78% of the swearing posts are non-bullying  Swear words based filtering will be irritating to the users in such SMPs where swear words are used casually  Swear word list based methods will also have a low recall as P(S INLINEFORM1 B) is not close to 1  For Twitter dataset, 82% of bullying posts do not use any swear words  Such passive-aggressive cyberbullying will go undetected with swear word list based methods  Anonymity is another clue that is used for detecting cyberbullying as bully might prefer to hide its identity  Anonymity definitely leads to increased use of swear words (P(S INLINEFORM2 A) INLINEFORM3 P(S)) and cyberbullying (P(B INLINEFORM4 A) INLINEFORM5 P(B), and P(B INLINEFORM6 A&S)) INLINEFORM7 P(B))  However, significant fraction of anonymous posts are non-bullying (P(B INLINEFORM8 A) not close to 1) and many of bullying posts are not anonymous (P(A INLINEFORM9 B) not close to 1)  Further, anonymity might not be allowed by many SMPs such as Twitter 
Related Work
Cyberbullying is recognized as a phenomenon at least since 2003 BIBREF5   Use of social media exploded with launching of multiple platforms such as Wikipedia (2001), MySpace (2003), Orkut (2004), Facebook (2004), and Twitter (2005)  By 2006, researchers had pointed that cyberbullying was as serious phenomenon as offline bullying BIBREF6   However, automatic detection of cyberbullying was addressed only since 2009 BIBREF7   As a research topic, cyberbullying detection is a text classification problem  Most of the existing works fit in the following template: get training dataset from single SMP, engineer variety of features with certain style of cyberbullying as the target, apply a few traditional machine learning methods, and evaluate success in terms of measures such as F1 score and accuracy  These works heavily rely on handcrafted features such as use of swear words  These methods tend to have low precision for cyberbullying detection as handcrafted features are not robust against variations in bullying style across SMPs and bullying topics  Only recently, deep learning has been applied for cyberbullying detection BIBREF8   Table TABREF27 summarizes important related work 
Deep Neural Network (DNN) Based Models
We experimented with four DNN based models for cyberbullying detection: CNN, LSTM, BLSTM, and BLSTM with attention  These models are listed in the increasing complexity of their neural architecture and amount of information used by these models  Please refer to Figure 1 for general architecture that we have used across four models  Various models differ only in the Neural Architecture layer while having identical rest of the layers  CNNs are providing state-of-the-results on extracting contextual feature for classification tasks in images, videos, audios, and text  Recently, CNNs were used for sentiment classification BIBREF9   Long Short Term Memory networks are a special kind of RNN, capable of learning long-term dependencies  Their ability to use their internal memory to process arbitrary sequences of inputs has been found to be effective for text classification BIBREF10   Bidirectional LSTMs BIBREF11 further increase the amount of input information available to the network by encoding information in both forward and backward direction  By using two directions, input information from both the past and future of the current time frame can be used  Attention mechanisms allow for a more direct dependence between the state of the model at different points in time  Importantly, attention mechanism lets the model learn what to attend to based on the input sentence and what it has produced so far 
The embedding layer processes a fixed size sequence of words  Each word is represented as a real-valued vector, also known as word embeddings  We have experimented with three methods for initializing word embeddings: random, GloVe BIBREF12 , and SSWE BIBREF13   During the training, model improves upon the initial word embeddings to learn task specific word embeddings  We have observed that these task specific word embeddings capture the SMP specific and topic specific style of cyberbullying  Using GloVe vectors over random vector initialization has been reported to improve performance for some NLP tasks  Most of the word embedding methods such as GloVe, consider only syntactic context of the word while ignoring the sentiment conveyed by the text  SSWE method overcomes this problem by incorporating the text sentiment as one of the parameters for word embedding generation  We experimented with various dimension size for word embeddings  Experimental results reported here are with dimension size as 50  There was no significant variation in results with dimension size ranging from 30 to 200 
To avoid overfitting, we used two dropout layers, one before the neural architecture layer and one after, with dropout rates of 0 25 and 0 5 respectively  Fully connected layer is a dense output layer with the number of neurons equal to the number of classes, followed by softmax layer that provides softmax activation  All our models are trained using backpropagation  The optimizer used for training is Adam and the loss function is categorical cross-entropy  Besides learning the network weights, these methods also learn task-specific word embeddings tuned towards the bullying labels (See Section SECREF21 )  Our code is available at: https://github com/sweta20/Detecting-Cyberbullying-Across-SMPs 
Experiments
Existing works have heavily relied on traditional machine learning models for cyberbullying detection  However, they do not study the performance of these models across multiple SMPs  We experimented with four models: logistic regression (LR), support vector machine (SVM), random forest (RF), and naive Bayes (NB), as these are used in previous works (Table TABREF27 )  We used two data representation methods: character n-gram and word unigram  Past work in the domain of detecting abusive language have showed that simple n-gram features are more powerful than linguistic and syntactic features, hand-engineered lexicons, and word and paragraph embeddings BIBREF14   As compared to DNN models, performance of all four traditional machine learning models was significantly lower  Please refer to Table TABREF11  
All DNN models reported here were implemented using Keras  We pre-process the data, subjecting it to standard operations of removal of stop words, punctuation marks and lowercasing, before annotating it to assigning respective labels to each comment  For each trained model, we report its performance after doing five-fold cross-validation  We use following short forms 
Effect of Oversampling Bullying Instances
The training datasets had a major problem of class imbalance with posts marked as bullying in the minority  As a result, all models were biased towards labeling the posts as non-bullying  To remove this bias, we oversampled the data from bullying class thrice  That is, we replicated bullying posts thrice in the training data  This significantly improved the performance of all DNN models with major leap in all three evaluation measures  Table TABREF17 shows the effect of oversampling for a variety of word embedding methods with BLSTM Attention as the detection model  Results for other models are similar BIBREF15   We can notice that oversampled datasets (F+, T+, W+) have far better performance than their counterparts (F, T, W respectively)  Oversampling particularly helps the smallest dataset Formspring where number of training instances for bullying class is quite small (825) as compared to other two datasets (about 5K and 13K)  We also experimented with varying the replication rate for bullying posts BIBREF15   However, we observed that for bullying posts, replication rate of three is good enough 
Choice of Initial Word Embeddings and Model
Initial word embeddings decide data representation for DNN models  However during the training, DNN models modify these initial word embeddings to learn task specific word embeddings  We have experimented with three methods to initialize word embeddings  Please refer to Table TABREF19   This table shows the effect of varying initial word embeddings for multiple DNN models across datasets  We can notice that initial word embeddings do not have a significant effect on cyberbullying detection when oversampling of bullying posts is done (rows corresponding to F+, T+, W+)  In the absence of oversampling (rows corresponding to F, T W), there is a gap in performance of simplest (CNN) and most complex (BLSTM with attention) models  However, this gap goes on reducing with the increase in the size of datasets 
Table TABREF20 compares the performance of four DNN models for three evaluation measures while using SSWE as the initial word embeddings  We have noticed that most of the time LSTM performs weaker than other three models  However, performance gap in the other three models is not significant 
Task Specific Word Embeddings
DNN models learn word embeddings over the training data  These learned embeddings across multiple datasets show the difference in nature and style of bullying across cyberbullying topics and SMPs  Here we report results for BLSTM with attention model  Results for other models are similar  We first verify that important words for each topic of cyberbullying form clusters in the learned embeddings  To enable the visualization of grouping, we reduced dimensionality with t-SNE BIBREF16 , a well-known technique for dimensionality reduction particularly well suited for visualization of high dimensional datasets  Please refer to Table TABREF22   This table shows important clusters observed in t-SNE projection of learned word embeddings  Each cluster shows that words most relevant to a particular topic of bullying form cluster 
We also observed changes in the meanings of the words across topics of cyberbullying  Table TABREF23 shows most similar words for a given query word for two datasets  Twitter dataset which is heavy on sexism and racism, considers word slave as similar to targets of racism and sexism  However, Wikipedia dataset that is about personal attacks does not show such bias 
Transfer Learning
We used transfer learning to check if the knowledge gained by DNN models on one dataset can be used to improve cyberbullying detection performance on other datasets  We report results where BLSTM with attention is used as the DNN model  Results for other models are similar BIBREF15   We experimented with following three flavors of transfer learning 
Complete Transfer Learning (TL1): In this flavor, a model trained on one dataset was directly used to detect cyberbullying in other datasets without any extra training  TL1 resulted in significantly low recall indicating that three datasets have different nature of cyberbullying with low overlap (Table TABREF25 )  However precision was relatively higher for TL1, indicating that DNN models are cautious in labeling a post as bully (Table TABREF25 )  TL1 also helps to measure similarity in nature of cyberbullying across three datasets  We can observe that bullying nature in Formspring and Wikipedia datasets is more similar to each other than the Twitter dataset  This can be inferred from the fact that with TL1, cyberbullying detection performance for Formspring dataset is higher when base model is Wikipedia (precision =0 51 and recall=0 66)as compared to Twitter as the base model (precision=0 38 and recall=0 04)  Similarly, for Wikipedia dataset, Formspring acts as a better base model than Twitter while using TL1 flavor of transfer learning  Nature of SMP might be a factor behind this similarity in nature of cyberbullying  Both Formspring and Wikipedia are task oriented social networks (Q&A and collaborative knowledge repository respectively) that allow anonymity and larger posts  Whereas communication on Twitter is short, free of anonymity and not oriented towards a particular task 
Feature Level Transfer Learning (TL2): In this flavor, a model was trained on one dataset and only learned word embeddings were transferred to another dataset for training a new model  As compared to TL1, recall score improved dramatically with TL2 (Table TABREF25 )  Improvement in precision was also significant (Table TABREF25 )  These improvements indicate that learned word embeddings are an essential part of knowledge transfer across datasets for cyberbullying detection 
Model Level Transfer Learning (TL3): In this flavor, a model was trained on one dataset and learned word embeddings, as well as network weights, were transferred to another dataset for training a new model  TL3 does not result in any significant improvement over TL2  This lack of improvement indicates that transfer of network weights is not essential for cyberbullying detection and learned word embeddings is the key knowledge gained by the DNN models 
DNN based models coupled with transfer learning beat the best-known results for all three datasets  Previous best F1 scores for Wikipedia BIBREF4 and Twitter BIBREF8 datasets were 0 68 and 0 93 respectively  We achieve F1 scores of 0 94 for both these datasets using BLSTM with attention and feature level transfer learning (Table TABREF25 )  For Formspring dataset, authors have not reported F1 score  Their method has accuracy score of 78 5% BIBREF2   We achieve F1 score of 0 95 with accuracy score of 98% for the same dataset 
Conclusion and Future Work
We have shown that DNN models can be used for cyberbullying detection on various topics across multiple SMPs using three datasets and four DNN models  These models coupled with transfer learning beat state of the art results for all three datasets  These models can be further improved with extra data such as information about the profile and social graph of users  Most of the current datasets do not provide any information about the severity of bullying  If such fine-grained information is made available, then cyberbullying detection models can be further improved to take a variety of actions depending on the perceived seriousness of the posts 

ground_truth: No

No

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How many layers does the UTCNN model have?
Context 2: 1 users were involved in one post  The maximum was 23,297 and the minimum was one (the author)  For comments, on average there were 3 comments per post  The maximum was 1,092 and the minimum was zero 
To test whether the assumption of this paper – posts attract users who hold the same stance to like them – is reliable, we examine the likes from authors of different stances  Posts in FBFans dataset are used for this analysis  We calculate the like statistics of each distinct author from these 32,595 posts  As the numbers of authors in the Sup, Neu and Uns stances are largely imbalanced, these numbers are normalized by the number of users of each stance  Table TABREF13 shows the results  Posts with stances (i e , not neutral) attract users of the same stance  Neutral posts also attract both supportive and neutral users, like what we observe in supportive posts, but just the neutral posts can attract even more neutral likers  These results do suggest that users prefer posts of the same stance, or at least posts of no obvious stance which might cause annoyance when reading, and hence support the user modeling in our approach 
The CreateDebate dataset was collected from an English online debate forum discussing four topics: abortion (ABO), gay rights (GAY), Obama (OBA), and marijuana (MAR)  The posts are annotated as for (F) and against (A)  Replies to posts in this dataset are also labeled with stance and hence use the same data format as posts  The labeling results are shown in the right-hand side of Table TABREF12   We observe that the dataset is more balanced than the FBFans dataset  In addition, there are 977 unique users in the dataset  To compare with Hasan and Ng's work, we conducted five-fold cross-validation and present the annotation results as the average number of all folds BIBREF9 , BIBREF5  
The FBFans dataset has more integrated functions than the CreateDebate dataset; thus our model can utilize all linguistic and extra-linguistic features  For the CreateDebate dataset, on the other hand, the like and comment features are not available (as there is a stance label for each reply, replies are evaluated as posts as other previous work) but we still implemented our model using the content, author, and topic information 
Settings
In the UTCNN training process, cross-entropy was used as the loss function and AdaGrad as the optimizer  For FBFans dataset, we learned the 50-dimensional word embeddings on the whole dataset using GloVe BIBREF21 to capture the word semantics; for CreateDebate dataset we used the publicly available English 50-dimensional word embeddings, pre-trained also using GloVe  These word embeddings were fixed in the training process  The learning rate was set to 0 03  All user and topic embeddings were randomly initialized in the range of [-0 1 0 1]  Matrix embeddings for users and topics were sized at 250 ( INLINEFORM0 ); vector embeddings for users and topics were set to length 10 
We applied the LDA topic model BIBREF22 on the FBFans dataset to determine the latent topics with which to build topic embeddings, as there is only one general known topic: nuclear power plants  We learned 100 latent topics and assigned the top three topics for each post  For the CreateDebate dataset, which itself constitutes four topics, the topic labels for posts were used directly without additionally applying LDA 
For the FBFans data we report class-based f-scores as well as the macro-average f-score ( INLINEFORM0 ) shown in equation EQREF19   DISPLAYFORM0
where INLINEFORM0 and INLINEFORM1 are the average precision and recall of the three class  We adopted the macro-average f-score as the evaluation metric for the overall performance because (1) the experimental dataset is severely imbalanced, which is common for contentious issues; and (2) for stance classification, content in minor-class posts is usually more important for further applications  For the CreateDebate dataset, accuracy was adopted as the evaluation metric to compare the results with related work BIBREF7 , BIBREF9 , BIBREF12  
Baselines
We pit our model against the following baselines: 1) SVM with unigram, bigram, and trigram features, which is a standard yet rather strong classifier for text features; 2) SVM with average word embedding, where a document is represented as a continuous representation by averaging the embeddings of the composite words; 3) SVM with average transformed word embeddings (the INLINEFORM0 in equation EQREF6 ), where a document is represented as a continuous representation by averaging the transformed embeddings of the composite words; 4) two mature deep learning models on text classification, CNN BIBREF3 and Recurrent Convolutional Neural Networks (RCNN) BIBREF0 , where the hyperparameters are based on their work; 5) the above SVM and deep learning models with comment information; 6) UTCNN without user information, representing a pure-text CNN model where we use the same user matrix and user embeddings INLINEFORM1 and INLINEFORM2 for each user; 7) UTCNN without the LDA model, representing how UTCNN works with a single-topic dataset; 8) UTCNN without comments, in which the model predicts the stance label given only user and topic information  All these models were trained on the training set, and parameters as well as the SVM kernel selections (linear or RBF) were fine-tuned on the development set  Also, we adopt oversampling on SVMs, CNN and RCNN because the FBFans dataset is highly imbalanced 
Results on FBFans Dataset
In Table TABREF22 we show the results of UTCNN and the baselines on the FBFans dataset  Here Majority yields good performance on Neu since FBFans is highly biased to the neutral class  The SVM models perform well on Sup and Neu but perform poorly for Uns, showing that content information in itself is insufficient to predict stance labels, especially for the minor class  With the transformed word embedding feature, SVM can achieve comparable performance as SVM with n-gram feature  However, the much fewer feature dimension of the transformed word embedding makes SVM with word embeddings a more efficient choice for modeling the large scale social media dataset  For the CNN and RCNN models, they perform slightly better than most of the SVM models but still, the content information is insufficient to achieve a good performance on the Uns posts  As to adding comment information to these models, since the commenters do not always hold the same stance as the author, simply adding comments and post contents together merely adds noise to the model 
Among all UTCNN variations, we find that user information is most important, followed by topic and comment information  UTCNN without user information shows results similar to SVMs — it does well for Sup and Neu but detects no Uns  Its best f-scores on both Sup and Neu among all methods show that with enough training data, content-based models can perform well; at the same time, the lack of user information results in too few clues for minor-class posts to either predict their stance directly or link them to other users and posts for improved performance  The 17 5% improvement when adding user information suggests that user information is especially useful when the dataset is highly imbalanced  All models that consider user information predict the minority class successfully  UCTNN without topic information works well but achieves lower performance than the full UTCNN model  The 4 9% performance gain brought by LDA shows that although it is satisfactory for single topic datasets, adding that latent topics still benefits performance: even when we are discussing the same topic, we use different arguments and supporting evidence  Lastly, we get 4 8% improvement when adding comment information and it achieves comparable performance to UTCNN without topic information, which shows that comments also benefit performance  For platforms where user IDs are pixelated or otherwise hidden, adding comments to a text model still improves performance  In its integration of user, content, and comment information, the full UTCNN produces the highest f-scores on all Sup, Neu, and Uns stances among models that predict the Uns class, and the highest macro-average f-score overall  This shows its ability to balance a biased dataset and supports our claim that UTCNN successfully bridges content and user, topic, and comment information for stance classification on social media text  Another merit of UTCNN is that it does not require a balanced training data  This is supported by its outperforming other models though no oversampling technique is applied to the UTCNN related experiments as shown in this paper  Thus we can conclude that the user information provides strong clues and it is still rich even in the minority class 
We also investigate the semantic difference when a user acts as an author/liker or a commenter  We evaluated a variation in which all embeddings from the same user were forced to be identical (this is the UTCNN shared user embedding setting in Table TABREF22 )  This setting yielded only a 2 5% improvement over the model without comments, which is not statistically significant  However, when separating authors/likers and commenters embeddings (i e , the UTCNN full model), we achieved much greater improvements (4 8%)  We attribute this result to the tendency of users to use different wording for different roles (for instance author vs commenter)  This is observed when the user, acting as an author, attempts to support her argument against nuclear power by using improvements in solar power; when acting as a commenter, though, she interacts with post contents by criticizing past politicians who supported nuclear power or by arguing that the proposed evacuation plan in case of a nuclear accident is ridiculous  Based on this finding, in the final UTCNN setting we train two user matrix embeddings for one user: one for the author/liker role and the other for the commenter role 
Results on CreateDebate Dataset
Table TABREF24 shows the results of UTCNN, baselines as we implemented on the FBFans datset and related work on the CreateDebate dataset  We do not adopt oversampling on these models because the CreateDebate dataset is almost balanced  In previous work, integer linear programming (ILP) or linear-chain conditional random fields (CRFs) were proposed to integrate text features, author, ideology, and user-interaction constraints, where text features are unigram, bigram, and POS-dependencies; the author constraint tends to require that posts from the same author for the same topic hold the same stance; the ideology constraint aims to capture inferences between topics for the same author; the user-interaction constraint models relationships among posts via user interactions such as replies BIBREF7 , BIBREF9  
The SVM with n-gram or average word embedding feature performs just similar to the majority  However, with the transformed word embedding, it achieves superior results  It shows that the learned user and topic embeddings really capture the user and topic semantics  This finding is not so obvious in the FBFans dataset and it might be due to the unfavorable data skewness for SVM  As for CNN and RCNN, they perform slightly better than most SVMs as we found in Table TABREF22 for FBFans 
Compared to the ILP BIBREF7 and CRF BIBREF9 methods, the UTCNN user embeddings encode author and user-interaction constraints, where the ideology constraint is modeled by the topic embeddings and text features are modeled by the CNN  The significant improvement achieved by UTCNN suggests the latent representations are more effective than overt model constraints 
The PSL model BIBREF12 jointly labels both author and post stance using probabilistic soft logic (PSL) BIBREF23 by considering text features and reply links between authors and posts as in Hasan and Ng's work  Table TABREF24 reports the result of their best AD setting, which represents the full joint stance/disagreement collective model on posts and is hence more relevant to UTCNN  In contrast to their model, the UTCNN user embeddings represent relationships between authors, but UTCNN models do not utilize link information between posts  Though the PSL model has the advantage of being able to jointly label the stances of authors and posts, its performance on posts is lower than the that for the ILP or CRF models  UTCNN significantly outperforms these models on posts and has the potential to predict user stances through the generated user embeddings 
For the CreateDebate dataset, we also evaluated performance when not using topic embeddings or user embeddings; as replies in this dataset are viewed as posts, the setting without comment embeddings is not available  Table TABREF24 shows the same findings as Table TABREF22 : the 21% improvement in accuracy demonstrates that user information is the most vital  This finding also supports the results in the related work: user constraints are useful and can yield 11 2% improvement in accuracy BIBREF7   Further considering topic information yields 3 4% improvement, suggesting that knowing the subject of debates provides useful information  In sum, Table TABREF22 together with Table TABREF24 show that UTCNN achieves promising performance regardless of topic, language, data distribution, and platform 
Conclusion
We have proposed UTCNN, a neural network model that incorporates user, topic, content and comment information for stance classification on social media texts  UTCNN learns user embeddings for all users with minimum active degree, i e , one post or one like  Topic information obtained from the topic model or the pre-defined labels further improves the UTCNN model  In addition, comment information provides additional clues for stance classification  We have shown that UTCNN achieves promising and balanced results  In the future we plan to explore the effectiveness of the UTCNN user embeddings for author stance classification 
Acknowledgements
Research of this paper was partially supported by Ministry of Science and Technology, Taiwan, under the contract MOST 104-2221-E-001-024-MY2 

ground_truth: eight layers

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What cyberbulling topics did they address?
Context 4: 
Among the extractive evaluation metrics, CS, which also only relies on the reference, has poor correlation in the hyperparams category  RG-F, and both variants of the PARENT metric achieve the highest correlation for both settings  There is no significant difference among these for the hyperparams category, but for systems, PARENT-W is significantly better than the other two  While RG-F needs a full information extraction pipeline in its implementation, PARENT-C only relies on co-occurrence counts, and PARENT-W can be used out-of-the-box for any dataset  To our knowledge, this is the first rigorous evaluation of using information extraction for generation evaluation 
On this dataset, the word-overlap model showed higher correlation than the co-occurrence model for entailment  In § SECREF47 we will show that for the WebNLG dataset, where more paraphrasing is involved between the table and text, the opposite is true  Lastly, we note that the heuristic for selecting INLINEFORM0 is sufficient to produce high correlations for PARENT, however, if human annotations are available, this can be tuned to produce significantly higher correlations (PARENT*-W/C) 
Analysis
In this section we further analyze the performance of PARENT-W under different conditions, and compare to the other best metrics from Table TABREF37  
To study the correlation as we vary the number of divergent references, we also collected binary labels from workers for whether a reference is entailed by the corresponding table  We define a reference as entailed when it mentions only information which can be inferred from the table  Each table and reference pair was judged by 3 independent workers, and we used the majority vote as the label for that pair  Overall, only INLINEFORM0 of the references were labeled as entailed by the table  Fleiss' INLINEFORM1 was INLINEFORM2 , which indicates a fair agreement  We found the workers sometimes disagreed on what information can be reasonably entailed by the table 
Figure FIGREF40 shows the correlations as we vary the percent of entailed examples in the evaluation set of WikiBio  Each point is obtained by fixing the desired proportion of entailed examples, and sampling subsets from the full set which satisfy this proportion  PARENT and RG-F remain stable and show a high correlation across the entire range, whereas BLEU and BLEU-T vary a lot  In the hyperparams category, the latter two have the worst correlation when the evaluation set contains only entailed examples, which may seem surprising  However, on closer examination we found that this subset tends to omit a lot of information from the tables  Systems which produce more information than these references are penalized by BLEU, but not in the human evaluation  PARENT overcomes this issue by measuring recall against the table in addition to the reference 
We check how different components in the computation of PARENT contribute to its correlation to human judgments  Specifically, we remove the probability INLINEFORM0 of an n-gram INLINEFORM1 being entailed by the table from Eqs  EQREF19 and EQREF23   The average correlation for PARENT-W drops to INLINEFORM5 in this case  We also try a variant of PARENT with INLINEFORM6 , which removes the contribution of Table Recall (Eq  EQREF22 )  The average correlation is INLINEFORM7 in this case  With these components, the correlation is INLINEFORM8 , showing that they are crucial to the performance of PARENT 
BIBREF28 point out that hill-climbing on an automatic metric is meaningless if that metric has a low instance-level correlation to human judgments  In Table TABREF46 we show the average accuracy of the metrics in making the same judgments as humans between pairs of generated texts  Both variants of PARENT are significantly better than the other metrics, however the best accuracy is only INLINEFORM0 for the binary task  This is a challenging task, since there are typically only subtle differences between the texts  Achieving higher instance-level accuracies will require more sophisticated language understanding models for evaluation 
WebNLG Dataset
To check how PARENT correlates with human judgments when the references are elicited from humans (and less likely to be divergent), we check its correlation with the human ratings provided for the systems competing in the WebNLG challenge BIBREF6   The task is to generate text describing 1-5 RDF triples (e g  John E Blaha, birthPlace, San Antonio), and human ratings were collected for the outputs of 9 participating systems on 223 instances  These systems include a mix of pipelined, statistical and neural methods  Each instance has upto 3 reference texts associated with the RDF triples, which we use for evaluation 
The human ratings were collected on 3 distinct aspects – grammaticality, fluency and semantics, where semantics corresponds to the degree to which a generated text agrees with the meaning of the underlying RDF triples  We report the correlation of several metrics with these ratings in Table TABREF48   Both variants of PARENT are either competitive or better than the other metrics in terms of the average correlation to all three aspects  This shows that PARENT is applicable for high quality references as well 
While BLEU has the highest correlation for the grammar and fluency aspects, PARENT does best for semantics  This suggests that the inclusion of source tables into the evaluation orients the metric more towards measuring the fidelity of the content of the generation  A similar trend is seen comparing BLEU and BLEU-T  As modern neural text generation systems are typically very fluent, measuring their fidelity is of increasing importance  Between the two entailment models, PARENT-C is better due to its higher correlation with the grammaticality and fluency aspects 
The INLINEFORM0 parameter in the calculation of PARENT decides whether to compute recall against the table or the reference (Eq  EQREF22 )  Figure FIGREF50 shows the distribution of the values taken by INLINEFORM1 using the heuristic described in § SECREF3 for instances in both WikiBio and WebNLG  For WikiBio, the recall of the references against the table is generally low, and hence the recall of the generated text relies more on the table  For WebNLG, where the references are elicited from humans, this recall is much higher (often INLINEFORM2 ), and hence the recall of the generated text relies more on the reference 
Related Work
Over the years several studies have evaluated automatic metrics for measuring text generation performance BIBREF29 , BIBREF30 , BIBREF31 , BIBREF32 , BIBREF33 , BIBREF34 , BIBREF35   The only consensus from these studies seems to be that no single metric is suitable across all tasks  A recurring theme is that metrics like BLEU and NIST BIBREF36 are not suitable for judging content quality in NLG  Recently, BIBREF37 did a comprehensive study of several metrics on the outputs of state-of-the-art NLG systems, and found that while they showed acceptable correlation with human judgments at the system level, they failed to show any correlation at the sentence level  Ours is the first study which checks the quality of metrics when table-to-text references are divergent  We show that in this case even system level correlations can be unreliable 
Hallucination BIBREF38 , BIBREF39 refers to when an NLG system generates text which mentions extra information than what is present in the source from which it is generated  Divergence can be viewed as hallucination in the reference text itself  PARENT deals with hallucination by discounting n-grams which do not overlap with either the reference or the table 
PARENT draws inspiration from iBLEU BIBREF26 , a metric for evaluating paraphrase generation, which compares the generated text to both the source text and the reference  While iBLEU penalizes texts which match the source, here we reward such texts since our task values accuracy of generated text more than the need for paraphrasing the tabular content BIBREF40   Similar to SARI for text simplification BIBREF41 and Q-BLEU for question generation BIBREF42 , PARENT falls under the category of task-specific metrics 
Conclusions
We study the automatic evaluation of table-to-text systems when the references diverge from the table  We propose a new metric, PARENT, which shows the highest correlation with humans across a range of settings with divergent references in WikiBio  We also perform the first empirical evaluation of information extraction based metrics BIBREF1 , and find RG-F to be effective  Lastly, we show that PARENT is comparable to the best existing metrics when references are elicited by humans on the WebNLG data 
Acknowledgements
Bhuwan Dhingra is supported by a fellowship from Siemens, and by grants from Google  We thank Maruan Al-Shedivat, Ian Tenney, Tom Kwiatkowski, Michael Collins, Slav Petrov, Jason Baldridge, David Reitter and other members of the Google AI Language team for helpful discussions and suggestions  We thank Sam Wiseman for sharing data for an earlier version of this paper  We also thank the anonymous reviewers for their feedback 
Information Extraction System
For evaluation via information extraction BIBREF1 we train a model for WikiBio which accepts text as input and generates a table as the output  Tables in WikiBio are open-domain, without any fixed schema for which attributes may be present or absent in an instance  Hence we employ the Pointer-Generator Network (PG-Net) BIBREF19 for this purpose  Specifically, we use a sequence-to-sequence model, whose encoder and decoder are both single-layer bi-directional LSTMs  The decoder is augmented with an attention mechanism over the states of the encoder  Further, it also uses a copy mechanism to optionally copy tokens directly from the source text  We do not use the coverage mechanism of BIBREF19 since that is specific to the task of summarization they study  The decoder is trained to produce a linearized version of the table where the rows and columns are flattened into a sequence, and separate by special tokens  Figure FIGREF53 shows an example 
Clearly, since the references are divergent, the model cannot be expected to produce the entire table, and we see some false information being hallucinated after training  Nevertheless, as we show in § SECREF36 , this system can be used for evaluating generated texts  After training, we can parse the output sequence along the special tokens INLINEFORM0 R INLINEFORM1 and INLINEFORM2 C INLINEFORM3 to get a set of (attribute, value) pairs  Table TABREF54 shows the precision, recall and F-score of these extracted pairs against the ground truth tables, where the attributes and values are compared using an exact string match 
Hyperparameters
After tuning we found the same set of hyperparameters to work well for both the table-to-text PG-Net, and the inverse information extraction PG-Net  The hidden state size of the biLSTMs was set to 200  The input and output vocabularies were set to 50000 most common words in the corpus, with additional special symbols for table attribute names (such as “birth-date”)  The embeddings of the tokens in the vocabulary were initialized with Glove BIBREF43   Learning rate of INLINEFORM0 was used during training, with the Adam optimizer, and a dropout of INLINEFORM1 was also applied to the outputs of the biLSTM  Models were trained till the loss on the dev set stopped dropping  Maximum length of a decoded text was set to 40 tokens, and that of the tables was set to 120 tokens  Various beam sizes and length normalization penalties were applied for the table-to-text system, which are listed in the main paper  For the information extraction system, we found a beam size of 8 and no length penalty to produce the highest F-score on the dev set 
Sample Outputs
Table TABREF55 shows some sample references and the corresponding predictions from the best performing model, PG-Net for WikiBio 

ground_truth: personal attack, racism, and sexism

racism, sexism, personal attack, not specifically about any single topic