prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What was their performance on emotion detection?
Context 1: Introduction
This work is licenced under a Creative Commons Attribution 4 0 International Licence  Licence details: http://creativecommons org/licenses/by/4 0/
In the spirit of the brevity of social media's messages and reactions, people have got used to express feelings minimally and symbolically, as with hashtags on Twitter and Instagram  On Facebook, people tend to be more wordy, but posts normally receive more simple “likes” than longer comments  Since February 2016, Facebook users can express specific emotions in response to a post thanks to the newly introduced reaction feature (see Section SECREF2 ), so that now a post can be wordlessly marked with an expression of say “joy" or “surprise" rather than a generic “like” 
It has been observed that this new feature helps Facebook to know much more about their users and exploit this information for targeted advertising BIBREF0 , but interest in people's opinions and how they feel isn't limited to commercial reasons, as it invests social monitoring, too, including health care and education BIBREF1   However, emotions and opinions are not always expressed this explicitly, so that there is high interest in developing systems towards their automatic detection  Creating manually annotated datasets large enough to train supervised models is not only costly, but also—especially in the case of opinions and emotions—difficult, due to the intrinsic subjectivity of the task BIBREF2 , BIBREF3   Therefore, research has focused on unsupervised methods enriched with information derived from lexica, which are manually created BIBREF3 , BIBREF4   Since go2009twitter have shown that happy and sad emoticons can be successfully used as signals for sentiment labels, distant supervision, i e  using some reasonably safe signals as proxies for automatically labelling training data BIBREF5 , has been used also for emotion recognition, for example exploiting both emoticons and Twitter hashtags BIBREF6 , but mainly towards creating emotion lexica  mohammad2015using use hashtags, experimenting also with highly fine-grained emotion sets (up to almost 600 emotion labels), to create the large Hashtag Emotion Lexicon  Emoticons are used as proxies also by hallsmarmulti, who use distributed vector representations to find which words are interchangeable with emoticons but also which emoticons are used in a similar context 
We take advantage of distant supervision by using Facebook reactions as proxies for emotion labels, which to the best of our knowledge hasn't been done yet, and we train a set of Support Vector Machine models for emotion recognition  Our models, differently from existing ones, exploit information which is acquired entirely automatically, and achieve competitive or even state-of-the-art results for some of the emotion labels on existing, standard evaluation datasets  For explanatory purposes, related work is discussed further and more in detail when we describe the benchmarks for evaluation (Section SECREF3 ) and when we compare our models to existing ones (Section SECREF5 )  We also explore and discuss how choosing different sets of Facebook pages as training data provides an intrinsic domain-adaptation method 
Facebook reactions as labels
For years, on Facebook people could leave comments to posts, and also “like” them, by using a thumbs-up feature to explicitly express a generic, rather underspecified, approval  A “like” could thus mean “I like what you said", but also “I like that you bring up such topic (though I find the content of the article you linked annoying)" 
In February 2016, after a short trial, Facebook made a more explicit reaction feature available world-wide  Rather than allowing for the underspecified “like” as the only wordless response to a post, a set of six more specific reactions was introduced, as shown in Figure FIGREF1 : Like, Love, Haha, Wow, Sad and Angry  We use such reactions as proxies for emotion labels associated to posts 
We collected Facebook posts and their corresponding reactions from public pages using the Facebook API, which we accessed via the Facebook-sdk python library  We chose different pages (and therefore domains and stances), aiming at a balanced and varied dataset, but we did so mainly based on intuition (see Section SECREF4 ) and with an eye to the nature of the datasets available for evaluation (see Section SECREF5 )  The choice of which pages to select posts from is far from trivial, and we believe this is actually an interesting aspect of our approach, as by using different Facebook pages one can intrinsically tackle the domain-adaptation problem (See Section SECREF6 for further discussion on this)  The final collection of Facebook pages for the experiments described in this paper is as follows: FoxNews, CNN, ESPN, New York Times, Time magazine, Huffington Post Weird News, The Guardian, Cartoon Network, Cooking Light, Home Cooking Adventure, Justin Bieber, Nickelodeon, Spongebob, Disney 
Note that thankful was only available during specific time spans related to certain events, as Mother's Day in May 2016 
For each page, we downloaded the latest 1000 posts, or the maximum available if there are fewer, from February 2016, retrieving the counts of reactions for each post  The output is a JSON file containing a list of dictionaries with a timestamp, the post and a reaction vector with frequency values, which indicate how many users used that reaction in response to the post (Figure FIGREF3 )  The resulting emotion vectors must then be turned into an emotion label 
In the context of this experiment, we made the simple decision of associating to each post the emotion with the highest count, ignoring like as it is the default and most generic reaction people tend to use  Therefore, for example, to the first post in Figure FIGREF3 , we would associate the label sad, as it has the highest score (284) among the meaningful emotions we consider, though it also has non-zero scores for other emotions  At this stage, we didn't perform any other entropy-based selection of posts, to be investigated in future work 
Emotion datasets
Three datasets annotated with emotions are commonly used for the development and evaluation of emotion detection systems, namely the Affective Text dataset, the Fairy Tales dataset, and the ISEAR dataset  In order to compare our performance to state-of-the-art results, we have used them as well  In this Section, in addition to a description of each dataset, we provide an overview of the emotions used, their distribution, and how we mapped them to those we obtained from Facebook posts in Section SECREF7   A summary is provided in Table TABREF8 , which also shows, in the bottom row, what role each dataset has in our experiments: apart from the development portion of the Affective Text, which we used to develop our models (Section SECREF4 ), all three have been used as benchmarks for our evaluation 
Affective Text dataset
Task 14 at SemEval 2007 BIBREF7 was concerned with the classification of emotions and valence in news headlines  The headlines where collected from several news websites including Google news, The New York Times, BBC News and CNN  The used emotion labels were Anger, Disgust, Fear, Joy, Sadness, Surprise, in line with the six basic emotions of Ekman's standard model BIBREF8   Valence was to be determined as positive or negative  Classification of emotion and valence were treated as separate tasks  Emotion labels were not considered as mututally exclusive, and each emotion was assigned a score from 0 to 100  Training/developing data amounted to 250 annotated headlines (Affective development), while systems were evaluated on another 1000 (Affective test)  Evaluation was done using two different methods: a fine-grained evaluation using Pearson's r to measure the correlation between the system scores and the gold standard; and a coarse-grained method where each emotion score was converted to a binary label, and precision, recall, and f-score were computed to assess performance  As it is done in most works that use this dataset BIBREF3 , BIBREF4 , BIBREF9 , we also treat this as a classification problem (coarse-grained)  This dataset has been extensively used for the evaluation of various unsupervised methods BIBREF2 , but also for testing different supervised learning techniques and feature portability BIBREF10  
Fairy Tales dataset
This is a dataset collected by alm2008affect, where about 1,000 sentences from fairy tales (by B  Potter, H C  Andersen and Grimm) were annotated with the same six emotions of the Affective Text dataset, though with different names: Angry, Disgusted, Fearful, Happy, Sad, and Surprised  In most works that use this dataset BIBREF3 , BIBREF4 , BIBREF9 , only sentences where all annotators agreed are used, and the labels angry and disgusted are merged  We adopt the same choices 
ISEAR
The ISEAR (International Survey on Emotion Antecedents and Reactions BIBREF11 , BIBREF12 ) is a dataset created in the context of a psychology project of the 1990s, by collecting questionnaires answered by people with different cultural backgrounds  The main aim of this project was to gather insights in cross-cultural aspects of emotional reactions  Student respondents, both psychologists and non-psychologists, were asked to report situations in which they had experienced all of seven major emotions (joy, fear, anger, sadness, disgust, shame and guilt)  In each case, the questions covered the way they had appraised a given situation and how they reacted  The final dataset contains reports by approximately 3000 respondents from all over the world, for a total of 7665 sentences labelled with an emotion, making this the largest dataset out of the three we use 
Overview of datasets and emotions
We summarise datasets and emotion distribution from two viewpoints  First, because there are different sets of emotions labels in the datasets and Facebook data, we need to provide a mapping and derive a subset of emotions that we are going to use for the experiments  This is shown in Table TABREF8 , where in the “Mapped” column we report the final emotions we use in this paper: anger, joy, sadness, surprise  All labels in each dataset are mapped to these final emotions, which are therefore the labels we use for training and testing our models 
Second, the distribution of the emotions for each dataset is different, as can be seen in Figure FIGREF9  
In Figure FIGREF9 we also provide the distribution of the emotions anger, joy, sadness, surprise per Facebook page, in terms of number of posts (recall that we assign to a post the label corresponding to the majority emotion associated to it, see Section SECREF2 )  We can observe that for example pages about news tend to have more sadness and anger posts, while pages about cooking and tv-shows have a high percentage of joy posts  We will use this information to find the best set of pages for a given target domain (see Section SECREF5 ) 
Model
There are two main decisions to be taken in developing our model: (i) which Facebook pages to select as training data, and (ii) which features to use to train the model, which we discuss below  Specifically, we first set on a subset of pages and then experiment with features  Further exploration of the interaction between choice of pages and choice of features is left to future work, and partly discussed in Section SECREF6   For development, we use a small portion of the Affective data set described in Section SECREF4 , that is the portion that had been released as development set for SemEval's 2007 Task 14 BIBREF7 , which contains 250 annotated sentences (Affective development, Section SECREF4 )  All results reported in this section are on this dataset  The test set of Task 14 as well as the other two datasets described in Section SECREF3 will be used to evaluate the final models (Section SECREF4 ) 
Selecting Facebook pages
Although page selection is a crucial ingredient of this approach, which we believe calls for further and deeper, dedicated investigation, for the experiments described here we took a rather simple approach  First, we selected the pages that would provide training data based on intuition and availability, then chose different combinations according to results of a basic model run on development data, and eventually tested feature combinations, still on the development set 
For the sake of simplicity and transparency, we first trained an SVM with a simple bag-of-words model and default parameters as per the Scikit-learn implementation BIBREF13 on different combinations of pages  Based on results of the attempted combinations as well as on the distribution of emotions in the development dataset (Figure FIGREF9 ), we selected a best model (B-M), namely the combined set of Time, The Guardian and Disney, which yields the highest results on development data  Time and The Guardian perform well on most emotions but Disney helps to boost the performance for the Joy class 
Features
In selecting appropriate features, we mainly relied on previous work and intuition  We experimented with different combinations, and all tests were still done on Affective development, using the pages for the best model (B-M) described above as training data  Results are in Table TABREF20   Future work will further explore the simultaneous selection of features and page combinations 
We use a set of basic text-based features to capture the emotion class  These include a tf-idf bag-of-words feature, word (2-3) and character (2-5) ngrams, and features related to the presence of negation words, and to the usage of punctuation 
This feature is used in all unsupervised models as a source of information, and we mainly include it to assess its contribution, but eventually do not use it in our final model 
We used the NRC10 Lexicon because it performed best in the experiments by BIBREF10 , which is built around the emotions anger, anticipation, disgust, fear, joy, sadness, and surprise, and the valence values positive and negative  For each word in the lexicon, a boolean value indicating presence or absence is associated to each emotion  For a whole sentence, a global score per emotion can be obtained by summing the vectors for all content words of that sentence included in the lexicon, and used as feature 
As additional feature, we also included Word Embeddings, namely distributed representations of words in a vector space, which have been exceptionally successful in boosting performance in a plethora of NLP tasks  We use three different embeddings:
Google embeddings: pre-trained embeddings trained on Google News and obtained with the skip-gram architecture described in BIBREF14   This model contains 300-dimensional vectors for 3 million words and phrases 
Facebook embeddings: embeddings that we trained on our scraped Facebook pages for a total of 20,000 sentences  Using the gensim library BIBREF15 , we trained the embeddings with the following parameters: window size of 5, learning rate of 0 01 and dimensionality of 100  We filtered out words with frequency lower than 2 occurrences 
Retrofitted embeddings: Retrofitting BIBREF16 has been shown as a simple but efficient way of informing trained embeddings with additional information derived from some lexical resource, rather than including it directly at the training stage, as it's done for example to create sense-aware BIBREF17 or sentiment-aware BIBREF18 embeddings  In this work, we retrofit general embeddings to include information about emotions, so that emotion-similar words can get closer in space  Both the Google as well as our Facebook embeddings were retrofitted with lexical information obtained from the NRC10 Lexicon mentioned above, which provides emotion-similarity for each token  Note that differently from the previous two types of embeddings, the retrofitted ones do rely on handcrafted information in the form of a lexical resource 
Results on development set
We report precision, recall, and f-score on the development set  The average f-score is reported as micro-average, to better account for the skewed distribution of the classes as well as in accordance to what is usually reported for this task BIBREF19  
From Table TABREF20 we draw three main observations  First, a simple tf-idf bag-of-word mode works already very well, to the point that the other textual and lexicon-based features don't seem to contribute to the overall f-score (0 368), although there is a rather substantial variation of scores per class  Second, Google embeddings perform a lot better than Facebook embeddings, and this is likely due to the size of the corpus used for training  Retrofitting doesn't seem to help at all for the Google embeddings, but it does boost the Facebook embeddings, leading to think that with little data, more accurate task-related information is helping, but corpus size matters most  Third, in combination with embeddings, all features work better than just using tf-idf, but removing the Lexicon feature, which is the only one based on hand-crafted resources, yields even better results  Then our best model (B-M) on development data relies entirely on automatically obtained information, both in terms of training data as well as features 
Results
In Table TABREF26 we report the results of our model on the three datasets standardly used for the evaluation of emotion classification, which we have described in Section SECREF3  
Our B-M model relies on subsets of Facebook pages for training, which were chosen according to their performance on the development set as well as on the observation of emotions distribution on different pages and in the different datasets, as described in Section SECREF4   The feature set we use is our best on the development set, namely all the features plus Google-based embeddings, but excluding the lexicon  This makes our approach completely independent of any manual annotation or handcrafted resource  Our model's performance is compared to the following systems, for which results are reported in the referred literature  Please note that no other existing model was re-implemented, and results are those reported in the respective papers 
Discussion, conclusions and future work
We have explored the potential of using Facebook reactions in a distant supervised setting to perform emotion classification  The evaluation on standard benchmarks shows that models trained as such, especially when enhanced with continuous vector representations, can achieve competitive results without relying on any handcrafted resource  An interesting aspect of our approach is the view to domain adaptation via the selection of Facebook pages to be used as training data 
We believe that this approach has a lot of potential, and we see the following directions for improvement  Feature-wise, we want to train emotion-aware embeddings, in the vein of work by tang:14, and iacobacci2015sensembed  Retrofitting FB-embeddings trained on a larger corpus might also be successful, but would rely on an external lexicon 
The largest room for yielding not only better results but also interesting insights on extensions of this approach lies in the choice of training instances, both in terms of Facebook pages to get posts from, as well as in which posts to select from the given pages  For the latter, one could for example only select posts that have a certain length, ignore posts that are only quotes or captions to images, or expand posts by including content from linked html pages, which might provide larger and better contexts BIBREF23   Additionally, and most importantly, one could use an entropy-based measure to select only posts that have a strong emotion rather than just considering the majority emotion as training label  For the former, namely the choice of Facebook pages, which we believe deserves the most investigation, one could explore several avenues, especially in relation to stance-based issues BIBREF24   In our dataset, for example, a post about Chile beating Colombia in a football match during the Copa America had very contradictory reactions, depending on which side readers would cheer for  Similarly, the very same political event, for example, would get very different reactions from readers if it was posted on Fox News or The Late Night Show, as the target audience is likely to feel very differently about the same issue  This also brings up theoretical issues related more generally to the definition of the emotion detection task, as it's strongly dependent on personal traits of the audience  Also, in this work, pages initially selected on availability and intuition were further grouped into sets to make training data according to performance on development data, and label distribution  Another criterion to be exploited would be vocabulary overlap between the pages and the datasets 
Lastly, we could develop single models for each emotion, treating the problem as a multi-label task  This would even better reflect the ambiguity and subjectivity intrinsic to assigning emotions to text, where content could be at same time joyful or sad, depending on the reader 
Acknowledgements
In addition to the anonymous reviewers, we want to thank Lucia Passaro and Barbara Plank for insightful discussions, and for providing comments on draft versions of this paper 
Context 2: Introduction
The irony is a kind of figurative language, which is widely used on social media BIBREF0   The irony is defined as a clash between the intended meaning of a sentence and its literal meaning BIBREF1   As an important aspect of language, irony plays an essential role in sentiment analysis BIBREF2 , BIBREF0 and opinion mining BIBREF3 , BIBREF4  
Although some previous studies focus on irony detection, little attention is paid to irony generation  As ironies can strengthen sentiments and express stronger emotions, we mainly focus on generating ironic sentences  Given a non-ironic sentence, we implement a neural network to transfer it to an ironic sentence and constrain the sentiment polarity of the two sentences to be the same  For example, the input is “I hate it when my plans get ruined" which is negative in sentiment polarity and the output should be ironic and negative in sentiment as well, such as “I like it when my plans get ruined"  The speaker uses “like" to be ironic and express his or her negative sentiment  At the same time, our model can preserve contents which are irrelevant to sentiment polarity and irony  According to the categories mentioned in BIBREF5 , irony can be classified into 3 classes: verbal irony by means of a polarity contrast, the sentences containing expression whose polarity is inverted between the intended and the literal evaluation; other types of verbal irony, the sentences that show no polarity contrast between the literal and intended meaning but are still ironic; and situational irony, the sentences that describe situations that fail to meet some expectations  As ironies in the latter two categories are obscure and hard to understand, we decide to only focus on ironies in the first category in this work  For example, our work can be specifically described as: given a sentence “I hate to be ignored", we train our model to generate an ironic sentence such as “I love to be ignored"  Although there is “love" in the generated sentence, the speaker still expresses his or her negative sentiment by irony  We also make some explorations in the transformation from ironic sentences to non-ironic sentences at the end of our work  Because of the lack of previous work and baselines on irony generation, we implement our model based on style transfer  Our work will not only provide the first large-scale irony dataset but also make our model as a benchmark for the irony generation 
Recently, unsupervised style transfer becomes a very popular topic  Many state-of-the-art studies try to solve the task with sequence-to-sequence (seq2seq) framework  There are three main ways to build up models  The first is to learn a latent style-independent content representation and generate sentences with the content representation and another style BIBREF6 , BIBREF7   The second is to directly transfer sentences from one style to another under the control of classifiers and reinforcement learning BIBREF8   The third is to remove style attribute words from the input sentence and combine the remaining content with new style attribute words BIBREF9 , BIBREF10   The first method usually obtains better performances via adversarial training with discriminators  The style-independent content representation, nevertheless, is not easily obtained BIBREF11 , which results in poor performances  The second method is suitable for complex styles which are difficult to model and describe  The model can learn the deep semantic features by itself but sometimes the model is sensitive to parameters and hard to train  The third method succeeds to preserve content but cannot work for some complex styles such as democratic and republican  Sentences with those styles usually do not have specific style attribute words  Unfortunately, due to the lack of large irony dataset and difficulties of modeling ironies, there has been little work trying to generate ironies based on seq2seq framework as far as we know  Inspired by methods for style transfer, we decide to implement a specifically designed model based on unsupervised style transfer to explore irony generation 
In this paper, in order to address the lack of irony data, we first crawl over 2M tweets from twitter to build a dataset with 262,755 ironic and 112,330 non-ironic tweets  Then, due to the lack of parallel data, we propose a novel model to transfer non-ironic sentences to ironic sentences in an unsupervised way  As ironic style is hard to model and describe, we implement our model with the control of classifiers and reinforcement learning  Different from other studies in style transfer, the transformation from non-ironic to ironic sentences has to preserve sentiment polarity as mentioned above  Therefore, we not only design an irony reward to control the irony accuracy and implement denoising auto-encoder and back-translation to control content preservation but also design a sentiment reward to control sentiment preservation 
Experimental results demonstrate that our model achieves a high irony accuracy with well-preserved sentiment and content  The contributions of our work are as follows:
Related Work
Style Transfer: As irony is a complicated style and hard to model with some specific style attribute words, we mainly focus on studies without editing style attribute words 
Some studies are trying to disentangle style representation from content representation  In BIBREF12 , authors leverage adversarial networks to learn separate content representations and style representations  In BIBREF13 and BIBREF6 , researchers combine variational auto-encoders (VAEs) with style discriminators 
However, some recent studies BIBREF11 reveal that the disentanglement of content and style representations may not be achieved in practice  Therefore, some other research studies BIBREF9 , BIBREF10 strive to separate content and style by removing stylistic words  Nonetheless, many non-ironic sentences do not have specific stylistic words and as a result, we find it difficult to transfer non-ironic sentences to ironic sentences through this way in practice 
Besides, some other research studies do not disentangle style from content but directly learn representations of sentences  In BIBREF8 , authors propose a dual reinforcement learning framework without separating content and style representations  In BIBREF7 , researchers utilize a machine translation model to learn a sentence representation preserving the meaning of the sentence but reducing stylistic properties  In this method, the quality of generated sentences relies on the performance of classifiers to a large extent  Meanwhile, such models are usually sensitive to parameters and difficult to train  In contrast, we combine a pre-training process with reinforcement learning to build up a stable language model and design special rewards for our task 
Irony Detection: With the development of social media, irony detection becomes a more important task  Methods for irony detection can be mainly divided into two categories: methods based on feature engineering and methods based on neural networks 
As for methods based on feature engineering, In BIBREF1 , authors investigate pragmatic phenomena and various irony markers  In BIBREF14 , researchers leverage a combination of sentiment, distributional semantic and text surface features  Those models rely on hand-crafted features and are hard to implement 
When it comes to methods based on neural networks, long short-term memory (LSTM) BIBREF15 network is widely used and is very efficient for irony detection  In BIBREF16 , a tweet is divided into two segments and a subtract layer is implemented to calculate the difference between two segments in order to determine whether the tweet is ironic  In BIBREF17 , authors utilize a recurrent neural network with Bi-LSTM and self-attention without hand-crafted features  In BIBREF18 , researchers propose a system based on a densely connected LSTM network 
Our Dataset
In this section, we describe how we build our dataset with tweets  First, we crawl over 2M tweets from twitter using GetOldTweets-python  We crawl English tweets from 04/09/2012 to /12/18/2018  We first remove all re-tweets and use langdetect to remove all non-English sentences  Then, we remove hashtags attached at the end of the tweets because they are usually not parts of sentences and will confuse our language model  After that, we utilize Ekphrasis to process tweets  We remove URLs and restore remaining hashtags, elongated words, repeated words, and all-capitalized words  To simplify our dataset, We replace all “ INLINEFORM0 money INLINEFORM1 " and “ INLINEFORM2 time INLINEFORM3 " tokens with “ INLINEFORM4 number INLINEFORM5 " token when using Ekphrasis  And we delete sentences whose lengths are less than 10 or greater than 40  In order to restore abbreviations, we download an abbreviation dictionary from webopedia and restore abbreviations to normal words or phrases according to the dictionary  Finally, we remove sentences which have more than two rare words (appearing less than three times) in order to constrain the size of vocabulary  Finally, we get 662,530 sentences after pre-processing 
As neural networks are proved effective in irony detection, we decide to implement a neural classifier in order to classify the sentences into ironic and non-ironic sentences  However, the only high-quality irony dataset we can obtain is the dataset of Semeval-2018 Task 3 and the dataset is pretty small, which will cause overfitting to complex models  Therefore, we just implement a simple one-layer RNN with LSTM cell to classify pre-processed sentences into ironic sentences and non-ironic sentences because LSTM networks are widely used in irony detection  We train the model with the dataset of Semeval-2018 Task 3  After classification, we get 262,755 ironic sentences and 399,775 non-ironic sentences  According to our observation, not all non-ironic sentences are suitable to be transferred into ironic sentences  For example, “just hanging out   watching   is it monday yet" is hard to transfer because it does not have an explicit sentiment polarity  So we remove all interrogative sentences from the non-ironic sentences and only obtain the sentences which have words expressing strong sentiments  We evaluate the sentiment polarity of each word with TextBlob and we view those words with sentiment scores greater than 0 5 or less than -0 5 as words expressing strong sentiments  Finally, we build our irony dataset with 262,755 ironic sentences and 102,330 non-ironic sentences 
[t] Irony Generation Algorithm
INLINEFORM0 pre-train with auto-encoder Pre-train INLINEFORM1 , INLINEFORM2 with INLINEFORM3 using MLE based on Eq  EQREF16 Pre-train INLINEFORM4 , INLINEFORM5 with INLINEFORM6 using MLE based on Eq  EQREF17 INLINEFORM7 pre-train with back-translation Pre-train INLINEFORM8 , INLINEFORM9 , INLINEFORM10 , INLINEFORM11 with INLINEFORM12 using MLE based on Eq  EQREF19 Pre-train INLINEFORM13 , INLINEFORM14 , INLINEFORM15 , INLINEFORM16 with INLINEFORM17 using MLE based on Eq  EQREF20
INLINEFORM0 train with RL each epoch e = 1, 2,    , INLINEFORM1 INLINEFORM2 train non-irony2irony with RL INLINEFORM3 in N INLINEFORM4 update INLINEFORM5 , INLINEFORM6 , using INLINEFORM7 based on Eq  EQREF29 INLINEFORM8 back-translation INLINEFORM9 INLINEFORM10 INLINEFORM11 update INLINEFORM12 , INLINEFORM13 , INLINEFORM14 , INLINEFORM15 using MLE based on Eq  EQREF19 INLINEFORM16 train irony2non-irony with RL INLINEFORM17 in I INLINEFORM18 update INLINEFORM19 , INLINEFORM20 , using INLINEFORM21 similar to Eq  EQREF29 INLINEFORM22 back-translation INLINEFORM23 INLINEFORM24 INLINEFORM25 update INLINEFORM26 , INLINEFORM27 , INLINEFORM28 , INLINEFORM29 using MLE based on Eq  EQREF20
Our Method
Given two non-parallel corpora: non-ironic corpus N={ INLINEFORM0 , INLINEFORM1 ,    , INLINEFORM2 } and ironic corpus I={ INLINEFORM3 , INLINEFORM4 ,    , INLINEFORM5 }, the goal of our irony generation model is to generate an ironic sentence from a non-ironic sentence while preserving the content and sentiment polarity of the source input sentence  We implement an encoder-decoder framework where two encoders are utilized to encode ironic sentences and non-ironic sentences respectively and two decoders are utilized to decode ironic sentences and non-ironic sentences from latent representations respectively  In order to enforce a shared latent space, we share two layers on both the encoder side and the decoder side  Our model architecture is illustrated in Figure FIGREF13   We denote irony encoder as INLINEFORM6 , irony decoder as INLINEFORM7 and non-irony encoder as INLINEFORM8 , non-irony decoder as INLINEFORM9   Their parameters are INLINEFORM10 , INLINEFORM11 , INLINEFORM12 and INLINEFORM13  
Our irony generation algorithm is shown in Algorithm SECREF3   We first pre-train our model using denoising auto-encoder and back-translation to build up language models for both styles (section SECREF14 )  Then we implement reinforcement learning to train the model to transfer sentences from one style to another (section SECREF21 )  Meanwhile, to achieve content preservation, we utilize back-translation for one time in every INLINEFORM0 time steps 
Pretraining
In order to build up our language model and preserve the content, we apply the auto-encoder model  To prevent the model from simply copying the input sentence, we randomly add some noises in the input sentence
Context 3: Introduction
The concept of message passing over graphs has been around for many years BIBREF0, BIBREF1, as well as that of graph neural networks (GNNs) BIBREF2, BIBREF3  However, GNNs have only recently started to be closely investigated, following the advent of deep learning  Some notable examples include BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10, BIBREF11, BIBREF12  These approaches are known as spectral  Their similarity with message passing (MP) was observed by BIBREF9 and formalized by BIBREF13 and BIBREF14 
The MP framework is based on the core idea of recursive neighborhood aggregation  That is, at every iteration, the representation of each vertex is updated based on messages received from its neighbors  All spectral GNNs can be described in terms of the MP framework 
GNNs have been applied with great success to bioinformatics and social network data, for node classification, link prediction, and graph classification  However, a few studies only have focused on the application of the MP framework to representation learning on text  This paper proposes one such application  More precisely, we represent documents as word co-occurrence networks, and develop an expressive MP GNN tailored to document understanding, the Message Passing Attention network for Document understanding (MPAD)  We also propose several hierarchical variants of MPAD  Evaluation on 10 document classification datasets show that our architectures learn representations that are competitive with the state-of-the-art  Furthermore, ablation experiments shed light on the impact of various architectural choices 
In what follows, we first provide some background about the MP framework (in sec  SECREF2), thoroughly describe and explain MPAD (sec  SECREF3), present our experimental framework (sec  SECREF4), report and interpret our results (sec  SECREF5), and provide a review of the relevant literature (sec  SECREF6) 
Message Passing Neural Networks
BIBREF13 proposed a MP framework under which many of the recently introduced GNNs can be reformulated  MP consists in an aggregation phase followed by a combination phase BIBREF14  More precisely, let $G(V,E)$ be a graph, and let us consider $v \in V$  At time $t+1$, a message vector $\mathbf {m}_v^{t+1}$ is computed from the representations of the neighbors $\mathcal {N}(v)$ of $v$:
The new representation $\mathbf {h}^{t+1}_v$ of $v$ is then computed by combining its current feature vector $\mathbf {h}^{t}_v$ with the message vector $\mathbf {m}_v^{t+1}$:
Messages are passed for $T$ time steps  Each step is implemented by a different layer of the MP network  Hence, iterations correspond to network depth  The final feature vector $\mathbf {h}_v^T$ of $v$ is based on messages propagated from all the nodes in the subtree of height $T$ rooted at $v$  It captures both the topology of the neighborhood of $v$ and the distribution of the vertex representations in it 
If a graph-level feature vector is needed, e g , for classification or regression, a READOUT pooling function, that must be invariant to permutations, is applied:
Next, we present the MP network we developed for document understanding 
Message Passing Attention network for Document understanding (MPAD) ::: Word co-occurrence networks
We represent a document as a statistical word co-occurrence network BIBREF18, BIBREF19 with a sliding window of size 2 overspanning sentences  Let us denote that graph $G(V,E)$  Each unique word in the preprocessed document is represented by a node in $G$, and an edge is added between two nodes if they are found together in at least one instantiation of the window  $G$ is directed and weighted: edge directions and weights respectively capture text flow and co-occurrence counts 
$G$ is a compact representation of its document  In $G$, immediate neighbors are consecutive words in the same sentence  That is, paths of length 2 correspond to bigrams  Paths of length more than 2 can correspond either to traditional $n$-grams or to relaxed $n$-grams, that is, words that never appear in the same sentence but co-occur with the same word(s)  Such nodes are linked through common neighbors 
Master node  Inspired by BIBREF3, our $G$ also includes a special document node, linked to all other nodes via unit weight bi-directional edges  In what follows, let us denote by $n$ the number of nodes in $G$, including the master node 
Message Passing Attention network for Document understanding (MPAD) ::: Message passing
We formulate our AGGREGATE function as:
where $\mathbf {H}^t \in \mathbb {R}^{n \times d}$ contains node features ($d$ is a hyperparameter), and $\mathbf {A} \in \mathbb {R}^{n \times n}$ is the adjacency matrix of $G$  Since $G$ is directed, $\mathbf {A}$ is asymmetric  Also, $\mathbf {A}$ has zero diagonal as we choose not to consider the feature of the node itself, only that of its incoming neighbors, when updating its representation  Since $G$ is weighted, the $i^{th}$ row of $A$ contains the weights of the edges incoming on node $v_i$  $\mathbf {D} \in \mathbb {R}^{n \times n}$ is the diagonal in-degree matrix of $G$  MLP denotes a multi-layer perceptron, and $\mathbf {M}^{t+1} \in \mathbb {R}^{n \times d}$ is the message matrix 
The use of a MLP was motivated by the observation that for graph classification, MP neural nets with 1-layer perceptrons are inferior to their MLP counterparts BIBREF14  Indeed, 1-layer perceptrons are not universal approximators of multiset functions  Note that like in BIBREF14, we use a different MLP at each layer 
Renormalization  The rows of $\mathbf {D}^{-1}\mathbf {A}$ sum to 1  This is equivalent to the renormalization trick of BIBREF9, but using only the in-degrees  That is, instead of computing a weighted sum of the incoming neighbors' feature vectors, we compute a weighted average of them  The coefficients are proportional to the strength of co-occurrence between words  One should note that by averaging, we lose the ability to distinguish between different neighborhood structures in some special cases, that is, we lose injectivity  Such cases include neighborhoods in which all nodes have the same representations, and neighborhoods of different sizes containing various representations in equal proportions BIBREF14  As suggested by the results of an ablation experiment, averaging is better than summing in our application (see subsection SECREF30)  Note that instead of simply summing/averaging, we also tried using GAT-like attention BIBREF11 in early experiments, without obtaining better results 
As far as our COMBINE function, we use the Gated Recurrent Unit BIBREF20, BIBREF21:
Omitting biases for readability, we have:
where the $\mathbf {W}$ and $\mathbf {U}$ matrices are trainable weight matrices not shared across time steps, $\sigma (\mathbf {x}) = 1/(1+\exp (-\mathbf {x}))$ is the sigmoid function, and $\mathbf {R}$ and $\mathbf {Z}$ are the parameters of the reset and update gates  The reset gate controls the amount of information from the previous time step (in $\mathbf {H}^t$) that should propagate to the candidate representations, $\tilde{\mathbf {H}}^{t+1}$  The new representations $\mathbf {H}^{t+1}$ are finally obtained by linearly interpolating between the previous and the candidate ones, using the coefficients returned by the update gate 
Interpretation  Updating node representations through a GRU should in principle allow nodes to encode a combination of local and global signals (low and high values of $t$, resp ), by allowing them to remember about past iterations  In addition, we also explicitly consider node representations at all iterations when reading out (see Eq  DISPLAY_FORM18) 
Message Passing Attention network for Document understanding (MPAD) ::: Readout
After passing messages and performing updates for $T$ iterations, we obtain a matrix $\mathbf {H}^T \in \mathbb {R}^{n \times d}$ containing the final vertex representations  Let $\hat{G}$ be graph $G$ without the special document node, and matrix $\mathbf {\hat{H}}^T \in \mathbb {R}^{(n-1) \times d}$ be the corresponding representation matrix (i e , $\mathbf {H}^T$ without the row of the document node) 
We use as our READOUT function the concatenation of self-attention applied to $\mathbf {\hat{H}}^T$ with the final document node representation  More precisely, we apply a global self-attention mechanism BIBREF22 to the rows of $\mathbf {\hat{H}}^T$  As shown in Eq  DISPLAY_FORM17, $\mathbf {\hat{H}}^T$ is first passed to a dense layer parameterized by matrix $\mathbf {W}_A^T \in \mathbb {R}^{d \times d}$  An alignment vector $\mathbf {a}$ is then derived by comparing, via dot products, the rows of the output of the dense layer $\mathbf {Y}^T \in \mathbb {R}^{(n-1) \times d}$ with a trainable vector $\mathbf {v}^T \in \mathbb {R}^d$ (initialized randomly) and normalizing with a softmax  The normalized alignment coefficients are finally used to compute the attentional vector $\mathbf {u}^T \in \mathbb {R}^d$ as a weighted sum of the final representations $\mathbf {\hat{H}}^T$ 
Note that we tried with multiple context vectors, i e , with a matrix $\mathbf {V}^T$ instead of a vector $\mathbf {v}^T$, like in BIBREF22, but results were not convincing, even when adding a regularization term to the loss to favor diversity among the rows of $\mathbf {V}^T$ 
Master node skip connection  $\mathbf {h}_G^T \in \mathbb {R}^{2d}$ is obtained by concatenating $\mathbf {u}^T$ and the final master node representation  That is, the master node vector bypasses the attention mechanism  This is equivalent to a skip or shortcut connection BIBREF23  The reason behind this choice is that we expect the special document node to learn a high-level summary about the document, such as its size, vocabulary, etc  (more details are given in subsection SECREF30)  Therefore, by making the master node bypass the attention layer, we directly inject global information about the document into its final representation 
Multi-readout  BIBREF14, inspired by Jumping Knowledge Networks BIBREF12, recommend to not only use the final representations when performing readout, but also that of the earlier steps  Indeed, as one iterates, node features capture more and more global information  However, retaining more local, intermediary information might be useful too  Thus, instead of applying the readout function only to $t=T$, we apply it to all time steps and concatenate the results, finally obtaining $\mathbf {h}_G \in \mathbb {R}^{T \times 2d}$ :
In effect, with this modification, we take into account features based on information aggregated from subtrees of different heights (from 1 to $T$), corresponding to local and global features 
Message Passing Attention network for Document understanding (MPAD) ::: Hierarchical variants of MPAD
Through the successive MP iterations, it could be argued that MPAD implicitly captures some soft notion of the hierarchical structure of documents (words $\rightarrow $ bigrams $\rightarrow $ compositions of bigrams, etc )  However, it might be beneficial to explicitly capture document hierarchy  Hierarchical architectures have brought significant improvements to many NLP tasks, such as language modeling and generation BIBREF24, BIBREF25, sentiment and topic classification BIBREF26, BIBREF27, and spoken language understanding BIBREF28, BIBREF29  Inspired by this line of research, we propose several hierarchical variants of MPAD, detailed in what follows  In all of them, we represent each sentence in the document as a word co-occurrence network, and obtain an embedding for it by applying MPAD as previously described 
MPAD-sentence-att  Here, the sentence embeddings are simply combined through self-attention 
MPAD-clique  In this variant, we build a complete graph where each node represents a sentence  We then feed that graph to MPAD, where the feature vectors of the nodes are initialized with the sentence embeddings previously obtained 
MPAD-path  This variant is similar to the clique one, except that instead of a complete graph, we build a path according to the natural flow of the text  That is, two nodes are linked by a directed edge if the two sentences they represent follow each other in the document 
Experiments ::: Datasets
We evaluate the quality of the document embeddings learned by MPAD on 10 document classification datasets, covering the topic identification, coarse and fine sentiment analysis and opinion mining, and subjectivity detection tasks  We briefly introduce the datasets next  Their statistics are reported in Table TABREF21 
(1) Reuters  This dataset contains stories collected from the Reuters news agency in 1987  Following common practice, we used the ModApte split and considered only the 10 classes with the highest number of positive training examples  We also removed documents belonging to more than one class and then classes left with no document (2 classes) 
(2) BBCSport BIBREF30 contains documents from the BBC Sport website corresponding to 2004-2005 sports news articles 
(3) Polarity BIBREF31 features positive and negative labeled snippets from Rotten Tomatoes 
(4) Subjectivity BIBREF32 contains movie review snippets from Rotten Tomatoes (subjective sentences), and Internet Movie Database plot summaries (objective sentences) 
(5) MPQA BIBREF33 is made of positive and negative phrases, annotated as part of the summer 2002 NRRC Workshop on Multi-Perspective Question Answering 
(6) IMDB BIBREF34 is a collection of highly polarized movie reviews from IMDB (positive and negative)  There are at most 30 reviews for each movie 
(7) TREC BIBREF35 consists of questions that are classified into 6 different categories 
(8) SST-1 BIBREF36 contains the same snippets as Polarity  The authors used the Stanford Parser to parse the snippets and split them into multiple sentences  They then used Amazon Mechanical Turk to annotate the resulting phrases according to their polarity (very negative, negative, neutral, positive, very positive) 
(9) SST-2 BIBREF36 is the same as SST-1 but with neutral reviews removed and snippets classified as positive or negative 
(10) Yelp2013 BIBREF26 features reviews obtained from the 2013 Yelp Dataset Challenge 
Experiments ::: Baselines
We evaluate MPAD against multiple state-of-the-art baseline models, including hierarchical ones, to enable fair comparison with the hierarchical MPAD variants 
doc2vec BIBREF37  Doc2vec (or paragraph vector) is an extension of word2vec that learns vectors for documents in a fully unsupervised manner  Document embeddings are then fed to a logistic regression classifier 
CNN BIBREF38  The convolutional neural network architecture, well-known in computer vision, is applied to text  There is one spatial dimension and the word embeddings are used as channels (depth dimensions) 
DAN BIBREF39  The Deep Averaging Network passes the unweighted average of the embeddings of the input words through multiple dense layers and a final softmax 
Tree-LSTM BIBREF40 is a generalization of the standard LSTM architecture to constituency and dependency parse trees 
DRNN BIBREF41  Recursive neural networks are stacked and applied to parse trees 
LSTMN BIBREF42 is an extension of the LSTM model where the memory cell is replaced by a memory network which stores word representations 
C-LSTM BIBREF43 combines convolutional and recurrent neural networks  The region embeddings provided by a CNN are fed to a LSTM 
SPGK BIBREF44 also models documents as word co-occurrence networks  It computes a graph kernel that compares shortest paths extracted from the word co-occurrence networks and then uses a SVM to categorize documents 
WMD BIBREF45 is an application of the well-known Earth Mover's Distance to text  A k-nearest neighbor classifier is used 
S-WMD BIBREF46 is a supervised extension of the Word Mover's Distance 
Semantic-CNN BIBREF47  Here, a CNN is applied to semantic units obtained by clustering words in the embedding space 
LSTM-GRNN BIBREF26 is a hierarchical model where sentence embeddings are obtained with a CNN and a GRU-RNN is fed the sentence representations to obtain a document vector 
HN-ATT BIBREF27 is another hierarchical model, where the same encoder architecture (a bidirectional GRU-RNN) is used for both sentences and documents, with different parameters  A self-attention mechanism is applied to the RNN annotations at each level 
Experiments ::: Model configuration and training
We preprocess all datasets using the code of BIBREF38  On Yelp2013, we also replace all tokens appearing strictly less than 6 times with a special UNK token, like in BIBREF27  We then build a directed word co-occurrence network from each document, with a window of size 2 
We use two MP iterations ($T$=2) for the basic MPAD, and two MP iterations at each level, for the hierarchical variants  We set $d$ to 64, except on IMDB and Yelp on which $d=128$, and use a two-layer MLP  The final graph representations are passed through a softmax for classification  We train MPAD in an end-to-end fashion by minimizing the cross-entropy loss function with the Adam optimizer BIBREF48 and an initial learning rate of 0 001 
To regulate potential differences in magnitude, we apply batch normalization after concatenating the feature vector of the master node with the self-attentional vector, that is, after the skip connection (see subsection SECREF16)  To prevent overfitting, we use dropout BIBREF49 with a rate of 0 5  We select the best epoch, capped at 200, based on the validation accuracy  When cross-validation is used (see 3rd column of Table TABREF21), we construct a validation set by randomly sampling 10% of the training set of each fold 
On all datasets except Yelp2013, we use the publicly available 300-dimensional pre-trained Google News vectors ($D$=300) BIBREF50 to initialize the node representations $\mathbf {H}^0$  On Yelp2013, we follow BIBREF27 and learn our own word vectors from the training and validation sets with the gensim implementation of word2vec BIBREF51 
MPAD was implemented in Python 3 6 using the PyTorch library BIBREF52  All experiments were run on a single machine consisting of a 3 4 GHz Intel Core i7 CPU with 16 GB of RAM and an NVidia GeForce Titan Xp GPU 
Results and ablations ::: Results
Experimental results are shown in Table TABREF28  For the baselines, the best scores reported in each original paper are shown  MPAD reaches best performance on 7 out of 10 datasets, and is close second elsewhere  Moreover, the 7 datasets on which MPAD ranks first widely differ in training set size, number of categories, and prediction task (topic, sentiment, subjectivity), which indicates that MPAD can perform well in different settings 
MPAD vs  hierarchical variants  On 9 datasets out of 10, one or more of the hierarchical variants outperform the vanilla MPAD architecture, highlighting the benefit of explicitly modeling the hierarchical nature of documents 
However, on Subjectivity, standard MPAD outperforms all hierarchical variants  On TREC, it reaches the same accuracy  We hypothesize that in some cases, using a different graph to separately encode each sentence might be worse than using one single graph to directly encode the document  Indeed, in the single document graph, some words that never appear in the same sentence can be connected through common neighbors, as was explained in subsection SECREF7  So, this way, some notion of cross-sentence context is captured while learning representations of words, bigrams, etc  at each MP iteration  This creates better informed representations, resulting in a better document embedding  With the hierarchical variants, on the other hand, each sentence vector is produced in isolation, without any contextual information about the other sentences in the document  Therefore, the final sentence embeddings might be of lower quality, and as a group might also contain redundant/repeated information  When the sentence vectors are finally combined into a document representation, it is too late to take context into account 
Results and ablations ::: Ablation studies
To understand the impact of some hyperparameters on performance, we conducted additional experiments on the Reuters, Polarity, and IMDB datasets, with the non-hierarchical version of MPAD  Results are shown in Table TABREF29 
Number of MP iterations  First, we varied the number of message passing iterations from 1 to 4  We can clearly see in Table TABREF29 that having more iterations improves performance  We attribute this to the fact that we are reading out at each iteration from 1 to $T$ (see Eq  DISPLAY_FORM18), which enables the final graph representation to encode a mixture of low-level and high-level features  Indeed, in initial experiments involving readout at $t$=$T$ only, setting $T\ge 2$ was always decreasing performance, despite the GRU-based updates (Eq  DISPLAY_FORM14)  These results were consistent with that of BIBREF53 and BIBREF9, who both are reading out only at $t$=$T$ too  We hypothesize that node features at $T\ge 2$ are too diffuse to be entirely relied upon during readout  More precisely, initially at $t$=0, node representations capture information about words, at $t$=1, about their 1-hop neighborhood (bigrams), at $t$=2, about compositions of bigrams, etc  Thus, pretty quickly, node features become general and diffuse  In such cases, considering also the lower-level, more precise features of the earlier iterations when reading out may be necessary 
Undirected edges  On Reuters, using an undirected graph leads to better performance, while on Polarity and IMDB, it is the opposite  This can be explained by the fact that Reuters is a topic classification task, for which the presence or absence of some patterns is important, but not necessarily the order in which they appear, while Polarity and IMDB are sentiment analysis tasks  To capture sentiment, modeling word order is crucial, e g , in detecting negation 
No master node  Removing the master node deteriorates performance across all datasets, clearly showing the value of having such a node  We hypothesize that since the special document node is connected to all other nodes, it is able to encode during message passing a summary of the document 
No renormalization  Here, we do not use the renormalization trick of BIBREF9 during MP (see subsection SECREF10)  That is, Eq  DISPLAY_FORM11 becomes $\mathbf {M}^{t+1} = \textsc {MLP}^{t+1}\big (\mathbf {A}\mathbf {H}^{t}\big )$  In other words, instead of computing a weighted average of the incoming neighbors' feature vectors, we compute a weighted sum of them  Unlike the mean, which captures distributions, the sum captures structural information BIBREF14  As shown in Table TABREF29, using sum instead of mean decreases performance everywhere, suggesting that in our application, capturing the distribution of neighbor representations is more important that capturing their structure  We hypothesize that this is the case because statistical word co-occurrence networks tend to have similar structural properties, regardless of the topic, polarity, sentiment, etc  of the corresponding documents 
Neighbors-only  In this experiment, we replaced the GRU combine function (see Eq  DISPLAY_FORM14) with the identity function  That is, we simply have $\mathbf {H}^{t+1}$=$\mathbf {M}^{t+1}$  Since $\mathbf {A}$ has zero diagonal, by doing so, we completely ignore the previous feature of the node itself when updating its representation  That is, the update is based entirely on its neighbors  Except on Reuters (almost no change), performance always suffers, stressing the need to take into account the root node during updates, not only its neighborhood 
Related work
In what follows, we offer a brief review of relevant studies, ranked by increasing order of similarity with our work 
BIBREF9, BIBREF54, BIBREF11, BIBREF10 conduct some node classification experiments on citation networks, where nodes are scientific papers, i e , textual data  However, text is only used to derive node feature vectors  The external graph structure, which plays a central role in determining node labels, is completely unrelated to text 
On the other hand, BIBREF55, BIBREF7 experiment on traditional document classification tasks  They both build $k$-nearest neighbor similarity graphs based on the Gaussian diffusion kernel  More precisely, BIBREF55 build one single graph where nodes are documents and distance is computed in the BoW space  Node features are then used for classification  Closer to our work, BIBREF7 represent each document as a graph  All document graphs are derived from the same underlying structure  Only node features, corresponding to the entries of the documents' BoW vectors, vary  The underlying, shared structure is that of a $k$-NN graph where nodes are vocabulary terms and similarity is the cosine of the word embedding vectors  BIBREF7 then perform graph classification  However they found performance to be lower than that of a naive Bayes classifier 
BIBREF56 use a GNN for hierarchical classification into a large taxonomy of topics  This task differs from traditional document classification  The authors represent documents as unweighted, undirected word co-occurrence networks with word embeddings as node features  They then use the spatial GNN of BIBREF15 to perform graph classification 
The work closest to ours is probably that of BIBREF53  The authors adopt the semi-supervised node classification approach of BIBREF9  They build one single undirected graph from the entire dataset, with both word and document nodes  Document-word edges are weighted by TF-IDF and word-word edges are weighted by pointwise mutual information derived from co-occurrence within a sliding window  There are no document-document edges  The GNN is trained based on the cross-entropy loss computed only for the labeled nodes, that is, the documents in the training set  When the final node representations are obtained, one can use that of the test documents to classify them and evaluate prediction performance 
There are significant differences between BIBREF53 and our work  First, our approach is inductive, not transductive  Indeed, while the node classification approach of BIBREF53 requires all test documents at training time, our graph classification model is able to perform inference on new, never-seen documents  The downside of representing documents as separate graphs, however, is that we lose the ability to capture corpus-level dependencies  Also, our directed graphs capture word ordering, which is ignored by BIBREF53  Finally, the approach of BIBREF53 requires computing the PMI for every word pair in the vocabulary, which may be prohibitive on datasets with very large vocabularies  On the other hand, the complexity of MPAD does not depend on vocabulary size 
Conclusion
We have proposed an application of the message passing framework to NLP, the Message Passing Attention network for Document understanding (MPAD)  Experiments conducted on 10 standard text classification datasets show that our architecture is competitive with the state-of-the-art  By processing weighted, directed word co-occurrence networks, MPAD is sensitive to word order and word-word relationship strength  To explicitly capture the hierarchical structure of documents, we also propose three hierarchical variants of MPAD, that we show bring improvements over the vanilla architecture 
Acknowledgments
We thank the NVidia corporation for the donation of a GPU as part of their GPU grant program 

ground_truth: Answer with content missing: (Table 3) Best author's model B-M average micro f-score is 0.409, 0.459, 0.411 on Affective, Fairy Tales and ISEAR datasets respectively. 

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What experiments are conducted?
Context 2:  Specifically, for every word in the input sentence, there is 10% chance that we delete it, 10 % chance that we duplicate it, 10% chance that we swap it with the next word, or it remains unchanged  We first encode the input sentence INLINEFORM0 or INLINEFORM1 with respective encoder INLINEFORM2 or INLINEFORM3 to obtain its latent representation INLINEFORM4 or INLINEFORM5 and reconstruct the input sentence with the latent representation and respective decoder  So we can get the reconstruction loss for auto-encoder INLINEFORM6 : DISPLAYFORM0 DISPLAYFORM1
In addition to denoising auto-encoder, we implement back-translation BIBREF19 to generate a pseudo-parallel corpus  Suppose our model takes non-ironic sentence INLINEFORM0 as input  We first encode INLINEFORM1 with INLINEFORM2 to obtain its latent representation INLINEFORM3 and decode the latent representation with INLINEFORM4 to get a transferred sentence INLINEFORM5   Then we encode INLINEFORM6 with INLINEFORM7 and decode its latent representation with INLINEFORM8 to reconstruct the original input sentence INLINEFORM9   Therefore, our reconstruction loss for back-translation INLINEFORM10 : DISPLAYFORM0
And if our model takes ironic sentence INLINEFORM0 as input, we can get the reconstruction loss for back-translation as: DISPLAYFORM0
Reinforcement Learning
Since the gold transferred result of input is unavailable, we cannot evaluate the quality of the generated sentence directly  Therefore, we implement reinforcement learning and elaborately design two rewards to describe the irony accuracy and sentiment preservation, respectively 
A pre-trained binary irony classifier based on CNN BIBREF20 is used to evaluate how ironic a sentence is  We denote the parameter of the classifier as INLINEFORM0 and it is fixed during the training process 
In order to facilitate the transformation, we design the irony reward as the difference between the irony score of the input sentence and that of the output sentence  Formally, when we input a non-ironic sentence INLINEFORM0 and transfer it to an ironic sentence INLINEFORM1 , our irony reward is defined as: DISPLAYFORM0
where INLINEFORM0 denotes ironic style and INLINEFORM1 is the probability of that a sentence INLINEFORM2 is ironic 
To preserve the sentiment polarity of the input sentence, we also need to use classifiers to evaluate the sentiment polarity of the sentences  However, the sentiment analysis of ironic sentences and non-ironic sentences are different  In the case of figurative languages such as irony, sarcasm or metaphor, the sentiment polarity of the literal meaning may differ significantly from that of the intended figurative meaning BIBREF0   As we aim to train our model to transfer sentences from non-ironic to ironic, using only one classifier is not enough  As a result, we implement two pre-trained sentiment classifiers for non-ironic sentences and ironic sentences respectively  We denote the parameter of the sentiment classifier for ironic sentences as INLINEFORM0 and that of the sentiment classifier for non-ironic sentences as INLINEFORM1  
A challenge, when we implement two classifiers to evaluate the sentiment polarity, is that the two classifiers trained with different datasets may have different distributions of scores  That means we cannot directly calculate the sentiment reward with scores applied by two classifiers  To alleviate this problem and standardize the prediction results of two classifiers, we set a threshold for each classifier and subtract the respective threshold from scores applied by the classifier to obtain the comparative sentiment polarity score  We get the optimal threshold by maximizing the ability of the classifier according to the distribution of our training data 
We denote the threshold of ironic sentiment classifier as INLINEFORM0 and the threshold of non-ironic sentiment classifier as INLINEFORM1   The standardized sentiment score is defined as INLINEFORM2 and INLINEFORM3 where INLINEFORM4 denotes the positive sentiment polarity and INLINEFORM5 is the probability of that a sentence is positive in sentiment polarity 
As mentioned above, the input sentence and the generated sentence should express the same sentiment  For example, if we input a non-ironic sentence “I hate to be ignored" which is negative in sentiment polarity, the generated ironic sentence should be also negative, such as “I love to be ignored"  To achieve sentiment preservation, we design the sentiment reward as that one minus the absolute value of the difference between the standardized sentiment score of the input sentence and that of the generated sentence  Formally, when we input a non-ironic sentence INLINEFORM0 and transfer it to an ironic sentence INLINEFORM1 , our sentiment reward is defined as: DISPLAYFORM0
To encourage our model to focus on both the irony accuracy and the sentiment preservation, we apply the harmonic mean of irony reward and sentiment reward: DISPLAYFORM0
Policy Gradient
The policy gradient algorithm BIBREF21 is a simple but widely-used algorithm in reinforcement learning  It is used to maximize the expected reward INLINEFORM0   The objective function to minimize is defined as: DISPLAYFORM0
where INLINEFORM0 , INLINEFORM1 is the reward of INLINEFORM2 and INLINEFORM3 is the input size 
Training Details
INLINEFORM0 , INLINEFORM1 , INLINEFORM2 and INLINEFORM3 in our model are Transformers BIBREF22 with 4 layers and 2 shared layers  The word embeddings of 128 dimensions are learned during the training process  Our maximum sentence length is set as 40  The optimizer is Adam BIBREF23 and the learning rate is INLINEFORM4   The batch size is 32 and harmonic weight INLINEFORM5 in Eq 9 is 0 5  We set the interval INLINEFORM6 as 200  The model is pre-trained for 6 epochs and trained for 15 epochs for reinforcement learning 
Irony Classifier: We implement a CNN classifier trained with our irony dataset  All the CNN classifiers we utilize in this paper use the same parameters as BIBREF20  
Sentiment Classifier for Irony: We first implement a one-layer LSTM network to classify ironic sentences in our dataset into positive and negative ironies  The LSTM network is trained with the dataset of Semeval 2015 Task 11 BIBREF0 which is used for the sentiment analysis of figurative language in twitter  Then, we use the positive ironies and negative ironies to train the CNN sentiment classifier for irony 
Sentiment Classifier for Non-irony: Similar to the training process of the sentiment classifier for irony, we first implement a one-layer LSTM network trained with the dataset for the sentiment analysis of common twitters to classify the non-ironies into positive and negative non-ironies  Then we use the positive and negative non-ironies to train the sentiment classifier for non-irony 
Baselines
We compare our model with the following state-of-art generative models:
BackTrans BIBREF7 : In BIBREF7 , authors propose a model using machine translation in order to preserve the meaning of the sentence while reducing stylistic properties 
Unpaired BIBREF10 : In BIBREF10 , researchers implement a method to remove emotional words and add desired sentiment controlled by reinforcement learning 
CrossAlign BIBREF6 : In BIBREF6 , authors leverage refined alignment of latent representations to perform style transfer and a cross-aligned auto-encoder is implemented 
CPTG BIBREF24 : An interpolated reconstruction loss is introduced in BIBREF24 and a discriminator is implemented to control attributes in this work 
DualRL BIBREF8 : In BIBREF8 , researchers use two reinforcement rewards simultaneously to control style accuracy and content preservation 
Evaluation Metrics
In order to evaluate sentiment preservation, we use the absolute value of the difference between the standardized sentiment score of the input sentence and that of the generated sentence  We call the value as sentiment delta (senti delta)  Besides, we report the sentiment accuracy (Senti ACC) which measures whether the output sentence has the same sentiment polarity as the input sentence based on our standardized sentiment classifiers  The BLEU score BIBREF25 between the input sentences and the output sentences is calculated to evaluate the content preservation performance  In order to evaluate the overall performance of different models, we also report the geometric mean (G2) and harmonic mean (H2) of the sentiment accuracy and the BLEU score  As for the irony accuracy, we only report it in human evaluation results because it is more accurate for the human to evaluate the quality of irony as it is very complicated 
We first sample 50 non-ironic input sentences and their corresponding output sentences of different models  Then, we ask four annotators who are proficient in English to evaluate the qualities of the generated sentences of different models  They are required to rank the output sentences of our model and baselines from the best to the worst in terms of irony accuracy (Irony), Sentiment preservation (Senti) and content preservation (Content)  The best output is ranked with 1 and the worst output is ranked with 6  That means that the smaller our human evaluation value is, the better the corresponding model is 
Results and Discussions
Table TABREF35 shows the automatic evaluation results of the models in the transformation from non-ironic sentences to ironic sentences  From the results, our model obtains the best result in sentiment delta  The DualRL model achieves the highest result in other metrics, but most of its outputs are the almost same as the input sentences  So it is reasonable that DualRL system outperforms ours in these metrics but it actually does not transfer the non-ironic sentences to ironic sentences at all  From this perspective, we cannot view DualRL as an effective model for irony generation  In contrast, our model gets results close to those of DualRL and obtains a balance between irony accuracy, sentiment preservation, and content preservation if we also consider the irony accuracy discussed below 
And from human evaluation results shown in Table TABREF36 , our model gets the best average rank in irony accuracy  And as mentioned above, the DualRL model usually does not change the input sentence and outputs the same sentence  Therefore, it is reasonable that it obtains the best rank in sentiment and content preservation and ours is the second  However, it still demonstrates that our model, instead of changing nothing, transfers the style of the input sentence with content and sentiment preservation at the same time 
Case Study
In the section, we present some example outputs of different models  Table TABREF37 shows the results of the transformation from non-ironic sentences to ironic sentences  We can observe that: (1) The BackTrans system, the Unpaired system, the CrossAlign system and the CPTG system tends to generate sentences which are towards irony but do not preserve content  (2) The DualRL system preserves content and sentiment very well but even does not change the input sentence  (3) Our model considers both aspects and achieves a better balance among irony accuracy, sentiment and content preservation 
Error Analysis
Although our model outperforms other style transfer baselines according to automatic and human evaluation results, there are still some failure cases because irony generation is still a very challenging task  We would like to share the issues we meet during our experiments and our solutions to some of them in this section 
No Change: As mentioned above, many style transfer models, such as DualRL, tend to make few changes to the input sentence and output the same sentence  Actually, this is a common issue for unsupervised style transfer systems and we also meet it during our experiments  The main reason for the issue is that rewards for content preservation are too prominent and rewards for style accuracy cannot work well  In contrast, in order to guarantee the readability and fluency of the output sentence, we also cannot emphasize too much on rewards for style accuracy because it may cause some other issues such as word repetition mentioned below  A method to solve the problem is tuning hyperparameters and this is also the method we implement in this work  As for content preservation, maybe MLE methods such as back-translation are not enough because they tend to force models to generate specific words  In the future, we should further design some more suitable methods to control content preservation for models without disentangling style and content representations, such as DualRL and ours 
Word Repetition: During our experiments, we observe that some of the outputs prefer to repeat the same word as shown in Table TABREF38   This is because reinforcement learning rewards encourage the model to generate words which can get high scores from classifiers and even back-translation cannot stop it  Our solution is that we can lower the probability of decoding a word in decoders if the word has been generated in the previous time steps during testing  We also try to implement this method during training time but obtain worse performances because it may limit the effects of training  Some previous studies utilize language models to control the fluency of the output sentence and we also try this method  Nonetheless, pre-training a language model with tweets and using it to generate rewards is difficult because tweets are more casual and have more noise  Rewards from that kind of language model are usually not accurate and may confuse the model  In the future, we should come up with better methods to model language fluency with the consideration of irony accuracy, sentiment and content preservation, especially for tweets 
Improper Words: As ironic style is hard for our model to learn, it may generate some improper words which make the sentence strange  As the example shown in the Table TABREF38 , the sentiment word in the input sentence is “wonderful" and the model should change it into a negative word such as “sad" to make the output sentence ironic  However, the model changes “friday" and “fifa" which are not related to ironic styles  We have not found a very effective method to address this issue and maybe we should further explore stronger models to learn ironic styles better 
Additional Experiments
In this section, we describe some additional experiments on the transformation from ironic sentences to non-ironic sentences  Sometimes ironies are hard to understand and may cause misunderstanding, for which our task also explores the transformation from ironic sentences to non-ironic sentences 
As shown in Table TABREF46 , we also conduct automatic evaluations and the conclusions are similar to those of the transformation from non-ironic sentences to ironic sentences  As for human evaluation results in Table TABREF47 , our model still can achieve the second-best results in sentiment and content preservation  Nevertheless, DualRL system and ours get poor performances in irony accuracy  The reason may be that the other four baselines tend to generate common and even not fluent sentences which are irrelevant to the input sentences and are hard to be identified as ironies  So annotators usually mark these output sentences as non-ironic sentences, which causes these models to obtain better performances than DualRL and ours but much poorer results in sentiment and content preservation  Some examples are shown in Table TABREF52  
Conclusion and Future Work
In this paper, we first systematically define irony generation based on style transfer  Because of the lack of irony data, we make use of twitter and build a large-scale dataset  In order to control irony accuracy, sentiment preservation and content preservation at the same time, we also design a combination of rewards for reinforcement learning and incorporate reinforcement learning with a pre-training process  Experimental results demonstrate that our model outperforms other generative models and our rewards are effective  Although our model design is effective, there are still many errors and we systematically analyze them  In the future, we are interested in exploring these directions and our work may extend to other kinds of ironies which are more difficult to model 

ground_truth: Irony Classifier, Sentiment Classifier for Irony, Sentiment Classifier for Non-irony, transformation from ironic sentences to non-ironic sentences

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which component is the least impactful?
Context 4:  Now instead of orthogonalizing INLINEFORM10 to INLINEFORM11 we could orthogonalize INLINEFORM12 to the hidden state of this GRU at time-step INLINEFORM13   In practice, we found this to work better as it accounts for all the field vectors in the history instead of only the previous field vector 
In summary, Equation provides a mechanism for remembering the current field vector when appropriate (thus capturing stay-on behavior) using a remember gate  On the other hand, Equation EQREF10 explicitly ensures that the field vector is very different (soft-orthogonal) from the previous field vectors once it is time to move on (thus capturing never look back behavior)  The value of INLINEFORM0 computed in Equation EQREF10 is then used in Equation   The INLINEFORM1 (macro) thus obtained is then concatenated with INLINEFORM2 (micro) and fed to the decoder (see Fig  FIGREF3 )
Experimental setup
We now describe our experimental setup:
Datasets
We use the WikiBio dataset introduced by lebret2016neural  It consists of INLINEFORM0 biography articles from English Wikipedia  A biography article corresponds to a person (sportsman, politician, historical figure, actor, etc )  Each Wikipedia article has an accompanying infobox which serves as the structured input and the task is to generate the first sentence of the article (which typically is a one-line description of the person)  We used the same train, valid and test sets which were made publicly available by lebret2016neural 
We also introduce two new biography datasets, one in French and one in German  These datasets were created and pre-processed using the same procedure as outlined in lebret2016neural  Specifically, we extracted the infoboxes and the first sentence from the corresponding Wikipedia article  As with the English dataset, we split the French and German datasets randomly into train (80%), test (10%) and valid (10%)  The French and German datasets extracted by us has been made publicly available  The number of examples was 170K and 50K and the vocabulary size was 297K and 143K for French and German respectively  Although in this work we focus only on generating descriptions in one language, we hope that this dataset will also be useful for developing models which jointly learn to generate descriptions from structured data in multiple languages 
Models compared
We compare with the following models:
1  BIBREF0 : This is a conditional language model which uses a feed-forward neural network to predict the next word in the description conditioned on local characteristics (i e , words within a field) and global characteristics (i e , overall structure of the infobox) 
2  BIBREF9 : This model was proposed in the context of the WeatherGov and RoboCup datasets which have a much smaller vocabulary  They use an improved attention model with additional regularizer terms which influence the weights assigned to the fields 
3  Basic Seq2Seq: This is the vanilla encode-attend-decode model BIBREF1   Further, to deal with the large vocabulary ( INLINEFORM0 400K words) we use a copying mechanism as a post-processing step  Specifically, we identify the time steps at which the decoder produces unknown words (denoted by the special symbol UNK)  For each such time step, we look at the attention weights on the input words and replace the UNK word by that input word which has received maximum attention at this timestep  This process is similar to the one described in BIBREF30   Even lebret2016neural have a copying mechanism tightly integrated with their model 
Hyperparameter tuning
We tuned the hyperparameters of all the models using a validation set  As mentioned earlier, we used a bidirectional GRU cell as the function INLINEFORM0 for computing the representation of the fields and the values (see Section SECREF4 )  For all the models, we experimented with GRU state sizes of 128, 256 and 512  The total number of unique words in the corpus is around 400K (this includes the words in the infobox and the descriptions)  Of these, we retained only the top 20K words in our vocabulary (same as BIBREF0 )  We initialized the embeddings of these words with 300 dimensional Glove embeddings BIBREF31   We used Adam BIBREF32 with a learning rate of INLINEFORM1 , INLINEFORM2 and INLINEFORM3   We trained the model for a maximum of 20 epochs and used early stopping with the patience set to 5 epochs 
Results and Discussions
We now discuss the results of our experiments 
Comparison of different models
Following lebret2016neural, we used BLEU-4, NIST-4 and ROUGE-4 as the evaluation metrics  We first make a few observations based on the results on the English dataset (Table TABREF15 )  The basic seq2seq model, as well as the model proposed by weather16, perform better than the model proposed by lebret2016neural  Our final model with bifocal attention and gated orthogonalization gives the best performance and does 10% (relative) better than the closest baseline (basic seq2seq) and 21% (relative) better than the current state of the art method BIBREF0   In Table TABREF16 , we show some qualitative examples of the output generated by different models 
Human Evaluations
To make a qualitative assessment of the generated sentences, we conducted a human study on a sample of 500 Infoboxes which were sampled from English dataset  The annotators for this task were undergraduate and graduate students  For each of these infoboxes, we generated summaries using the basic seq2seq model and our final model with bifocal attention and gated orthogonalization  For each description and for each model, we asked three annotators to rank the output of the systems based on i) adequacy (i e  does it capture relevant information from the infobox), (ii) fluency (i e  grammar) and (iii) relative preference (i e , which of the two outputs would be preferred)  Overall the average fluency/adequacy (on a scale of 5) for basic seq2seq model was INLINEFORM0 and INLINEFORM1 for our model respectively 
The results from Table TABREF17 suggest that in general gated orthogonalization model performs better than the basic seq2seq model  Additionally, annotators were asked to verify if the generated summaries look natural (i e, as if they were generated by humans)  In 423 out of 500 cases, the annotators said “Yes” suggesting that gated orthogonalization model indeed produces good descriptions 
Performance on different languages
The results on the French and German datasets are summarized in Tables TABREF20 and TABREF20 respectively  Note that the code of BIBREF0 is not publicly available, hence we could not report numbers for French and German using their model  We observe that our final model gives the best performance - though the bifocal attention model performs poorly as compared to the basic seq2seq model on French  However, the overall performance for French and German are much smaller than those for English  There could be multiple reasons for this  First, the amount of training data in these two languages is smaller than that in English  Specifically, the amount of training data available in French (German) is only INLINEFORM0 ( INLINEFORM1 )% of that available for English  Second, on average the descriptions in French and German are longer than that in English (EN: INLINEFORM2 words, FR: INLINEFORM3 words and DE: INLINEFORM4 words)  Finally, a manual inspection across the three languages suggests that the English descriptions have a more consistent structure than the French descriptions  For example, most English descriptions start with name followed by date of birth but this is not the case in French  However, this is only a qualitative observation and it is hard to quantify this characteristic of the French and German datasets 
Visualizing Attention Weights
If the proposed model indeed works well then we should see attention weights that are consistent with the stay on and never look back behavior  To verify this, we plotted the attention weights in cases where the model with gated orthogonalization does better than the model with only bifocal attention  Figure FIGREF21 shows the attention weights corresponding to infobox in Figure FIGREF25   Notice that the model without gated orthogonalization has attention on both name field and article title while rendering the name  The model with gated orthogonalization, on the other hand, stays on the name field for as long as it is required but then moves and never returns to it (as expected) 
Due to lack of space, we do not show similar plots for French and German but we would like to mention that, in general, the differences between the attention weights learned by the model with and without gated orthogonalization were more pronounced for the French/German dataset than the English dataset  This is in agreement with the results reported in Table TABREF20 and TABREF20 where the improvements given by gated orthogonalization are more for French/German than for English 
Out of domain results
What if the model sees a different INLINEFORM0 of person at test time? For example, what if the training data does not contain any sportspersons but at test time we encounter the infobox of a sportsperson  This is the same as seeing out-of-domain data at test time  Such a situation is quite expected in the products domain where new products with new features (fields) get frequently added to the catalog  We were interested in three questions here  First, we wanted to see if testing the model on out-of-domain data indeed leads to a drop in the performance  For this, we compared the performance of our best model in two scenarios (i) trained on data from all domains (including the target domain) and tested on the target domain (sports, arts) and (ii) trained on data from all domains except the target domain and tested on the target domain  Comparing rows 1 and 2 of Table TABREF32 we observed a significant drop in the performance  Note that the numbers for sports domain in row 1 are much better than the Arts domain because roughly 40% of the WikiBio training data contains sportspersons 
Next, we wanted to see if we can use a small amount of data from the target domain to fine tune a model trained on the out of domain data  We observe that even with very small amounts of target domain data the performance starts improving significantly (see rows 3 and 4 of Table TABREF32 )  Note that if we train a model from scratch with only limited data from the target domain instead of fine-tuning a model trained on a different source domain then the performance is very poor  In particular, training a model from scratch with 10K training instances we get a BLEU score of INLINEFORM0 and INLINEFORM1 for arts and sports respectively  Finally, even though the actual words used for describing a sportsperson (footballer, cricketer, etc ) would be very different from the words used to describe an artist (actor, musician, etc ) they might share many fields (for example, date of birth, occupation, etc )  As seen in Figure FIGREF28 (attention weights corresponding to the infobox in Figure FIGREF27 ), the model predicts the attention weights correctly for common fields (such as occupation) but it is unable to use the right vocabulary to describe the occupation (since it has not seen such words frequently in the training data)  However, once we fine tune the model with limited data from the target domain we see that it picks up the new vocabulary and produces a correct description of the occupation 
Conclusion
We present a model for generating natural language descriptions from structured data  To address specific characteristics of the problem we propose neural components for fused bifocal attention and gated orthogonalization to address stay on and never look back behavior while decoding  Our final model outperforms an existing state of the art model on a large scale WikiBio dataset by 21%  We also introduce datasets for French and German and demonstrate that our model gives state of the art results on these datasets  Finally, we perform experiments with an out-of-domain model and show that if such a model is fine-tuned with small amounts of in domain data then it can give an improved performance on the target domain 
Given the multilingual nature of the new datasets, as future work, we would like to build models which can jointly learn to generate natural language descriptions from structured data in multiple languages  One idea is to replace the concepts in the input infobox by Wikidata concept ids which are language agnostic  A large amount of input vocabulary could thus be shared across languages thereby facilitating joint learning 
Acknowledgements
We thank Google for supporting Preksha Nema through their Google India Ph D  Fellowship program  We also thank Microsoft Research India for supporting Shreyas Shetty through their generous travel grant for attending the conference 

ground_truth: Based on table results provided changing directed to undirected edges had least impact - max abs difference of 0.33 points on all three datasets.