prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What evaluation metrics are looked at for classification tasks?
Context 1: Introduction
Performance appraisal (PA) is an important HR process, particularly for modern organizations that crucially depend on the skills and expertise of their workforce  The PA process enables an organization to periodically measure and evaluate every employee's performance  It also provides a mechanism to link the goals established by the organization to its each employee's day-to-day activities and performance  Design and analysis of PA processes is a lively area of research within the HR community BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3  
The PA process in any modern organization is nowadays implemented and tracked through an IT system (the PA system) that records the interactions that happen in various steps  Availability of this data in a computer-readable database opens up opportunities to analyze it using automated statistical, data-mining and text-mining techniques, to generate novel and actionable insights / patterns and to help in improving the quality and effectiveness of the PA process BIBREF4 , BIBREF5 , BIBREF6   Automated analysis of large-scale PA data is now facilitated by technological and algorithmic advances, and is becoming essential for large organizations containing thousands of geographically distributed employees handling a wide variety of roles and tasks 
A typical PA process involves purposeful multi-step multi-modal communication between employees, their supervisors and their peers  In most PA processes, the communication includes the following steps: (i) in self-appraisal, an employee records his/her achievements, activities, tasks handled etc ; (ii) in supervisor assessment, the supervisor provides the criticism, evaluation and suggestions for improvement of performance etc ; and (iii) in peer feedback (aka INLINEFORM0 view), the peers of the employee provide their feedback  There are several business questions that managers are interested in  Examples:
In this paper, we develop text mining techniques that can automatically produce answers to these questions  Since the intended users are HR executives, ideally, the techniques should work with minimum training data and experimentation with parameter setting  These techniques have been implemented and are being used in a PA system in a large multi-national IT company 
The rest of the paper is organized as follows  Section SECREF2 summarizes related work  Section SECREF3 summarizes the PA dataset used in this paper  Section SECREF4 applies sentence classification algorithms to automatically discover three important classes of sentences in the PA corpus viz , sentences that discuss strengths, weaknesses of employees and contain suggestions for improving her performance  Section SECREF5 considers the problem of mapping the actual targets mentioned in strengths, weaknesses and suggestions to a fixed set of attributes  In Section SECREF6 , we discuss how the feedback from peers for a particular employee can be summarized  In Section SECREF7 we draw conclusions and identify some further work 
Related Work
We first review some work related to sentence classification  Semantically classifying sentences (based on the sentence's purpose) is a much harder task, and is gaining increasing attention from linguists and NLP researchers  McKnight and Srinivasan BIBREF7 and Yamamoto and Takagi BIBREF8 used SVM to classify sentences in biomedical abstracts into classes such as INTRODUCTION, BACKGROUND, PURPOSE, METHOD, RESULT, CONCLUSION  Cohen et al  BIBREF9 applied SVM and other techniques to learn classifiers for sentences in emails into classes, which are speech acts defined by a verb-noun pair, with verbs such as request, propose, amend, commit, deliver and nouns such as meeting, document, committee; see also BIBREF10   Khoo et al  BIBREF11 uses various classifiers to classify sentences in emails into classes such as APOLOGY, INSTRUCTION, QUESTION, REQUEST, SALUTATION, STATEMENT, SUGGESTION, THANKING etc  Qadir and Riloff BIBREF12 proposes several filters and classifiers to classify sentences on message boards (community QA systems) into 4 speech acts: COMMISSIVE (speaker commits to a future action), DIRECTIVE (speaker expects listener to take some action), EXPRESSIVE (speaker expresses his or her psychological state to the listener), REPRESENTATIVE (represents the speaker's belief of something)  Hachey and Grover BIBREF13 used SVM and maximum entropy classifiers to classify sentences in legal documents into classes such as FACT, PROCEEDINGS, BACKGROUND, FRAMING, DISPOSAL; see also BIBREF14   Deshpande et al  BIBREF15 proposes unsupervised linguistic patterns to classify sentences into classes SUGGESTION, COMPLAINT 
There is much work on a closely related problem viz , classifying sentences in dialogues through dialogue-specific categories called dialogue acts BIBREF16 , which we will not review here  Just as one example, Cotterill BIBREF17 classifies questions in emails into the dialogue acts of YES_NO_QUESTION, WH_QUESTION, ACTION_REQUEST, RHETORICAL, MULTIPLE_CHOICE etc 
We could not find much work related to mining of performance appraisals data  Pawar et al  BIBREF18 uses kernel-based classification to classify sentences in both performance appraisal text and product reviews into classes SUGGESTION, APPRECIATION, COMPLAINT  Apte et al  BIBREF6 provides two algorithms for matching the descriptions of goals or tasks assigned to employees to a standard template of model goals  One algorithm is based on the co-training framework and uses goal descriptions and self-appraisal comments as two separate perspectives  The second approach uses semantic similarity under a weak supervision framework  Ramrakhiyani et al  BIBREF5 proposes label propagation algorithms to discover aspects in supervisor assessments in performance appraisals, where an aspect is modelled as a verb-noun pair (e g  conduct training, improve coding) 
Dataset
In this paper, we used the supervisor assessment and peer feedback text produced during the performance appraisal of 4528 employees in a large multi-national IT company  The corpus of supervisor assessment has 26972 sentences  The summary statistics about the number of words in a sentence is: min:4 max:217 average:15 5 STDEV:9 2 Q1:9 Q2:14 Q3:19 
Sentence Classification
The PA corpus contains several classes of sentences that are of interest  In this paper, we focus on three important classes of sentences viz , sentences that discuss strengths (class STRENGTH), weaknesses of employees (class WEAKNESS) and suggestions for improving her performance (class SUGGESTION)  The strengths or weaknesses are mostly about the performance in work carried out, but sometimes they can be about the working style or other personal qualities  The classes WEAKNESS and SUGGESTION are somewhat overlapping; e g , a suggestion may address a perceived weakness  Following are two example sentences in each class 
STRENGTH:
WEAKNESS:
SUGGESTION:
Several linguistic aspects of these classes of sentences are apparent  The subject is implicit in many sentences  The strengths are often mentioned as either noun phrases (NP) with positive adjectives (Excellent technology leadership) or positive nouns (engineering strength) or through verbs with positive polarity (dedicated) or as verb phrases containing positive adjectives (delivers innovative solutions)  Similarly for weaknesses, where negation is more frequently used (presentations are not his forte), or alternatively, the polarities of verbs (avoid) or adjectives (poor) tend to be negative  However, sometimes the form of both the strengths and weaknesses is the same, typically a stand-alone sentiment-neutral NP, making it difficult to distinguish between them; e g , adherence to timing or timely closure  Suggestions often have an imperative mood and contain secondary verbs such as need to, should, has to  Suggestions are sometimes expressed using comparatives (better process compliance)  We built a simple set of patterns for each of the 3 classes on the POS-tagged form of the sentences  We use each set of these patterns as an unsupervised sentence classifier for that class  If a particular sentence matched with patterns for multiple classes, then we have simple tie-breaking rules for picking the final class  The pattern for the STRENGTH class looks for the presence of positive words / phrases like takes ownership, excellent, hard working, commitment, etc  Similarly, the pattern for the WEAKNESS class looks for the presence of negative words / phrases like lacking, diffident, slow learner, less focused, etc  The SUGGESTION pattern not only looks for keywords like should, needs to but also for POS based pattern like “a verb in the base form (VB) in the beginning of a sentence” 
We randomly selected 2000 sentences from the supervisor assessment corpus and manually tagged them (dataset D1)  This labelled dataset contained 705, 103, 822 and 370 sentences having the class labels STRENGTH, WEAKNESS, SUGGESTION or OTHER respectively  We trained several multi-class classifiers on this dataset  Table TABREF10 shows the results of 5-fold cross-validation experiments on dataset D1  For the first 5 classifiers, we used their implementation from the SciKit Learn library in Python (scikit-learn org)  The features used for these classifiers were simply the sentence words along with their frequencies  For the last 2 classifiers (in Table TABREF10 ), we used our own implementation  The overall accuracy for a classifier is defined as INLINEFORM0 , where the denominator is 2000 for dataset D1  Note that the pattern-based approach is unsupervised i e , it did not use any training data  Hence, the results shown for it are for the entire dataset and not based on cross-validation 
Comparison with Sentiment Analyzer
We also explored whether a sentiment analyzer can be used as a baseline for identifying the class labels STRENGTH and WEAKNESS  We used an implementation of sentiment analyzer from TextBlob to get a polarity score for each sentence  Table TABREF13 shows the distribution of positive, negative and neutral sentiments across the 3 class labels STRENGTH, WEAKNESS and SUGGESTION  It can be observed that distribution of positive and negative sentiments is almost similar in STRENGTH as well as SUGGESTION sentences, hence we can conclude that the information about sentiments is not much useful for our classification problem 
Discovering Clusters within Sentence Classes
After identifying sentences in each class, we can now answer question (1) in Section SECREF1   From 12742 sentences predicted to have label STRENGTH, we extract nouns that indicate the actual strength, and cluster them using a simple clustering algorithm which uses the cosine similarity between word embeddings of these nouns  We repeat this for the 9160 sentences with predicted label WEAKNESS or SUGGESTION as a single class  Tables TABREF15 and TABREF16 show a few representative clusters in strengths and in weaknesses, respectively  We also explored clustering 12742 STRENGTH sentences directly using CLUTO BIBREF19 and Carrot2 Lingo BIBREF20 clustering algorithms  Carrot2 Lingo discovered 167 clusters and also assigned labels to these clusters  We then generated 167 clusters using CLUTO as well  CLUTO does not generate cluster labels automatically, hence we used 5 most frequent words within the cluster as its labels  Table TABREF19 shows the largest 5 clusters by both the algorithms  It was observed that the clusters created by CLUTO were more meaningful and informative as compared to those by Carrot2 Lingo  Also, it was observed that there is some correspondence between noun clusters and sentence clusters  E g  the nouns cluster motivation expertise knowledge talent skill (Table TABREF15 ) corresponds to the CLUTO sentence cluster skill customer management knowledge team (Table TABREF19 )  But overall, users found the nouns clusters to be more meaningful than the sentence clusters 
PA along Attributes
In many organizations, PA is done from a predefined set of perspectives, which we call attributes  Each attribute covers one specific aspect of the work done by the employees  This has the advantage that we can easily compare the performance of any two employees (or groups of employees) along any given attribute  We can correlate various performance attributes and find dependencies among them  We can also cluster employees in the workforce using their supervisor ratings for each attribute to discover interesting insights into the workforce  The HR managers in the organization considered in this paper have defined 15 attributes (Table TABREF20 )  Each attribute is essentially a work item or work category described at an abstract level  For example, FUNCTIONAL_EXCELLENCE covers any tasks, goals or activities related to the software engineering life-cycle (e g , requirements analysis, design, coding, testing etc ) as well as technologies such as databases, web services and GUI 
In the example in Section SECREF4 , the first sentence (which has class STRENGTH) can be mapped to two attributes: FUNCTIONAL_EXCELLENCE and BUILDING_EFFECTIVE_TEAMS  Similarly, the third sentence (which has class WEAKNESS) can be mapped to the attribute INTERPERSONAL_EFFECTIVENESS and so forth  Thus, in order to answer the second question in Section SECREF1 , we need to map each sentence in each of the 3 classes to zero, one, two or more attributes, which is a multi-class multi-label classification problem 
We manually tagged the same 2000 sentences in Dataset D1 with attributes, where each sentence may get 0, 1, 2, etc  up to 15 class labels (this is dataset D2)  This labelled dataset contained 749, 206, 289, 207, 91, 223, 191, 144, 103, 80, 82, 42, 29, 15, 24 sentences having the class labels listed in Table TABREF20 in the same order  The number of sentences having 0, 1, 2, or more than 2 attributes are: 321, 1070, 470 and 139 respectively  We trained several multi-class multi-label classifiers on this dataset  Table TABREF21 shows the results of 5-fold cross-validation experiments on dataset D2 
Precision, Recall and F-measure for this multi-label classification are computed using a strategy similar to the one described in BIBREF21   Let INLINEFORM0 be the set of predicted labels and INLINEFORM1 be the set of actual labels for the INLINEFORM2 instance  Precision and recall for this instance are computed as follows: INLINEFORM3
It can be observed that INLINEFORM0 would be undefined if INLINEFORM1 is empty and similarly INLINEFORM2 would be undefined when INLINEFORM3 is empty  Hence, overall precision and recall are computed by averaging over all the instances except where they are undefined  Instance-level F-measure can not be computed for instances where either precision or recall are undefined  Therefore, overall F-measure is computed using the overall precision and recall 
Summarization of Peer Feedback using ILP
The PA system includes a set of peer feedback comments for each employee  To answer the third question in Section SECREF1 , we need to create a summary of all the peer feedback comments about a given employee  As an example, following are the feedback comments from 5 peers of an employee 
The individual sentences in the comments written by each peer are first identified and then POS tags are assigned to each sentence  We hypothesize that a good summary of these multiple comments can be constructed by identifying a set of important text fragments or phrases  Initially, a set of candidate phrases is extracted from these comments and a subset of these candidate phrases is chosen as the final summary, using Integer Linear Programming (ILP)  The details of the ILP formulation are shown in Table TABREF36   As an example, following is the summary generated for the above 5 peer comments 
humble nature, effective communication, technical expertise, always supportive, vast knowledge
Following rules are used to identify candidate phrases:
Various parameters are used to evaluate a candidate phrase for its importance  A candidate phrase is more important:
A complete list of parameters is described in detail in Table TABREF36  
There is a trivial constraint INLINEFORM0 which makes sure that only INLINEFORM1 out of INLINEFORM2 candidate phrases are chosen  A suitable value of INLINEFORM3 is used for each employee depending on number of candidate phrases identified across all peers (see Algorithm SECREF6 )  Another set of constraints ( INLINEFORM4 to INLINEFORM5 ) make sure that at least one phrase is selected for each of the leadership attributes  The constraint INLINEFORM6 makes sure that multiple phrases sharing the same headword are not chosen at a time  Also, single word candidate phrases are chosen only if they are adjectives or nouns with lexical category noun attribute  This is imposed by the constraint INLINEFORM7   It is important to note that all the constraints except INLINEFORM8 are soft constraints, i e  there may be feasible solutions which do not satisfy some of these constraints  But each constraint which is not satisfied, results in a penalty through the use of slack variables  These constraints are described in detail in Table TABREF36  
The objective function maximizes the total importance score of the selected candidate phrases  At the same time, it also minimizes the sum of all slack variables so that the minimum number of constraints are broken 
INLINEFORM0 : No  of candidate phrases INLINEFORM1 : No  of phrases to select as part of summary
INLINEFORM0 INLINEFORM1 INLINEFORM2 INLINEFORM3 INLINEFORM4 INLINEFORM5 INLINEFORM6 INLINEFORM7 INLINEFORM8
INLINEFORM0 and INLINEFORM1 INLINEFORM2 INLINEFORM3 INLINEFORM4 INLINEFORM5 INLINEFORM6
INLINEFORM0 (For determining number of phrases to select to include in summary)
Evaluation of auto-generated summaries
We considered a dataset of 100 employees, where for each employee multiple peer comments were recorded  Also, for each employee, a manual summary was generated by an HR personnel  The summaries generated by our ILP-based approach were compared with the corresponding manual summaries using the ROUGE BIBREF22 unigram score  For comparing performance of our ILP-based summarization algorithm, we explored a few summarization algorithms provided by the Sumy package  A common parameter which is required by all these algorithms is number of sentences keep in the final summary  ILP-based summarization requires a similar parameter K, which is automatically decided based on number of total candidate phrases  Assuming a sentence is equivalent to roughly 3 phrases, for Sumy algorithms, we set number of sentences parameter to the ceiling of K/3  Table TABREF51 shows average and standard deviation of ROUGE unigram f1 scores for each algorithm, over the 100 summaries  The performance of ILP-based summarization is comparable with the other algorithms, as the two sample t-test does not show statistically significant difference  Also, human evaluators preferred phrase-based summary generated by our approach to the other sentence-based summaries 
Conclusions and Further Work
In this paper, we presented an analysis of the text generated in Performance Appraisal (PA) process in a large multi-national IT company  We performed sentence classification to identify strengths, weaknesses and suggestions for improvements found in the supervisor assessments and then used clustering to discover broad categories among them  As this is non-topical classification, we found that SVM with ADWS kernel BIBREF18 produced the best results  We also used multi-class multi-label classification techniques to match supervisor assessments to predefined broad perspectives on performance  Logistic Regression classifier was observed to produce the best results for this topical classification  Finally, we proposed an ILP-based summarization technique to produce a summary of peer feedback comments for a given employee and compared it with manual summaries 
The PA process also generates much structured data, such as supervisor ratings  It is an interesting problem to compare and combine the insights from discovered from structured data and unstructured text  Also, we are planning to automatically discover any additional performance attributes to the list of 15 attributes currently used by HR 
Context 2: Introduction
Decoding intended speech or motor activity from brain signals is one of the major research areas in Brain Computer Interface (BCI) systems BIBREF0 , BIBREF1   In particular, speech-related BCI technologies attempt to provide effective vocal communication strategies for controlling external devices through speech commands interpreted from brain signals BIBREF2   Not only do they provide neuro-prosthetic help for people with speaking disabilities and neuro-muscular disorders like locked-in-syndrome, nasopharyngeal cancer, and amytotropic lateral sclerosis (ALS), but also equip people with a better medium to communicate and express thoughts, thereby improving the quality of rehabilitation and clinical neurology BIBREF3 , BIBREF4   Such devices also have applications in entertainment, preventive treatments, personal communication, games, etc  Furthermore, BCI technologies can be utilized in silent communication, as in noisy environments, or situations where any sort of audio-visual communication is infeasible 
Among the various brain activity-monitoring modalities in BCI, electroencephalography (EEG) BIBREF5 , BIBREF6 has demonstrated promising potential to differentiate between various brain activities through measurement of related electric fields  EEG is non-invasive, portable, low cost, and provides satisfactory temporal resolution  This makes EEG suitable to realize BCI systems  EEG data, however, is challenging: these data are high dimensional, have poor SNR, and suffer from low spatial resolution and a multitude of artifacts  For these reasons, it is not particularly obvious how to decode the desired information from raw EEG signals  Although the area of BCI based speech intent recognition has received increasing attention among the research community in the past few years, most research has focused on classification of individual speech categories in terms of discrete vowels, phonemes and words BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 , BIBREF15   This includes categorization of imagined EEG signal into binary vowel categories like /a/, /u/ and rest BIBREF7 , BIBREF8 , BIBREF9 ; binary syllable classes like /ba/ and /ku/ BIBREF1 , BIBREF10 , BIBREF11 , BIBREF12 ; a handful of control words like 'up', 'down', 'left', 'right' and 'select' BIBREF15 or others like 'water', 'help', 'thanks', 'food', 'stop' BIBREF13 , Chinese characters BIBREF14 , etc  Such works mostly involve traditional signal processing or manual feature handcrafting along with linear classifiers (e g , SVMs)  In our recent work BIBREF16 , we introduced deep learning models for classification of vowels and words that achieved 23 45% improvement of accuracy over the baseline 
Production of articulatory speech is an extremely complicated process, thereby rendering understanding of the discriminative EEG manifold corresponding to imagined speech highly challenging  As a result, most of the existing approaches failed to achieve satisfactory accuracy on decoding speech tokens from the speech imagery EEG data  Perhaps, for these reasons, very little work has been devoted to relating the brain signals to the underlying articulation  The few exceptions include BIBREF17 , BIBREF18   In BIBREF17 , Zhao et al  used manually handcrafted features from EEG data, combined with speech audio and facial features to achieve classification of the phonological categories varying based on the articulatory steps  However, the imagined speech classification accuracy based on EEG data alone, as reported in BIBREF17 , BIBREF18 , are not satisfactory in terms of accuracy and reliability  We now turn to describing our proposed models 
Proposed Framework
Cognitive learning process underlying articulatory speech production involves incorporation of intermediate feedback loops and utilization of past information stored in the form of memory as well as hierarchical combination of several feature extractors  To this end, we develop our mixed neural network architecture composed of three supervised and a single unsupervised learning step, discussed in the next subsections and shown in Fig  FIGREF1   We formulate the problem of categorizing EEG data based on speech imagery as a non-linear mapping INLINEFORM0 of a multivariate time-series input sequence INLINEFORM1 to fixed output INLINEFORM2 , i e, mathematically INLINEFORM3 : INLINEFORM4 , where c and t denote the EEG channels and time instants respectively 
Preprocessing step
We follow similar pre-processing steps on raw EEG data as reported in BIBREF17 (ocular artifact removal using blind source separation, bandpass filtering and subtracting mean value from each channel) except that we do not perform Laplacian filtering step since such high-pass filtering may decrease information content from the signals in the selected bandwidth 
Joint variability of electrodes
Multichannel EEG data is high dimensional multivariate time series data whose dimensionality depends on the number of electrodes  It is a major hurdle to optimally encode information from these EEG data into lower dimensional space  In fact, our investigation based on a development set (as we explain later) showed that well-known deep neural networks (e g , fully connected networks such as convolutional neural networks, recurrent neural networks and autoencoders) fail to individually learn such complex feature representations from single-trial EEG data  Besides, we found that instead of using the raw multi-channel high-dimensional EEG requiring large training times and resource requirements, it is advantageous to first reduce its dimensionality by capturing the information transfer among the electrodes  Instead of the conventional approach of selecting a handful of channels as BIBREF17 , BIBREF18 , we address this by computing the channel cross-covariance, resulting in positive, semi-definite matrices encoding the connectivity of the electrodes  We define channel cross-covariance (CCV) between any two electrodes INLINEFORM0 and INLINEFORM1 as: INLINEFORM2   Next, we reject the channels which have significantly lower cross-covariance than auto-covariance values (where auto-covariance implies CCV on same electrode)  We found this measure to be essential as the higher cognitive processes underlying speech planning and synthesis involve frequent information exchange between different parts of the brain  Hence, such matrices often contain more discriminative features and hidden information than mere raw signals  This is essentially different than our previous work BIBREF16 where we extract per-channel 1-D covariance information and feed it to the networks  We present our sample 2-D EEG cross-covariance matrices (of two individuals) in Fig  FIGREF2  
CNN & LSTM
In order to decode spatial connections between the electrodes from the channel covariance matrix, we use a CNN BIBREF19 , in particular a four-layered 2D CNN stacking two convolutional and two fully connected hidden layers  The INLINEFORM0 feature map at a given CNN layer with input INLINEFORM1 , weight matrix INLINEFORM2 and bias INLINEFORM3 is obtained as: INLINEFORM4   At this first level of hierarchy, the network is trained with the corresponding labels as target outputs, optimizing a cross-entropy cost function  In parallel, we apply a four-layered recurrent neural network on the channel covariance matrices to explore the hidden temporal features of the electrodes  Namely, we exploit an LSTM BIBREF20 consisting of two fully connected hidden layers, stacked with two LSTM layers and trained in a similar manner as CNN 
Deep autoencoder for spatio-temporal information
As we found the individually-trained parallel networks (CNN and LSTM) to be useful (see Table TABREF12 ), we suspected the combination of these two networks could provide a more powerful discriminative spatial and temporal representation of the data than each independent network  As such, we concatenate the last fully-connected layer from the CNN with its counterpart in the LSTM to compose a single feature vector based on these two penultimate layers  Ultimately, this forms a joint spatio-temporal encoding of the cross-covariance matrix 
In order to further reduce the dimensionality of the spatio-temporal encodings and cancel background noise effects BIBREF21 , we train an unsupervised deep autoenoder (DAE) on the fused heterogeneous features produced by the combined CNN and LSTM information  The DAE forms our second level of hierarchy, with 3 encoding and 3 decoding layers, and mean squared error (MSE) as the cost function 
Classification with Extreme Gradient Boost
At the third level of hierarchy, the discrete latent vector representation of the deep autoencoder is fed into an Extreme Gradient Boost based classification layer BIBREF22 , BIBREF23 motivated by BIBREF21   It is a regularized gradient boosted decision tree that performs well on structured problems
Context 3: Introduction
Natural text generation, as a key task in NLP, has been advanced substantially thanks to the flourish of neural models BIBREF0 , BIBREF1   Typical frameworks such as sequence-to-sequence (seq2seq) have been applied to various generation tasks, including machine translation BIBREF2 and dialogue generation BIBREF3   The standard paradigm to train such neural models is maximum likelihood estimation (MLE), which maximizes the log-likelihood of observing each word in the text given the ground-truth proceeding context BIBREF4  
Although widely used, MLE suffers from the exposure bias problem BIBREF5 , BIBREF6 : during test, the model sequentially predicts the next word conditioned on its previous generated words while during training conditioned on ground-truth words  To tackle this problem, generative adversarial networks (GAN) with reinforcement learning (RL) training approaches have been introduced to text generation tasks BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , where the discriminator is trained to distinguish real and generated text samples to provide reward signals for the generator, and the generator is optimized via policy gradient BIBREF7  
However, recent studies have shown that potential issues of training GANs on discrete data are more severe than exposure bias BIBREF14 , BIBREF15   One of the fundamental issues when generating discrete text samples with GANs is training instability  Updating the generator with policy gradient always leads to an unstable training process because it's difficult for the generator to derive positive and stable reward signals from the discriminator even with careful pre-training BIBREF8   As a result, the generator gets lost due to the high variance of reward signals and the training process may finally collapse BIBREF16  
In this paper, we propose a novel adversarial training framework called Adversarial Reward Augmented Maximum Likelihood (ARAML) to deal with the instability issue of training GANs for text generation  At each iteration of adversarial training, we first train the discriminator to assign higher rewards to real data than to generated samples  Then, inspired by reward augmented maximum likelihood (RAML) BIBREF17 , the generator is updated on the samples acquired from a stationary distribution with maximum likelihood estimation (MLE), weighted by the discriminator's rewards  This stationary distribution is designed to guarantee that training samples are surrounding the real data, thus the exploration space of our generator is indeed restricted by the MLE training objective, resulting in more stable training  Compared to other text GANs with RL training techniques, our framework acquires samples from the stationary distribution rather than the generator's distribution, and uses RAML training paradigm to optimize the generator instead of policy gradient  Our contributions are mainly as follows:
Related Work
Recently, text generation has been widely studied with neural models trained with maximum likelihood estimation BIBREF4   However, MLE tends to generate universal text BIBREF18   Various methods have been proposed to enhance the generation quality by refining the objective function BIBREF18 , BIBREF19 or modifying the generation distribution with external information like topic BIBREF20 , sentence type BIBREF21 , emotion BIBREF22 and knowledge BIBREF23  
As mentioned above, MLE suffers from the exposure bias problem BIBREF5 , BIBREF6   Thus, reinforcement learning has been introduced to text generation tasks such as policy gradient BIBREF6 and actor-critic BIBREF24   BIBREF17 proposed an efficient and stable approach called Reward Augmented Maximum Likelihood (RAML), which connects the log-likelihood and expected rewards to incorporate MLE training objective into RL framework 
Since some text generation tasks have no explicit metrics to be directly optimized, adversarial training has been applied to generating discrete text samples with a discriminator to learn a proper reward  For instance, SeqGAN BIBREF7 devised a discriminator to distinguish the real data and generated samples, and a generator to maximize the reward from the discriminator via policy gradient  Other variants of GANs have been proposed to improve the generator or the discriminator  To improve the generator, MaliGAN BIBREF8 developed a normalized maximum likelihood optimization target for the generator to stably model the discrete sequences  LeakGAN BIBREF11 guided the generator with reward signals leaked from the discriminator at all generation steps to deal with long text generation task  MaskGAN BIBREF10 employed an actor-critic architecture to make the generator fill in missing text conditioned on the surrounding context, which is expected to mitigate the problem of mode collapse  As for the discriminator, RankGAN BIBREF9 replaced traditional discriminator with a ranker to learn the relative ranking information between the real texts and generated ones  Inverse reinforcement learning BIBREF12 used a trainable reward approximator as the discriminator to provide dense reward signals at each generation step  DPGAN BIBREF13 introduced a language model based discriminator and regarded cross-entropy as rewards to promote the diversity of generation results 
The most similar works to our model are RAML BIBREF17 and MaliGAN BIBREF8 : 1) Compared with RAML, our model adds a discriminator to learn the reward signals instead of choosing existing metrics as rewards  We believe that our model can adapt to various text generation tasks, particularly those without explicit evaluation metrics  2) Unlike MaliGAN, we acquire samples from a fixed distribution near the real data rather than the generator's distribution, which is expected to make the training process more stable 
Task Definition and Model Overview
Text generation can be formulated as follows: given the real data distribution INLINEFORM0 , the task is to train a generative model INLINEFORM1 where INLINEFORM2 can fit INLINEFORM3 well  In this formulation, INLINEFORM4 and INLINEFORM5 denotes a word in the vocabulary INLINEFORM6  
Figure FIGREF3 shows the overview of our model ARAML  This adversarial training framework consists of two phases: 1) The discriminator is trained to assign higher rewards to real data than to generated data  2) The generator is trained on the samples acquired from a stationary distribution with reward augmented MLE training objective  This training paradigm of the generator indeed constrains the search space with the MLE training objective, which alleviates the issue of unstable training 
Discriminator
The discriminator INLINEFORM0 aims to distinguish real data and generated data like other GANs  Inspired by Least-Square GAN BIBREF25 , we devise the loss function as follows: DISPLAYFORM0
This loss function forces the discriminator to assign higher rewards to real data than to generated data, so the discriminator can learn to provide more proper rewards as the training proceeds 
Generator
The training objective of our generator INLINEFORM0 is derived from the objective of other discrete GANs with RL training method: DISPLAYFORM0
where INLINEFORM0 denotes the rewards from the discriminator INLINEFORM1 and the entropy regularized term INLINEFORM2 encourages INLINEFORM3 to generate diverse text samples  INLINEFORM4 is a temperature hyper-parameter to balance these two terms 
As mentioned above, discrete GANs suffer from the instability issue due to policy gradient, thus they are consequently difficult to train  Inspired by RAML BIBREF17 , we introduce an exponential payoff distribution INLINEFORM0 to connect RL loss with RAML loss: DISPLAYFORM0
where INLINEFORM0   Thus, we can rewrite INLINEFORM1 with INLINEFORM2 and INLINEFORM3 as follows: DISPLAYFORM0
Following RAML, we remove the constant term and optimize the KL divergence in the opposite direction: DISPLAYFORM0
where INLINEFORM0 is a constant in the training phase of the generator  It has been proved that INLINEFORM1 and INLINEFORM2 are equivalent up to their first order Taylor approximations, and they have the same global optimum BIBREF17   INLINEFORM3 can be trained in a MLE-like fashion but sampling from the distribution INLINEFORM4 is intractable in the adversarial setting, because INLINEFORM5 varies with the discriminator INLINEFORM6   Thus, we introduce importance sampling to separate sampling process from INLINEFORM7 and obtain the final loss function: DISPLAYFORM0
where INLINEFORM0 denotes a stationary distribution and INLINEFORM1   To optimize this loss function, we first construct the fixed distribution INLINEFORM2 to get samples, and devise the proper reward function INLINEFORM3 to train the generator in a stable and effective way 
We construct the distribution INLINEFORM0 based on INLINEFORM1 : DISPLAYFORM0
In this way, INLINEFORM0 can be designed to guarantee that INLINEFORM1 is near INLINEFORM2 , leading to a more stable training process  To obtain a new sample INLINEFORM3 from a real data sample INLINEFORM4 , we can design three steps which contain sampling an edit distance INLINEFORM5 , the positions INLINEFORM6 for substitution and the new words INLINEFORM7 filled into the corresponding positions  Thus, INLINEFORM8 can be decomposed into three terms: DISPLAYFORM0
The first step is to sample an edit distance based on a real data sample INLINEFORM0 , where INLINEFORM1 is a sequence of length INLINEFORM2   The number of sentences which have the edit distance INLINEFORM3 to some input sentence can be computed approximately as below: DISPLAYFORM0
where INLINEFORM0 denotes the number of sentences which have an edit distance INLINEFORM1 to a sentence of length INLINEFORM2 , and INLINEFORM3 indicates the size of vocabulary  We then follow BIBREF17 to re-scale the counts by INLINEFORM4 and do normalization, so that we can sample an edit distance INLINEFORM5 from: DISPLAYFORM0
where INLINEFORM0 , as a temperature hyper-parameter, restricts the search space surrounding the original sentence  Larger INLINEFORM1 brings more samples with long edit distances 
The next step is to select positions for substitution based on the sampled edit distance INLINEFORM0   Intuitively, we can randomly choose INLINEFORM1 distinct positions in INLINEFORM2 to be replaced by new words  The probability of choosing the position INLINEFORM3 is calculated as follows: DISPLAYFORM0
Following this sampling strategy, we can obtain the position set INLINEFORM0   This strategy approximately guarantees that the edit distance between a new sentence and the original sentence is INLINEFORM1  
At the final step, our model determines new words for substitution at each sampled position INLINEFORM0   We can formulate this sampling process from the original sequence INLINEFORM1 to a new sample INLINEFORM2 as a sequential transition INLINEFORM3   At each step from INLINEFORM4 to INLINEFORM5 INLINEFORM6 , we first sample a new word INLINEFORM7 from the distribution INLINEFORM8 , then replace the old word at position INLINEFORM9 of INLINEFORM10 to obtain INLINEFORM11   The whole sampling process can be decomposed as follows: DISPLAYFORM0
There are two common sampling strategies to model INLINEFORM0 , i e  random sampling and constrained sampling  Random sampling strategy samples a new word INLINEFORM1 according to the uniform distribution over the vocabulary INLINEFORM2 BIBREF17 , while constrained sampling strategy samples INLINEFORM3 to maximize the language model score of the target sentence INLINEFORM4 BIBREF26 , BIBREF27   Here, we adopt constrained sampling in our model and compare the performances of two strategies in the experiment 
We devise the reward function INLINEFORM0 according to the discriminator's output INLINEFORM1 and the stationary distribution INLINEFORM2 : DISPLAYFORM0
Intuitively, this reward function encourages the generator to generate sentences with large sampling probability and high rewards from the discriminator  Thus, the weight of samples INLINEFORM0 can be calculated as follows: DISPLAYFORM0
So far, we can successfully optimize the generator's loss INLINEFORM0 via Equation EQREF12   This training paradigm makes our generator avoid possible variances caused by policy gradient and get more stable reward signals from the discriminator, because our generator is restricted to explore the training samples near the real data 
[htb] Adversarial Reward Augmented Maximum Likelihood [1]
Total adversarial training iterations: INLINEFORM0
Steps of training generator: INLINEFORM0
Steps of training discriminator: INLINEFORM0
Pre-train the generator INLINEFORM0 with MLE loss Generate samples from INLINEFORM1 Pre-train the discriminator INLINEFORM2 via Eq ( EQREF6 ) Construct INLINEFORM3 via Eq ( EQREF14 ) - Eq ( EQREF19 ) each INLINEFORM4 each INLINEFORM5 Update INLINEFORM6 via Eq ( EQREF12 ) each INLINEFORM7 Update INLINEFORM8 via Eq ( EQREF6 )
Extension to Conditional Text Generation
We have shown our adversarial training framework for text generation tasks without an input  Actually, it can also be extended to conditional text generation tasks like dialogue generation  Given the data distribution INLINEFORM0 where INLINEFORM1 denote contexts and responses respectively, the objective function of ARAML's generator can be modified as below: DISPLAYFORM0
where INLINEFORM0 and INLINEFORM1 is trained to distinguish whether INLINEFORM2 is the true response to INLINEFORM3  
Comparison with RAML and MaliGAN
The most similar works to our framework are RAML BIBREF17 and MaliGAN BIBREF8   The main difference among them is the training objective of their generators  We have shown different objective functions in Table TABREF26   For comparison, we use the form with no input for all the three models 
Our model is greatly inspired by RAML, which gets samples from a non-parametric distribution INLINEFORM0 constructed based on a specific reward  Compared to RAML, our reward comes from a learnable discriminator which varies as the adversarial training proceeds rather than a specific reward function  This difference equips our framework with the ability to adapt to the text generation tasks with no explicit evaluation metrics as rewards 
Our model is also similar to MaliGAN, which gets samples from the generator's distribution  In MaliGAN's training objective, INLINEFORM0 also indicates the generator's distribution but it's used in the sampling phase and fixed at each optimization step  The weight of samples INLINEFORM1   Different from our model, MaliGAN acquires samples from the generator's distribution INLINEFORM2 , which usually brings samples with low rewards even with careful pre-training for the generator, leading to training instability  Instead, our framework gets samples from a stationary distribution INLINEFORM3 around real data, thus our training process is more stable 
Datasets
We evaluated ARAML on three datasets: COCO image caption dataset BIBREF28 , EMNLP2017 WMT dataset and WeiboDial single-turn dialogue dataset BIBREF29 

ground_truth: Precision, Recall, F-measure, accuracy

Precision, Recall and F-measure

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What data was presented to the subjects to elicit event-related responses?
Context 2:  Since our EEG-phonological pairwise classification has an internal structure involving individual phonemes and words, it seems to be a reasonable choice of classifier  The classifier receives its input from the latent vectors of the deep autoencoder and is trained in a supervised manner to output the final predicted classes corresponding to the speech imagery 
Dataset
We evaluate our model on a publicly available dataset, KARA ONE BIBREF17 , composed of multimodal data for stimulus-based, imagined and articulated speech state corresponding to 7 phonemic/syllabic ( /iy/, /piy/, /tiy/, /diy/, /uw/, /m/, /n/ ) as well as 4 words(pat, pot, knew and gnaw)  The dataset consists of 14 participants, with each prompt presented 11 times to each individual  Since our intention is to classify the phonological categories from human thoughts, we discard the facial and audio information and only consider the EEG data corresponding to imagined speech  It is noteworthy that given the mixed nature of EEG signals, it is reportedly challenging to attain a pairwise EEG-phoneme mapping BIBREF18   In order to explore the problem space, we thus specifically target five binary classification problems addressed in BIBREF17 , BIBREF18 , i e presence/absence of consonants, phonemic nasal, bilabial, high-front vowels and high-back vowels 
Training and hyperparameter selection
We performed two sets of experiments with the single-trial EEG data  In PHASE-ONE, our goals was to identify the best architectures and hyperparameters for our networks with a reasonable number of runs  For PHASE-ONE, we randomly shuffled and divided the data (1913 signals from 14 individuals) into train (80%), development (10%) and test sets (10%)  In PHASE-TWO, in order to perform a fair comparison with the previous methods reported on the same dataset, we perform a leave-one-subject out cross-validation experiment using the best settings we learn from PHASE-ONE 
The architectural parameters and hyperparameters listed in Table TABREF6 were selected through an exhaustive grid-search based on the validation set of PHASE-ONE  We conducted a series of empirical studies starting from single hidden-layered networks for each of the blocks and, based on the validation accuracy, we increased the depth of each given network and selected the optimal parametric set from all possible combinations of parameters  For the gradient boosting classification, we fixed the maximum depth at 10, number of estimators at 5000, learning rate at 0 1, regularization coefficient at 0 3, subsample ratio at 0 8, and column-sample/iteration at 0 4  We did not find any notable change of accuracy while varying other hyperparameters while training gradient boost classifier 
Performance analysis and discussion
To demonstrate the significance of the hierarchical CNN-LSTM-DAE method, we conducted separate experiments with the individual networks in PHASE-ONE of experiments and summarized the results in Table TABREF12 From the average accuracy scores, we observe that the mixed network performs much better than individual blocks which is in agreement with the findings in BIBREF21   A detailed analysis on repeated runs further shows that in most of the cases, LSTM alone does not perform better than chance  CNN, on the other hand, is heavily biased towards the class label which sees more training data corresponding to it  Though the situation improves with combined CNN-LSTM, our analysis clearly shows the necessity of a better encoding scheme to utilize the combined features rather than mere concatenation of the penultimate features of both networks 
The very fact that our combined network improves the classification accuracy by a mean margin of 14 45% than the CNN-LSTM network indeed reveals that the autoencoder contributes towards filtering out the unrelated and noisy features from the concatenated penultimate feature set  It also proves that the combined supervised and unsupervised neural networks, trained hierarchically, can learn the discriminative manifold better than the individual networks and it is crucial for improving the classification accuracy  In addition to accuracy, we also provide the kappa coefficients BIBREF24 of our method in Fig  FIGREF14   Here, a higher mean kappa value corresponding to a task implies that the network is able to find better discriminative information from the EEG data beyond random decisions  The maximum above-chance accuracy (75 92%) is recorded for presence/absence of the vowel task and the minimum (49 14%) is recorded for the INLINEFORM0  
To further investigate the feature representation achieved by our model, we plot T-distributed Stochastic Neighbor Embedding (tSNE) corresponding to INLINEFORM0 and V/C classification tasks in Fig  FIGREF8   We particularly select these two tasks as our model exhibits respectively minimum and maximum performance for these two  The tSNE visualization reveals that the second set of features are more easily separable than the first one, thereby giving a rationale for our performance 
Next, we provide performance comparison of the proposed approach with the baseline methods for PHASE-TWO of our study (cross-validation experiment) in Table TABREF15   Since the model encounters the unseen data of a new subject for testing, and given the high inter-subject variability of the EEG data, a reduction in the accuracy was expected  However, our network still managed to achieve an improvement of 18 91, 9 95, 67 15, 2 83 and 13 70 % over BIBREF17   Besides, our best model shows more reliability compared to previous works: The standard deviation of our model's classification accuracy across all the tasks is reduced from 22 59% BIBREF17 and 17 52% BIBREF18 to a mere 5 41% 
Conclusion and future direction
In an attempt to move a step towards understanding the speech information encoded in brain signals, we developed a novel mixed deep neural network scheme for a number of binary classification tasks from speech imagery EEG data  Unlike previous approaches which mostly deal with subject-dependent classification of EEG into discrete vowel or word labels, this work investigates a subject-invariant mapping of EEG data with different phonological categories, varying widely in terms of underlying articulator motions (eg: involvement or non-involvement of lips and velum, variation of tongue movements etc)  Our model takes an advantage of feature extraction capability of CNN, LSTM as well as the deep learning benefit of deep autoencoders  We took BIBREF17 , BIBREF18 as the baseline works investigating the same problem and compared our performance with theirs  Our proposed method highly outperforms the existing methods across all the five binary classification tasks by a large average margin of 22 51% 
Acknowledgments
This work was funded by the Natural Sciences and Engineering Research Council (NSERC) of Canada and Canadian Institutes for Health Research (CIHR) 

ground_truth: 7 phonemic/syllabic ( /iy/, /piy/, /tiy/, /diy/, /uw/, /m/, /n/ ) as well as 4 words(pat, pot, knew and gnaw)

KARA ONE BIBREF17 , composed of multimodal data for stimulus-based, imagined and articulated speech state corresponding to 7 phonemic/syllabic ( /iy/, /piy/, /tiy/, /diy/, /uw/, /m/, /n/ ) as well as 4 words(pat, pot, knew and gnaw)

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How much improvement is gained from Adversarial Reward Augmented Maximum Likelihood (ARAML)?
Context 3:  COCO and EMNLP2017 WMT are the common benchmarks with no input to evaluate the performance of discrete GANs, and we followed the existing works to preprocess these datasets BIBREF12 , BIBREF11   WeiboDial, as a dialogue dataset, was applied to test the performance of our model with input trigger  We simply removed post-response pairs containing low-frequency words and randomly selected a subset for our training/test set  The statistics of three datasets are presented in Table TABREF28  
Baselines
We compared our model with MLE, RL and GAN baselines  Since COCO and EMNLP2017 WMT don't have input while WeiboDial regards posts as input, we chose the following baselines respectively:
MLE: a RNN model trained with MLE objective BIBREF4   Its extension, Seq2Seq, can work on the dialogue dataset BIBREF2  
SeqGAN: The first text GAN model that updates the generator with policy gradient based on the rewards from the discriminator BIBREF7  
LeakGAN: A variant of SeqGAN that provides rewards based on the leaked information of the discriminator for the generator BIBREF11  
MaliGAN: A variant of SeqGAN that optimizes the generator with a normalized maximum likelihood objective BIBREF8  
IRL: This inverse reinforcement learning method replaces the discriminator with a reward approximator to provide dense rewards BIBREF12  
RAML: A RL approach to incorporate MLE objective into RL training framework, which regards BLEU as rewards BIBREF17  
DialogGAN: An extension of SeqGAN tuned to dialogue generation task with MLE objective added to the adversarial objective BIBREF16  
DPGAN: A variant of DialogGAN which uses a language model based discriminator and regards cross-entropy as rewards BIBREF13  
Note that MLE, SeqGAN, LeakGAN, MaliGAN and IRL are the baselines on COCO and EMNLP2017 WMT, while MLE, RAML, DialogGAN, and DPGAN on WeiboDial  The original codes are used to test the baselines 
Implementation Details
The implementation details of our model are shown in Table TABREF31   For COCO / EMNLP2017, the generator is a LSTM unit BIBREF30 with 128 cells, and the discriminator is implemented based on BIBREF7   For WeiboDial, the generator is an encoder-decoder structure with attention mechanism, where both the encoder and the decoder consist of a two-layer GRU BIBREF31 with 128 cells  The discriminator is implemented based on BIBREF32   The language model used in the constrained sampling of ARAML is implemented in the same setting as the generators, and is pre-trained on the training set of each dataset  The codes and the datasets are available at https://github com/kepei1106/ARAML 
As for the details of the baselines, the generators of all the baselines except LeakGAN are the same as ours  Note that the generator of LeakGAN consists of a hierarchical LSTM unit, thus we followed the implementation in the original paper  In terms of the differences, the discriminators of GAN baselines are implemented based on the original papers  Other hyper-parameters of baselines including batch size, learning rate, and pre-training epochs, were set based on the original codes, because the convergence of baselines is sensitive to these hyper-parameters 
Language Generation on COCO and EMNLP2017 WMT
We adopted forward/reverse perplexity BIBREF33 and Self-BLEU BIBREF34 to evaluate the quality of generated texts  Forward perplexity (PPL-F) indicates the perplexity on the generated data provided by a language model trained on real data to measure the fluency of generated samples  Reverse perplexity (PPL-R) switches the roles of generated data and real data to reflect the discrepancy between the generated distribution and the data distribution  Self-BLEU (S-BLEU) regards each sentence in the generated collection as hypothesis and the others as reference to obtain BLEU scores, which evaluates the diversity of generated results 
Results are shown in Table TABREF33   LeakGAN performs best on forward perplexity because it can generate more fluent samples  As for reverse perplexity, our model ARAML beats other baselines, showing that our model can fit the data distribution better  Other GANs, particularly LeakGAN, obtain high reverse perplexity due to mode collapse BIBREF12 , thus they only capture limited fluent expressions, resulting in large discrepancy between the generated distribution and data distribution  ARAML also outperforms the baselines in terms of Self-BLEU, indicating that our model doesn't fall into mode collapse with the help of the MLE training objective and has the ability to generate more diverse sentences 
We also provide standard deviation of each metric in Table TABREF33 , reflecting the stability of each model's performance  Our model ARAML nearly achieves the smallest standard deviation in all the metrics, indicating that our framework outperforms policy gradient in the stability of adversarial training 
Dialogue Generation on WeiboDial
Dialogue evaluation is an open problem and existing works have found that automatic metrics have low correlation to human evaluation BIBREF35 , BIBREF36 , BIBREF37   Thus, we resorted to manual evaluation to assess the generation quality on WeiboDial  We randomly sampled 200 posts from the test set and collected the generated results from all the models  For each pair of responses (one from ARAML and the other from a baseline, given the same input post), five annotators were hired to label which response is better (i e  win, lose or tie) in terms of grammaticality (whether a response itself is grammatical and logical) and relevance (whether a response is appropriate and relevant to the post)  The two metrics were evaluated independently 
The evaluation results are shown in Table TABREF35   To measure the inter-annotator agreement, we calculated Fleiss' kappa BIBREF38 for each pair-wise comparison where results show moderate agreement ( INLINEFORM0 )  We also conducted sign test to check the significance of the differences 
As shown in Table TABREF35 , ARAML performs significantly better than other baselines in all the cases  This result indicates that the samples surrounding true responses provide stable rewards for the generator, and stable RAML training paradigm significantly enhances the performance in both metrics 
Further Analysis on Stability
To verify the training stability, we conducted experiments on COCO many times and chose the best 5 trials for SeqGAN, LeakGAN, IRL, MaliGAN and ARAML, respectively  Then, we presented the forward/reverse perplexity in the training process in Figure FIGREF38   We can see that our model with smaller standard deviation is more stable than other GAN baselines in both metrics  Although LeakGAN reaches the best forward perplexity, its standard deviation is extremely large and it performs badly in reverse perplexity, indicating that it generates limited expressions that are grammatical yet divergent from the data distribution 
Ablation Study
The temperature INLINEFORM0 controls the search space surrounding the real data as we analyze in Section UID13   To investigate its impact on the performance of our model, we fixed all the other hyper-parameters and test ARAML with different temperatures on COCO 
The experimental results are shown in Figure FIGREF41   We can see that as the temperature becomes larger, forward perplexity increases gradually while Self-BLEU decreases  As mentioned in Section UID13 , large temperatures encourage our generator to explore the samples that are distant from real data distribution, thus the diversity of generated results will be improved  However, these samples distant from the data distribution are more likely to be poor in fluency, leading to worse forward perplexity  Reverse perplexity is influenced by both generation quality and diversity, so the correlation between temperature and reverse perplexity is not intuitive  We can observe that the model with INLINEFORM0 reaches the best reverse perplexity 
We have mentioned two common sampling strategies in Section UID13 , i e  random sampling and constrained sampling  To analyze their impact, we keep all the model structures and hyper-parameters fixed and test ARAML with these two strategies on COCO 
Table TABREF45 shows the results  It's obvious that random sampling hurts the model performance except Self-BLEU-1, because it indeed allows low-quality samples available to the generator  Exploring these samples degrades the quality and diversity of generated results  Despite the worse performance on automatic metrics, random sampling doesn't affect the training stability of our framework  The standard deviation of ARAML-R is still smaller than other GAN baselines 
Case Study
Table TABREF47 presents the examples generated by the models on COCO  We can find that other baselines suffer from grammatical errors (e g  “in front of flying her kite" from MLE), repetitive expressions (e g  “A group of people" from IRL) and incoherent statements (e g  “A group of people sitting on a cell phone” from IRL)  By contrast, our model performs well in these sentences and has the ability to generate grammatical and coherent results 
Table TABREF48 shows the generated examples on WeiboDial  It's obvious that other baselines don't capture the topic word “late" in the post, thus generate irrelevant responses  ARAML can provide a response that is grammatical and closely relevant to the post 
Conclusion
We propose a novel adversarial training framework to deal with the instability problem of current GANs for text generation  To address the instability issue caused by policy gradient, we incorporate RAML into the advesarial training paradigm to make our generator acquire stable rewards  Experiments show that our model performs better than several state-of-the-art GAN baselines with lower training variance, yet producing better performance on three text generation tasks 
Acknowledgments
This work was supported by the National Science Foundation of China (Grant No  61936010/61876096) and the National Key R&D Program of China (Grant No  2018YFC0830200)  We would like to thank THUNUS NExT Joint-Lab for the support 

ground_truth: ARAM has achieved improvement over all baseline methods using reverese perplexity and  slef-BLEU metric.  The maximum reverse perplexity improvement 936,16 is gained for EMNLP2017  WMT  dataset and  48,44 for COCO dataset.

Compared to the baselines, ARAML does not do better in terms of perplexity on COCO and EMNLP 2017 WMT datasets, but it does by up to 0.27 Self-BLEU points on COCO and 0.35 Self-BLEU on EMNLP 2017 WMT. In terms of Grammaticality and Relevance, it scores better than the baselines on up to 75.5% and 73% of the cases respectively.