Introduction
The challenge in Natural Language Inference (NLI), also known as Recognizing Textual Entailment (RTE), is to correctly decide whether a sentence (referred to as a premise) entails or contradicts or is neutral in respect to another sentence (a hypothesis). This classification task requires various natural language comprehension skills. In this paper, we are focused on the following natural language generation task based on NLI. Given the premise the goal is to generate a stream of hypotheses that comply with the label (entailment, contradiction or neutral). In addition to reading capabilities this task also requires language generation capabilities.
The Stanford Natural Language Inference (SNLI) Corpus BIBREF0 is a NLI dataset that contains over a half a million examples. The size of the dataset is sufficient to train powerful neural networks. Several successful classification neural networks have already been proposed BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 . In this paper, we utilize SNLI to train generative neural networks. Each example in the dataset consist of two human-written sentences, a premise and a hypothesis, and a corresponding label that describes the relationship between them. Few examples are presented in Table TABREF1 .
The proposed generative networks are trained to generate a hypothesis given a premise and a label, which allow us to construct new, unseen examples. Some generative models are build to generate a single optimal response given the input. Such models have been applied to machine translation BIBREF5 , image caption generation BIBREF6 , or dialogue systems BIBREF7 . Another type of generative models are autoencoders that generate a stream of random samples from the original distribution. For instance, autoencoders have been used to generate text BIBREF8 , BIBREF9 , and images BIBREF10 . In our setting we combine both approaches to generate a stream of random responses (hypotheses) that comply with the input (premise, label).
But what is a good stream of hypotheses? We argue that a good stream contains diverse, comprehensible, accurate and non-trivial hypotheses. A hypothesis is comprehensible if it is grammatical and semantically makes sense. It is accurate if it clearly expresses the relationship (signified by the label) with the premise. Finally, it is non-trivial if it is not trivial to determine the relationship (label) between the hypothesis and premise. For instance, given a premise ”A man drives a red car” and label entailment, the hypothesis ”A man drives a car” is more trivial than ”A person is sitting in a red vehicle”.
The next question is how to automatically measure the quality of generated hypotheses. One way is to use metrics that are standard in text generation tasks, for instance ROUGE BIBREF11 , BLEU BIBREF12 , METEOR BIBREF13 . These metrics estimate the similarity between the generated text and the original reference text. In our task they can be used by comparing the generated and reference hypotheses with the same premise and label. The main issue of these metrics is that they penalize the diversity since they penalize the generated hypotheses that are dissimilar to the reference hypothesis. An alternative metric is to use a NLI classifier to test the generated hypothesis if the input label is correct in respect to the premise. A perfect classifier would not penalize diverse hypotheses and would reward accurate and (arguably to some degree) comprehensible hypotheses. However, it would not reward non-trivial hypotheses.
Non-trivial examples are essential in a dataset for training a capable machine learning model. Furthermore, we make the following hypothesis.
A good dataset for training a NLI classifier consists of a variety of accurate, non-trivial and comprehensible examples.
Based on this hypothesis, we propose the following approach for evaluation of generative models, which is also presented in Figure FIGREF2 . First, the generative model is trained on the original training dataset. Then, the premise and label from an example in the original dataset are taken as the input to the generative model to generate a new random hypothesis. The generated hypothesis is combined with the premise and the label to form a new unseen example. This is done for every example in the original dataset to construct a new dataset. Next, a classifier is trained on the new dataset. Finally, the classifier is evaluated on the original test set. The accuracy of the classifier is the proposed quality metric for the generative model. It can be compared to the accuracy of the classifier trained on the original training set and tested on the original test set.
The generative models learn solely from the original training set to regenerate the dataset. Thus, the model learns the distribution of the original dataset. Furthermore, the generated dataset is just a random sample from the estimated distribution. To determine how well did the generative model learn the distribution, we observe how close does the accuracy of the classifier trained on the generated dataset approach the accuracy of classifier trained on the original dataset.
Our flagship generative network EmbedDecoder works in a similar fashion as the encoder-decoder networks, where the encoder is used to transform the input into a low-dimensional latent representation, from which the decoder reconstructs the input. The difference is that EmbedDecoder consists only of the decoder, and the latent representation is learned as an embedding for each training example separately. In our models, the latent representation represents the mapping between the premise and the label on one side and the hypothesis on the other side.
Our main contributions are i) a novel generative neural network, which consist of the decoder that learns a mapping embedding for each training example separately, ii) a procedure for generating NLI datasets automatically, iii) and a novel evaluation metric for NLI generative models – the accuracy of the classifier trained on the generated dataset.
In Section SECREF2 we present the related work. In Section SECREF3 the considered neural networks are presented. Besides the main generative networks, we also present classification and discriminative networks, which are used for evaluation. The results are presented in Section SECREF5 , where the generative models are evaluated and compared. From the experiments we can see that the best dataset was generated by the attention-based model EmbedDecoder. The classifier on this dataset achieved accuracy of INLINEFORM0 , which is INLINEFORM1 less than the accuracy achieved on the original dataset. We also investigate the influence of latent dimensionality on the performance, compare different evaluation metrics, and provide deeper insights of the generated datasets. The conclusion is presented in Section SECREF6 .
Related Work
NLI has been the focal point of Recognizing Textual Entailment (RTE) Challenges, where the goal is to determine if the premise entails the hypothesis or not. The proposed approaches for RTE include bag-of-words matching approach BIBREF14 , matching predicate argument structure approach BIBREF15 and logical inference approach BIBREF16 , BIBREF17 . Another rule-based inference approach was proposed by BIBREF18 . This approach allows generation of new hypotheses by transforming parse trees of the premise while maintaining entailment. BIBREF19 proposes an approach for constructing training datasets by extracting sentences from news articles that tend to be in an entailment relationship.
After SNLI dataset was released several neural network approaches for NLI classification have emerged. BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 . The state-of-the-art model BIBREF4 achieves INLINEFORM0 accuracy on the SNLI dataset. A similar generation approach to ours was proposed by BIBREF20 , The goal of this work is generating entailment inference chains, where only examples with entailment label are used.
Natural Lanuguage Generation (NLG) is a task of generating natural language from a structured form such as knowledge base or logic form BIBREF21 , BIBREF22 , BIBREF23 . The input in our task is unstructured text (premise) and label. On the other side of this spectrum, there are tasks that deal solely with unstructured text, like machine translation BIBREF24 , BIBREF25 , BIBREF26 , summarization BIBREF27 , BIBREF28 and conversational dialogue systems BIBREF7 , BIBREF29 . Another recently popular task is generating captions from images BIBREF30 , BIBREF31 .
With the advancement of deep learning, many neural network approaches have been introduced for generating sequences. The Recurrent Neural Network Language Model (RNNLM) BIBREF32 is one of the simplest neural architectures for generating text. The approach was extended by BIBREF5 , which use encoder-decoder architecture to generate a sequence from the input sequence. The Hierarchical Recurrent Encoder-Decoder (HRED) architecture BIBREF7 generates sequences from several input sequences. These models offer very little variety of output sequences. It is obtained by modeling the output distribution of the language model. To introduce more variety, models based on variational autoencoder (VAE) BIBREF33 have been proposed. These models use stochastic random variables as a source of variety. In BIBREF8 a latent variable is used to initial the RNN that generates sentences, while the variational recurrent neural network (VRNN) BIBREF34 models the dependencies between latent variables across subsequent steps of RNN. The Latent Variable Hierarchical Recurrent Encoder-Decoder (VHRED) BIBREF35 extends the HRED by incorporating latent variables, which are learned similarly than in VAE. The latent variables are, like in some of our models, used to represent the mappings between sequences. Conditional variational autoencoders (CVAEs) BIBREF36 were used to generate images from continuous visual attributes. These attributes are conditional information that is fed to the models, like the discrete label is in our models.
As recognized by BIBREF37 , the evaluation metrics of text-generating models fall into three categories: manual evaluation, automatic evaluation metrics, task-based evaluation. In evaluation based on human judgment each generated textual example is inspected manually. The automatic evaluation metrics, like ROUGE, BLEU and METEOR, compare human texts and generated texts. BIBREF38 shows METEOR has the strongest correlation with human judgments in image description evaluation. The last category is task-based evaluation, where the impact of the generated texts on a particular task is measured. This type of evaluation usually involves costly and lengthy human involvement, like measuring the effectiveness of smoking-cessation letters BIBREF39 . On the other hand, the task in our evaluation, the NLI classification, is automatic. In BIBREF40 ranking was used as an automatic task-based evaluation for associating images with captions.
Models
In this section, we present several neural networks used in the experiments. We start with variants of Recurrent Neural Networks, which are essential layers in all our models. Then, we present classification networks, which are needed in evaluation of generative neural networks presented in the following section. Next, we present how to use generative networks to generate hypothesis. Finally, we present discriminative networks, which are used for evaluation and analysis of the hypotheses.
The premise INLINEFORM0 and hypothesis INLINEFORM1 are represented with word embeddings INLINEFORM2 and INLINEFORM3 respectively. Each INLINEFORM4 is a INLINEFORM5 -dimensional vector that represents the corresponding word, INLINEFORM6 is the length of premise, and INLINEFORM7 is the length of hypothesis. The labels (entailment, contradiction, neutral) are represented by a 3-dimensional vector INLINEFORM8 if the label is the output of the model, or INLINEFORM9 if the label is the input to the model.
Recurrent Neural Networks
The Recurrent Neural Networks (RNNs) are neural networks suitable for processing sequences. They are the basic building block in all our networks. We use two variants of RNNs – Long short term memory (LSTM) network BIBREF41 and an attention-based extension of LSTM, the mLSTM BIBREF2 . The LSTM tends to learn long-term dependencies better than vanilla RNNs. The input to the LSTM is a sequence of vectors INLINEFORM0 , and the output is a sequence of vectors INLINEFORM1 . At each time point INLINEFORM2 , input gate INLINEFORM3 , forget gate INLINEFORM4 , output gate INLINEFORM5 , cell state INLINEFORM6 and one output vector INLINEFORM7 are calculated. DISPLAYFORM0
where INLINEFORM0 is a sigmoid function, INLINEFORM1 is the element-wise multiplication operator, INLINEFORM2 and INLINEFORM3 are parameter matrices, INLINEFORM4 parameter vectors, INLINEFORM5 is the input vector dimension, and INLINEFORM6 is the output vector dimension. The vectors INLINEFORM7 and INLINEFORM8 are set to zero in the standard setting, however, in some cases in our models, they are set to a value that is the result of previous layers.
The mLSTM is an attention-based model with two input sequences – premise and hypothesis in case of NLI. Each word of the premise is matched against each word of the hypothesis to find the soft alignment between the sentences. The mLSTM is based on LSTM in such a way that it remembers the important matches and forgets the less important. The input to the LSTM inside the mLSTM at each time step is INLINEFORM0 , where INLINEFORM1 is an attention vector that represents the weighted sum of premise sequence, where the weights present the degree to which each token of the premise is aligned with the INLINEFORM2 -th token of the hypothesis INLINEFORM3 , and INLINEFORM4 is the concatenation operator. More details about mLSTM are presented in BIBREF2 .
Classification model
The classification model predicts the label of the example given the premise and the hypothesis. We use the mLSTM-based model proposed by BIBREF2 .
The architecture of the model is presented in Figure FIGREF9 . The embeddings of the premise INLINEFORM0 and hypothesis INLINEFORM1 are the input to the first two LSTMs to obtain the hidden states of the premise INLINEFORM2 and hypothesis INLINEFORM3 . DISPLAYFORM0
All the hidden states in our models are INLINEFORM0 -dimensional unless otherwise noted. The hidden states INLINEFORM1 and INLINEFORM2 are the input to the mLSTM layer. The output of mLSTM are hidden states INLINEFORM3 , although only the last state INLINEFORM4 is further used. A fully connected layer transforms it into a 3-dimensional vector, on top of which softmax function is applied to obtain the probabilities INLINEFORM5 of labels. DISPLAYFORM0
where INLINEFORM0 represents the fully connected layer, whose output size is INLINEFORM1 .
Generative models
The goal of the proposed generative models, is to generate a diverse stream of hypotheses given the premise and the label. In this section, we present four variants of generative models, two variants of EmbedDecoder model presented in Figure FIGREF11 , and two variants of EncoderDecoder model presented in Figure FIGREF11 .
All models learn a latent representation INLINEFORM0 that represents the mapping between the premise and the label on one side, and the hypothesis on the other side. The EmbedDecoder models learn the latent representation by learning an embedding of the mapping for each training example separately. The embedding for INLINEFORM1 -th training example INLINEFORM2 is a INLINEFORM3 -dimensional trainable parameter vector. Consequentely, INLINEFORM4 is a parameter matrix of all embeddings, where INLINEFORM5 is the number of training examples. On the other hand, in EncoderDecoder models latent representation is the output of the decoder.
The EmbedDecoder models are trained to predict the next word of the hypothesis given the previous words of hypothesis, the premise, the label, and the latent representation of the example. DISPLAYFORM0
where INLINEFORM0 represent parameters other than INLINEFORM1 , and INLINEFORM2 is the length of the hypothesis INLINEFORM3 .
The AttEmbedDecoder, presented in Figure FIGREF26 , is attention based variant of EmbedDecoder. The same mLSTM layer is used as in classification model. However, the initial cell state INLINEFORM0 of mLSTM is constructed from the latent vector and the label input. DISPLAYFORM0
For the sake of simplifying the notation, we dropped the superscript INLINEFORM0 from the equations, except in INLINEFORM1 , where we explicitly want to state that the embedding vector is used.
The premise and the hypothesis are first processed by LSTM and then fed into the mLSTM, like in the classification model, however here the hypothesis is shifted. The first word of the hypothesis input is an empty token INLINEFORM0 null INLINEFORM1 , symbolizing the empty input sequence when predicting the first word. The output of the mLSTM is a hidden state INLINEFORM2 , where each INLINEFORM3 represents an output word. To obtain the probabilities for all the words in the vocabulary INLINEFORM4 for the position INLINEFORM5 in the output sequence, INLINEFORM6 is first transformed into a vocabulary-sized vector, then the softmax function is applied. DISPLAYFORM0
where V is the size of the vocabulary. But, due to the large size of the vocabulary, a two-level hierarchical softmax BIBREF42 was used instead of a regular softmax to reduce the number of parameters updated during each training step. DISPLAYFORM0
In the training step, the last output word INLINEFORM0 is set to INLINEFORM1 null INLINEFORM2 , while in the generating step, it is ignored.
In the EmbedDecoder model without attention, BaseEmbedDecoder, the mLSTM is replaced by a regular LSTM. The input to this LSTM is the shifted hypothesis. But, here the premise is provided through the initial cell state INLINEFORM0 . Specifically, last hidden state of the premise is merged with class input and the latent representation, then fed to the LSTM. DISPLAYFORM0
In order to not lose information INLINEFORM0 was picked to be equal to sum of the sizes of INLINEFORM1 , INLINEFORM2 and INLINEFORM3 . Thus, INLINEFORM4 . Since the size of INLINEFORM5 is INLINEFORM6 , the output vectors of the LSTM are also the size of INLINEFORM7 .
We also present two variants of EncoderDecoder models, a regular one BaseEncodeDecoder, and a regularized one VarEncoderDecoder, which is based on Variational Bayesian approach. As presented in Figure FIGREF11 , all the information (premise, hypothesis, label) is available to the encoder, whose output is the latent representation INLINEFORM0 . On the other hand, the decoder is provided with the same premise and label, but the hypothesis is shifted. This forces the encoder to learn to encode only the missing information – the mapping between premise-label pair and the hypothesis. The encoder has a similar structure as the classification model in Figure FIGREF9 . Except that the label is connected to the initial cell state of the mLSTM DISPLAYFORM0
and the output of mLSTM INLINEFORM0 is transformed into latent representation INLINEFORM1 DISPLAYFORM0
The decoder is the same as in EmbedDecoder.
The VarEncoderDecoder models is based on Variational Autoencoder from BIBREF33 . Instead of using single points for latent representation as in all previous models, the latent representation in VarEncoderDecoder is presented as a continuous variable INLINEFORM0 . Thus, the mappings are presented as a soft elliptical regions in the latent space, instead of a single points, which forces the model to fill up the latent space BIBREF8 . Both INLINEFORM1 and INLINEFORM2 are calculated form the output of the encoder using two different fully connected layers. INLINEFORM3
To sample from the distribution the reparametrization trick is applied DISPLAYFORM0
When training, a single sample is generated per example to generate INLINEFORM0 .
As in BIBREF33 , the following regularization term is added to the loss function DISPLAYFORM0
Generating hypotheses
In the generation phase only decoder of a trained generative model is used. It generates a hypothesis given the premise, label, and a randomly selected latent vector INLINEFORM0 . A single word is generated in each step, and it becomes the hypothesis input in the next step. DISPLAYFORM0
We also used beam search to optimize hypothesis generation. Similarly as in BIBREF5 , a small number of hypotheses are generated given a single input, then the best is selected. In INLINEFORM0 -beam search, in each time step INLINEFORM1 best partial hypotheses are expanded by all the words in the vocabulary producing INLINEFORM2 partial hypothesis. Out of these INLINEFORM3 best partial hypotheses are selected for the next step according to the joint probability of each partial hypothesis. Thus, when INLINEFORM4 is 1, the procedure is the same as the one presented in Eq EQREF24 . The generation ends when INLINEFORM5 null INLINEFORM6 symbol is encountered or maximum hypothesis length is reached. The random latent vector INLINEFORM10 is selected randomly from a normal distribution INLINEFORM11 , where INLINEFORM12 is the standard deviation of INLINEFORM13 .
Discriminative model
The discriminative model is used to measure the distinguishability between the original human written sentences and the generated ones. Higher error rate of the model means that the generative distribution is similar to the original distribution, which is one of the goals on the generative model. The model is based on Generative Adversarial Nets BIBREF10 , where in a single network the generative part tires to trick the discriminative part by generating images that are similar to the original images, and the discriminative part tries to distinguish between the original and generated images. Due to the discreteness of words (the output of our generative model) it is difficult to connect both the discriminative and generative part in a single differentiable network, thus we construct them separately. The generative models have already been defined in Section SECREF10 . Here we define the discriminative model.
The discriminative model INLINEFORM0 takes sequence INLINEFORM1 and process it with LSTM and fully connected layer DISPLAYFORM0
In the training step, one original sequence INLINEFORM0 and one generated sequence INLINEFORM1 are processed by the discriminative model. The optimization function maximizes the following objective DISPLAYFORM0
In the testing step, the discriminative model predicts correctly if DISPLAYFORM0
Dataset Generation
To construct a new dataset, first a generative model is trained on the training set of the original dataset. Then, a new dataset is constructed by generating a new hypotheses with a generative model. The premises and labels from the examples of the original dataset are taken as an input for the generative model. The new hypotheses replace the training hypotheses in the new dataset.
Next, the classifier, presented in Section SECREF6 , is trained on the generated dataset. The accuracy of the new classifier is the main metric for evaluating the quality of the generated dataset.
Experiment details
All the experiments are performed on the SNLI dataset. There are 549,367 examples in the dataset, divided into training, development and test set. Both the development and test set contain around 10.000 examples. Some examples are labeled with '-', which means there was not enough consensus on them. These examples are excluded. Also, to speed up the computation we excluded examples, which have the premise longer than 25 words, or the hypothesis longer than 15 words. There were still INLINEFORM0 remaining examples. Both premises and hypothesis were padded with INLINEFORM1 null INLINEFORM2 symbols (empty words), so that all premises consisted of 25 words, and all hypotheses consisted of 15 tokens.
We use 50-dimensional word vectors trained with GloVe BIBREF43 . For words without pretrained embeddings, the embeddings are randomly selected from the normal distribution. Word embeddings are not updated during training.
For optimization Adam method BIBREF44 was used with suggested hyperparameters.
Classification models are trained until the loss on the validation set does not improve for three epochs. The model with best validation loss is retained.
Generative models are trained for 20 epochs, since it turned out that none of the stopping criteria were useful. With each generative model a new dataset is created. The new dataset consists of training set, which is generated using examples from the original training set, and a development set, which is generated from the original development set. The beam size for beam search was set to 1. The details of the decision are presented in Section SECREF35 .
Some datasets were constructed by filtering the generated datasets according to various thresholds. Thus, the generated datasets were constructed to contain enough examples, so that the filtered datasets had at least the number of examples as the original dataset. In the end, all the datasets were trimmed down to the size of the original dataset by selecting the samples sequentially from the beginning until the dataset had the right size. Also, the datasets were filtered so that each of the labels was represented equally. All the models, including classification and discriminative models, were trained with hidden dimension INLINEFORM0 set to 150, unless otherwise noted.
Our implementation is accessible at http://github.com/jstarc/nli_generation. It is based on libraries Keras and Theano BIBREF45 .
Results
First, the classification model OrigClass was trained on the original dataset. This model was then used throughout the experiments for filtering the datasets, comparison, etc. Notice that we have assumed OrigClass to be ground truth for the purpose of our experiments. However, the accuracy of this model on the original test set was INLINEFORM0 , which is less than INLINEFORM1 , which was attained by mLSTM (d=150) model in BIBREF2 . Both models are very similar, including the experimental settings, however ours was trained and evaluated on a slightly smaller dataset.
Preliminary evaluation
Several AttEmbedDecoder models with various latent dimensions INLINEFORM0 were first trained and then used to generate new datasets. A couple of generated examples are presented in Table TABREF36 .
Figure FIGREF37 shows the accuracies of the generated development datasets evaluated by the OrigClass. The maximum accuracy of INLINEFORM0 was achieved by EmbedDecoder (z=2), and the accuracy is decreasing with the number of dimensions in the latent variable. The analysis for each label shows that the accuracy of contradiction and neutral labels is quite stable, while the accuracy of the entailment examples drops significantly with latent dimensionality. One reason for this is that the hypothesis space of the entailment label is smaller than the spaces of other two labels. Thus, when the dimensionality is higher, more creative examples are generated, and these examples less often comply with the entailment label.
Since none of the generated datasets' accuracies is as high as the accuracy of the OrigClass on the original test set, we used OrigClass to filter the datasets subject to various prediction thresholds. The examples from the generated dataset were classified by OrigClass and if the probability of the label of the example exceeded the threshold INLINEFORM0 , then the example was retained.
For each filtered dataset a classifier was trained. Figure FIGREF38 shows the accuracies of these classifiers on the original test set. Filtering out the examples that have incorrect labels (according to the OrigClass) improves the accuracy of the classifier. However, if the threshold is set too high, the accuracy drops, since the dataset contains examples that are too trivial. Figure FIGREF38 , which represents the accuracy of classifiers on their corresponding generated development sets, further shows the trade-off between the accuracy and triviality of the examples. The classifiers trained on datasets with low latent dimension or high filtering threshold have higher accuracies. Notice that the training dataset and test dataset were generated by the same generative model.
The unfiltered datasets have been evaluated with five other metrics besides classification accuracy. The results are presented in Figure FIGREF41 . The whole figure shows the effect of latent dimensionality of the models on different metrics. The main purpose of the figure is not show absolute values for each of the metrics, but to compare the metrics' curves to the curve of our main metric, the accuracy of the classifier.
The first metric – Premise-Hypothesis Distance – represents the average Jaccard distance between the premise and the generated hypothesis. Datasets generated with low latent dimensions have hypotheses more similar to premises, which indicates that the generated hypotheses are more trivial and less diverse than hypothesis generated with higher latent dimensions.
We also evaluated the models with standard language generation metrics ROUGE-L and METEOR. The metrics are negatively correlated with the accuracy of the classifier. We believe this is because the two metrics reward hypotheses that are similar to their reference (original) hypothesis. However, the classifier is better if trained on more diverse hypotheses.
The next metric is the log-likelihood of hypotheses in the development set. This metric is the negative of the training loss function. The log-likelihood improves with dimensionality since it is easier to fit the hypotheses in the training step having more dimensions. Consequently, the hypothesis in the generating step are more confident – they have lower log-likelihood.
The last metric – discriminative error rate – is calculated with the discriminative model. The model is trained on the hypotheses from the unfiltered generated dataset on one side and the original hypotheses on the other side. Error rate is calculated on the (generated and original) development sets. Higher error rate indicates that it is more difficult for discriminative model to distinguish between the generated and the original hypotheses, which suggests that the original generating distribution and the distribution of the generative model are more similar. The discriminative model detects that low dimensional generative models generate more trivial examples as also indicated by the distance between premise and hypotheses. On the other hand, it also detects the hypotheses of high dimensional models, which more frequently contain grammatic or semantic errors.
There is a positive correlation between the discriminative error rate and the accuracy of the classifier. This observation led us to the experiment, where the generated dataset was filtered according to the prediction probability of the discriminative model. Two disjoint filtered datasets were created. One with hypotheses that had high probability that they come from the original distribution and the other one with low probability. However, the accuracies of classifiers trained on these datasets were very similar to the accuracy of the classifier on the unfiltered dataset. Similar test was also done with the log-likelihood metric. The examples with higher log-likelihood had similar performance than the ones with lower log-likelihood. This also lead us to set the size of the beam to 1. Also, the run time of generating hypothesis is INLINEFORM0 , where INLINEFORM1 is beam size. Thus, with lower beam sizes much more hypotheses can be generated.
To accept the hypothesis from Section SECREF1 we have shown that a quality dataset requires accurate examples by showing that filtering the dataset with the original classifier improves the performance (Figure FIGREF38 ). Next, we have shown that non-trivial examples are also required. If the filtering threshold is set too high, these examples are excluded, and the accuracy drops. Also, the more trivial examples are produced by low-dimensional models, which is indicated by lower premise-hypothesis distances, and lower discriminative error rate (Figure FIGREF41 ). Finally, a quality dataset requires more comprehensible examples. The high dimensional models produce less comprehensible hypotheses. They are detected by the discriminative model (see discriminator error rate in Figure FIGREF41 ).
Other models
We also compared AttEmbedDecoder model to all other models. Table TABREF43 presents the results. For all the models the latent dimension INLINEFORM0 is set to 8, as it was previously shown to be one of the best dimensions.
For all the models the number of total parameters is relatively high, however only a portion of parameters get updated each time. The AttEmbedDecoder model was the best model according to our main metric – the accuracy of the classifier trained on the generated dataset.
The hidden dimension INLINEFORM0 of the BaseEmbedDecoder was selected so that the model was comparable to AttEmbedDecoder in terms of the number of parameters INLINEFORM1 . The accuracies of classifiers generated by BaseEmbedDecoder are still lower than the accuracies of classifiers generated by AttEmbedDecoder, which shows that the attention mechanism helps the models.
Table TABREF44 shows the performance of generated datasets compared to the original one. The best generated dataset was generated by AttEmbedDecoder. The accuracy of its classifier is only 2.7 % lower than the accuracy of classifier generated on the original human crafted dataset. The comparison of the best generated dataset to the original dataset shows that the datasets had only INLINEFORM0 of identical examples. The average length of the hypothesis was INLINEFORM1 and INLINEFORM2 in the original dataset and in the generated dataset, respectively. In another experiment the generated dataset and the original dataset were merged to train a new classifier. Thus, the merged dataset contained twice as many examples as other datasets. The accuracy of this classifier was 82.0%, which is 0.8 % better than the classifier trained solely on the original training set. However, the lowest average loss is achieved by the classifier trained on the original dataset.
Qualitative evaluation
We also did a qualitative evaluation of the generated hypothesis. Hypotheses are mostly grammatically sound. Sometimes the models incorrectly use indefinite articles, for instance ”an phone”, or possessive pronouns ”a man uses her umbrella”. These may be due to the fact the system must learn the right indefinite article for every word separately. On the other hand, the models sometimes generate hypotheses that showcase more advanced grammatical patterns. For instance, hypothesis ”The man and woman have a cake for their family” shows that the model can correctly use plural in a non-trivial setting. Generative neural networks have a tendency to repeat words, which sometimes make sentences meaningless, like ”A cup is drinking from a cup of coffee” or even ungrammatical, like ”Several people in a car car”.
As shown previously the larger is the latent dimension more creative hypotheses are generated. However, with more creativity semantic errors emerge. Some hypotheses are correct, just unlikely to be written by a human, like ”A shirtless man is holding a guitar with a woman and a woman”. Others present improbable events, like ”The girls were sitting in the park watching tv”, or even impossible events, for instance ”The child is waiting for his wife”. This type of errors arise because the models have not learned enough common sense logic. Finally, there are hypotheses, which make no sense. For instance, ”Two women with grassy beach has no tennis equipment”. On the contrary, the models are able to generate some non-trivial hypotheses. From the original premise ”A band performing with a girl singing and a guy next to her singing as well while playing the guitar”, the model has generated some hypotheses that do not contain concepts explicitly found in the premise. For instance, ”People are playing instruments” (entailment), ”The band was entirely silent” (contradiction), or ”The girl is playing at the concert” (neutral).
Regarding the compliance of the hypotheses with the label and premise, we observed that many generated hypotheses are not complying with the label, however they would be a very good example with a different label. For instance, the generated hypotheses represent entailment instead of contradiction. This also explains why the accuracy of the generated dataset measured by the original classifier is low in Figure FIGREF37 . On the other hand, the models generate examples that are more ambiguous and not as clear as those in the original dataset. These examples are harder to classify even for a human. For instance, the relationship between premise ”A kid hitting a baseball in a baseball field” and hypothesis ”The baseball player is trying to get the ball” can be either interpreted either as an entailment if verb get is intepreted as not to miss or contradiction if get is intepreted as possess. For a deeper insight into generated hypothesis more examples are presented in SECREF7 .
The gap between the discriminative error rates (disc-er) of EncoderDecoder models and EmbedDecoder models in Table TABREF43 is significant. To further investigate, the same experiment was performed again by a human evaluator and the discriminative model. This time on a sample of 200 examples. To recap, both the model and human were asked to select the generated hypothesis given a random original and generated hypothesis without knowing which one is which.
Human evaluation confirms that AttEmbedDecoder hypotheses are more difficult to separate from the original one than the hypotheses of VaeEncoderDecoder. Table TABREF46 presents the results. The discriminative model discriminates better than the human evaluator. This may be due to the fact that the discriminative model has learned from a large training set, while the human was not shown any training examples. Human evaluation has shown that generated hypotheses are positively recognized if they contain a grammatical or semantic error. But even if the generated hypothesis does not contain these errors, it sometimes reveals itself by not being as sophisticated as the original example. On the other hand, the discriminative model does not always recognize these discrepancies. It relies more on the differences in distributions learned form a big training set. The true number of non-distinguishable examples may be even higher than indicated by the human discriminator error rate since the human may have correctly guessed some of the examples he could not distinguish.
Conclusion
In this paper, we have proposed several generative neural networks for generating hypothesis using NLI dataset. To evaluate these models we propose the accuracy of classifier trained on the generated dataset as the main metric. The best model achieved INLINEFORM0 accuracy, which is only INLINEFORM1 less than the accuracy of the classifier trained on the original human written dataset, while the best dataset combined with the original dataset has achieved the highest accuracy. This model learns a decoder and a mapping embedding for each training example. It outperforms the more standard encoder-decoder networks. Although more parameters are needed to be trained, less are updated on each batch. We have also shown that the attention mechanism improves the model. The analysis has confirmed our hypothesis that a good dataset contains accurate, non-trivial and comprehensible examples. To further examine the quality of generated hypothesis, they were compared against the original human written hypotheses. The discriminative evaluation shows that in INLINEFORM2 of cases the human evaluator incorrectly distinguished between the original and the generated hypothesis. The discriminative model was actually better in distinguishing. We have also compared the accuracy of classifier to other metrics. The standard text generation metrics ROUGE and METEOR do not indicate if a generated dataset is good for training a classifier.
To obtain higher accuracies of the generated datasets, they need to be filtered, because the generative models produce examples, whose label is not always accurate. Thus, we propose for future work incorporating the classifier into the generative model, in a similar fashion that it was done on images by BIBREF46 . This network could also include the discriminative model to generate examples from a distribution that is more similar to the original training distribution. Finally, constructing a dataset requires a lot of intensive manual work that mainly consists of writing text with some creativity. To extend the original dataset human users could just validate or correct the generated examples. On top of that we would like to develop active learning methods to identify incorrect generated examples that would most improve the dataset if corrected.
Acknowledgements
This work was supported by the Slovenian Research Agency and the ICT Programme of the EC under XLike (ICT-STREP-288342) and XLime (FP7-ICT-611346).
More Examples
In this section more generated hypotheses are presented. Each example starts with the original example data. Then, several hypotheses generated with from the original example with our best model are displayed.

Question: What is the highest accuracy score achieved?
Answer: 82.0%

Question: What is the size range of the datasets?
Answer: Unanswerable