Fake Sentence Detection as a Training Task for Sentence Encoding

1 INTRODUCTION . Universal sentence encoding is a way to scale language processing tasks , especially when the target training data is limited . Solutions to sentence-level language processing tasks can be seen as consisting of two parts – one that creates a generic representation which approximates the meaning of the sentence , and another that uses this representations to make the target classifications . The idea behind universal sentence encoding is to learn generic sentence representations from unlabeled texts , which are often much larger and easier to obtain than the training data for the target tasks . A good universal encoding is one which is effective for training downstream target tasks . The success of language modeling ideas for learning word representations has inspired similar ideas for learning universal sentence representations . Skip-thought model ( Kiros et al. , 2015 ) uses a Bidirectional LSTM ( BiLSTM ) encoder , which encodes a given sentence , and uses the encoding to generate neighboring sentences . Following up on this general idea , the FastSent model ( Hill et al. , 2016 ) simplifies the task and the encoder further by learning to output the average word embeddings so that they can predict words in the neighboring sentences ( but not necessarily in a sequence ) . This language modeling based training is undesirable in two respects . ( 1 ) Predicting neighboring sentences is a difficult generative training task . The task requires having a large output space , at least the size of the vocabulary . It also requires a complex decoding model such as an LSTM decoder with millions of parameters . Effective training requires a large amount of training data with long training cycles . Indeed Skip-thought requires tens of millions of training sentences and it takes more than a week to train . ( 2 ) Even minor changes to a sentence can drastically change its meaning , a phenomenon that needs to be modeled for many NLP tasks . Language model based training essentially relies on the training sample for this kind of generalization , i.e. , the training text collection should include many instances of sentences that have only minor lexical differences but found in completely different contexts . Larger the dataset the more likely it is for the encoder to learn how to model these fine grained distinctions . In this work , we introduce a discriminative training task fake sentence detection to address these challenges . The main idea is to generate fake sentences by corrupting an original sentence . We use two methods to generate fake sentences : word shuffling where we swap the positions of two words at random and word dropping , where we drop a word at random from the original sentence . The resulting fake sentences are mostly similar to the original sentences—a fake sentence differs from its source in at most two word positions . We create the training corpus from a source corpus of unlabeled sentences . For each sentence in the source corpus we add multiple fake sentences by corrupting the source sentence . This training task has three key advantages : ( 1 ) This binary classification task can be modeled with fewer parameters in the output layer and can be trained more efficiently compared to language modeling based training tasks . ( 2 ) From a language standpoint , the task forces the encoder to track both syntax and semantics . For instance , swapping words within a sentence can break the syntax ( e.g. , “ John landed in Chicago on Friday ” versus “ John landed Chicago in on Friday ” ) , and break or alter the semantics ; it can lead to an incoherent or less plausible sentence ( e.g. , “ John landed in Chicago on Friday ” versus “ Chicago landed in John on Friday ” ) . ( 3 ) The task explicitly forces the encoder to capture big shifts in meaning that arise from small changes to a sentence . In particular , since the discrimination is really dependent on a small change to an original sentence , encoders can not get away with simple aggregation—they need to model compositional aspects well enough to be able to detect small but semantically shifts in sentence constructions . In overview , we train a bidirectional LSTM ( BiLSTM ) as our universal encoder and use a threelayer feed-forward network that uses the encoded sentence to predict if it is fake or real . We then evaluate this trained encoder without any further tuning on multiple sentence-level tasks and use probing tasks ( Conneau et al. , 2018 ) to study the syntactic and semantic properties in the encoded representations . In summary , this paper makes the following main contributions : 1 . Introduces fake sentence detection as an unsupervised training task for learning sentence encoders that can distinguish between small changes in mostly similar sentences . 2 . Provides an empirical evaluation on multiple sentence-level tasks showing representations trained on the fake sentence tasks outperform a strong baseline model trained on language modeling tasks , even when training on small amounts of data ( 1M vs. 64M sentences ) reducing training time from weeks to within 20 hours . 3 . Demonstrates that the fake sentence training produces encoded representations that are better at capturing many linguistic properties compared to language model training . 2 MOTIVATION . Many sentence-level prediction tasks require learning a function to estimate the probability of a label y given a sentence x . This function can be parameterized as P ( y|x , θ ) , and the parameter vector θ can be learned by minimizing the negative conditional log likelihood of the training data D , i.e. , minθ − ∑ ( x , y ) ∈D log ( P ( y|x , θ ) ) . However , the amount of training data is limited in many cases , so learning θ by optimizing the conditional probability leads to overfitting , especially if one uses a deep neural network with millions of parameters . One approach to address this problem is to break the task into two sub-tasks each with its own set of parameters : ( 1 ) sentence encoding—produce an effective low-dimensional representation of the sentence enc ( x , θ1 ) , ( 2 ) target prediction—estimate probability of the label y conditioned on the encoding P ( y|x , enc ( x , θ1 ) , θ2 ) . The dimension of θ2 is much smaller than the dimension of θ and it can be learned with the provided labeled training data D. For the encoder function enc ( x , θ1 ) , it is hoped that the parameter vector θ1 can be learned or pre-trained with a different sentence-level task using other data , which is possibly unlabeled , but can be easily collected in large quantity . The encoder can be trained on the auxiliary task as part of a prediction function f which has its own set of parameters θ3 . The encoder parameters θ1 and the auxiliary task parameters θ3 are learned to minimize the loss on the auxiliary task : ∑ x∈U Laux ( f ( enc ( x , θ1 ) , θ3 ) ) . In this work , we are interested in learning an universal sentence encoder that can be subsequently used for various downstream sentence-level tasks . One reasonable choice for the auxiliary loss function would be the negative log likelihood of the data , i.e. , − logP ( x|θ1 , θ3 ) or equivalently − log ( P ( enc ( x , θ1 ) |θ1 , θ3 ) ) . Recall for the downstream task , we will seek the parameter vector θ2 to minimize the negative conditional log likelihood − log ( P ( y|enc ( x , θ1 ) , θ1 , θ2 ) ) . The combination of two loss functions is : − log ( P ( y|enc ( x , θ1 ) , θ1 , θ2 ) ) − log ( P ( enc ( x , θ1 ) |θ1 , θ3 ) ) . This is equivalent to − log ( P ( y , enc ( x , θ1 ) |θ1 , θ2 , θ3 ) ) . In other words , we are maximizing the joint probability of observing the label y and the encoding vector enc ( x , θ1 ) . Optimizing the joint probability of the label and data corresponds to having a generative classifier , as opposed to the discriminative classifier that optimizes a conditional probability . As shown by ( Ng & Jordan , 2002 ) , a generative classifier converges faster to its asymptotic behavior than a discriminative classifier does . A generative approach is particularly useful when there is limited amount of training data . Thus , we propose to learn a sentence encoder by maximizing the data likelihood P ( enc ( x , θ1 ) |θ1 , θ3 ) , and we expect it to be an useful universal sentence encoder , especially when there is limited amount of labeled training data for downstream sentence-level task . It is surprisingly simple to learn an encoder to maximize the data likelihood P ( enc ( x , θ1 ) |θ1 , θ3 ) , and this paper proposes exactly that . We can parameterize this probability using a simple function such as a log-linear model : P ( enc ( x , θ1 ) |θ1 , θ3 ) = 1 Z exp ( −θT3 enc ( x , θ1 ) ) ( 1 ) where the functional form of the encoder enc is an BiLSTM , and Z is a normalization constant to ensure a valid probability function . Z = ∑ x exp ( −θT3 enc ( x , θ1 ) ) = ∑ x∈U exp ( −θT3 enc ( x , θ1 ) ) + ∑ x∈V exp ( −θT3 enc ( x , θ1 ) ) . ( 2 ) where U is the set of real sentences , and V is the set of fake sentences . Our objective is to learn θ1 , θ3 to maximize the quantity in Equation 1 . This is equivalent to maximizing∑ x∈U exp ( −θT3 enc ( x , θ1 ) ) while minimizing ∑ x∈V exp ( −θT3 enc ( x , θ1 ) ) . This can simply be done by training a classifier to separate real sentences from fake sentences . In this work , we approximate the space of fake sentences by randomly swapping or dropping words from real sentences . Our method has some connection to the GAN framework ( Goodfellow et al. , 2014 ) , where there is a generator that attempts to generate realistic data and a discriminator that distinguishes between real and generated data instances . As shown by Goodfellow et al . ( 2014 ) , this approach will converge to a solution where the output of the discriminator is the estimated value for a data point to be real . In our case , we do not need to learn the generator and the discriminator in an adversarial manner because we already have a ‘ strong ’ generator that generates highly realistic sentences , simply by shuffling or dropping words . Furthermore , our method can be thought as a generative model , defined and trained using a discriminative function . This aspect is related to the recently proposed Introspective Neural Networks ( Lazarow et al. , 2017 ) . There are some similarities between the proposed method and Skip-thought sentence encoder . The training task of Skip-thought involves predicting the next or previous sentence z that follows or precedes the current encoded sentence x . This sentence language model is realized using an LSTM decoder , which sequentially generates a word at each position i conditioning on the previous encoded sentence enc ( x ) , and the words that have been decoded so far ( z1 , · · · , zi−1 ) . This model can be seen as maximizing the conditional probability : P ( z|enc ( x , θ1 ) , θ3 ) = |z|−1∏ i=1 P ( zi|z1 : i−1 , enc ( x , θ1 ) , θ3 ) . ( 3 ) Interestingly , both Skip-thought and our proposed method model the unlabeled data U , a sequence of sentences u1 , u2 , · · · , um , and aim to optimize for the joint probability distribution : P ( U|θ1 , θ3 ) . In the case of Skip-thought , there is an underlying Markovian assumption that the probability of a sentence only depends on the previous sentence . That is to approximate P ( U|θ1 , θ3 ) by∏ k P ( u k|enc ( uk−1 , θ1 ) , θ3 ) . In the case of the proposed method , we assume the sentences are independent of one another , so P ( U|θ1 , θ3 ) can be approximated as ∏ k P ( enc ( u k , θ1 ) |θ1 , θ3 ) . Both methods are generative models of the data , and both are useful sentence encoders for the downstream tasks , as will be seen in the experiment sections . However , Skip-thought model has several disadvantages compared to the proposed method : ( 1 ) The architecture of Skip-thought model is much more complex . Skip-thought model requires an LSTM decoder with millions of parameters . Furthermore , the need for computing the probability of individual words requires having a large output space , as large as the size of the vocabulary . This further increases then number of parameters and the complexity of the overall architecture . ( 2 ) The modeling task of Skip-thought is much harder than that of the proposed method . Skipthought requires a probability model for both complete and incomplete sentences , and this much harder than modeling the probability of complete sentences alone ( as proposed here ) . ( 3 ) Due to the complexity of the architecture and the difficulty of the modeling task , Skip-thought requires much more training data and longer training time . ( 4 ) Skip-thought is a purely generative model and it is only trained using the real data . Meanwhile , the proposed method is a generative model defined via a discriminative function . This function can be trained using both real and fake data , where the fake data can be selectively chosen to be very hard examples . Specifically , we use word shuffling and word dropping to generate hard examples that do not differ from the original sentences . This enables the sentence encoder to focus on the subtle semantics information .

Summary: The paper proposes a discriminative training formulation for learning sentence representations, where a classifier is required to distinguish between real and fake sentences. The sentences are encoded with a Bi-LSTM and the resulting sentence representations are then used in a number of sentence-level tasks (classification, entailment, and retrieval). The experiments show benefits on most tasks compared to Skip-Thought and FastSent baselines, and the information captured by the representations is analyzed with probing tasks, showing that they are better at capturing certain kinds of information like the presence or order of words.