Fake Sentence Detection as a Training Task for Sentence Encoding

1 INTRODUCTION . Universal sentence encoding is a way to scale language processing tasks , especially when the target training data is limited . Solutions to sentence-level language processing tasks can be seen as consisting of two parts – one that creates a generic representation which approximates the meaning of the sentence , and another that uses this representations to make the target classifications . The idea behind universal sentence encoding is to learn generic sentence representations from unlabeled texts , which are often much larger and easier to obtain than the training data for the target tasks . A good universal encoding is one which is effective for training downstream target tasks . The success of language modeling ideas for learning word representations has inspired similar ideas for learning universal sentence representations . Skip-thought model ( Kiros et al. , 2015 ) uses a Bidirectional LSTM ( BiLSTM ) encoder , which encodes a given sentence , and uses the encoding to generate neighboring sentences . Following up on this general idea , the FastSent model ( Hill et al. , 2016 ) simplifies the task and the encoder further by learning to output the average word embeddings so that they can predict words in the neighboring sentences ( but not necessarily in a sequence ) . This language modeling based training is undesirable in two respects . ( 1 ) Predicting neighboring sentences is a difficult generative training task . The task requires having a large output space , at least the size of the vocabulary . It also requires a complex decoding model such as an LSTM decoder with millions of parameters . Effective training requires a large amount of training data with long training cycles . Indeed Skip-thought requires tens of millions of training sentences and it takes more than a week to train . ( 2 ) Even minor changes to a sentence can drastically change its meaning , a phenomenon that needs to be modeled for many NLP tasks . Language model based training essentially relies on the training sample for this kind of generalization , i.e. , the training text collection should include many instances of sentences that have only minor lexical differences but found in completely different contexts . Larger the dataset the more likely it is for the encoder to learn how to model these fine grained distinctions . In this work , we introduce a discriminative training task fake sentence detection to address these challenges . The main idea is to generate fake sentences by corrupting an original sentence . We use two methods to generate fake sentences : word shuffling where we swap the positions of two words at random and word dropping , where we drop a word at random from the original sentence . The resulting fake sentences are mostly similar to the original sentences—a fake sentence differs from its source in at most two word positions . We create the training corpus from a source corpus of unlabeled sentences . For each sentence in the source corpus we add multiple fake sentences by corrupting the source sentence . This training task has three key advantages : ( 1 ) This binary classification task can be modeled with fewer parameters in the output layer and can be trained more efficiently compared to language modeling based training tasks . ( 2 ) From a language standpoint , the task forces the encoder to track both syntax and semantics . For instance , swapping words within a sentence can break the syntax ( e.g. , “ John landed in Chicago on Friday ” versus “ John landed Chicago in on Friday ” ) , and break or alter the semantics ; it can lead to an incoherent or less plausible sentence ( e.g. , “ John landed in Chicago on Friday ” versus “ Chicago landed in John on Friday ” ) . ( 3 ) The task explicitly forces the encoder to capture big shifts in meaning that arise from small changes to a sentence . In particular , since the discrimination is really dependent on a small change to an original sentence , encoders can not get away with simple aggregation—they need to model compositional aspects well enough to be able to detect small but semantically shifts in sentence constructions . In overview , we train a bidirectional LSTM ( BiLSTM ) as our universal encoder and use a threelayer feed-forward network that uses the encoded sentence to predict if it is fake or real . We then evaluate this trained encoder without any further tuning on multiple sentence-level tasks and use probing tasks ( Conneau et al. , 2018 ) to study the syntactic and semantic properties in the encoded representations . In summary , this paper makes the following main contributions : 1 . Introduces fake sentence detection as an unsupervised training task for learning sentence encoders that can distinguish between small changes in mostly similar sentences . 2 . Provides an empirical evaluation on multiple sentence-level tasks showing representations trained on the fake sentence tasks outperform a strong baseline model trained on language modeling tasks , even when training on small amounts of data ( 1M vs. 64M sentences ) reducing training time from weeks to within 20 hours . 3 . Demonstrates that the fake sentence training produces encoded representations that are better at capturing many linguistic properties compared to language model training . 2 MOTIVATION . Many sentence-level prediction tasks require learning a function to estimate the probability of a label y given a sentence x . This function can be parameterized as P ( y|x , θ ) , and the parameter vector θ can be learned by minimizing the negative conditional log likelihood of the training data D , i.e. , minθ − ∑ ( x , y ) ∈D log ( P ( y|x , θ ) ) . However , the amount of training data is limited in many cases , so learning θ by optimizing the conditional probability leads to overfitting , especially if one uses a deep neural network with millions of parameters . One approach to address this problem is to break the task into two sub-tasks each with its own set of parameters : ( 1 ) sentence encoding—produce an effective low-dimensional representation of the sentence enc ( x , θ1 ) , ( 2 ) target prediction—estimate probability of the label y conditioned on the encoding P ( y|x , enc ( x , θ1 ) , θ2 ) . The dimension of θ2 is much smaller than the dimension of θ and it can be learned with the provided labeled training data D. For the encoder function enc ( x , θ1 ) , it is hoped that the parameter vector θ1 can be learned or pre-trained with a different sentence-level task using other data , which is possibly unlabeled , but can be easily collected in large quantity . The encoder can be trained on the auxiliary task as part of a prediction function f which has its own set of parameters θ3 . The encoder parameters θ1 and the auxiliary task parameters θ3 are learned to minimize the loss on the auxiliary task : ∑ x∈U Laux ( f ( enc ( x , θ1 ) , θ3 ) ) . In this work , we are interested in learning an universal sentence encoder that can be subsequently used for various downstream sentence-level tasks . One reasonable choice for the auxiliary loss function would be the negative log likelihood of the data , i.e. , − logP ( x|θ1 , θ3 ) or equivalently − log ( P ( enc ( x , θ1 ) |θ1 , θ3 ) ) . Recall for the downstream task , we will seek the parameter vector θ2 to minimize the negative conditional log likelihood − log ( P ( y|enc ( x , θ1 ) , θ1 , θ2 ) ) . The combination of two loss functions is : − log ( P ( y|enc ( x , θ1 ) , θ1 , θ2 ) ) − log ( P ( enc ( x , θ1 ) |θ1 , θ3 ) ) . This is equivalent to − log ( P ( y , enc ( x , θ1 ) |θ1 , θ2 , θ3 ) ) . In other words , we are maximizing the joint probability of observing the label y and the encoding vector enc ( x , θ1 ) . Optimizing the joint probability of the label and data corresponds to having a generative classifier , as opposed to the discriminative classifier that optimizes a conditional probability . As shown by ( Ng & Jordan , 2002 ) , a generative classifier converges faster to its asymptotic behavior than a discriminative classifier does . A generative approach is particularly useful when there is limited amount of training data . Thus , we propose to learn a sentence encoder by maximizing the data likelihood P ( enc ( x , θ1 ) |θ1 , θ3 ) , and we expect it to be an useful universal sentence encoder , especially when there is limited amount of labeled training data for downstream sentence-level task . It is surprisingly simple to learn an encoder to maximize the data likelihood P ( enc ( x , θ1 ) |θ1 , θ3 ) , and this paper proposes exactly that . We can parameterize this probability using a simple function such as a log-linear model : P ( enc ( x , θ1 ) |θ1 , θ3 ) = 1 Z exp ( −θT3 enc ( x , θ1 ) ) ( 1 ) where the functional form of the encoder enc is an BiLSTM , and Z is a normalization constant to ensure a valid probability function . Z = ∑ x exp ( −θT3 enc ( x , θ1 ) ) = ∑ x∈U exp ( −θT3 enc ( x , θ1 ) ) + ∑ x∈V exp ( −θT3 enc ( x , θ1 ) ) . ( 2 ) where U is the set of real sentences , and V is the set of fake sentences . Our objective is to learn θ1 , θ3 to maximize the quantity in Equation 1 . This is equivalent to maximizing∑ x∈U exp ( −θT3 enc ( x , θ1 ) ) while minimizing ∑ x∈V exp ( −θT3 enc ( x , θ1 ) ) . This can simply be done by training a classifier to separate real sentences from fake sentences . In this work , we approximate the space of fake sentences by randomly swapping or dropping words from real sentences . Our method has some connection to the GAN framework ( Goodfellow et al. , 2014 ) , where there is a generator that attempts to generate realistic data and a discriminator that distinguishes between real and generated data instances . As shown by Goodfellow et al . ( 2014 ) , this approach will converge to a solution where the output of the discriminator is the estimated value for a data point to be real . In our case , we do not need to learn the generator and the discriminator in an adversarial manner because we already have a ‘ strong ’ generator that generates highly realistic sentences , simply by shuffling or dropping words . Furthermore , our method can be thought as a generative model , defined and trained using a discriminative function . This aspect is related to the recently proposed Introspective Neural Networks ( Lazarow et al. , 2017 ) . There are some similarities between the proposed method and Skip-thought sentence encoder . The training task of Skip-thought involves predicting the next or previous sentence z that follows or precedes the current encoded sentence x . This sentence language model is realized using an LSTM decoder , which sequentially generates a word at each position i conditioning on the previous encoded sentence enc ( x ) , and the words that have been decoded so far ( z1 , · · · , zi−1 ) . This model can be seen as maximizing the conditional probability : P ( z|enc ( x , θ1 ) , θ3 ) = |z|−1∏ i=1 P ( zi|z1 : i−1 , enc ( x , θ1 ) , θ3 ) . ( 3 ) Interestingly , both Skip-thought and our proposed method model the unlabeled data U , a sequence of sentences u1 , u2 , · · · , um , and aim to optimize for the joint probability distribution : P ( U|θ1 , θ3 ) . In the case of Skip-thought , there is an underlying Markovian assumption that the probability of a sentence only depends on the previous sentence . That is to approximate P ( U|θ1 , θ3 ) by∏ k P ( u k|enc ( uk−1 , θ1 ) , θ3 ) . In the case of the proposed method , we assume the sentences are independent of one another , so P ( U|θ1 , θ3 ) can be approximated as ∏ k P ( enc ( u k , θ1 ) |θ1 , θ3 ) . Both methods are generative models of the data , and both are useful sentence encoders for the downstream tasks , as will be seen in the experiment sections . However , Skip-thought model has several disadvantages compared to the proposed method : ( 1 ) The architecture of Skip-thought model is much more complex . Skip-thought model requires an LSTM decoder with millions of parameters . Furthermore , the need for computing the probability of individual words requires having a large output space , as large as the size of the vocabulary . This further increases then number of parameters and the complexity of the overall architecture . ( 2 ) The modeling task of Skip-thought is much harder than that of the proposed method . Skipthought requires a probability model for both complete and incomplete sentences , and this much harder than modeling the probability of complete sentences alone ( as proposed here ) . ( 3 ) Due to the complexity of the architecture and the difficulty of the modeling task , Skip-thought requires much more training data and longer training time . ( 4 ) Skip-thought is a purely generative model and it is only trained using the real data . Meanwhile , the proposed method is a generative model defined via a discriminative function . This function can be trained using both real and fake data , where the fake data can be selectively chosen to be very hard examples . Specifically , we use word shuffling and word dropping to generate hard examples that do not differ from the original sentences . This enables the sentence encoder to focus on the subtle semantics information .

Summary: Derive sentence representations from a bidirectional LSTM encoder trained to distinguish real sentences from fake ones. Fake sentences are derived from real ones by swapping two words or dropping a single word (yielding two different models). The resulting representations are applied to various sentence classification tasks by using them as input to a logistic regression classifier trained for the task. Results are generally better than similar experiments performed with SkipThought vectors trained on the same Toronto BookCorpus.