Unsupervised Word Discovery with Segmental Neural Language Models

1 INTRODUCTION . How infants discover the words of their native languages is a long-standing question in developmental psychology ( Saffran et al. , 1996 ) . Machine learning has contributed much to this discussion by showing that predictive models of language are capable of inferring the existence of word boundaries solely based on statistical properties of the input ( Elman , 1990 ; Brent & Cartwright , 1996 ; Goldwater et al. , 2009 ) . Unfortunately , the best language models , measured in terms of their ability to model language , segment quite poorly ( Chung et al. , 2017 ; Wang et al. , 2017 ) , while the strongest models in terms of word segmentation are far too weak to adequately predict language ( Goldwater et al. , 2009 ; Berg-Kirkpatrick et al. , 2010 ) . Moreover , since language acquisition is ultimately a multimodal process , neural models which simplify working with multimodal data offer opportunities for future research . However , as Kádár et al . ( 2018 ) have argued , current neural models ’ inability to discover meaningful words is too far behind the current ( non-neural ) state-of-the-art to be a useful foundation . In this paper , we close this gap by introducing a neural model ( §2 ) of natural language sentences that explicitly discovers and models word-like units from completely unsegmented sequences of characters . The model generates text as a sequence of segments , where each segment is generated either character-by-character from a sequence model or as a single draw from a lexical memory of multi-character units . The segmentation decisions and decisions about the generation mechanism for each segment are latent . In order to efficiently deal with an exponential number of possible segmentations , we use a conditional semi-Markov model . The the characters inside each segment are generated using non-Markovian processes , conditional on the previously generated characters ( the previous segmentation decisions are forgotten ) . This conditional independence assumption— forgetting previous segmenation decisions—enables us to calculate and differentiate exact marginal likelihood over all possible discrete segmentation decisions with a dynamic programming algorithm , while letting the model retain the most relevant information about the generation history . There are two components to make the model work . One is a lexical memory . The memory stores pairs of a vector ( key ) and a string ( value ) appearing in the training set and the vector representation of each strings are randomly initialized and learned during training . The other is a regularizer ( §3 ) to prevent the model from overfitting to the training data . Since the lexical memory stores strings that appeared in the training data , each sentence could be generated as a single unit , thus the model can fit to the training data perfectly while generalizing poorly . The regularizer penalizes based on the expectation of the powered length of each segment . Although the length of each segment is not differentiable , the expectation is differentiable and can be computed efficiently together with the marginal likelihood for each sentence in a single forward pass . Our evaluation ( §4–§6 ) , therefore , looks at both language modeling performance and the quality of the induced segmentations . First , we look at the segmentations induced by our model . We find that these correspond closely to human intuitions about word segments , competitive with the best existing models . These segments are obtained in models whose hyperparameters are tuned to optimize validation likelihood , whereas tuning the hyperparameters based on likelihood on our benchmark models produces poor segmentations . Second , we confirm findings ( Kawakami et al. , 2017 ; Mielke & Eisner , 2018 ) that show that word segmentation information leads to better language models compared to pure character models . However , in contrast to previous work , we do so without observing the segment boundaries , including in Chinese , where word boundaries are not part of the orthography . Finally , we find that both the lexicon and the regularizer are crucial for good performance , particularly in word segmentation—removing either or both significantly harms performance . 2 MODEL . We now describe the segmental neural language model ( SNLM ) . Refer to Figure 1 for an illustration . The SNLM generates a character sequence x = x1 , . . . , xn , where each xi is a character in a finite character set Σ . Each sequence x is the concatenation of a sequence of segments s = s1 , . . . , s|s| where |s| ≤ n measures the length of the sequence in segments and each segment si ∈ Σ+ is a sequence of characters , si,1 , . . . , si , |si| . Intuitively , each si corresponds to one word . Let π ( s1 , . . . , si ) represent the concatenation of the characters of the segments s1 to si , discarding segmentation information ; thus x = π ( s ) . For example if x = anapple , the underlying segmentation might be s = an apple ( with s1 = an and s2 = apple ) , or s = a nap ple , or any of the 2|x|−1 segmentation possibilities for x . The SNLM defines the distribution over x as the marginal distribution over all segmentations that give rise to x , i.e. , p ( x ) = ∑ s : π ( s ) =x p ( s ) . ( 1 ) To define the probability of p ( s ) , we use the chain rule , rewriting this in terms of a product of the series of conditional probabilities , p ( st | s < t ) . The process stops when a special end-sequence segment 〈/S〉 is generated . To ensure that the summation in Eq . 1 is tractable , we assume the following : p ( st | s < t ) ≈ p ( st | π ( s < t ) ) = p ( st | x < t ) , ( 2 ) which amounts to a conditional semi-Markov assumption—i.e. , non-Markovian generation happens inside each segment , but the segment generation probability does not depend on memory of the previous segmentation decisions , only upon the sequence of characters π ( s < t ) corresponding to the prefix character sequence x < t . This assumption has been employed in a number of related models to permit the use of LSTMs to represent rich history while retaining the convenience of dynamic programming inference algorithms ( Wang et al. , 2017 ; Ling et al. , 2017 ; Graves , 2012 ) . 2.1 SEGMENT GENERATION . We model p ( st | x < t ) as a mixture of two models , one that generates the segment using a sequence model and the other that generates multi-character sequences as a single event . Both are conditional on a common representation of the history , as is the mixture proportion . Representing history To represent x < t , we use an LSTM encoder to read the sequence of characters , where each character type σ ∈ Σ has a learned vector embedding vσ . Thus the history representation at time t is ht = LSTMenc ( vx1 , . . . , vxt ) . This corresponds to the standard history representation for a character-level language model , although in general we assume that our modeled data is not delimitered by whitespace . Character-by-character generation The first component model , pchar ( st | ht ) , generates st by sampling a sequence of characters from a LSTM language model over Σ and a two extra special symbols , an end-of-word symbol 〈/W〉 /∈ Σ and the end-of-sequence symbol 〈/S〉 discussed above . The initial state of the LSTM is a learned transformation of ht , the initial cell is 0 , and different parameters than the history encoding LSTM are used . During generation , each letter that is sampled ( i.e. , each st , i ) is fed back into the LSTM in the usual way and the probability of the character sequence decomposes according to the chain rule . The end-of-sequence symbol can never be generated in the initial position . Lexical generation The second component model , plex ( st | ht ) , samples full segments from lexical memory . Lexical memory is a key-value memory containing M entries , where each key , ki , a vector , is associated with a value vi ∈ Σ+ . The generation probability of st is defined as h′t = MLP ( ht ) m = softmax ( Kh′t + b ) plex ( st | ht ) = M∑ i=1 mi [ vi = st ] , where [ vi = st ] is 1 if the ith value in memory is st and 0 otherwise , and K is a matrix obtained by stacking the k > i ’ s . Note that this generation process will assign zero probability to most strings , but the alternate character model can generate anything in Σ+ . In this work , we fix the vi ’ s to be subsequences of at least length 2 , and up to a maximum length L that are observed at least F times in the training data . These values are tuned as hyperparameters ( See Appendix B for details of the reported experiments ) . Mixture proportion The mixture proportion , gt , determines how likely the character generator is to be used at time t ( the lexicon is used with probability 1−gt ) . It is defined by as gt = σ ( MLP ( ht ) ) . Total segment probability The total generation probability of st is thus : p ( st | x < t ) = gtpchar ( st | ht ) + ( 1− gt ) plex ( st | ht ) . 2.2 INFERENCE . We are interested in two inference questions : first , given a sequence x , evaluate its ( log ) marginal likelihood ; second , given x , find the most likely decomposition into segments s∗ . Marginal likelihood To efficiently compute the marginal likelihood , we use a variant of the forward algorithm for semi-Markov models ( Yu , 2010 ) , which incrementally computes a sequence of probabilities , αi , where αi is the marginal likelihood of generating x≤i and concluding a segment at time i . Although there are an exponential number of segmental decompositions of x , these values can be computed using O ( |x| ) space and O ( |x|2 ) time as : α0 = 1 , αt = t−1∑ j=t−L αjp ( s = xj : t | x < j ) . ( 3 ) By letting xt+1 = 〈/S〉 , then p ( x ) = αt+1 . Most probable segmentation The most probable segmentation of a sequence x can be computed by replacing the summation with a max operator in Eq . 3 and maintaining backpointers . 3 EXPECTED LENGTH REGULARIZATION . When the lexical memory contains all the substrings in the training data , the model easily overfits by copying the longest continuation from the memory . To prevent overfitting , we introduce a regularizer that penalizes based on the expectation of the exponentiated ( by a hyperparameter β ) length of each segment : R ( x , β ) = ∑ s : π ( s ) =x p ( s | x ) ∑ s∈s |s|β . This can be understood as a regularizer based on the double exponential prior identified to be effective in previous work ( Liang & Klein , 2009 ; Berg-Kirkpatrick et al. , 2010 ) . This expectation is a differentiable function of the model parameters . Because of the linearity of the penalty across segments , it can be computed efficiently using the above dynamic programming algorithm under the expectation semiring ( Eisner , 2002 ) . This is particular efficient since the expectation semiring jointly computes the expectation and marginal likelihood .

Summary: This paper presented a novel approach for modeling a sequence of characters as a sequence of latent segmentations. The challenge here was how to efficiently compute the marginal likelihood of a character sequence (exponential number different of segmentations). The author(s) overcame this by having a segment generation process independent from the previous segment (only depends on a sequence of characters). The inference is then required a forward algorithm. To generate a segment, a model can either select a lexical unit (pre-processed from a training corpus) or generate character by character.