Natural Language Generation Part 1: Back to Basics
By: George Dittmar
Tags: Naturallanguageprocessing, Machine Learning, Open Source, Apache Spark

Have you ever come across those Facebook or Twitter posts showing the output of an AI that was“forced” to watch TV or read books and it comes up with new output similar to what it saw or read? They are usually pretty hilarious and don't follow exactly how someone would actually say things or write, but they are examples of Natural Language Generation. NLG is a really interesting area of ML that can be fun to play around with and come up with your own models. Maybe you want to make a Rick and Morty Star Trek cross over script, or just create tweets that sounds similar to another persons tweets.

Using ML to generate text, images and video is becoming more widespread as research and hardware advances. With recent advancements in deep learning based systems, such as OpenAI’s GPT-2 model, we are now seeing language models that can be used to generate very real sounding text from a large set of other examples. I’ve had an interest in building a system to generate fake text in the style of another genre or person, so I decided to focus on learning the different ML approaches and give an overview of what I learned using these different techniques.

Getting Back to Basics

Instead of just jumping straight into the fancy deep learning techniques, lets look at a technique that is fairly straight forward to understand and easy to implement as a starting point. One of the most common methods used for language generation for many years has been Markov chains which are surprisingly powerful for as simple of a technique as they can be. Markov chains are a stochastic process that are used to describe the next event in a sequence given the previous event only. This is cool because it means we don’t really need to keep track of all the previous states in a sequence to be able to infer what the next possible state could be.

In our case the state will be the previous word (unigram) or 2 words (bigram) or 3 (trigram). These are more generally known as ngrams since we will be using the last n words to generate the next possible word in the sequence. A Markov chain usually picks the next state via a probabilistic weighting but in our case that would just create text that would be too deterministic in structure and word choice. You could play with the weighting of the probabilities, but really having a random choice helps make the generated text feel original.

Building the Language Model

Creating the language model is fairly straightforward. First we need some example text as our corpus to build our language model from. It can be any kind of text such as book passages, tweets, reddit posts, you name it. In my corpus I used presidential speeches from multiple presidents.

Once we have the text collected we need to do the following steps. As a note I add the token “#END#” to my language model to make it easy to determine an ending state in any of the sample speeches.

tokenize the text

do any preprocessing over the tokens (I don't in my case)

generate ngrams from the tokens

create a mapping from ngram to next possible words found in the text

To better understand how this model is built lets look at a super simple example. Assume we have a simple text “The dog jumped over the moon. The dog is funny.”. Using the above process, we will generate the following language model.

(The, dog) -> [jumped, is]

(dog, jumped) -> [over]

(jumped, over) -> [the]

(over, the) -> [moon.]

(dog, is) -> [funny.]

(is, funny) -> [#END#]

Once we have the ngram mappings completed, the model is ready to be used to generate some new text. This can be done fairly easily now by either passing in a seed of an example ngram, like “The dog”, or having the system randomly choose a starting point from the extracted ngram keys. Once we have a seed ngram chosen, we then randomly choose the next word from the possible list of words, then take the next ngram from the generated text so far and pick the next state etc etc…

Simple Python Markov Chain

Now that have we have gone over how it works conceptually, lets look at the full code for training and generating text. Below is a python script I cobbled together from other examples online that builds a basic Markov model in python.

Lets dig deeper into some of the code. Its mainly just two functions. The learn function and the generate function. Lets first look at the learn function which builds the model from a list of tokens and ngrams of size n.

def learn(self,tokens,n=2):

model = {}



for i in range(0,len(tokens)-n):

gram = tuple(tokens[i:i+n])

token = tokens[i+n]



if gram in model:

model[gram].append(token)

else:

model[gram] = [token]



final_gram = tuple(tokens[len(tokens) - n:])

if final_gram in model:

model[final_gram].append("#END#")

else:

model[final_gram] = ["#END#"]

self.model = model

return model

We start with the loop which goes from the first token to the length of the list minus n. As we go along we build up the dictionary of ngrams to adjacent words found in the tokenized text. After looping, we stop just before the last n words of the input text are left and create that final token variable which we then add to the model with a “#END#” to signify we have reached the end of the documents.

One limitation I will point out with this approach is that I am putting all text together into one list so we will only really have one end state. A further improvement is to have end states for each document we process, or could go further and add end states at the end of sentences so we know better when to start a new sentence etc. Next we have the generate function.

def generate(self,n=2,seed=None, max_tokens=100):

if seed is None:

seed = random.choice(list(self.model.keys()))



output = list(seed)

output[0] = output[0].capitalize()

current = seed



for i in range(n, max_tokens):

# get next possible set of words from the seed word

if current in self.model:

possible_transitions = self.model[current]

choice = random.choice(possible_transitions)

if choice is "#END#" : break



if choice == '.':

output[-1] = output[-1] + choice

else:

output.append(choice)

current = tuple(output[-n:])

else:

# should return ending punctuation of some sort

if current not in string.punctuation:

output.append('.')

return output

The code to generate new text takes in the size of the ngrams we trained on and how long we want the generated text to be. It also takes in an optional seed parameter which if it is not set will randomly pick a starting seed from the possible ngrams learned in the model. On each iteration of the loop we look at the previous ngram and randomly select the next possible transition word until we hit one of the ending states or hit the max length of the text.

Here is an example of the output from the script using bigrams as the language model.

Us from carrying out even the dishonest media report the facts! my hit was on the 1st of december, 1847, being the great reviews & will win on the front lines of freedom. we are now saying you will never forget the rigged system that is what we do at a 15 year high. i can perceive no good reason why the civil and religious freedom we enjoy and by the secretary of war would be 0.0 ratings if not.

As you can see it makes sense but its also a little random sounding. What if we try going to a trigram language model?

Was $7,842,306.90, and during the seven months under the act of the 3d of march last i caused an order to be issued to our military occupation during the war, and may have calculated to gain much by protracting it, and, indeed, that we might ultimately abandon it altogether without insisting on any indemnity, territorial or otherwise. whatever may be the least trusted name in news if they continue to cover me inaccurately and with a group, it’s going to be open, and the land will be left in better shape than it is right now. is that right? better shape. (applause.) we declined to certify the terrible one-sided iran nuclear deal. that was a horrible deal. (applause.) whoever heard you give $150 billion to a nation

Well this sound a lot better…but wait when digging into the sample corpus I noticed that its lifting large chunks of text out of the corpus. In reality, unless you have a ton of data to build off of, most models tend to show this behavior once you start using trigrams or higher. This is one downside to the Markov model approach. The bigram model, while more random sounding, seems to generate fairly unique output on each run and not lift sections of text from the corpus.

Spark Distributed Markov Chain

Let’s say we create a much larger data set to pull from like a whole subreddit or years of tweets. The python script as is probably could process the data set if we wanted to let it run long enough and have enough memory on our machine, but it might not easily scale still in the end. Lets do the same thing but using Apache Spark and use its distributed computing abilities to build and store the model. Below is the full code of the spark based model and we will dig deeper into its operations as well.

The biggest hurdle was trying to figure out how to generate the ngram model in Spark, create the dictionary like structure and query against it. Luckily Sparks mllib already has ngram feature extraction functionality built into the framework so that park was taken care of. It just takes in a Spark dataframe object, our tokenized document rows, and then outputs in another column the ngrams to a new dataframe object.

ngram = NGram(n=self.n, inputCol='tokenized_text',outputCol='ngram')

ngram_df = ngram.transform(text_df)

Using Sparks ngram module let me then create a function to map over each row in the dataframe and process the text to generate the adjacent words to each ngram. This is the function called in the first map call of the Spark process. Its goal is to just take the list of ngrams for the document and loop through producing per document a list of tuples in the form [(ngram, adjacent term)]. Right now there will potentially be duplicate (ngram, adjacent term) tuples in the list.

def generate_adjacent_terms(ngrams):

adjacent_list = []

for i in range(0, len(ngrams)):

if(i == len(ngrams) - 1):

adjacent_tuple = (ngrams[i], "#END#")

adjacent_list.append(adjacent_tuple)

else:

adjacent_tuple = (ngrams[i], ngrams[i+1].split(" ")[-1])

adjacent_list.append(adjacent_tuple)

return adjacent_list

The flat map is to put all the lists of tuples into one flat rdd instead of each rdd element being a list from each document. The next map is to setup for the reduceByKey so we take each element and modify it into a tuple of (ngram, list object) which then can be used to combine the ngrams keys together to finally create the model in the form (ngram, [adjacent term list]). It is important to note there will be duplicate terms in the adjacency list. I kept the dupes in to act like a weighting on how likely our algorithm will randomly choose particular next states.

self.ngram_model = ngram_df.rdd.map(lambda x: PreProcess

.generate_adjacent_terms(x.asDict()['ngram'])) \

.flatMap(lambda xs: [x for x in xs]) \

.map(lambda y: (y[0], [y[1]])) \

.reduceByKey(lambda a, b: a + b)

Below is the full code I used to create the Spark based Markov chain code. The text generation logic is then very similar to the other script, except that instead of querying a dictionary we are querying an rdd to get the next term in the sequence. In practice this would most likely be behind an api call but for now we can just call the rdd directly.

The Spark code will generate similar output as the first python script but in theory should scale much more nicely when ran over a large data set on a cluster.

Feel free to check out the code and scripts yourself, and let me know what you think / what improvements I should look into! https://github.com/GeorgeDittmar/MarkovTextGenerator. Next round of work is to be able to save models to extend them with new data, as well as add some more example scripts for people to work from.

Thanks for reading!