Creating a Spell Checker with TensorFlow | by David Currie | Towards Data Science

One of the most important aspects of machine learning is working with good, clean data. Natural Language Progressing projects have the issue of using text that is written by humans, and we are unfortunately bad at writing. Just think about the many spelling mistakes that would be in a dataset about posts and comments from Reddit. For this reason, I thought a very worthwhile project would be to make a spell checker, which would help alleviate some of these problems.
The model that we will use for this project is very similar to the one I wrote about in my article, “Text Summarization with Amazon Reviews” (both are seq2seq models), but I have added some extra lines of code so that the architecture and hyperparameters can be tuned using grid search, and the results can be analyzed with TensorBoard. If you would like a more detailed walkthrough about how to add TensorBoard to your code, then check out “Predicting Movie Review Sentiment with TensorFlow and TensorBoard”.
The main focus of this article will be how to prepare the data for the model, and I will also talk about a few other features of the model. We will be using Python 3 and TensorFlow 1.1 in this project. The data is composed of twenty popular books from Project Gutenberg. If you are interested in scaling up this project to make it more accurate, there are hundreds of books that you can download on Project Gutenberg. Plus, it would be really interesting to see how good of a spell checker someone could make with this model.
To see the full code, here is its GitHub page.
To give you a preview of what this model is capable of, here are some curated examples:
Spellin is difficult, whch is wyh you need to study everyday.
Spelling is difficult, which is why you need to study everyday.
The first days of her existence in th country were vrey hard for Dolly.
The first days of her existence in the country were very hard for Dolly.
Thi is really something impressiv thaat we should look into right away!
This is really something impressive that we should look into right away!
To make things a little more organized, I have put all of the books that we will use in their own folder, called “books”. Here is the function that we will use to load all of the books:
def load_book(path):    input_file = os.path.join(path)    with open(input_file) as f:        book = f.read()    return book
We will also need the unique file name for each of the books.
path = './books/'book_files = [f for f in listdir(path) if isfile(join(path, f))]book_files = book_files[1:]
When we put these two code blocks together, we will be able to load the text from all of our books into a list.
books = []for book in book_files:    books.append(load_book(path+book))
If you are interested in knowing how many words are in each book, you can use these lines of code:
for i in range(len(books)):    print("There are {} words in {}.".format(len(books[i].split()), book_files[i]))
Note: If you do not include .split(), it will return the number of characters in each book.
To clean the text of these books is rather simple. Since we will be using characters instead of words as the input to our model, we do not need to worry about removing stop words, or shorten words down to their stems. We only need to remove the characters that we do not want to include and extra spaces.
def clean_text(text):    '''Remove unwanted characters and extra spaces from the text'''    text = re.sub(r'\n', ' ', text)     text = re.sub(r'[{}@_*>()\\#%+=\[\]]','', text)    text = re.sub('a0','', text)    text = re.sub('\'92t','\'t', text)    text = re.sub('\'92s','\'s', text)    text = re.sub('\'92m','\'m', text)    text = re.sub('\'92ll','\'ll', text)    text = re.sub('\'91','', text)    text = re.sub('\'92','', text)    text = re.sub('\'93','', text)    text = re.sub('\'94','', text)    text = re.sub('\.','. ', text)    text = re.sub('\!','! ', text)    text = re.sub('\?','? ', text)    text = re.sub(' +',' ', text) # Removes extra spaces    return text
I’m going to skip over how to make the vocab_to_int and int_to_vocab dictionaries, since that’s pretty standard stuff and you can find it on this project’s GitHub page. However, I think it’s worth showing you the characters that are included in the input data:
The vocabulary contains 78 characters.[' ', '!', '"', '$', '&', "'", ',', '-', '.', '/', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', ':', ';', '<EOS>', '<GO>', '<PAD>', '?', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']
We could have removed some more of the special characters, or made the text all lower case, but I wanted to make this spell checker as useful as possible.
The data will be organized into sentences before it is fed into the model. We will split the data at each period that is followed by a space (“. ”). One issue with this is that some sentences end in a question mark or exclamation mark, but we are not accounting for that. Fortunately, our model will still be able to learn about the use of question marks and exclamation marks, just as long as that and the following sentence, combined, are not as long as the maximum sentence length.
Just to give an example:
Today is a lovely day. I want to go to the beach. (This would be split into two input sentences)
Is today a lovely day? I want to go to the beach. (This would be one long input sentence)
sentences = []for book in clean_books:    for sentence in book.split('. '):        sentences.append(sentence + '.')
I used a GPU on floydhub.com to train my model (I highly recommend their services), which saved me hours training time. Nonetheless, to properly tune this model, it still takes 30–60 minutes to run an iteration, which is why I am limiting the data, so that it does not take even longer. This will of course reduce the accuracy of our model, but since this is just a personal project, I don’t mind the trade-off.
max_length = 92min_length = 10good_sentences = []for sentence in int_sentences:    if len(sentence) <= max_length and len(sentence) >= min_length:        good_sentences.append(sentence)
To track the performance of this model, I will be spliting the data into a training and testing set. The testing set will be composed of 15% of the data.
training, testing = train_test_split(good_sentences,                                      test_size = 0.15,                                      random_state = 2)
Just like with some of my recent projects, I will be sorting the data by length. This results in sentences of a batch being of similar length, thus less padding is used, and the model will train faster.
training_sorted = []testing_sorted = []for i in range(min_length, max_length+1):    for sentence in training:        if len(sentence) == i:            training_sorted.append(sentence)    for sentence in testing:        if len(sentence) == i:            testing_sorted.append(sentence)
Perhaps the most interesting/important part of this project is the function that will convert the sentences to sentences with mistakes, which will be used as the input data. Mistakes are created within this function in one of three ways:
the order of two characters will be swapped (hlelo ~hello)
an extra letter will be added (heljlo ~ hello)
a character will not be typed (helo ~hello)
The likelihood of either of the three errors occurring is equal, and the likelihood of any error occurring is 5%. Therefore, one in every 20 characters, on average, will include a mistake.
letters = ['a','b','c','d','e','f','g','h','i','j','k','l','m',           'n','o','p','q','r','s','t','u','v','w','x','y','z',]def noise_maker(sentence, threshold):        noisy_sentence = []    i = 0    while i < len(sentence):        random = np.random.uniform(0,1,1)        if random < threshold:            noisy_sentence.append(sentence[i])        else:            new_random = np.random.uniform(0,1,1)            if new_random > 0.67:                if i == (len(sentence) - 1):                    continue                else:                    noisy_sentence.append(sentence[i+1])                    noisy_sentence.append(sentence[i])                    i += 1            elif new_random < 0.33:                random_letter = np.random.choice(letters, 1)[0]                noisy_sentence.append(vocab_to_int[random_letter])                noisy_sentence.append(sentence[i])            else:                pass             i += 1    return noisy_sentence
The last thing that I want to show you in this article is how to create the batches. Typically one would create their input data before training their model, which would mean they have a fixed amount of training data. However, we are going to be creating new input data as we train our model, by applying noise_maker to each batch. This means that for every epoch the target (correct) sentence will be fed back through noise_maker and should receive a new input sentence. Using this method, we have, in a slightly exaggerated sense, an infinite amount of training data.
def get_batches(sentences, batch_size, threshold):        for batch_i in range(0, len(sentences)//batch_size):        start_i = batch_i * batch_size        sentences_batch = sentences[start_i:start_i + batch_size]                sentences_batch_noisy = []        for sentence in sentences_batch:            sentences_batch_noisy.append(                noise_maker(sentence, threshold))                    sentences_batch_eos = []        for sentence in sentences_batch:            sentence.append(vocab_to_int['<EOS>'])            sentences_batch_eos.append(sentence)                    pad_sentences_batch = np.array(            pad_sentence_batch(sentences_batch_eos))        pad_sentences_noisy_batch = np.array(            pad_sentence_batch(sentences_batch_noisy))                pad_sentences_lengths = []        for sentence in pad_sentences_batch:            pad_sentences_lengths.append(len(sentence))                pad_sentences_noisy_lengths = []        for sentence in pad_sentences_noisy_batch:            pad_sentences_noisy_lengths.append(len(sentence))                yield (pad_sentences_noisy_batch,                pad_sentences_batch,                pad_sentences_noisy_lengths,                pad_sentences_lengths)
That’s all for this project! Although the results are encouraging, there are still limitations to this model. I would really appreciate if someone would scale this model up or improve upon its design! Please post about it in the comments if you do. One thought for new design would be to apply FAIR’s new CNN model (it achieves state-of-the-art results for translation).
Thanks for reading and I hope that you learned something new!