Machine Learning: Word Embedding and Predicting | by Anuradha Wickramarachchi | Towards Data Science

Word embedding is a popular technique of converting sparse representation vectors into dense smaller vectors. This increases computation times by a significant factor and saves resources. In this article let’s see how we can develop a prediction engine and utilize the knowledge of word embedding in the workflow.
This is where individual words are adjusted to the closest matching correct word. In most of the cases, this approach tries to fix spelling. For example, if you type “helo”, it is quite likely that your phone would make it “hello”.
This is a newer approach where you get your grammar corrected. Words are replaced to make sentences more natural. For example, if you say “You’re a beautiful per” we can have a rational logic that the word might turn up to be “person”. In fact, this shall be completed as “You’re a beautiful person”. In this article, we will focus on this auto-correction which uses the context information. This is somewhat complex than typo fixing. I hope you all are victims/users of this feature on mobile phones. Let us build one ourselves!
Once I stumbled upon this URL which directed me to the eBook named Metamorphosis. This is available free to download as a text document in ASCII encoding (UTF-8). Henceforth, I will be referring to this source as my dataset for training. First things first! Let us quickly preprocess and tokenize so we are ready to go. I will be using the following python libraries for preprocessing (In case you don’t have them already).
numpytensorflowkeras (comes as the API for tensorflow backend)
One of the first steps we shall perform is the tokenization of words in the training set. This is because we should fit all the words in the entire corpus in a finite space for downstream processing. Usually, neural networks perform better within a range of values. Thus, we should embed all those data in dense vectors. It is rather a compression of space that our word tokens like. Let’s see how it works.
TF-IDF vectorization
TF-IDF vectorization
This is a very common method of embedding words by considering the frequency of a word in a document and its occurrence in the corpus. The size of the vector will be equal to the number of unique words considered. Usually implemented using a sparse matrix. Let’s have a look at the sample code below.
from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizerimport pandas as pdtexts = ["Data Science is a popular field.",          "Data Science is useful in predicting financial performance of economies",          "Performance is important in data science"]vectorizer = CountVectorizer()counts = vectorizer.fit_transform(texts)tfidf = TfidfVectorizer()features = tfidf.fit_transform(texts)pd.DataFrame(counts.todense(),columns=vectorizer.get_feature_names())pd.DataFrame(features.todense(),columns=tfidf.get_feature_names())
This will print the following outputs in the terminal (Read more about theory here).
By the look of this vector you’ll see that they are high dimensional, thus requires a lot of resources for computations.
This is where we use an embedding layer, which will condense sparse representations into a target dimension making a smaller representation. Let’s have a look at an example.
from tensorflow.keras.preprocessing.text import Tokenizerfrom tensorflow.keras.models import Sequentialfrom tensorflow.keras.layers import Embeddingdocs = ["Data Science is a popular field.",          "Data Science is useful in predicting financial performance of economies",          "Performance is important in data science"]tokenizer = Tokenizer()tokenizer.fit_on_texts(docs)vocab_size = len(t.word_index) + 1model = Sequential()model.add(Embedding(vocab_size, 3, input_length=1))model.compile('rmsprop', 'mse')input_array = tokenizer.texts_to_sequences(["Data"])output_array = model.predict(input_array)output_array
This will be followed by the output;
array([[[0.02056189, 0.04136733, 0.03566055]]], dtype=float32)
You could see that we see a tensor of shape N, 1, 3 this is because our target dimension is 3, we have one word. Multiple words would increase the second dimension of the tensor. These will be hyperparameters of your learning model. Now you will have sentences with words in 3 dimensions in contrast to the previous dimension which was the vocabulary size.
Now that we have good know-how about the preliminary steps, let us build the model.
from numpy as npimport tensorflow as tffrom tensorflow.keras.preprocessing.text import Tokenizerfrom tensorflow.keras.utils import to_categorical
We will load data and tokenize words using the aforementioned book Metamorphosis as the corpus. We will consider 3 consecutive words. Inputs to the model are the first 2 words and output will be the last word for each set of 3 words.
data = ""with open("./metamorphosis.txt") as f:    data = f.read().strip()tokenizer = Tokenizer()tokenizer.fit_on_texts([data])# determine the vocabulary sizevocab_size = len(tokenizer.word_index) + 1print('Vocabulary Size: %d' % vocab_size)# create line-based sequencessequences = list()for encoded in tokenizer.texts_to_sequences(data.split("\n")):    if len(encoded) > 0:        for i in range(0, len(encoded) - 2):            sequences.append(encoded[i:i+3])    print('Total Sequences: %d' % len(sequences))sequences = np.array(sequences)X, y = sequences[:,:-1], to_categorical(sequences[:,-1], num_classes=vocab_size)
Let us build a simple model for our training. In this article I will develop a very simple model, that will consider two preceding words to predict or correct the next word. My output will be in a categorical binary format to predict the next word. I will use the Keras functional API which I love using.
# define modeli = tf.keras.layers.Input(shape=(X.shape[1]))e = tf.keras.layers.Embedding(vocab_size, 10, input_length=max_length)(i)l = tf.keras.layers.LSTM(10)(e)d = tf.keras.layers.Dense(vocab_size, activation='softmax')(l)model = tf.keras.Model(inputs=i, outputs=[d])print(model.summary())# compile networkmodel.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])tf.keras.utils.plot_model(model)
Note that we use categorical cross-entropy since we are training to predict binary categorical values. One bit will indicate one item in a specific index.
We will be able to see the below model when we run our code.
Model: "model"_________________________________________________________________Layer (type)                 Output Shape              Param #   =================================================================input_1 (InputLayer)         [(None, 2)]               0         _________________________________________________________________embedding_4 (Embedding)      (None, 2, 10)             2700      _________________________________________________________________lstm (LSTM)                  (None, 10)                840       _________________________________________________________________dense (Dense)                (None, 270)               2970      =================================================================Total params: 6,510Trainable params: 6,510Non-trainable params: 0
We can initiate the training program using the following lines of code.
model.fit(X, y, epochs=1000, verbose=2)
Now that we have trained the model we can start predicting the next word and correcting. Let us see how we do the prediction part from the trained model.
text = " there's still so"text = " ".join(text.split(" ")[:3])encoded = tokenizer.texts_to_sequences([text])[0]encoded = array([encoded])next = model.predict(encoded, verbose=0)for x in next:    next_word_token = np.argmax(x)    # map predicted word index to word    for word, index in tokenizer.word_index.items():        if index == next_word_token:            print(word + " ")
Here in bold notation, I have put so, which is the last word typed half (for simulation of user input). Note that I consider the first 2 words for prediction. Our predictor will predict the word “some”. So the complete sentence would be “there’s still some”. We are using argmax to pick the predicted category and look the word up in the tokenizer dictionary.
Making the workflow further smart
We are naively picking the most probable item. However, if we had “there’s still some fo”, the next word could easily be “food”. The simplest workaround for this is picking the set of most probable items without argmax. Then look the words up and assign the closest match. You can use cosine distance for this purpose.
Hints
Try to plot your Keras model for clarity.Find a good corpus for training (What I have chosen is naive)Where categorical encoding would not fit, use binary encoding with binary cross-entropy.
I hope you enjoy reading this! G’day!