Understanding Word Embeddings from scratch | LSTM model | | by Manik Soni | Towards Data Science

Yo reader! I am Manik. What’s up?.
Hope you’re doing great and working hard for your goals. If not, it’s never late. Start now, at this very moment.
With this piece of information, you’ll walk away with a clear explanation on Sequence and Text processing for Deep Neural Networks which includes:
What’s one-hot Encoding?OneHot Encoding with keras.What are word embeddings and their advantage over One-Hot encoding?What are word embeddings trying to say?A complete example of converting raw text to word embeddings in keras with an LSTM and GRU layer.
What’s one-hot Encoding?
OneHot Encoding with keras.
What are word embeddings and their advantage over One-Hot encoding?
What are word embeddings trying to say?
A complete example of converting raw text to word embeddings in keras with an LSTM and GRU layer.
if you want to learn about LSTMs, you can go here
binarykeys.in
“Yours and my ancestors had run after a mastodon or wild boar, like an Olympic sprinter, with a spear in hand covering themselves with leaves and tiger skin, for their breakfast” — History
The above sentence is in textual form and for neural networks to understand and ingest it, we need to convert it into some numeric form. Two ways of doing that are One-hot encoding and the other is Word embeddings.
This is a way of representing each word by an array of 0s and 1. In the array, only one index has ‘1’ present and rest all are 0s.
Example: The following vector represents only one word, in a sentence with 6 unique words.
Let’s find all the unique words in our sentence.
array(['Yours', 'a', 'after', 'an', 'ancestors', 'and', 'boar,',       'breakfast', 'covering', 'for', 'had', 'hand', 'in','leaves',       'like', 'mastodons', 'mine', 'olympic', 'or', 'run', 'skin,',       'spear', 'sprinter,', 'their', 'themselves', 'tiger', 'wild',       'with'], dtype='<U10')shape: (28,)
Now, give each of them an index i.e. create a word_index where each word has an index attached to it in a dictionary.
You might have observed above in the code that 0 is not assigned to any word. It’s a reserved index in Keras(We’ll get here later).
{'Yours': 1, 'a': 2, 'after': 3, 'an': 4, 'ancestors': 5, 'and': 6, 'boar,': 7, 'breakfast': 8, 'covering': 9, 'for': 10, 'had': 11, 'hand': 12, 'in': 13, 'leaves': 14, 'like': 15, 'mastodons': 16, 'mine': 17, 'olympic': 18, 'or': 19, 'run': 20, 'skin,': 21, 'spear': 22, 'sprinter,': 23, 'their': 24, 'themselves': 25, 'tiger': 26, 'wild': 27, 'with': 28}
Now, let’s create one-hot encoding for them.
Example output: This is how “yours” is represented.
Yours [0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
text_to_matrix is the method used to return one-hot encoding.
You can see that, to represent a word, we are actually wasting a lot of memory to just set 0s(sparse matrix). These one-hot encodings also doesn’t reflect any relation between similar words. They are just representation of some word with ‘1’. Two similar words such as “accurate” and “exact” might be at very different positions in one-hot encodings.
What if we can represent a word with less space and have a meaning of its representation with which we can learn something.
Word embeddings also represent words in an array, not in the form of 0s and 1s but continuous vectors.
They can represent any word in few dimensions, mostly based on the number of unique words in our text.
They are dense, low dimensional vectors
Not hardcoded but are “learned” through data.
Geometric relationship between words in a word embeddings can represent semantic relationship between words. Words closer to each other have a strong relation compared to words away from each other.
Vectors/words closer to each other means the cosine distance or geometric distance between them is less compared to others.
There could be vector “male to female” which represents the relation between a word and its feminine. That vector may help us in predicting “king” when “he” is used and “Queen” when she is used in the sentence.
Below is a single row of embedding matrix representing the word ‘the’ in 100 dimensions from a text having 100K unique words.
Such matrices are learned from data and can represent any text with millions of words in 100, 200, 1000 or more dimensions (The same would require 1MM dimensions if one-hot encoding is used).
Let’s see how to create embeddings of our text in keras with a recurrent neural network.
Load text data in array.Process the data.Convert the text to sequence and using the tokenizer and pad them with keras.preprocessing.text.pad_sequences method.Initialise a model with Embedding layer of dimensions (max_words, representation_dimensions, input_size))
Load text data in array.
Process the data.
Convert the text to sequence and using the tokenizer and pad them with keras.preprocessing.text.pad_sequences method.
Initialise a model with Embedding layer of dimensions (max_words, representation_dimensions, input_size))
max_words: It is the no. of unique words in your data
representation_dimension: It is the no. of dimensions in which you want to represent a word. Usually, it is number of (unique words)^(1/4)
input_size: size of your padded sequence(maxlen)
5 . Run the model
Let’s follow the above steps for IMDB raw data. All the code below is present in my Kaggle notebook.
loading the text data with pandas.
Marking 1 for positive movie review and 0 for negative review.
Creating an instance of keras’s Tokenizer class and padding the sequence to ‘maxlen’.
A simple recurrent neural network with embedding as first layer.
All the above code is present here.
If this piece of writing helped you in any way, do share your knowledge with others!
Thanks for reaching till here! Great!