ELMo helps to further improve your sentence embeddings | by Edward Ma | Towards Data Science

In the last story, Contextualized Word Vectors (CoVe) is introduced which is the enhanced version of word embeddings. Peters et al. introduces Deep Contextualized Word Representations which aim at providing a better word representation for NLP tasks under different context. They call it as ELMo (Embeddings from Language Models).
After reading this post, you will understand:
Embeddings from Language Models Design
Architecture
Implementation
Take Away
ELMo use bidirectional language model (biLM) to learn both word (e.g., syntax and semantics) and linguistic context (i.e., to model polysemy). After pre-training, an internal state of vectors can be transferred to downstream NLP tasks. Peters et al. used 6 NLP tasks to evaluate the outcome from biLM.
Those tasks are Question Answering, Textual Entailment, Semantic Role Labeling, Coreference Resolution, Named Entity Extraction and Sentiment Analysis. All of them got a outperform result.
Different from traditional word embeddings, ELMo produced multiple word embeddings per single word for different scenarios. Higher-level layers capture context-dependent aspects of word embeddings while lower-level layers capture model aspects of syntax. In the simplest case, we only use top layer (1 layer only) from ELMo while we can also combine all layers into a single vector.
We can concatenate ELMo vector and token embeddings (word embeddings and/or character embeddings) to form a new embeddings as follow:
In the experiment, Peters et al. use L=2 (2 biLSTM layer) with 4096 units and 512 output dimension for context-dependent part while 2048 character n-gram constitutional filters and 512 output dimension for context insensitive part to build contextualized word embeddings.
Original authors are McCann et al. who implemented ELMo by Pytorch with python 3.6. There are Tensorflow, chainer and Keras versions available. I will use Keras version to demonstrate how we can convert text to vectors. For others, you can check out those githubs which is mentioned in Reference section.
The following examples show how we can use keras to implement it . Tensorflow Hub is a model hub which storing lots of different model. If you only need a pre-trained embeddings, you can leverage use the following code to retrieve it from Tensorflow Hub and transferring to Keras.
2 embeddings layer (there are 5 layers in total) can be chosen from Tensorflow Hub pre-trained model. The first one “elmo” which is the weighed sum of the 3 other layers. The second one is “default” which is a fixed mean-polling of all layers. You may find more information from here.
In my demo, there are 3 approaches to use ELMo pre-trained model. They are:
Weighted sum of the 3 layers with word embeddingsWeighted sum of the 3 layers without word embeddingsFixed mean-pooling without word embeddings
Weighted sum of the 3 layers with word embeddings
Weighted sum of the 3 layers without word embeddings
Fixed mean-pooling without word embeddings
1. Weighted sum of the 3 layers with word embeddings
# Input Layersword_input_layer = Input(shape=(None, ), dtype='int32')elmo_input_layer = Input(shape=(None, ), dtype=tf.string)# Output Layersword_output_layer = Embedding(    input_dim=vocab_size, output_dim=256)(word_input_layer)elmo_output_layer = Lambda(    elmo_embs.to_keras_layer,     output_shape=(None, 1024))(elmo_input_layer)output_layer = Concatenate()(    [word_output_layer, elmo_output_layer])output_layer = BatchNormalization()(output_layer)output_layer = LSTM(    256, dropout=0.2, recurrent_dropout=0.2)(output_layer)output_layer = Dense(4, activation='sigmoid')(output_layer)# Build Modelmodel = Model(    inputs=[word_input_layer, elmo_input_layer],     outputs=output_layer)model.compile(    loss='sparse_categorical_crossentropy',     optimizer='adam', metrics=['accuracy'])model.summary()model.fit(    [x_train_words, x_train_sentences], y_train,#     validation_data=([x_test_words, x_test_sentences], y_test),     epochs=10, batch_size=32)
Result
Accuracy:77.10%Average Precision: 0.78Average Recall: 0.77Average f1: 0.76
2. Weighted sum of the 3 layers without word embeddings
# Input Layerselmo_input_layer = Input(shape=(None, ), dtype=tf.string)# Output Layersoutput_layer = Lambda(    elmo_embs.to_keras_layer,     output_shape=(None, 1024))(elmo_input_layer)output_layer = BatchNormalization()(output_layer)output_layer = LSTM(    256, dropout=0.2, recurrent_dropout=0.2)(output_layer)output_layer = Dense(4, activation='sigmoid')(output_layer)# Build Modelmodel = Model(    inputs=elmo_input_layer, outputs=output_layer)model.compile(    loss='sparse_categorical_crossentropy',     optimizer='adam', metrics=['accuracy'])model.summary()model.fit(    x_train_sentences, y_train,#     validation_data=([x_test_words, x_test_sentences], y_test),     epochs=10, batch_size=32)
Result
Accuracy:76.83%Average Precision: 0.78Average Recall: 0.77Average f1: 0.77
3. Fixed mean-pooling without word embeddings
# Input Layersinput_layer = Input(shape=(None,), dtype=tf.string)# Output Layersoutput_layer = Lambda(    elmo_embs.to_keras_layer,     output_shape=(1024,))(input_layer)output_layer = Dense(    256, activation='relu')(output_layer)output_layer = Dense(4, activation='sigmoid')(output_layer)model = Model(inputs=[input_layer], outputs=output_layer)model.compile(    loss='sparse_categorical_crossentropy',     optimizer='adam', metrics=['accuracy'])model.summary()model.fit(    x_train_sentences, y_train,#     validation_data=([x_test_words, x_test_sentences], y_test),     epochs=10, batch_size=32)
Result
Accuracy:74.37%Average Precision: 0.79Average Recall: 0.74Average f1: 0.75
To access all code, you can visit this github repo
CoVe needs label data to get the contextual word vectors while ELMo go for unsupervised way.
CoVe use only the last layer while ELMo use multiple layer for contextual word representations.
CoVe unable to resolve OOV issue. It suggests to use zero vectors to represent unknown word. ELMO can handle the OOV problem as it uses character embeddings to build word embeddings.
ELMo is quite time consuming to compute vectors. Per authors suggested, you can precompute token offline and lookup it during online prediction to reduce time spending.
I am Data Scientist in Bay Area. Focusing on state-of-the-art in Data Science, Artificial Intelligence , especially in NLP and platform related. You can reach me from Medium Blog, LinkedIn or Github.
Peters M. E., Neumann M., Uyyer M., Gardner M., Clark C., Lee K., Zettlemoyer L.. Deep contextualized word representations. 2018. https://arxiv.org/pdf/1802.05365.pdf
ELMo in Pytorch (Original)
ELMo in Keras
ELMo in Tensorflow
ELMo in chainer