Dynamic Meta Embeddings in Keras. Learn a valuable combination of... | by Marco Cerliani | Towards Data Science

Many NLP solutions make use of pre-trained word embeddings. The choice of which one to use is often related to the final performances and is achieved after a lot of trials and manual tuning. At Facebook AI Lab agreed that the best way to make this kind of selection is to let neural networks to figure out by themselves.
They introduced dynamic meta-embeddings, a simple yet effective method for the supervised learning of embedding ensembles, which leads to state of the art performance within the same model class on a variety of tasks. This simple, but extremely efficient, method permits to learn a linear combination of a set of selected word embeddings which outperforms the naive concatenation of various embeddings.
As mentioned, the authors proved the validity of their solution on various tasks in NLP domain. We limited ourselves to adopt these techniques in a text classification problem, where we have 2 pre-trained embeddings and want to combine them intelligently to boost the final performances.
I found a valuable dataset on Kaggle containing the full text of articles (2225 in total) from BBC archive. There are 5 thematic areas to which news belong to:
Our aims is to classify them correctly, and to do this we want to train our different types of embeddings, combine them intelligently and build on this merge our classifier. In the beginning, a standard cleaning procedure is applied to the raw corpus. As embedding model, I’ve selected the most common types: Word2Vec and FastText. We can train them easily with Gensim to ‘drag and drop’ them into Keras architecture (remember to maintain the same embedding size for every selected embedding framework). I compute this procedure carefully by hand in order to control the process of padding if needed: the Tokenizer object and the pad_sequence function from Keras make all the things easy.
When we end with multiple trained embedding (also pretrained form of model like Glove or similar are perfect) and a sequential corpus we are ready to combine our weights.
In the original paper two different kinds of techniques are introduced:
Dynamic Meta-Embeddings (DME): the original embeddings are projected in a new space adding extra learnable weights through an LSTM encoder, following an attention mechanism. Then they are linearly combined with their original format. In Keras language:
def DME(maxlen):    inp = Input(shape=(maxlen, 100, 2))    x = Reshape((maxlen, -1))(inp)    x = LSTM(2, return_sequences=True)(x)    x = Activation('sigmoid')(x)    x = Reshape((maxlen, 1, 2))(x)    x = multiply([inp, x])    out = Lambda(lambda t: K.sum(t, axis=-1))(x)    return Model(inp, out)
Contextual Dynamic Meta-Embeddings (CDME): as above, the original embeddings are projected in a new space adding extra learnable weights; but now a context-dependent system is applied through a BiLSTM-Max encoder. In the end, the self-attention mechanism and the weighted combination with the original format are pursued. In Keras language (the MaxPooling merge in Bidirectional Layer of Keras is not supplied so we have to code it by ourself):
def CDME(maxlen, latent_dim=2):    inp = Input(shape=(maxlen, 100, 2))    x = Reshape((maxlen, -1))(inp)    x = Bidirectional(LSTM(latent_dim, return_sequences=True))(x)    x = Lambda(lambda t: [t[:,:,:int(latent_dim/2+1)],                            t[:,:,int(latent_dim/2+1):]])(x)    x = Maximum()(x)    x = Activation('sigmoid')(x)    x = Reshape((maxlen, 1, 2))(x)    x = multiply([inp, x])    out = Lambda(lambda t: K.sum(t, axis=-1))(x)    return Model(inp, out)
We recreate two general code blocks which carry out the embeddings combination through a dynamic procedure. Both solutions can be placed at the beginning of a network, immediately after the reading and concatenation of our embeddings. On them, we can stack normal layers for different purposes. In our case, we add some recurrent layers to correctly classify our news articles. We ended with these two architectures:
For DME:
concat_inp = Concat_Emb([embedding_matrix_w2v, embedding_matrix_ft], maxlen=max_len)dme = DME(max_len)x = dme(concat_inp.output)x = GRU(128, dropout=0.2, return_sequences=True)(x)x = GRU(32, dropout=0.2)(x)out = Dense(y.shape[1], activation='softmax')(x)dme_model = Model(concat_inp.input, out)dme_model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
For CDME:
concat_inp = Concat_Emb([embedding_matrix_w2v, embedding_matrix_ft], maxlen=max_len)cdme = CDME(max_len)x = cdme(concat_inp.output)x = GRU(128, dropout=0.2, return_sequences=True)(x)x = GRU(32, dropout=0.2)(x)out = Dense(y.shape[1], activation='softmax')(x)cdme_model = Model(concat_inp.input, out)cdme_model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
Now we are ready to execute the training and see some results. Both models are able to achieve an overall accuracy of around 93% on test data with a great recall score for every class.
Giving access to multiple types of embeddings (it doesn’t matter if they are pre-trained or built ad hoc), we allow a NN to learn which embeddings it prefers by predicting a weight for each embedding type (DME), optionally depending on the context (CDME). We make this happen in an NLP task for text classification, simply combining this procedure with the normal approach for this kind of problem.
CHECK MY GITHUB REPO
Keep in touch: Linkedin
REFERENCES
Dynamic Meta-Embeddings for Improved Sentence Representations: Douwe Kiela, Changhan Wang and Kyunghyun Cho; Facebook AI Research; New York University; CIFAR Global Scholar.