Package: simple-elmo
Simple_elmois a Python library to work with pre-trainedELMo contextualized language modelsin TensorFlow.This is a significantly updated wrapper to theoriginal ELMo implementation.
The main changes are:more convenient and transparent data loading (including from compressed files)code adapted to modern TensorFlow versions (including TensorFlow 2).Installationpip install --upgrade simple_elmoMake sure to update the package regularly.Usagefrom simple_elmo import ElmoModelmodel = ElmoModel()LoadingFirst, let's load a pretrained model:model.load(PATH_TO_ELMO)Required argumentsPATH_TO_ELMOis either a ZIP archive downloaded from theNLPL vector repository,
OR a directory containing 2 files:*.hdf5, pre-trained ELMo weights in HDF5 format (simple_elmoassumes the file is namedmodel.hdf5;
if it is not found, the first existing file with the.hdf5extension will be used);options.json, description of the model architecture in JSON;One can also provide avocab.txt/vocab.txt.gzfile in the same directory:
a one-word-per-line vocabulary of words to be cached (as character id representations) before inference.
Even if it is not present at all, ELMo will still process all words normally.
However, providing the vocabulary file can slightly increase inference speed when working with very large corpora (by reducing the amount of word to char ids conversions).Optional argumentsmax_batch_size:integer, default 32;the maximum number of sentences/documents in a batch during inference;
your input will be automatically split into chunks of the respective size;
if your computational resources allow, you might want to increase this value.limit:integer, default 100;the number of words from the vocabulary file to actually cache (counted from the first line).
Increase the default value if you are sure these words occur in your data much more often than 1 or 2 times.full:boolean, default False;if True, will try to load the full model from TensorFlow checkpoints, together with the vocabulary.
Models loaded this way can be used for language modeling.Working with modelsCurrently, we provide three methods for loaded models (will be expanded in the future):model.get_elmo_vectors(SENTENCES)model.get_elmo_vector_average(SENTENCES)model.get_elmo_substitutes(RAW_SENTENCES)SENTENCESis a list of input sentences (lists of words).RAW_SENTENCESis a list of input sentences as strings.Theget_elmo_vectors()method produces a tensor of contextualized word embeddings.
Its shape is (number of sentences, the length of the longest sentence, ELMo dimensionality).Theget_elmo_vector_average()method produces a tensor with one vector per each input sentence,
constructed by averaging individual contextualized word embeddings.
Its shape is (number of sentences, ELMo dimensionality).Both these methods can be used with thelayersargument, which takes one of the three values:average(default): return the average of all ELMo layers for each word;top: return only the top (last) layer for each word;all: return all ELMo layers for each word
(an additional dimension appears in the produced tensor, with the shape equal to the number of layers in the model, 3 as a rule)Use these tensors for your downstream tasks.Another argument for these methods issession.
It defaults toNonewhich means a new TensorFlow session is created automatically when the method is called.
This is convenient, since one does not have to worry about initializing the computational graph.
However, in some cases, you might want to re-use an existing session (for example, to call the method multiple times without the initialization overhead).For this to work, one must do all the initialization manually before the method is called, for example:import tensorflow as tf
from simple_elmo import ElmoModel

graph = tf.Graph()
with graph.as_default() as elmo_graph:
    elmo_model = ElmoModel()
    elmo_model.load(PATH_TO_ELMO)
...
with elmo_graph.as_default() as current_graph:
    tf_session = tf.compat.v1.Session(graph=elmo_graph)
        with tf_session.as_default() as sess:
            elmo_model.elmo_sentence_input = simple_elmo.elmo.weight_layers("input", elmo_model.sentence_embeddings_op)
            sess.run(tf.compat.v1.global_variables_initializer())
...
elmo_model.get_elmo_vectors(SENTENCES, session=tf_session)
elmo_model.get_elmo_vectors(SENTENCES2, session=tf_session)
...Theget_elmo_substitutes()method currently works only with the models loaded  withfull=True.
For each input sentence, it  produces a list of lexical substitutes (LM predictions) for each word token in the sentence, produced by the forward and backward ELMo language models.
The substitutes are yielded as dictionaries containing the vocabulary identifiers of the most probable LM predictions, their lexical forms and their logit scores.
NB: this method is still experimental!Example scriptsWe provide three example scripts to make it easier to start usingsimple_elmoright away:Inferring token embeddingspython3 get_elmo_vectors.py -i test.txt -e ~/PATH_TO_ELMO/This script simply returns contextualized ELMo embeddings for the words in your input sentences.Text pairs classificationpython3 text_classification.py -i paraphrases_lemm.tsv.gz -e ~/PATH_TO_ELMO/This script can be used to perform document pair classification (like in text entailment or paraphrase detection).
Simple average of ELMo embeddings for all words in a document is used;
then, the cosine similarity between two documents is calculated and used as a single classifier feature.
Evaluated with macro F1 score and 10-fold cross-validation.Example paraphrase dataset for English (adapted fromMRPC):https://rusvectores.org/static/testsets/mrpc.tsv.gzExample paraphrase datasets for Russian (adapted fromhttp://paraphraser.ru/):https://rusvectores.org/static/testsets/paraphrases.tsv.gzhttps://rusvectores.org/static/testsets/paraphrases_lemm.tsv.gz(lemmatized)Word sense disambiguationpython3 wsd_eval.py -i senseval3.tsv -e ~/PATH_TO_ELMO/This script takes as an input a word sense disambiguation (WSD) dataset and a pre-trained ELMo model.
It extracts token embeddings for ambiguous words and trains a simple Logistic Regression classifier  to predict word senses.
Averaged macro F1 score across all words in the test set is used as the evaluation measure (with 5-fold cross-validation).Example WSD datasets for English (adapted fromSenseval 3):https://rusvectores.org/static/testsets/senseval3.tsvhttps://rusvectores.org/static/testsets/senseval3_lemm.tsv(lemmatized)Example WSD datasets for Russian (adapted fromRUSSE'18):https://rusvectores.org/static/testsets/russe_wsd.tsvhttps://rusvectores.org/static/testsets/russe_wsd_lemm.tsv(lemmatized)Frequently Asked QuestionsWhere can I find pre-trained ELMo models?Several repositories are available where one can download ELMo models compatible withsimple_elmo:NLPL vector repository(exceptELMoForManyLangsmodels trained on the CoNLL17 corpus; see below)AllenNLPCLARINCan I loadELMoForManyLangsmodels?Unfortunately not. These models are trained using a slightly different architecture.
Therefore, they arenot compatibleneither withAllenNLPnor withsimple_elmo.
You should use the originalELMoForManyLangscode to work with these models.I see a lot of warnings about deprecated methodsThis is normal. Thesimple_elmolibrary is based on theoriginal ELMo implementationwhich was aimed at the versions of TensorFlow which are very outdated today.
We significantly updated the code and fixed many warnings - but not all of them yet. The work continues (and will eventually lead to a complete switch to TensorFlow 2).Meaniwhile, these warnings can be ignored: they do not harm the resulting embeddings in any way.Can I train my own ELMo with this library?Currently we provide ELMo training code (updated and improved in the same way compared to the original implementation)
in aseparate repository.
It will be integrated into thesimple_elmopackage at some point.