Repository: flairNLP/flair

Issue:
Using custom word embeddings
Hi there,

I am experimenting with training a SequenceTagger for NER in french, and I would like to use custom word embeddings for the language.

Is there a simple way to do this, that doesn't involve changing the code of `WordEmbeddings` ?

Cheers, and thank you for the great work !


Discussion:
Hi @mhham!

Thanks!

What do you mean by custom word embeddings? Are you looking for something like this: https://github.com/zalandoresearch/flair/issues/81?
Exactly, I was wondering if there was a way to do it just by passing the embedding path (correctly formatted to gensim) to the WordEmbeddings constructor
No, not yet. But as you are the second person asking about this, we will add it. 
I tried modifying the WordEmbeddings class (as suggested in #81) to input my custom embedding. 
The embedding seems to work fine, but when training a sequence tagger I get a torch serialization error, right after the first epoch. This code works with `WordEmbeddings('glove')`.

Code:

```
from flair.data_fetcher import NLPTask
from flair.embeddings import CharacterEmbeddings, CharLMEmbeddings, TokenEmbeddings, WordEmbeddings, StackedEmbeddings
from typing import List

# 1. get the corpus
corpus = downsampled_corpus 

# 2. what tag do we want to predict?
tag_type = 'ner'

# 3. make the tag dictionary from the corpus
tag_dictionary = corpus.make_tag_dictionary(tag_type=tag_type)
print(tag_dictionary.idx2item)


# 4. initialize embeddings
embedding_types: List[TokenEmbeddings] = [

    WordEmbeddings('fr-fasttext') #Defined in the WordEmbeddings class

    # comment in this line to use character embeddings
    # CharacterEmbeddings(),

    # comment in these lines to use contextual string embeddings
    #CharLMEmbeddings('news-forward'),
    #CharLMEmbeddings('news-backward'),
]

embeddings: StackedEmbeddings = StackedEmbeddings(embeddings=embedding_types)

# 5. initialize sequence tagger
from flair.models import SequenceTagger

tagger: SequenceTagger = SequenceTagger(hidden_size=256,
                                        embeddings=embeddings,
                                        tag_dictionary=tag_dictionary,
                                        tag_type=tag_type,
                                        use_crf=True)

# 6. initialize trainer
from flair.trainers import SequenceTaggerTrainer

trainer: SequenceTaggerTrainer = SequenceTaggerTrainer(tagger, corpus)

# 7. start training
trainer.train('resources/taggers/example-ner',
              learning_rate=0.1,
              mini_batch_size=32,
              max_epochs=10)
```

Gives the following error :

```
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-6-de6d08c3eb5d> in <module>()
     47               learning_rate=0.1,
     48               mini_batch_size=32,
---> 49               max_epochs=10)

.../lib/python3.7/site-packages/flair/trainers/sequence_tagger_trainer.py in train(self, base_path, learning_rate, mini_batch_size, max_epochs, anneal_factor, patience, train_with_dev, embeddings_in_memory, checkpoint, save_final_model, anneal_with_restarts)
    161                 # if we use dev data, remember best model based on dev evaluation score
    162                 if not train_with_dev and dev_score == scheduler.best:
--> 163                     self.model.save(base_path + "/best-model.pt")
    164 
    165             # if we do not use dev data for model selection, save final model

.../lib/python3.7/site-packages/flair/models/sequence_tagger_model.py in save(self, model_file)
    149             'rnn_layers': self.rnn_layers,
    150         }
--> 151         torch.save(model_state, model_file, pickle_protocol=4)
    152 
    153     @classmethod

.../lib/python3.7/site-packages/torch/serialization.py in save(obj, f, pickle_module, pickle_protocol)
    207         >>> torch.save(x, buffer)
    208     """
--> 209     return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickle_protocol))
    210 
    211 

.../lib/python3.7/site-packages/torch/serialization.py in _with_file_like(f, mode, body)
    132         f = open(f, mode)
    133     try:
--> 134         return body(f)
    135     finally:
    136         if new_fd:

.../lib/python3.7/site-packages/torch/serialization.py in <lambda>(f)
    207         >>> torch.save(x, buffer)
    208     """
--> 209     return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickle_protocol))
    210 
    211 

.../lib/python3.7/site-packages/torch/serialization.py in _save(obj, f, pickle_module, pickle_protocol)
    280     pickler = pickle_module.Pickler(f, protocol=pickle_protocol)
    281     pickler.persistent_id = persistent_id
--> 282     pickler.dump(obj)
    283 
    284     serialized_storage_keys = sorted(serialized_storages.keys())

OSError: [Errno 22] Invalid argument
```
Ok, this error seems to happen with `WordEmbeddings('de-fasttext')` too. This seems like a bug, I will open a new issue
Thanks for reporting the issue!

Wrt to the word embeddings: You could also convert your embeddings into gemsim keyedvectors format and then just pass the path to the keyedvectors to the `WordEmbeddings` class. If you use fasttext embeddings, you can use the code from #81 : 

```python
import os
from gensim.models.fasttext import FastText

# load fasttext embeddings by pointing this method to the ft embeddings txt file
model = FastText.load_fasttext_format('/path/to/fasttext/embeddings')
word_vectors = model.wv

# save as KeyedVectors
from gensim.models import KeyedVectors
word_vectors.save('/path/to/converted')
```

You can then init `WordEmbeddings` by passing this path, i.e. 
```python
embeddings = WordEmbeddings('/path/to/converted')
```

Does this work for you? 



Thank you for your answer. 

In fact using `FastText.load_fasttext_format('/path/to/fasttext/embeddings')` with a .txt file does not work ; it needs the fasttext **.bin** file.

What I did to make it work with the .txt file was the following : 
```python
import gensim

word_vectors = gensim.models.KeyedVectors.load_word2vec_format('/path/to/fasttext/embeddings.txt', binary=False)

word_vectors.save('/path/to/converted')
```

This works since the text file is exactly in the same format as word2vec text files

Then I ended up using the modified WordEmbeddings class with the custom embedding path

Resolving patch:
diff --git a/flair/embeddings.py b/flair/embeddings.py
--- a/flair/embeddings.py
+++ b/flair/embeddings.py
@@ -137,42 +137,43 @@ class WordEmbeddings(TokenEmbeddings):
     def __init__(self, embeddings: str):
         """
         Initializes classic word embeddings. Constructor downloads required files if not there.
-        :param embeddings: one of: 'glove', 'extvec', 'crawl' or two-letter language code
+        :param embeddings: one of: 'glove', 'extvec', 'crawl' or two-letter language code.
+        If you want to use a custom embedding file, just pass the path to the embeddings as embeddings variable.
         """
 
-        base_path = 'https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings/'
+        old_base_path = 'https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings/'
+        base_path = 'https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings-v0.3/'
 
         # GLOVE embeddings
         if embeddings.lower() == 'glove' or embeddings.lower() == 'en-glove':
-            cached_path(os.path.join(base_path, 'glove.gensim.vectors.npy'), cache_dir='embeddings')
-            embeddings = cached_path(os.path.join(base_path, 'glove.gensim'), cache_dir='embeddings')
+            cached_path(os.path.join(old_base_path, 'glove.gensim.vectors.npy'), cache_dir='embeddings')
+            embeddings = cached_path(os.path.join(old_base_path, 'glove.gensim'), cache_dir='embeddings')
 
         # KOMNIOS embeddings
-        if embeddings.lower() == 'extvec' or embeddings.lower() == 'en-extvec':
-            cached_path(os.path.join(base_path, 'extvec.gensim.vectors.npy'), cache_dir='embeddings')
-            embeddings = cached_path(os.path.join(base_path, 'extvec.gensim'), cache_dir='embeddings')
-
-        base_path = 'https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings-v0.3/'
+        elif embeddings.lower() == 'extvec' or embeddings.lower() == 'en-extvec':
+            cached_path(os.path.join(old_base_path, 'extvec.gensim.vectors.npy'), cache_dir='embeddings')
+            embeddings = cached_path(os.path.join(old_base_path, 'extvec.gensim'), cache_dir='embeddings')
 
         # FT-CRAWL embeddings
-        if embeddings.lower() == 'crawl' or embeddings.lower() == 'en-crawl':
-            cached_path(os.path.join(base_path, 'en-fasttext-crawl-300d-1M.vectors.npy'.format(embeddings)),
-                        cache_dir='embeddings')
+        elif embeddings.lower() == 'crawl' or embeddings.lower() == 'en-crawl':
+            cached_path(os.path.join(base_path, 'en-fasttext-crawl-300d-1M.vectors.npy'), cache_dir='embeddings')
             embeddings = cached_path(os.path.join(base_path, 'en-fasttext-crawl-300d-1M'), cache_dir='embeddings')
 
         # FT-CRAWL embeddings
-        if embeddings.lower() == 'news' or embeddings.lower() == 'en-news' or embeddings.lower() == 'en':
-            cached_path(os.path.join(base_path, 'en-fasttext-news-300d-1M.vectors.npy'.format(embeddings)),
-                        cache_dir='embeddings')
+        elif embeddings.lower() == 'news' or embeddings.lower() == 'en-news' or embeddings.lower() == 'en':
+            cached_path(os.path.join(base_path, 'en-fasttext-news-300d-1M.vectors.npy'), cache_dir='embeddings')
             embeddings = cached_path(os.path.join(base_path, 'en-fasttext-news-300d-1M'), cache_dir='embeddings')
 
         # other language fasttext embeddings
-        if len(embeddings.lower()) == 2 and not embeddings.lower() == 'en':
+        elif len(embeddings.lower()) == 2 and not embeddings.lower() == 'en':
             cached_path(os.path.join(base_path, '{}-fasttext-300d-1M.vectors.npy'.format(embeddings)),
                         cache_dir='embeddings')
             embeddings = cached_path(os.path.join(base_path, '{}-fasttext-300d-1M'.format(embeddings)),
                                      cache_dir='embeddings')
 
+        elif not os.path.exists(embeddings):
+            raise ValueError(f'The given embeddings "{embeddings}" is not available or is not a valid path.')
+
         self.name = embeddings
         self.static_embeddings = True
 
@@ -347,8 +348,9 @@ class CharLMEmbeddings(TokenEmbeddings):
     def __init__(self, model, detach: bool = True, use_cache: bool = True, cache_directory: str = None):
         """
         initializes contextual string embeddings using a character-level language model.
-        :param model: model string, one of 'news-forward', 'news-backward', 'mix-forward', 'mix-backward', 'german-forward',
-                'german-backward' depending on which character language model is desired
+        :param model: model string, one of 'news-forward', 'news-backward', 'news-forward-fast', 'news-backward-fast',
+                'mix-forward', 'mix-backward', 'german-forward', 'german-backward', 'polish-backward', 'polish-forward'
+                depending on which character language model is desired.
         :param detach: if set to False, the gradient will propagate into the language model. this dramatically slows down
                 training and often leads to worse results, so not recommended.
         :param use_cache: if set to False, will not write embeddings to file for later retrieval. this saves disk space but will
@@ -364,50 +366,53 @@ def __init__(self, model, detach: bool = True, use_cache: bool = True, cache_dir
             model = cached_path(base_path, cache_dir='embeddings')
 
         # news-english-backward
-        if model.lower() == 'news-backward':
+        elif model.lower() == 'news-backward':
             base_path = 'https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings/lm-news-english-backward-v0.2rc.pt'
             model = cached_path(base_path, cache_dir='embeddings')
 
         # news-english-forward
-        if model.lower() == 'news-forward-fast':
+        elif model.lower() == 'news-forward-fast':
             base_path = 'https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings/lm-news-english-forward-1024-v0.2rc.pt'
             model = cached_path(base_path, cache_dir='embeddings')
 
         # news-english-backward
-        if model.lower() == 'news-backward-fast':
+        elif model.lower() == 'news-backward-fast':
             base_path = 'https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings/lm-news-english-backward-1024-v0.2rc.pt'
             model = cached_path(base_path, cache_dir='embeddings')
 
         # mix-english-forward
-        if model.lower() == 'mix-forward':
+        elif model.lower() == 'mix-forward':
             base_path = 'https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings/lm-mix-english-forward-v0.2rc.pt'
             model = cached_path(base_path, cache_dir='embeddings')
 
         # mix-english-backward
-        if model.lower() == 'mix-backward':
+        elif model.lower() == 'mix-backward':
             base_path = 'https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings/lm-mix-english-backward-v0.2rc.pt'
             model = cached_path(base_path, cache_dir='embeddings')
 
         # mix-german-forward
-        if model.lower() == 'german-forward':
+        elif model.lower() == 'german-forward':
             base_path = 'https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings/lm-mix-german-forward-v0.2rc.pt'
             model = cached_path(base_path, cache_dir='embeddings')
 
         # mix-german-backward
-        if model.lower() == 'german-backward':
+        elif model.lower() == 'german-backward':
             base_path = 'https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings/lm-mix-german-backward-v0.2rc.pt'
             model = cached_path(base_path, cache_dir='embeddings')
 
         # common crawl Polish forward
-        if model.lower() == 'polish-forward':
+        elif model.lower() == 'polish-forward':
             base_path = 'https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings/lm-polish-forward-v0.2.pt'
             model = cached_path(base_path, cache_dir='embeddings')
 
         # common crawl Polish backward
-        if model.lower() == 'polish-backward':
+        elif model.lower() == 'polish-backward':
             base_path = 'https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings/lm-polish-backward-v0.2.pt'
             model = cached_path(base_path, cache_dir='embeddings')
 
+        elif not os.path.exists(model):
+            raise ValueError(f'The given model "{model}" is not available or is not a valid path.')
+
         self.name = model
         self.static_embeddings = detach
 
diff --git a/train.py b/train.py
--- a/train.py
+++ b/train.py
@@ -6,11 +6,11 @@
 from flair.visual.training_curves import Plotter
 
 # 1. get the corpus
-corpus: TaggedCorpus = NLPTaskDataFetcher.fetch_data(NLPTask.CONLL_03).downsample(0.1)
+corpus: TaggedCorpus = NLPTaskDataFetcher.fetch_data(NLPTask.UD_ENGLISH)
 print(corpus)
 
 # 2. what tag do we want to predict?
-tag_type = 'ner'
+tag_type = 'pos'
 
 # 3. make the tag dictionary from the corpus
 tag_dictionary = corpus.make_tag_dictionary(tag_type=tag_type)


Test patch:
diff --git a/tests/test_embeddings.py b/tests/test_embeddings.py
--- a/tests/test_embeddings.py
+++ b/tests/test_embeddings.py
@@ -8,43 +8,33 @@
 
 
 @pytest.mark.slow
-def test_en_glove():
-    load_and_apply_word_embeddings('en-glove')
+def test_glove():
+    load_and_apply_word_embeddings('glove')
 
 
 @pytest.mark.slow
-def test_en_numberbatch():
-    load_and_apply_word_embeddings('en-numberbatch')
+def test_extvec():
+    load_and_apply_word_embeddings('extvec')
 
 
 @pytest.mark.slow
-def test_en_extvec():
-    load_and_apply_word_embeddings('en-extvec')
+def test_crawl():
+    load_and_apply_word_embeddings('crawl')
 
 
 @pytest.mark.slow
-def test_en_crawl():
-    load_and_apply_word_embeddings('en-crawl')
+def test_news():
+    load_and_apply_word_embeddings('news')
 
 
 @pytest.mark.slow
-def test_en_news():
-    load_and_apply_word_embeddings('en-news')
+def test_fr():
+    load_and_apply_word_embeddings('fr')
 
 
 @pytest.mark.slow
-def test_de_fasttext():
-    load_and_apply_word_embeddings('de-fasttext')
-
-
-@pytest.mark.slow
-def test_de_numberbatch():
-    load_and_apply_word_embeddings('de-numberbatch')
-
-
-@pytest.mark.slow
-def test_sv_fasttext():
-    load_and_apply_word_embeddings('sv-fasttext')
+def test_it():
+    load_and_apply_word_embeddings('it')
 
 
 @pytest.mark.slow
@@ -77,6 +67,19 @@ def test_german_backward():
     load_and_apply_char_lm_embeddings('german-backward')
 
 
+def test_loading_not_existing_embedding():
+    with pytest.raises(ValueError):
+        WordEmbeddings('other')
+
+    with pytest.raises(ValueError):
+        WordEmbeddings('not/existing/path/to/embeddings')
+
+
+def test_loading_not_existing_char_lm_embedding():
+    with pytest.raises(ValueError):
+        CharLMEmbeddings('other')
+
+
 @pytest.mark.integration
 def test_stacked_embeddings():
     sentence, glove, charlm = init_document_embeddings()