Repository: flairNLP/flair

Issue:
Tagger complains that some input tokens are not in the dictionary (despite add_unk=True)
**Describe the bug**
I've trained flair vectors for Ukrainian Language and POS model on UD data using following embeddings:
```python
embedding_types = [
    FastTextEmbeddings('/gdrive/MyDrive/WordVectors/FastText/binary/cc.uk.300.bin'),

    # comment in this line to use character embeddings
    CharacterEmbeddings("/gdrive/MyDrive/ubertext1.0/flair_dictionary.pkl"),

    # comment in these lines to use flair embeddings
    FlairEmbeddings('/gdrive/MyDrive/ubertext1.0/resources/language_model_forward2/best-lm.pt'),
    FlairEmbeddings('/gdrive/MyDrive/ubertext1.0/resources/language_model_backward2/best-lm.pt'),
]
```

But when I'm trying to tag a sentence using that model it raises following error:
```
2021-08-30 14:13:51,839 The string '«' is not in dictionary! Dictionary contains only: ['<unk>', ' ', 'о', 'а', 'н', 'і', 'и', 'р', 'в', 'т', 'е', 'с', 'к', 'л', 'у', 'д', 'м', 'п', 'з', 'я', 'ь', 'г', 'б', '\n', 'й', 'ч', 'ц', 'х', 'ї', 'ж', 'ю', 'ш', 'є', '1', "'", '0', 'щ', '2', 'П', 'К', 'ф', 'У', 'С', 'В', '-', 'Н', 'А', 'М', '9', 'e', 'Р', 'i', 'a', 'Д', 'Т', '5', '3', 'О', 'З', '4', 'Б', 'o', '8', 'r', '6', 'n', 'І', '7', 't', 'Г', 'Л', 's', '.', 'l', 'Я', 'Ф', 'u', 'c', ',', 'Ч', 'h', 'd', 'Ш', 'Є', 'Е', 'Х', 'm', 'Ц', 'p', '%', 'g', 'I', 'S', 'Ю', 'A', 'y', 'C', 'b', ':', 'k', 'B', 'T', 'M', 'P', 'D', 'f', 'v', 'И', 'w', 'F', 'Ж', 'R', 'ы', 'E', 'L', 'N', 'G', 'Щ', 'V', 'H', 'Й', '=', 'x', 'X', 'z', 'W', '–', 'O', '_', '—', 'U', 'j', 'ґ', 'K', '+', 'J', 'Ь', 'Ї', '№', '°', 'Z', 'Y', '\xad', 'э', '$', 'Ґ', 'q', '&', 'Q', '*', 'Э', '#', '²', '(', ')', 'ü', '½', 'ъ', '−', 'ä', 'ё', 'ł', '@', 'ö', '†', 'Ъ', 'è', 'ă', '³', '×', 'ą', 'ę', 'â', 'ż', 'ș', 'š', 'ā', '€', 'ç', 'ў', 'č', 'à', '₴', 'ı', 'ß', '~', 'ë', 'ž', '·', '•', 'ѣ', 'å', 'ñ', '、', 'Ы', 'Š', 'ō', 'Ö', 'ě', 'ī', 'æ', 'ş', '^', 'ô', 'ê', 'é', 'º', 'ț', 'ə', '±', 'ū', 'ј', 'α', 'Ł', 'ø', 'ã', 'î']
2021-08-30 14:13:51,841 You can create a Dictionary that handles unknown items with an <unk>-key by setting add_unk = True in the construction.
```

I've double checked how I instantiate dicts:
```python
from flair.data import Dictionary
dictionary = Dictionary.load("/gdrive/MyDrive/ubertext1.0/flair_dictionary.pkl")
print(dictionary.add_unk)

from flair.embeddings import CharacterEmbeddings

cm = CharacterEmbeddings("/gdrive/MyDrive/ubertext1.0/flair_dictionary.pkl")
print(cm.char_dictionary.add_unk)
```
and it says `True` both times.

**To Reproduce**
Here is the model
https://drive.google.com/drive/folders/1fQdLsgmdCtle5sCShtqYZjB9ERWL0jbe?usp=sharing
Try tagging sentence like this:
```python
from flair.data import Sentence
from flair.models import SequenceTagger

tagger = SequenceTagger.load('/gdrive/MyDrive/FlairResults/POS/ukrainian.fasttext.fbvecs.tokenized.plusdev.plusflairemb.uk_charemb.300d/final-model.pt')

sentence = Sentence(['«', '!'])
tagger.predict(sentence)
```

**Expected behavior**
Well, it should respect unknown tokens/characters

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Environment (please complete the following information):**
flair==0.9
Python 3.7.11
Google Colab Pro Plus

**Additional context**
I'm really keen to publish my flair models for Ukrainian Language (flair embeddings, NER and POS) but I don't really want to publish something, that might be not usable by others.



Discussion:
@dchaplinsky is this error new with Flair 0.9 or did it already exist in Flair 0.8.1?
I believe, that I've used 0.8 to train embeddings and POS model.

I'll try installing 0.8.1 and comment on results.
Huh, it worked flawlessly with 0.8 (0.8.1 is not available in pip)
Even more, it correctly identified both unknown characters (« and !) as punct
@dchaplinsky I've loaded the model but it seems the `CharacterEmbeddings` have `add_unk` set to False. Any idea why this might be the case?

```python
forward: CharacterEmbeddings = tagger.embeddings.embeddings[1]

print(forward.char_dictionary)
print(forward.char_dictionary.add_unk)
forward.char_dictionary.get_idx_for_item('«')
```
Nope, no idea. As I said, I've checked that all dicts I initialized during
training of embeddings and POS models had it turned on by default (I didn't
set add_unk to True explicitly, but for example when you are loading dict
from the pickled file, add_unk is set to True for me by default)

On Tue, Aug 31, 2021 at 10:13 AM Alan Akbik ***@***.***>
wrote:

> @dchaplinsky <https://github.com/dchaplinsky> I've loaded the model but
> it seems the CharacterEmbeddings have add_unk set to False. Any idea why
> this might be the case?
>
> forward: CharacterEmbeddings = tagger.embeddings.embeddings[1]
>
>
> print(forward.char_dictionary)
> print(forward.char_dictionary.add_unk)
> forward.char_dictionary.get_idx_for_item('«')
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/flairNLP/flair/issues/2407#issuecomment-908964660>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AABAA4XUKNYUBW5F5IAJ3B3T7R6JZANCNFSM5DCDSFBQ>
> .
>

Very strange, I create a tagger with `CharacterEmbeddings`, save it out and load it again. Everything seems to work as it should: 

```python
# initialize character embeddings and check add_unk
character_embeddings = CharacterEmbeddings()
print(character_embeddings.char_dictionary.add_unk)

# create test tagger and save it
tagger = SequenceTagger(hidden_size=8, embeddings=character_embeddings, tag_type='ner', tag_dictionary=Dictionary())
tagger.save("test_model.pt")

# load the model
loaded = SequenceTagger.load("test_model.pt")

# check if add_unk is still True (it is)
print(loaded.embeddings.char_dictionary.add_unk)
```

That's how I did the dictionary (it's pretty much vanilla code from the example in repo):

```python
from flair.data import Dictionary
char_dictionary: Dictionary = Dictionary()

# counter object
import collections
counter = collections.Counter()

processed = 0

import glob
files = glob.glob('/gdrive/MyDrive/ubertext1.0/flair_data/train/*') + glob.glob('/gdrive/MyDrive/ubertext1.0/flair_data/*.txt')

print(files)

for file in files:
    print(file)

    with open(file, 'r', encoding='utf-8') as f:
        tokens = 0
        for line in f:

            processed += 1            
            chars = list(line)
            tokens += len(chars)

            # Add chars to the dictionary
            counter.update(chars)

            # comment this line in to speed things up (if the corpus is too large)
            # if tokens > 50000000: break

    # break

total_count = 0
for letter, count in counter.most_common():
    total_count += count

print(total_count)
print(processed)

sum = 0
idx = 0
for letter, count in counter.most_common():
    sum += count
    percentile = (sum / total_count)

    # comment this line in to use only top X percentile of chars, otherwise filter later
    if percentile > 1 - 0.000025: break

    char_dictionary.add_item(letter)
    idx += 1
    print('%d\t%s\t%7d\t%7d\t%f' % (idx, letter, count, sum, percentile))

print(char_dictionary.item2idx)

import pickle
with open('/gdrive/MyDrive/ubertext1.0/flair_dictionary.pkl', 'wb') as f:
    mappings = {
        'idx2item': char_dictionary.idx2item,
        'item2idx': char_dictionary.item2idx
    }
    pickle.dump(mappings, f)
```

and that's how I trained forward and backward flair embeddings:
```python
import os.path
from flair.data import Dictionary
from flair.models import LanguageModel
from flair.trainers.language_model_trainer import LanguageModelTrainer, TextCorpus


def train_flair_embeddings(
    corpus_path="/gdrive/MyDrive/ubertext1.0/flair_data/",
    dictionary_path="/gdrive/MyDrive/ubertext1.0/flair_dictionary.pkl",
    lm_file="/gdrive/MyDrive/ubertext1.0/resources/language_model_forward2",
    is_forward_lm=True,
    hidden_size=1024,
    sequence_length=250,
    mini_batch_size=100,
    max_epochs=20,
):
    # are you training a forward or backward LM?
    is_forward_lm = True

    # load the default character dictionary
    dictionary: Dictionary = Dictionary.load(dictionary_path)

    # get your corpus, process forward and at the character level
    corpus = TextCorpus(corpus_path, dictionary, is_forward_lm, character_level=True)

    # instantiate your language model, set hidden size and number of layers
    language_model = LanguageModel(dictionary, is_forward_lm, hidden_size=hidden_size, nlayers=1)

    checkpoint = os.path.join(lm_file, "checkpoint.pt")
    if os.path.exists(checkpoint):
        trainer = LanguageModelTrainer.load_from_checkpoint(checkpoint, corpus)
    else:
        # train your language model
        trainer = LanguageModelTrainer(language_model, corpus)

    trainer.train(
        lm_file,
        sequence_length=sequence_length,
        mini_batch_size=mini_batch_size,
        max_epochs=max_epochs,
        checkpoint=True,
    )
```

and that's the code I've used to train POS tagger (with some imports ommited):

```python
# # 2. what tag do we want to predict?
tag_type = 'upos'

# # 3. make the tag dictionary from the corpus
tag_dictionary = corpus.make_tag_dictionary(tag_type=tag_type)
print(tag_dictionary)

from flair.embeddings import CharacterEmbeddings, FlairEmbeddings, FastTextEmbeddings
embedding_types = [
    FastTextEmbeddings('/gdrive/MyDrive/WordVectors/FastText/binary/cc.uk.300.bin'),

    # comment in this line to use character embeddings
    CharacterEmbeddings("/gdrive/MyDrive/ubertext1.0/flair_dictionary.pkl"),

    # comment in these lines to use flair embeddings
    FlairEmbeddings('/gdrive/MyDrive/ubertext1.0/resources/language_model_forward2/best-lm.pt'),
    FlairEmbeddings('/gdrive/MyDrive/ubertext1.0/resources/language_model_backward2/best-lm.pt'),
]

embeddings: StackedEmbeddings = StackedEmbeddings(embeddings=embedding_types)

# 5. initialize sequence tagger
from flair.models import SequenceTagger

tagger: SequenceTagger = SequenceTagger(hidden_size=256,
                                        embeddings=embeddings,
                                        tag_dictionary=tag_dictionary,
                                        tag_type=tag_type,
                                        use_crf=True)

# 6. initialize trainer
from flair.trainers import ModelTrainer

trainer: ModelTrainer = ModelTrainer(tagger, corpus)

# 7. start training
trainer.train('/gdrive/MyDrive/FlairResults/POS/ukrainian.fasttext.fbvecs.tokenized.plusdev.plusflairemb.uk_charemb.300d',
              learning_rate=0.1,
              mini_batch_size=32,
              train_with_dev=True,
              embeddings_storage_mode='cpu',
              max_epochs=200)
```

as you can see both times I'm initializing a Dictionary class with pickled dict I saved before and `add_unk=True` is default value in the Dictionary class:
https://github.com/flairNLP/flair/blob/master/flair/data.py#L24
https://github.com/flairNLP/flair/blob/master/flair/data.py#L116

https://github.com/flairNLP/flair/blob/master/flair/embeddings/token.py#L282

So in my understanding, `add_unk=True` was used everywhere by default. Yet in 0.9 model throws an exception on unknown tokens, while in 0.8 the same code works flawlessly.
Could you also share the dictionary file to help us reproduce?
Sure, here goes:

https://drive.google.com/file/d/1xk-KEwNUZ6kewNyGBkDh-Co9QTM4aroA/view?usp=sharing

Resolving patch:
diff --git a/flair/data.py b/flair/data.py
--- a/flair/data.py
+++ b/flair/data.py
@@ -6,11 +6,11 @@
 from operator import itemgetter
 from typing import List, Dict, Union, Optional
 
-import flair
 import torch
 from torch.utils.data import Dataset
 from torch.utils.data.dataset import ConcatDataset, Subset
 
+import flair
 from flair.file_utils import Tqdm
 
 log = logging.getLogger("flair")
@@ -62,7 +62,8 @@ def get_idx_for_item(self, item: str) -> int:
             return 0
         else:
             log.error(f"The string '{item}' is not in dictionary! Dictionary contains only: {self.get_items()}")
-            log.error("You can create a Dictionary that handles unknown items with an <unk>-key by setting add_unk = True in the construction.")
+            log.error(
+                "You can create a Dictionary that handles unknown items with an <unk>-key by setting add_unk = True in the construction.")
             raise IndexError
 
     def get_idx_for_items(self, items: List[str]) -> List[int]:
@@ -102,24 +103,29 @@ def save(self, savefile):
         with open(savefile, "wb") as f:
             mappings = {"idx2item": self.idx2item, "item2idx": self.item2idx}
             pickle.dump(mappings, f)
-            
+
     def __setstate__(self, d):
         self.__dict__ = d
+        # set 'add_unk' if the dictionary was created with a version of Flair older than 0.9
         if 'add_unk' not in self.__dict__.keys():
-            self.__dict__['add_unk'] = False
-        
+            self.__dict__['add_unk'] = True if b'<unk>' in self.__dict__['idx2item'] else False
 
     @classmethod
     def load_from_file(cls, filename: str):
         import pickle
 
-        dictionary: Dictionary = Dictionary()
-        with open(filename, "rb") as f:
-            mappings = pickle.load(f, encoding="latin1")
-            idx2item = mappings["idx2item"]
-            item2idx = mappings["item2idx"]
-            dictionary.item2idx = item2idx
-            dictionary.idx2item = idx2item
+        f = open(filename, "rb")
+        mappings = pickle.load(f, encoding="latin1")
+        idx2item = mappings["idx2item"]
+        item2idx = mappings["item2idx"]
+        f.close()
+
+        # set 'add_unk' depending on whether <unk> is a key
+        add_unk = True if b'<unk>' in idx2item else False
+
+        dictionary: Dictionary = Dictionary(add_unk=add_unk)
+        dictionary.item2idx = item2idx
+        dictionary.idx2item = idx2item
         return dictionary
 
     @classmethod
diff --git a/flair/embeddings/token.py b/flair/embeddings/token.py
--- a/flair/embeddings/token.py
+++ b/flair/embeddings/token.py
@@ -279,9 +279,7 @@ def __init__(
         if path_to_char_dict is None:
             self.char_dictionary: Dictionary = Dictionary.load("common-chars")
         else:
-            self.char_dictionary: Dictionary = Dictionary.load_from_file(
-                path_to_char_dict
-            )
+            self.char_dictionary: Dictionary = Dictionary.load_from_file(path_to_char_dict)
 
         self.char_embedding_dim: int = char_embedding_dim
         self.hidden_size_char: int = hidden_size_char


Test patch: