Repository: flairNLP/flair

Issue:
CharacterEmbeddings: missing characters in vocab
Hi,

there's one major issue with the `CharacterEmbeddings` class: by default it uses a "common chars" dictionary (fetched from [here](https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/models/common_characters)).

I manually loaded these common characters with:

```python
from flair.data import Dictionary
d = Dictionary.load(name="chars")

len(d.get_items()) # Outputs 275
```

Then I used a Bulgarian word to check if all characters appear in the common characters dictionary:

```python
for char in "оглуша":
  if char not in d.get_items():
    print(char)
```

Unfortunately, the `ш` character is not found.

I think we should update these common characters dictionary :)


Discussion:
Hi @stefan-it yes that's a good point. Common chars should at very least contain all cyrillic characters. We could additionally offer other standard dictionaries, or include a method that distills a dictionary from a corpus.
I created a larger vocab file on the latest Universal Dependencies v2.4 containing a character vocab for  83 languages.

Creation steps:

* Download Universal Dependencies v2.4 from [here](http://hdl.handle.net/11234/1-2988)

* Extract archive and adjust path to the extracted folder in the following script:

```python
import collections
import os

from pathlib import Path
import pickle

from flair.data import Dictionary
char_dictionary: Dictionary = Dictionary()

ud_base = 'ud-treebanks-v2.4/'

datasets = [Path(d) / Path(f) for d,_,fl in os.walk(ud_base) for f in fl if f.endswith('txt')]

vocab = collections.Counter()

for dataset in datasets:
    with open(dataset, 'rt') as f_p:
        for line in f_p:
            for char in line:
                vocab[char] += 1

for letter, _ in vocab.most_common():
    char_dictionary.add_item(letter)

with open('common-chars-large', 'wb') as f:
    mappings = {
        'idx2item': char_dictionary.idx2item,
        'item2idx': char_dictionary.item2idx
    }
    pickle.dump(mappings, f)
```

Using all `*.txt` is much more faster than parsing the CoNLL-U format for each language.

The resulting vocab contains 9001 characters.

It can be downloaded here:

```bash
wget https://schweter.eu/cloud/flair-misc/common-chars-large
```

Test it with:

```python
from flair.data import Dictionary
d = Dictionary.load_from_file(filename='common-chars-large')

print(len(d.get_items()))
```

Resolving patch:
diff --git a/flair/data.py b/flair/data.py
--- a/flair/data.py
+++ b/flair/data.py
@@ -118,6 +118,16 @@ def load(cls, name: str):
             char_dict = cached_path(base_path, cache_dir="datasets")
             return Dictionary.load_from_file(char_dict)
 
+        if name == "chars-large" or name == "common-chars-large":
+            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/models/common_characters_large"
+            char_dict = cached_path(base_path, cache_dir="datasets")
+            return Dictionary.load_from_file(char_dict)
+
+        if name == "chars-xl" or name == "common-chars-xl":
+            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/models/common_characters_xl"
+            char_dict = cached_path(base_path, cache_dir="datasets")
+            return Dictionary.load_from_file(char_dict)
+
         return Dictionary.load_from_file(name)
 
 


Test patch: